You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
ML-For-Beginners/8-Reinforcement/1-QLearning/solution/assignment-solution.ipynb

413 lines
215 KiB

{
"metadata": {
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
},
"orig_nbformat": 2,
"kernelspec": {
"name": "python3",
"display_name": "Python 3.7.0 64-bit ('3.7')"
},
"interpreter": {
"hash": "70b38d7a306a849643e446cd70466270a13445e5987dfa1344ef2b127438fa4d"
}
},
"nbformat": 4,
"nbformat_minor": 2,
"cells": [
{
"source": [
"# Peter and the Wolf: Realistic Environment\n",
"\n",
"In our situation, Peter was able to move around almost without getting tired or hungry. In more realistic world, we has to sit down and rest from time to time, and also to feed himself. Let's make our world more realistic, by implementing the following rules:\n",
"\n",
"1. By moving from one place to another, Peter loses **energy** and gains some **fatigue**.\n",
"2. Peter can gain more energy by eating apples.\n",
"3. Peter can get rid of fatigue by resting under the tree or on the grass (i.e. walking into a board location with a tree or grass - green field)\n",
"4. Peter needs to find and kill the wolf\n",
"5. In order to kill the wolf, Peter needs to have certain levels of energy and fatigue, otherwise he loses the battle.\n"
],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import random\n",
"import math\n",
"from rlboard import *"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": "<Figure size 792x432 with 1 Axes>",
"image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"360.646265pt\" version=\"1.1\" viewBox=\"0 0 366.6475 360.646265\" width=\"366.6475pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <defs>\n <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;white-space:pre;}\n </style>\n </defs>\n <g id=\"figure_1\">\n <g id=\"patch_1\">\n <path d=\"M 0 360.646265 \nL 366.6475 360.646265 \nL 366.6475 0 \nL 0 0 \nz\n\" style=\"fill:none;\"/>\n </g>\n <g id=\"axes_1\">\n <g id=\"patch_2\">\n <path d=\"M 33.2875 336.76814 \nL 359.4475 336.76814 \nL 359.4475 10.60814 \nL 33.2875 10.60814 \nz\n\" style=\"fill:#ffffff;\"/>\n </g>\n <g clip-path=\"url(#pb71df41202)\">\n <image height=\"327\" id=\"imagec1d069daa2\" transform=\"scale(1 -1)translate(0 -327)\" width=\"327\" x=\"33.2875\" xlink:href=\"data:image/png;base64,\niVBORw0KGgoAAAANSUhEUgAAAUcAAAFHCAYAAAAySY5rAAAABHNCSVQICAgIfAhkiAAAIABJREFUeJzs3XmcZFV99/HPOefeW7dubb3NzjLAAINsgiLiggoaJC4RohKVaAI+uAOCrzzRJ+YRCYlbQmJIRBJNAopoRBCNEpAdBgUU2ZdBZoAZmJneu7r2e895/rhdlxmpxmdgXk/1PPN7Q72qp6u6+szpqW+f7Z6jTjzxRLdixQoWqlarxezsLMPDw/0uyrymp6fxPI9CodDvosxr8+bNbN48gnOm30WZ18DARvbcc+H+W0yShPHxcRYvXtzvosyrVquRJAnlcrnfRZnX+Pg4xWKRXC7X76LMa+PGjXDGGWe4hWzt2rXuoosu6ncxntcVV1zh1qxZ0+9iPK9zzz3X5XITDtwCvVn37nd/vN/V9LzGx8fdeeed1+9iPK9bb73VXXnllf0uxvO68MIL3dq1a/tdjOd1+umnO93vhBZCiIXI63cBhNgZWWsBUErhnPud9845Oo0GSaeTvYbxfaxNsInNPucFAZ12e5vvFYQhQRjO+/rdcuxsuvXy23+XXp/b+mvme2xH2Po1JRyFeAEmJiZIkoRisUi1WmV4eJjp6WmGh4eZnJykVCqxZcsWBgcHmZ2tcu9P/ovbL/kPHrvrDiAdSNj7yCPZ8uSTTGzcSGIdiYOXvP4N/Pq664mtw6II8nle90fv5g8+/glKixYxsmgR9XqdcrnM+Pg4Q0NDKKV2ynBsNps0Gg1yuRytVgutNcYY2u02URTRaDQIgoBOp4NSCs/zaLVaRFFEvV4nCAKSJMH3fZxzAHieR7PZJJ/P02w2CYIAay3OOTzPo9FoEEVR9phzjiRJMMaQz+e3KZ+EoxDb6bffWFu/2WZnZ/F9n2azSalUwlrLHZdezI//6hyctYSm2zqCjXfdQeIgNJpEORLnWHvzjeQ0GKWIHYRRnl9d+zMmNj7NB7/8ZVrlMlprZmdnyeVy2X0Yhn2ule0XxzG+72OtxfO8LOCDIMge64YapK1G3/eJ45ggCAAwxmSteOccnU4nC1itNXEcZ9+v0+nged42j3VboEmSPKd8MuYoxHYyxpAkCUmSoLUmSRI8z8Nam73Zfd+n3W5zyzcu5Ka//wqhhrynyXua0DOEvib0NHmjCD1FOPf5nKfJeYacUeS0IzCKyuAg9eoMXz/jDOzcGzqXy5EkCUEQZOGxs+m2BJVSNJvNLKyazSbGGJrNJgDtdjsLue5jjUYDay3tdnuboQXf9+l0OtsE7Nah232s+zMzxuCcw5jnruKQcBRiO2mdvm263dluSHbfZM45tNasu+N27rr4X/E6TfK+zm6Rl97ynib0NXnPpI8ZyBs1F6BpSHamJ5l9ZgNRPo9zlv/5pt/LymCtzcqyM+rW2dZ/h+544tb1CM+ONVpr2bJlCz/96U/ZY4892GuvvVixYgW77bYb3/jGN9i0aRNAFnZaa7TWKKW2+Rl1n9N9rNewxM5bs0L0SavVwvd9fN8nSRLy+Xw2PjY7O0sQBFQnJxi7/Wbs6DPkPZUGotk2GLufz3uavNGEviFndNqSNJqcMQRa0Z6ZZstjaxlevISoVOTxX/+aarVKLpejVqvR2WqSZ2fSbDYpFArEcUyhUMi6ysViMRuWSJKEMAzJ5XI453jooYc47LDD+JM/+RNmZmaoVqvMzMwwMzPDJz/5Sfbff39uv/12qtUqQRBQr9ezFmmtViMMQ6rVKp7nUa/XabVaQNo6/W0SjkJsp3w+T71ep9FoYIyhXq9nb+hKpUKz2UTFbe655F/TVqEx5D1D5Kf3YRaOJm0lGjXXUky71oFJP84ZRTB3m9m0kcfvuoPlK1fys3/7NyqlIo1Gg4GBgZ1yvBEgiiJmZmbwfZ+ZmZmsGz01NUUURUxNTWGMoVar0Ww2ue222/jABz7A1NTUvK/Zbrf54Ac/yI033kir1aJUKmVd63K5TKPRYHBwkE6nQ7FYzCZhetWhhKMQ26lWq1EqlSgWi8RxTLFYZHp6miiKGB8fJ4oifvqxPyXnp13j/Fwo5j2V3gdp9/rZQEwDMm8g9Byhp9OANOnnfaMJPMPkxidZe9stNOs1rrnsMkqlEmNjY9Tr9X5XyQtSrVYZHByk3W4zODiYBdXIyAjVapWRkZHsap8oirjzzjt56qmnfufrbtq0iRtuuIEwDJmamqLVamGtZWJigkKhwNjYGEEQMD09Ta1WA+hZhzvnSK4QfWSMIY7jbKwqjmM8zyNJEnK5HFOP/4b6k+vIewajFFortAOlNA6wgLUOqx0WRxw7Yg2x1Rhl8QCNQykLiZ77CkAbmjNT2Habdffcw8vf9HvkCwV83+9fZbwIpVKJiYkJyuUyU1NTBEGA7/tMTEwwNDTE+Pg4pVKJarWK1pooirKvXbRoEbvvvvs245IbNmxgy5YtOOeypTyVSoVOp0OSJAwODjI5OcnQ0BAzMzOUy+VsUue3l/GAhKMQ280YQ6fT2WaSoDtBopTiri//JbpVI/B0Go4K9Fbj/daB1WCdInHQUQ7POTqJRScGRYJDgVI4p8GBdRZnoFWdYctjj/DkQw9y7PtOZunee2dr/HY2rVaLMAyJ45h8Pp9NiuTz+Wyto7WWXC7H6Ogot9xyCwCFQoELLriA448/fpvF4D/+8Y/5+Mc/zsTEBL/85S955plnsnFM51w2Llyv1zHG0Gq1srqL4/g513pLt1qI7dRqtcjn8+Tzeay12yw4fuK2G2lufDKdYPE0odHZ7HPoeYSeN9fFNs8+bhQ5rQi0zsYYAwO+Tj/2jcLTCl8rPK0ZW7+OvQ86kB//8wXMzMz0nEzYGWz9CyVJkiyotl4i1f3ls27dOn74wx8C6fjgG9/4RoCs5aiU4m1ve1u2Qc11113H/fffv81sdHdN5NYz4c93hZGEoxDbKYoiqtVqNutZrVazwX67fi129GlCTxPlAgr5gLyniTwzN1Ot5sYe0
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAFpCAYAAAC8p8I3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzdeZxcVZ3//9fn1l7V3dV7J2QjIexBwhaIC6MgyKACg47iyogzqD9QZ8YZdUZnXJDBr8vgMF8V40hEXFBHWYavy2AGR1lEQCEkbAkkgSSdpbu6u/a6yzm/P+p209F09k5VJZ8nj3pU1b23qj65Tb9zcu45p8Rai1JKqdbhNLoApZRSe0eDWymlWowGt1JKtRgNbqWUajEa3Eop1WI0uJVSqsVMW3CLyAUi8rSIrBWRj07X5yil1OFGpmMct4hEgGeA84CNwEPAW6y1TxzwD1NKqcPMdLW4lwBrrbXPWWtd4Fbg4mn6LKWUOqxMV3DPAl6Y9HxjuE0ppdR+ijbqg0XkSuBKgFgsdtpLXvKS/Xq/kZERPM+b/P709fXt13uOq9VqFAoFent7D8j7TYfR0VFisRiZTKbRpUxpcHCQ/v5+IpFIo0uZ0vPPP8/cuXMbXcaUfN9n+/btzJw5s9GlTKlYLOL7Pp2dnY0uZUrbt2+no6ODRCLR6FKmtHr1aiqViux0p7X2gN+ApcDPJz3/B+Afpjq+v7/f7o9bbrnF9vT0WGDiFo1G7T/90z/t1/uOW7NmjV22bNkBea/pctttt9n777+/0WXs0jXXXGNzuVyjy5iSMcZeffXVjS5jl4aHh+21117b6DJ26d5777W33357o8vYpRtvvNGuWbOm0WXsUpiLO83M6WpxPwQcLSLzgU3AZcBbD/SH+L7P97//fT7wgQ8wMjLyR/u+8IUvAPCRj3yEdDqNyM7/8lJKqVYyLX3c1lofuBr4OfAk8ANr7eoD/TmbNm3iHe94xx+F9rhKpcJnPvMZfvGLXxzoj1ZKqYaZtj5ua+1PgJ9M1/sDbN68GcdxCIJgymNEhK1btxIEAdFow7r0lVLqgGnpmZMPPvjgLkMbwBjD73//+x0uXCqlVCtr6eB+wxvesNsRCo7jcOGFF5JMJg9SVUopNb1aOrhjsRiLFy/e5THz58+np6fnIFWklFLTr6WDu6+vj6uuumqXx1x44YWceuqpOqJEKXXIaOngdhyHiy++mDvvvJNjjz12h32ZTIbvfe97fPjDH27qQfZKKbW3Wjq4oT5q5Ic//CHPPPPMDttLpRKf//znxycATdwrpVSra+ngXrt2LR/60Ie45ZZbdhrMv/vd77jiiit48MEHMcY0oEKllDrwWjK4jTE8/fTTfPCDH2T58uW7PPYXv/gF733ve/nNb36z26GDSinVCloquK21VKtVPvvZz/Lyl7+cn//853v0uscee4yLLrqIt7/97eTz+clrqiilVMtpqamErutyww038LGPfWyvX5vL5bj11ltJpVJ87nOf0yGCSqmW1VLB/bnPfY5PfOIT+/Uey5cvJxaL8eUvf1mnwCulWlJLdJVYa7nuuuu47rrrDkgXx/Lly/mLv/gLvWCplGpJTR/cruvyb//2b3zyk5+kUqnssO/kk0/eoynvxx9//A6ta8/zuPXWW7nyyivJ5/PTUrdSSk2Xpg5uay1f/vKX+fCHP4zrujvsO+ecc/jhD3+4R8H9pS99ife9730sWbJkYnsQBHzzm9/kox/9KMVicVrqV0qp6dDUwf2///u/fPzjH99hZb+5c+dy3XXX8fWvf51sNrvH73X99ddz/fXXc9ZZZ01Mfw+CgBtvvJFbb71VR5kopVpG0wa3MYbvfve7VKvViW19fX3ccMMN/O3f/i3z58/fq/dzHIclS5bwxS9+kcnfb2mtZfny5drfrZRqGU0b3CLCW97yFhYtWgTAMcccw3/8x3/w+te/nng8vteLRokI0WiUpUuXsnz5cpYsWYKIMHfuXK688kocp2lPhVJK7aBpx8OJCK985StZtmwZd9xxB5deeimnn376Hx23t10cIsIpp5zCHXfcwb//+7+zdOlSzj//fF09UCnVMpo2uMctWbKEM844Y6fBWi6X8X1/l68PgoByuYy1dof3GBgY4JprrtHAVkq1nKbuHxARRATHcXYasDNnzuTTn/70Lt/jLW95Cy9/+ct3+t7j76vhrZRqJU0d3LsTiUTo6ura5TFtbW0kk0kNZ6XUIaOlg1sppQ5HGtxKKdViNLiVUqrFaHArpVSL0eBWSqkWo8GtlFItRoNbKaVajAa3Ukq1mJYObmvtbqe8G2N05T+l1CFlv4JbRNaLyOMi8qiIPBxu6xaRu0VkTXi/66mN+yEajTJv3ryJWZHJZJJFixbtMJuyv7+f9vb26SpBKaUOugOxyNSrrLVDk55/FFhhrf2siHw0fP6RA/A5O3XkkUdy+eWXU6vVWLBgAZ/5zGe46aabWLFiBZFIhDPOOGO6PloppRpiOlYHvBh4Zfj4ZuCXTFNwiwiLFy9m+fLlO2y/4ooruOKKK6bjI5VSquH2t4/bAv8tIo+IyJXhtgFr7WD4eAswsJ+foZRSapL9bXG/3Fq7SUT6gbtF5KnJO621VkR2+k0HYdBfCfUV/NasWbOfpUyfjRs3Mjo62tQ1Dg0NYYxp6hpLpRLr1q1jaGho9wc3iOu6TX0O8/k8pVKpqWvcsmVL0/++jI6O8sILLzT1d83ualDFfgW3tXZTeL9NRG4DlgBbRWSmtXZQRGYC26Z47TJgGUBPT4/95S9/uT+lTKvR0VE2btxIM9f47LPPkk6nGR4ebnQpUxoaGuL+++8nkUg0upQpFYvFpv45V6tVHtj+AHf88o5GlzKl9GCacyvnNvVork2bNvHII4+wdu3aRpcypV2eP2vtPt2ADNA+6fH9wAXA54GPhts/Cnxud+/V399vm9maNWvssmXLGl3GLt122232/vvvb3QZu3TNNdfYXC7X6DKmZIyxV199daPL2KXh4WF72rWnWZr4vxn3zrC33357o0/VLt144412zZo1jS5jl8Jc3Glm7k+LewC4LRyKFwW+a639mYg8BPxARN4NbADetB+foZRS6g/sc3Bba58DTt7J9mHg3P0pSiml1NRaeuakUkodjjS4lVKqxWhwK6VUi9HgVkqpFqPBrZRSLUaDWymlWowGt1JKtRgNbqWUajEa3Eop1WI0uJVSqsVocCulVIvR4FZKqRajwa2UUi1Gg1sppVqMBrdSSrUYDW6llGoxGtxKKdViNLiVUqrFaHArpVSL0eBWSqkWo8GtlFItRoNbKaVajAa3Ukq1GA1upZRqMRrcSinVYjS4lVKqxWhwK6VUi9HgVkqpFqPBrZRSLUaDWymlWsxug1tEbhKRbSKyatK2bhG5W0TWhPdd4XYRkRtEZK2IrBSRU6ezeKWUOhztSYv7m8AFf7Dto8AKa+3RwIrwOcCfAkeHtyuBrx6YMpVSrUREGl3CIW23wW2t/RWQ+4PNFwM3h49vBi6ZtP1btu43QKeIzDxQxSqlWoO1ttElHNL2tY97wFo7GD7eAgyEj2cBL0w6bmO4TSml1AGy3xcnbf2v1r3+61VErhSRh0Xk4Uqlsr9lKKXUYWNfg3vreBdIeL8t3L4JmDPpuNnhtj9irV1mrT3dWnt6KpXaxzKUUurwE93H190JXA58Nry/Y9L2q0XkVuBMYGxSl8qUgiDg9ttv38dSpt/Q0BDPPvtsU9e4atUqNmzYwNatWxtdypS2bNnCz372M5r5L+p8Pt/UP+dyuUxmMMOC2xc0upQpta9vZ1VpVVP3cz/33HNEo1FWrVq1+4MbJAiCKfftNrhF5HvAK4FeEdkIfIJ6YP9ARN4NbADeFB7+E+BCYC1QBt61JwW6rvC+9w3s/sAGSacNl1+eZmCgeWvcsGEDN96YZXS0eWtcuDDBJZf0kclkGl3KlKLRaFP/nIvFImckzuCzA59tdClTemrkKQpOoanPYzqd5l+6/4XyQLnRpUzJFXfKfbsNbmvtW6bYde5OjrXAVXtc2cTrHLZ
},
"metadata": {
"needs_background": "light"
}
}
],
"source": [
"width, height = 8,8\n",
"m = Board(width,height)\n",
"m.randomize(seed=13)\n",
"m.plot()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"actions = { \"U\" : (0,-1), \"D\" : (0,1), \"L\" : (-1,0), \"R\" : (1,0) }\n",
"action_idx = { a : i for i,a in enumerate(actions.keys()) }"
]
},
{
"source": [
"## Defining state\n",
"\n",
"In our new game rules, we need to keep track of energy and fatigue at each board state. Thus we will create an object `state` that will carry all required information about current problem state, including state of the board, current levels of energy and fatigue, and whether we can win the wolf while at terminal state:"
],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"class state:\n",
" def __init__(self,board,energy=10,fatigue=0,init=True):\n",
" self.board = board\n",
" self.energy = energy\n",
" self.fatigue = fatigue\n",
" self.dead = False\n",
" if init:\n",
" self.board.random_start()\n",
" self.update()\n",
"\n",
" def at(self):\n",
" return self.board.at()\n",
"\n",
" def update(self):\n",
" if self.at() == Board.Cell.water:\n",
" self.dead = True\n",
" return\n",
" if self.at() == Board.Cell.tree:\n",
" self.fatigue = 0\n",
" if self.at() == Board.Cell.apple:\n",
" self.energy = 10\n",
"\n",
" def move(self,a):\n",
" self.board.move(a)\n",
" self.energy -= 1\n",
" self.fatigue += 1\n",
" self.update()\n",
"\n",
" def is_winning(self):\n",
" return self.energy > self.fatigue"
]
},
{
"source": [
"Let's try to solve the problem using random walk and see if we succeed:"
],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"tags": []
},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0"
]
},
"metadata": {},
"execution_count": 5
}
],
"source": [
"def random_policy(state):\n",
" return random.choice(list(actions))\n",
"\n",
"def walk(board,policy):\n",
" n = 0 # number of steps\n",
" s = state(board)\n",
" while True:\n",
" if s.at() == Board.Cell.wolf:\n",
" if s.is_winning():\n",
" return n # success!\n",
" else:\n",
" return -n # failure!\n",
" if s.at() == Board.Cell.water:\n",
" return 0 # died\n",
" a = actions[policy(m)]\n",
" s.move(a)\n",
" n+=1\n",
"\n",
"walk(m,random_policy)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Killed by wolf = 5, won: 1 times, drown: 94 times\n"
]
}
],
"source": [
"def print_statistics(policy):\n",
" s,w,n = 0,0,0\n",
" for _ in range(100):\n",
" z = walk(m,policy)\n",
" if z<0:\n",
" w+=1\n",
" elif z==0:\n",
" n+=1\n",
" else:\n",
" s+=1\n",
" print(f\"Killed by wolf = {w}, won: {s} times, drown: {n} times\")\n",
"\n",
"print_statistics(random_policy)"
]
},
{
"source": [
"## Reward Function\n"
],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"def reward(s):\n",
" r = s.energy-s.fatigue\n",
" if s.at()==Board.Cell.wolf:\n",
" return 100 if s.is_winning() else -100\n",
" if s.at()==Board.Cell.water:\n",
" return -100\n",
" return r"
]
},
{
"source": [
"## Q-Learning algorithm\n",
"\n",
"The actual learning algorithm stays pretty much unchanged, we just use `state` instead of just board position."
],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"Q = np.ones((width,height,len(actions)),dtype=np.float)*1.0/len(actions)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"def probs(v,eps=1e-4):\n",
" v = v-v.min()+eps\n",
" v = v/v.sum()\n",
" return v"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
""
]
}
],
"source": [
"\n",
"from IPython.display import clear_output\n",
"\n",
"lpath = []\n",
"\n",
"for epoch in range(10000):\n",
" clear_output(wait=True)\n",
" print(f\"Epoch = {epoch}\",end='')\n",
"\n",
" # Pick initial point\n",
" s = state(m)\n",
" \n",
" # Start travelling\n",
" n=0\n",
" cum_reward = 0\n",
" while True:\n",
" x,y = s.board.human\n",
" v = probs(Q[x,y])\n",
" while True:\n",
" a = random.choices(list(actions),weights=v)[0]\n",
" dpos = actions[a]\n",
" if s.board.is_valid(s.board.move_pos(s.board.human,dpos)):\n",
" break \n",
" s.move(dpos)\n",
" r = reward(s)\n",
" if abs(r)==100: # end of game\n",
" print(f\" {n} steps\",end='\\r')\n",
" lpath.append(n)\n",
" break\n",
" alpha = np.exp(-n / 3000)\n",
" gamma = 0.5\n",
" ai = action_idx[a]\n",
" Q[x,y,ai] = (1 - alpha) * Q[x,y,ai] + alpha * (r + gamma * Q[x+dpos[0], y+dpos[1]].max())\n",
" n+=1"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": "<Figure size 792x432 with 1 Axes>",
"image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"360.646265pt\" version=\"1.1\" viewBox=\"0 0 366.6475 360.646265\" width=\"366.6475pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <defs>\n <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;white-space:pre;}\n </style>\n </defs>\n <g id=\"figure_1\">\n <g id=\"patch_1\">\n <path d=\"M 0 360.646265 \nL 366.6475 360.646265 \nL 366.6475 0 \nL 0 0 \nz\n\" style=\"fill:none;\"/>\n </g>\n <g id=\"axes_1\">\n <g id=\"patch_2\">\n <path d=\"M 33.2875 336.76814 \nL 359.4475 336.76814 \nL 359.4475 10.60814 \nL 33.2875 10.60814 \nz\n\" style=\"fill:#ffffff;\"/>\n </g>\n <g clip-path=\"url(#p3155e0caf7)\">\n <image height=\"327\" id=\"image3956b3e364\" transform=\"scale(1 -1)translate(0 -327)\" width=\"327\" x=\"33.2875\" xlink:href=\"data:image/png;base64,\niVBORw0KGgoAAAANSUhEUgAAAUcAAAFHCAYAAAAySY5rAAAABHNCSVQICAgIfAhkiAAAIABJREFUeJzsnXecVNXZx7+3zp07ZWcbvayAgqIoKKDGrokaG9iIDRMglmiEiMaoSV4VsWDDvCS2qG8sRGOLLRoNgghiEFERBKVJk4XtZfrce94/ZufK6mICzDILnC/MZ3Zm7pz73Dtzf/Oc5zznOcqZZ54punfvTkclmUzS3NxMaWlpoU3ZKg0NDei6TiAQKLQpW2XTpk1s2lSGEFqhTdkqkcgGevfe8e+i4zisWbOGPn36sHLlSioqKmhsbMR13R36HjmOQ01NDZ06ddphG9uLaDSK4ziEw+FCm7JVampqCAaD+Hy+QpuyVTZs2ADjx48XHZnly5eLhx9+uNBmfC8vvfSSeP/99wttxvcyadIk4fPVChAd9OaKc8+9Mi/HGo1Gxa9//Wsxf/58cdxxx4m1a9eKyZMni/fee2+H2q2pqRGTJ0/Oi43txZw5c8Tf//73QpvxvTz44INi+fLlhTbje7nqqquEWmiFlkjyjW3bHH/88YwZM4bFixfz85//nCVLlnDEEUcU2jTJLoReaAMkkvbgyCOP5PXXX+e8887jnnvuoXPnznlt33VdABRFQQjxH++FEKTjcZx02mtDMwxc18F1XO853TRJp1Kt9mVaFqZlbbX9nB27Grnz8u1jaeu5Ld+ztdfywZZtSnGU7Jb4/X569eqFZVn06NGDoqKivLZfW1uL4zgEg0GampooLS2loaGB0tJS6urqCIVCbN68meLiYpqbm1j0j9eZ9+RfWLFgPpANJPQZPpzNa9dSu2EDjitwBOx3zLF8MuMdMq7ARcH0+zn6J+dyxpW/JFReTll5ObFYjHA4TE1NDSUlJSiKskuKYyKRIB6P4/P5SCaTqKqKpmmkUils2yYej2OaJul0GkVR0HWdZDKJbdvEYjFM08RxHAzDQAgBgK7rJBIJ/H4/iUQC0zRxXRchBLquE4/HsW3be00IgeM4aJqG3+9vZZ8UR4lkG/n2hbXlxdbc3IxhGCQSCUKhEK7rMn/6E7x2680I18XSct4RbFgwH0eApak4isARguWzZ+FTQVMUMgIs28/Ct/9F7YavGXfXXSTDYVRVpbm5GZ/P591bllXgs7LtZDIZDMPAdV10XfcE3jRN77WcqEHWazQMg0wmg2maAGia5nnxQgjS6bQnsKqqkslkvP2l02l0XW/1Ws4DdRznO/bJmKNEso1omobjODiOg6qqOI6Druu4rutd7IZhkEqleO/RB3l36t1YKvh1Fb+uYukalqFi6Sp+TcHSFayW5326ik/X8GkKPlVgagpFxcXEmhp5aPx43JYL2ufz4TgOpml64rGrkfMEFUUhkUh4YpVIJNA0jUQiAUAqlfJELvdaPB7HdV1SqVSr0IJhGKTT6VYCu6Xo5l7LfWaapiGEQNO+m8UhxVEi2UZUNXvZ5LqzOZHMXWRCCFRVZfX8eSx44s/o6QR+Q/Vutp69+XUVy1Dx61r2NQ38mtIioFmRTDfU0bxxPbbfjxAu1/3wR54Nrut6trQHtbW1nkC1B7lztuUx5OKJW55H+CbW6Loumzdv5o033qBXr17stddedO/enR49evDoo49SWVkJ4ImdqqqoqoqiKK0+o9w2udfaCktIcZRItpFkMolhGBiGgeM4+P1+Lz7W3NyMaZo01dVSPW82btVG/LqSFUSttTDmnvfrKn5NxTI0fJqa9SQ1FZ+mYaoKqcYGNq9YTmmnztihIKs++YSmpiZ8Ph/RaJT0FoM8+eSuu+5i/vz57dI2ZL3AQCBAJpMhEAh4XeVgMOiFJRzHwbIsfD4fQgiWLl3K4MGD+elPf0pjYyNNTU00NjbS2NjIr371K/r378+8efNoamrCNE1isZjnkUajUSzLoqmpCV3XicViJJNJIOudfhspjhLJNuL3+4nFYsTjcTRNIxaLeRd0UVERiUQCJZPi0yf/nPUKNQ2/rmEb2XvLE0ct6yVqSounmO1am1r2b5+mYLbcGis3sGrBfLpVVPCvxx+nKBQkHo8TiUR2yXgjZFOuGhsbMQyDxsZGz0utr6/Htm3q6+vRNI1oNEoikWDu3LlcfPHF1NfXb7XNVCrFuHHjmDVrFslkklAo5HWtw+Ew8Xic4uJi0uk0wWDQG4Rp6xxKceygOI7Tbh7BnkRuFDOfRKNRQqEQwWCQTCZDMBikoaEB27apqanBtm3euOJn+Ixs19jfIop+Xcnem9nu9TeCmBVIvwaWLrB0NSuQWvZ5Q1MxdY26DWtZPvc9ErEobz3zDKFQiOrqamKxWN6PEbIxwZzX1R40NTVRXFxMKpWiuLjYE6qysjKampooKyvzZvvYts2HH37IunXr/mO7lZWVzJw5E8uyqK+vJ5lM4routbW1BAIBqqurMU2ThoYGotEoQJvncNeM5O7mfPrppyxevJgVK1Zw8sknM2TIkF026F5o2iPFRdM0MpmMF6vKZDLouo7jOPh8PupXrSS2djV+XUNTFFRVQRWgKCoCcAHXFbiqwEWQyQgyKmRcFU1x0QEVgaK44Kgt7wBUjURjPW4qxepPP+WQH/4IfyCAYRh5P0aAm266iR/+8IcMGzaMYDCY9/ZDoRC1tbWEw2Hq6+sxTRPDMKitraWkpISamhpCoRBNTU2oqopt2957y8vL6dmzZ6u45Pr169m8eTNCCC+Vp6ioiHQ6jeM4FBcXU1dXR0lJCY2NjYTDYW9Q59tpPCA9xw7JqFGjqK+v59BDD+Xcc8+lqamp0CZJtiA3WOC6rjdIkBsgURSFBXf9HjUZ9Uan/ZqC38iOSvt1pWXQpWXgRdPwGdnRa5+u4NM0DA1MVcGnKRiqiqGqmKqCoSkkmxrZvOILPnr1FWo3bvRy+NrrOHNpMu1BMpnEsiwymQx+v99zAPx+v5fr6Lpu9genvp733nsPgEAgwLRp05g1axazZs1i5syZzJw5k
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAFpCAYAAAC8p8I3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzdd3xUVd7H8c+Zmt5J6CC9qVQpKoqIoggIuouCoojCIkXQXUV3F0XEh10bYAFBUQFBwRXXroggIjYiKkWE0EsI6Zle7j3PH5mMsBBqkjsJ580rr2Tu3Mz9zoT8cubcc88RUkoURVGU6sNkdABFURTlzKjCrSiKUs2owq0oilLNqMKtKIpSzajCrSiKUs2owq0oilLNVFrhFkL0FUL8LoTIEkJMrqzjKIqinG9EZYzjFkKYge1AH+AA8CNwq5Rya4UfTFEU5TxTWS3uS4AsKeUuKaUfeAsYWEnHUhRFOa9UVuGuB+w/6vaB0DZFURTlHFmMOrAQYhQwCsBqtXa66KKLjIpySj6fD4fDQVpamtFRylVUVITVaiU2NtboKOXKzs4mPT0ds9lsdJRy7du3j4YNGxodo1zBYJDc3Fzq1KljdJRyOZ1OgsEgSUlJRkcpV25uLgkJCdjtdqOjlGvLli14PB5xwjullBX+AXQHPjvq9sPAw+Xtn56eLiPZjh075Lx584yOcVIrVqyQ69evNzrGSU2bNk0WFBQYHaNcuq7LcePGGR3jpPLz8+X06dONjnFS69atk++9957RMU5q7ty5cseOHUbHOKlQXTxhzaysrpIfgeZCiAuEEDbgFuD9SjqWoijKeaVSukqklEEhxDjgM8AMLJBSbqmMYymKopxvKq2PW0r5MfBxZT2+oijK+UpdOakoilLNqMKtKIpSzajCrSiKUs2owq0oilLNqMKtKIpSzajCrSiKUs3U2MK9cOHCsqs2FaXKeb1eli9fbnSM0/L222/j9/uNjlGut956K6LzGaHGFe6VK1cycOBAgsEggwYNYvHixYbkkFJy3333GXJsxVhPPfUUw4cPJzc3l4EDB7Jp0yajI53QTz/9xMCBAyksLGTo0KHMnDnT6EjHyMzMZODAgRQVFTF06FBmzZpldKSIUaMKdzAYJCsrixtuuIF+/foxatQotm3bhs/nq9IcL7zwAq1ateKWW26hZcuWzJkzp0qPX9mklBQUFBgdIyK53W42bdrEI488wk033cTFF1/Mnj170HXd6GjH0HWdPXv20LlzZwYNGsTkyZPZvHkzbrfb6GhAab7du3fTtWtXBg0axIMPPsjmzZvxeDxGR4sINapwHz58mF27dtG3b1/69u1Lly5dsNlsbN68ucoyFBQUUFRUxKJFi/D5fCxcuJCCggIKCwurLENlW716NdOnTzc6RkRavXo1HTt2JD09ndtvv52hQ4fy0Ucf4fV6jY52DJfLxRdffMGQIUO45ZZbqF+/Pq1bt2bdunVGRwNKZxhcvXo1f/rTnxgyZAiNGjWiVatWfP3110ZHiwiGTetaGerXr0/Lli0ZPXo0hw4dYvz48TRt2pROnTpVWQaPx4OmacTFxfH999+zb98+4uLi8Hg8JCcnV1mOyrJixQq2b9/Ov//9b6OjRKR+/foxZMgQfv/9d6644gruuusupk+fTkxMjNHRjhEfH8+AAQMYOXIke/fu5aGHHkJKyQMPPGB0NAASEhLo168fI0eOZM+ePTz44IMAEZPPaDWqcAP079+fNm3acOeddzJ+/HgaNGhQpcevV68eMTEx/OlPf2LZsmVcdNFFNG7cmMGDB5ORkRHRc1GXR0qJrut8+eWX7Ny5k3HjxlXJ8whPYWmqXm8Mp02bxv79+5k4cSLdunWjY8eORkc6oa5du/Kvf/2LcePGcfvtt9OoUSOjIx2je/fuzJgxg3HjxvHll1/y5ZdfGh0pYlSv34jTkJGRQY8ePYiNjaV79+6GTIo/ceJEfvjhB6ZOnYrD4WDTpk38+9//ZtCgQWRnZ1d5nnO1f/9+rrzySj7//HPuu+++KlmsYf/+/Xz33Xfce++9bNu2rVr1bbZo0YKrrrqKWbNm0aBBAyyWyGwfJScn06NHDxITE+natSvNmzc3OtIxyvKtW7eOtLS0iMhXUFCAw+EwOkbNa3FHAqvVitVqZdmyZeFtixYt4uDBgzz//PM0adKEP//5zyQkJBiY8vT89NNPvPnmm3z66adVurrOqFGjaNq0Kffccw8333wzr7/+Op07d66y458rIQRXXXUVq1ev5ocffqBXr15GR6q2oqKijI5AMBhk4cKFHDx4ELvdTt26dRk2bBhCnHiBmsqmCncVqlevHnfeeSfffvstjz76KM8++6xhP/jTsXXrVt5//30eeOCBKi3aH3zwAX369GHo0KE89NBD6LrOY489Rmpq6nH7Dho0iBtvvLHKsinnp2AwyIwZM3jzzTdxOp2MGjWKYcOGHbPP119/zSuvvHLC7+/cuTPjx4+vsDw1snA/+OCDzJ49OyKLYosWLWjRogVdu3Zl5MiRzJo1i+jo6Ih7O33kyBFmzJjBc889d8KCWZkuvfRSlixZwk033cTkyZN54oknuPnmm2nXrt1x+y5fvpypU6ee9PE+++wz0tPTKyvuSU2cOJHRo0fTuXNn4uPjDclwKgsXLuTWW2/lww8/NDrKCQkhmDVrFg8++KBhJ8VHjBjBf//7XzZs2MCzzz7LgQMHjjt30atXL/7xj3+c8Pvj4uIqNE9kVYsKkpOTQ+3atSOycJdp2bIl48ePp3///gwfPpzbb78dq9VqdCwAtm3bxrRp01i0aJEhJwZTUlKoX78+48aN4/rrr+fAgQO0adPmhH2cDz/8MJMnTz7p4xn5/yA1NZXCwsKIG8d9tHr16nHw4EGjY5RLCEHt2rXJyckxLMOrr75K9+7deffdd+nQoQM333wzmZmZx+1XVb8vNbJwVwdCCDp06MCaNWt4/fXX+de//kWbNm0YPHiwYZmOHDnC/Pnz8fv9PP/884aO5njqqacoKSnhww8/5Kuvvip3PyFERP+BBhg6dCiLFi1i3LhxRkdRzpLVamXAgAHhqTSGDx9u6P89VbgjwB133MGqVavCfcr9+/evkv8QUkrWr1/PkSNH6N+/P5MnT+ZPf/oTF198MSkpKZV+/FNJSEhg6NChRsc4ZyNGjKBjx46qcJ8jIQRSSkOKpdVqZdq0aezevZuYmBgyMjKqPMPRatxwwOpICMHVV1/N6NGjyczMpFOnTpX+1lVKSZs2bVi6dCnr1q2jXr16/N///R/XXXcddevWrdRjn2+EEKSmpqppAs5B06ZNueyyy3jzzTcNzXHBBRcYXrRBFe6IYrfbmTp1Kj/88APPP/98pR5r/fr19O7dm4cffpioqCh69uzJxo0bK/WY5yshBPPnz6/QUQXnG7PZjNVqjbipA4xSI7tKhg0bRq1atYyOcdYsFgszZsyo1GPk5eWRlpaGx+Ph5ptvZvXq1eTm5lbqMc9nqamp3H777UbHOCEhBH/729+MjnFKXbp0UVM1h9TIwn3NNdcYHSHiDRgwgAkTJuDxeMjIyODJJ5/kwIEDRseqsRITE+nbt6/RMU5ICFEtziW0adPG6AgRo0YWbuXUhBDs2LGDNWvWkJOTw8GDB7HZbEbHUhTlNKjCfR6z2Wzq3YmiVEPq5KSiKEo1owq3oihKNaMKt6IoSjWjCreiKEo1owq3oihKNXNOo0qEEHsAB6ABQSllZyFECvA20BjYA/xZSllzVspVFEUxWEW0uHtJKdtLKcuWJ5kMrJJSNgdWhW4riqIoFaQyukoGAm+Evn4DUMuTKIqiVKBzLdwS+FwIkSmEGBXaliGlLFsR9zBg/FRaiqIoNci5Xjl5mZTyoBAiHVgphNh29J1SSimEOOGsMKFCPwpKl/XZsWPHOUapPAcOHKCoqCiiM+bl5aHrekRndLlc7N69m7y8PKOjlMvv90f0a1hSUoLL5YrojIcPH47435eioiL2798f0ZNWnWzVpHMq3FLKg6HPR4QQK4BLgBwhRB0pZbYQog5wpJz
},
"metadata": {
"needs_background": "light"
}
}
],
"source": [
"m.plot(Q)"
]
},
{
"source": [
"## Results\n",
"\n",
"Let's see if we were successful training Peter to fight the wolf!"
],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Killed by wolf = 1, won: 9 times, drown: 90 times\n"
]
}
],
"source": [
"def qpolicy(m):\n",
" x,y = m.human\n",
" v = probs(Q[x,y])\n",
" a = random.choices(list(actions),weights=v)[0]\n",
" return a\n",
"\n",
"print_statistics(qpolicy)"
]
},
{
"source": [
"We now see much less cases of drowning, but Peter is still not always able to kill the wolf. Try to experiment and see if you can improve this result by playing with hyperparameters."
],
"cell_type": "markdown",
"metadata": {}
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"[<matplotlib.lines.Line2D at 0x7fe0987e7dd8>]"
]
},
"metadata": {},
"execution_count": 13
},
{
"output_type": "display_data",
"data": {
"text/plain": "<Figure size 432x288 with 1 Axes>",
"image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"248.518125pt\" version=\"1.1\" viewBox=\"0 0 395.093508 248.518125\" width=\"395.093508pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <defs>\n <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;white-space:pre;}\n </style>\n </defs>\n <g id=\"figure_1\">\n <g id=\"patch_1\">\n <path d=\"M 0 248.518125 \nL 395.093508 248.518125 \nL 395.093508 0 \nL 0 0 \nz\n\" style=\"fill:none;\"/>\n </g>\n <g id=\"axes_1\">\n <g id=\"patch_2\">\n <path d=\"M 52.375 224.64 \nL 387.175 224.64 \nL 387.175 7.2 \nL 52.375 7.2 \nz\n\" style=\"fill:#ffffff;\"/>\n </g>\n <g id=\"matplotlib.axis_1\">\n <g id=\"xtick_1\">\n <g id=\"line2d_1\">\n <defs>\n <path d=\"M 0 0 \nL 0 3.5 \n\" id=\"m95b0a9fb88\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n </defs>\n <g>\n <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"67.593182\" xlink:href=\"#m95b0a9fb88\" y=\"224.64\"/>\n </g>\n </g>\n <g id=\"text_1\">\n <!-- 0 -->\n <defs>\n <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n </defs>\n <g transform=\"translate(64.411932 239.238437)scale(0.1 -0.1)\">\n <use xlink:href=\"#DejaVuSans-48\"/>\n </g>\n </g>\n </g>\n <g id=\"xtick_2\">\n <g id=\"line2d_2\">\n <g>\n <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"128.471997\" xlink:href=\"#m95b0a9fb88\" y=\"224.64\"/>\n </g>\n </g>\n <g id=\"text_2\">\n <!-- 2000 -->\n <defs>\n <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n </defs>\n <g transform=\"translate(115.746997 239.238437)scale(0.1 -0.1)\">\n <use xlink:href=\"#DejaVuSans-50\"/>\n <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n <use x=\"190.869141\" xlink:href=\"#DejaVuSans-48\"/>\n </g>\n </g>\n </g>\n <g id=\"xtick_3\">\n <g id=\"line2d_3\">\n <g>\n <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"189.350812\" xlink:href=\"#m95b0a9fb88\" y=\"224.64\"/>\n </g>\n </g>\n <g id=\"text_3\">\n <!-- 4000 -->\n <defs>\n <path d=\"M 37.796875 64.3125 \nL 12.890625 25.390625 \nL 37.796875 25.390625 \nz\nM 35.203125 72.90625 \nL 47.609375 72.90625 \nL 47.609375 25.390625 \nL 58.015625 25.390625 \nL 58.015625 17.1875 \nL 47.6093
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAD4CAYAAADy46FuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAcD0lEQVR4nO3df3BV553f8fdHPxHCQgJkIEACrIkd4raxrcZksrOTCVmM3UzwH04Gz86azXrDtHHa7HZnEryZ1tMknkna7XrDrONdN2aDM1kTrzetaYpLKHYm05nasRw7trFNkLExkrGRkQAbJPTjfvvHeSQuQjrge4UkxOc1c9E53/Occ5/nHul+OD+kq4jAzMxsLBWT3QEzM5vaHBRmZpbLQWFmZrkcFGZmlstBYWZmuaomuwPjbd68ebF06dLJ7oaZ2UXlmWeeeScimkdbNu2CYunSpbS2tk52N8zMLiqSDoy1zKeezMwsl4PCzMxyOSjMzCyXg8LMzHI5KMzMLJeDwszMcjkozMwsl4NihOO9/Tz6XMdkd8PMbMqYdr9wV66vP/I8j734FlcuuIyrFjRMdnfMzCadjyhGePNYLwA9fYOT3BMzs6nBQWFmZrkcFGZmlstBYWZmuRwUZmaWy0FhZma5HBRmZpbLQWFmZrkcFGZmlstBYWZmuRwUZmaWy0ExhpjsDpiZTRHnDApJWyQdlvTiKMv+XFJImpfmJWmzpDZJz0u6tqjtBkn70mNDUf06SS+kdTZLUqrPkbQrtd8lqWl8hnyO8U7Ek5iZXUTO54jih8DakUVJS4A1wBtF5RuBFemxEbgvtZ0D3AVcD3wcuKvojf8+4EtF6w091yZgd0SsAHaneTMzm2DnDIqI+CXQNcqie4CvceZZmnXAg5F5EmiUtBC4AdgVEV0R0Q3sAtamZQ0R8WREBPAgcHPRtram6a1FdTMzm0AlXaOQtA7oiIjfjFi0CDhYNN+eann19lHqAPMj4lCafguYn9OfjZJaJbV2dna+3+GYmVmO9x0UkmYCfwH8x/HvzujS0caY15cj4v6IaImIlubm5onqlpnZJaGUI4rfAZYBv5H0OrAY+LWkBUAHsKSo7eJUy6svHqUO8HY6NUX6eriEvpqZWZned1BExAsRcXlELI2IpWSni66NiLeA7cBt6e6nVcCxdPpoJ7BGUlO6iL0G2JmWHZe0Kt3tdBvwaHqq7cDQ3VEbiupmZjaBzuf22IeA/wdcKald0u05zXcA+4E24L8BXwaIiC7gW8DT6fHNVCO1+UFa51XgsVT/DvD7kvYBn0nzZmY2warO1SAibj3H8qVF0wHcMUa7LcCWUeqtwNWj1I8Aq8/VPzMzu7D8m9lmZpbLQWFmZrkcFGZmlstBYWZmuRwUZmaWy0FhZma5HBRmZpbLQWFmZrkcFGZmlstBYWZmuRwUZmaWy0FhZma5HBRmZpbLQWFmZrkcFGZmlstBMYYY8xO6zcwuLQ6KEaTJ7oGZ2dRyPh+FukXSYUkvFtX+i6RXJD0v6b9LaixadqekNkl7Jd1QVF+bam2SNhXVl0l6KtV/Iqkm1WvTfFtavnS8Bm1mZufvfI4ofgisHVHbBVwdEf8c+C1wJ4CklcB64KNpne9LqpRUCdwL3AisBG5NbQG+C9wTEVcA3cDQZ3LfDnSn+j2pnZmZTbBzBkVE/BLoGlH7eUQMpNkngcVpeh2wLSJORcRrQBvw8fRoi4j9EdEHbAPWSRLwaeCRtP5W4OaibW1N048Aq1N7MzObQONxjeKPgcfS9CLgYNGy9lQbqz4XOFoUOkP1M7aVlh9L7c8iaaOkVkmtnZ2dZQ/IzMxOKysoJH0DGAB+PD7dKU1E3B8RLRHR0tzcPJldMTObdqpKXVHSHwGfBVZHDN9M2gEsKWq2ONUYo34EaJRUlY4aitsPbatdUhUwO7U3M7MJVNIRhaS1wNeAz0XEyaJF24H16Y6lZcAK4FfA08CKdIdTDdkF7+0pYJ4AbknrbwAeLdrWhjR9C/B4USCZmdkEOecRhaSHgE8B8yS1A3eR3eVUC+xK15efjIh/HRF7JD0MvER2SuqOiBhM2/kKsBOoBLZExJ70FF8Htkn6NvAs8ECqPwD8SFIb2cX09eMwXjMze5/OGRQRceso5QdGqQ21vxu4e5T6DmDHKPX9ZHdFjaz3Ap8/V//MzOzC8m9mm5lZLgeFmZnlclCYmVkuB4WZmeVyUJiZWS4HhZmZ5XJQmJlZLgeFmZnlclCYmVkuB4WZmeVyUJiZWS4HhZmZ5XJQmJlZLgeFmZnlclCMyZ+RZGYGDoqzaLI7YGY2xTgozMws1zmDQtIWSYclvVhUmyNpl6R96WtTqkvSZkltkp6XdG3ROhtS+32SNhTVr5P0Qlpns9Jnq471HGZmNrHO54jih8DaEbVNwO6IWAHsTvMANwIr0mMjcB9kb/pkn7V9PdnHnt5V9MZ/H/ClovXWnuM5zMxsAp0zKCLil0DXiPI6YGua3grcXFR/MDJPAo2SFgI3ALsioisiuoFdwNq0rCEinoyIAB4csa3RnsPMzCZQqdco5kfEoTT9FjA/TS8CDha1a0+1vHr7KPW85ziLpI2SWiW1dnZ2ljAcMzMbS9kXs9ORwAW9l/RczxER90dES0S0NDc3X8iumJldckoNirfTaSPS18Op3gEsKWq3ONXy6otHqec9h5mZTaBSg2I7MHTn0gbg0aL6benup1XAsXT6aCewRlJTuoi9BtiZlh2XtCrd7XTbiG2N9hxmZjaBqs7VQNJDwKeAeZLaye5e+g7wsKTbgQPAF1LzHcBNQBtwEvgiQER0SfoW8HRq982IGLpA/mWyO6vqgMfSg5znMDOzCXTOoIiIW8dYtHqUtgHcMcZ2tgBbRqm3AlePUj8y2nOYmdnE8m9mm5lZLgeFmZnlclCYmVkuB4WZmeVyUJiZWS4HhZmZ5XJQmJlZLgeFmZnlclCYmVkuB4WZmeVyUJiZWS4HhZmZ5XJQjCEu6EcxmZldPBwUI2Qfi2FmZkMcFGZmlstBYWZmuRwUZmaWq6ygkPRnkvZIelHSQ5JmSFom6SlJbZJ+Iqkmta1N821p+dKi7dyZ6nsl3VBUX5tqbZI2ldNXMzMrTclBIWkR8O+Aloi4GqgE1gPfBe6JiCuAbuD2tMrtQHeq35PaIWllWu+jwFrg+5IqJVUC9wI3AiuBW1NbMzObQOWeeqoC6iRVATOBQ8CngUfS8q3AzWl6XZonLV+t7BajdcC2iDgVEa8BbcDH06MtIvZHRB+wLbU1M7MJVHJQREQH8JfAG2QBcQx4BjgaEQOpWTuwKE0vAg6mdQdS+7nF9RHrjFU/i6SNkloltXZ2dpY6JDMzG0U5p56ayP6Hvwz4AFBPdupowkXE/RHREhEtzc3Nk9EFM7Npq5xTT58BXouIzojoB34KfBJoTKeiABYDHWm6A1gCkJbPBo4U10esM1bdzMwmUDlB8QawStLMdK1hNfAS8ARwS2qzAXg0TW9P86Tlj0dEpPr6dFfUMmAF8CvgaWBFuouqhuyC9/Yy+mtmZiWoOneT0UXEU5IeAX4NDADPAvcD/wvYJunbqfZAWuUB4EeS2oAusjd+ImKPpIfJQmYAuCMiBgEkfQXYSXZH1ZaI2FNqf83MrDQlBwVARNwF3DWivJ/sjqWRbXuBz4+xnbuBu0ep7wB2lNNHMzMrj38z28zMcjkozMwsl4PCzMxyOSjMzCyXg8LMzHI5KMzMLJeDwszMcjkozMwsl4PCzMxyOSjMzCyXg8LMzHI5KMYQk90BM7MpwkExgia7A2ZmU4yDwszMcjkozMwsl4PCzMxyOSjMzCxXWUEhqVHSI5JekfSypE9ImiNpl6R96WtTaitJmyW1SXpe0rVF29mQ2u+TtKGofp2kF9I6m9Nnc19QrQe6L/RTmJldVMo9ovge8L8j4irgXwAvA5uA3RGxAtid5gFuBFakx0bgPgBJc8g+TvV6so9QvWsoXFKbLxWtt7bM/pqZ2ftUclBImg38HvAAQET0RcRRYB2wNTXbCtycptcBD0bmSaBR0kLgBmB
},
"metadata": {
"needs_background": "light"
}
}
],
"source": [
"plt.plot(lpath)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
]
}