ML-For-Beginners/8-Reinforcement/1-QLearning/solution/assignment-solution.ipynb

{
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "orig_nbformat": 2,
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.7.0 64-bit ('3.7')"
  },
  "interpreter": {
   "hash": "70b38d7a306a849643e446cd70466270a13445e5987dfa1344ef2b127438fa4d"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2,
 "cells": [
  {
   "source": [
    "# Peter and the Wolf: Realistic Environment\n",
    "\n",
    "In our situation, Peter was able to move around almost without getting tired or hungry. In more realistic world, we has to sit down and rest from time to time, and also to feed himself. Let's make our world more realistic, by implementing the following rules:\n",
    "\n",
    "1. By moving from one place to another, Peter loses **energy** and gains some **fatigue**.\n",
    "2. Peter can gain more energy by eating apples.\n",
    "3. Peter can get rid of fatigue by resting under the tree or on the grass (i.e. walking into a board location with a tree or grass - green field)\n",
    "4. Peter needs to find and kill the wolf\n",
    "5. In order to kill the wolf, Peter needs to have certain levels of energy and fatigue, otherwise he loses the battle.\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import random\n",
    "import math\n",
    "from rlboard import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": "<Figure size 792x432 with 1 Axes>",
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"360.646265pt\" version=\"1.1\" viewBox=\"0 0 366.6475 360.646265\" width=\"366.6475pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;white-space:pre;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 360.646265 \nL 366.6475 360.646265 \nL 366.6475 0 \nL 0 0 \nz\n\" style=\"fill:none;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 33.2875 336.76814 \nL 359.4475 336.76814 \nL 359.4475 10.60814 \nL 33.2875 10.60814 \nz\n\" style=\"fill:#ffffff;\"/>\n   </g>\n   <g clip-path=\"url(#pb71df41202)\">\n    <image height=\"327\" id=\"imagec1d069daa2\" transform=\"scale(1 -1)translate(0 -327)\" width=\"327\" x=\"33.2875\" xlink:href=\"data:image/png;base64,\niVBORw0KGgoAAAANSUhEUgAAAUcAAAFHCAYAAAAySY5rAAAABHNCSVQICAgIfAhkiAAAIABJREFUeJzs3XmcZFV99/HPOefeW7dubb3NzjLAAINsgiLiggoaJC4RohKVaAI+uAOCrzzRJ+YRCYlbQmJIRBJNAopoRBCNEpAdBgUU2ZdBZoAZmJneu7r2e895/rhdlxmpxmdgXk/1PPN7Q72qp6u6+szpqW+f7Z6jTjzxRLdixQoWqlarxezsLMPDw/0uyrymp6fxPI9CodDvosxr8+bNbN48gnOm30WZ18DARvbcc+H+W0yShPHxcRYvXtzvosyrVquRJAnlcrnfRZnX+Pg4xWKRXC7X76LMa+PGjXDGGWe4hWzt2rXuoosu6ncxntcVV1zh1qxZ0+9iPK9zzz3X5XITDtwCvVn37nd/vN/V9LzGx8fdeeed1+9iPK9bb73VXXnllf0uxvO68MIL3dq1a/tdjOd1+umnO93vhBZCiIXI63cBhNgZWWsBUErhnPud9845Oo0GSaeTvYbxfaxNsInNPucFAZ12e5vvFYQhQRjO+/rdcuxsuvXy23+XXp/b+mvme2xH2Po1JRyFeAEmJiZIkoRisUi1WmV4eJjp6WmGh4eZnJykVCqxZcsWBgcHmZ2tcu9P/ovbL/kPHrvrDiAdSNj7yCPZ8uSTTGzcSGIdiYOXvP4N/Pq664mtw6II8nle90fv5g8+/glKixYxsmgR9XqdcrnM+Pg4Q0NDKKV2ynBsNps0Gg1yuRytVgutNcYY2u02URTRaDQIgoBOp4NSCs/zaLVaRFFEvV4nCAKSJMH3fZxzAHieR7PZJJ/P02w2CYIAay3OOTzPo9FoEEVR9phzjiRJMMaQz+e3KZ+EoxDb6bffWFu/2WZnZ/F9n2azSalUwlrLHZdezI//6hyctYSm2zqCjXfdQeIgNJpEORLnWHvzjeQ0GKWIHYRRnl9d+zMmNj7NB7/8ZVrlMlprZmdnyeVy2X0Yhn2ule0XxzG+72OtxfO8LOCDIMge64YapK1G3/eJ45ggCAAwxmSteOccnU4nC1itNXEcZ9+v0+nged42j3VboEmSPKd8MuYoxHYyxpAkCUmSoLUmSRI8z8Nam73Zfd+n3W5zyzcu5Ka//wqhhrynyXua0DOEvib0NHmjCD1FOPf5nKfJeYacUeS0IzCKyuAg9eoMXz/jDOzcGzqXy5EkCUEQZOGxs+m2BJVSNJvNLKyazSbGGJrNJgDtdjsLue5jjUYDay3tdnuboQXf9+l0OtsE7Nah232s+zMzxuCcw5jnruKQcBRiO2mdvm263dluSHbfZM45tNasu+N27rr4X/E6TfK+zm6Rl97ynib0NXnPpI8ZyBs1F6BpSHamJ5l9ZgNRPo9zlv/5pt/LymCtzcqyM+rW2dZ/h+544tb1CM+ONVpr2bJlCz/96U/ZY4892GuvvVixYgW77bYb3/jGN9i0aRNAFnZaa7TWKKW2+Rl1n9N9rNewxM5bs0L0SavVwvd9fN8nSRLy+Xw2PjY7O0sQBFQnJxi7/Wbs6DPkPZUGotk2GLufz3uavNGEviFndNqSNJqcMQRa0Z6ZZstjaxlevISoVOTxX/+aarVKLpejVqvR2WqSZ2fSbDYpFArEcUyhUMi6ysViMRuWSJKEMAzJ5XI453jooYc47LDD+JM/+RNmZmaoVqvMzMwwMzPDJz/5Sfbff39uv/12qtUqQRBQr9ezFmmtViMMQ6rVKp7nUa/XabVaQNo6/W0SjkJsp3w+T71ep9FoYIyhXq9nb+hKpUKz2UTFbe655F/TVqEx5D1D5Kf3YRaOJm0lGjXXUky71oFJP84ZRTB3m9m0kcfvuoPlK1fys3/7NyqlIo1Gg4GBgZ1yvBEgiiJmZmbwfZ+ZmZmsGz01NUUURUxNTWGMoVar0Ww2ue222/jABz7A1NTUvK/Zbrf54Ac/yI033kir1aJUKmVd63K5TKPRYHBwkE6nQ7FYzCZhetWhhKMQ26lWq1EqlSgWi8RxTLFYZHp6miiKGB8fJ4oifvqxPyXnp13j/Fwo5j2V3gdp9/rZQEwDMm8g9Byhp9OANOnnfaMJPMPkxidZe9stNOs1rrnsMkqlEmNjY9Tr9X5XyQtSrVYZHByk3W4zODiYBdXIyAjVapWRkZHsap8oirjzzjt56qmnfufrbtq0iRtuuIEwDJmamqLVamGtZWJigkKhwNjYGEEQMD09Ta1WA+hZhzvnSK4QfWSMIY7jbKwqjmM8zyNJEnK5HFOP/4b6k+vIewajFFortAOlNA6wgLUOqx0WRxw7Yg2x1Rhl8QCNQykLiZ77CkAbmjNT2Habdffcw8vf9HvkCwV83+9fZbwIpVKJiYkJyuUyU1NTBEGA7/tMTEwwNDTE+Pg4pVKJarWK1pooirKvXbRoEbvvvvs245IbNmxgy5YtOOeypTyVSoVOp0OSJAwODjI5OcnQ0BAzMzOUy+VsUue3l/GAhKMQ280YQ6fT2WaSoDtBopTiri//JbpVI/B0Go4K9Fbj/daB1WCdInHQUQ7POTqJRScGRYJDgVI4p8GBdRZnoFWdYctjj/DkQw9y7PtOZunee2dr/HY2rVaLMAyJ45h8Pp9NiuTz+Wyto7WWXC7H6Ogot9xyCwCFQoELLriA448/fpvF4D/+8Y/5+Mc/zsTEBL/85S955plnsnFM51w2Llyv1zHG0Gq1srqL4/g513pLt1qI7dRqtcjn8+Tzeay12yw4fuK2G2lufDKdYPE0odHZ7HPoeYSeN9fFNs8+bhQ5rQi0zsYYAwO+Tj/2jcLTCl8rPK0ZW7+OvQ86kB//8wXMzMz0nEzYGWz9CyVJkiyotl4i1f3ls27dOn74wx8C6fjgG9/4RoCs5aiU4m1ve1u2Qc11113H/fffv81sdHdN5NYz4c93hZGEoxDbKYoiqtVqNutZrVazwX67fi129GlCTxPlAgr5gLyniTwzN1Ot5sYe0
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAFpCAYAAAC8p8I3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzdeZxcVZ3//9fn1l7V3dV7J2QjIexBwhaIC6MgyKACg47iyogzqD9QZ8YZdUZnXJDBr8vgMF8V40hEXFBHWYavy2AGR1lEQCEkbAkkgSSdpbu6u/a6yzm/P+p209F09k5VJZ8nj3pU1b23qj65Tb9zcu45p8Rai1JKqdbhNLoApZRSe0eDWymlWowGt1JKtRgNbqWUajEa3Eop1WI0uJVSqsVMW3CLyAUi8rSIrBWRj07X5yil1OFGpmMct4hEgGeA84CNwEPAW6y1TxzwD1NKqcPMdLW4lwBrrbXPWWtd4Fbg4mn6LKWUOqxMV3DPAl6Y9HxjuE0ppdR+ijbqg0XkSuBKgFgsdtpLXvKS/Xq/kZERPM+b/P709fXt13uOq9VqFAoFent7D8j7TYfR0VFisRiZTKbRpUxpcHCQ/v5+IpFIo0uZ0vPPP8/cuXMbXcaUfN9n+/btzJw5s9GlTKlYLOL7Pp2dnY0uZUrbt2+no6ODRCLR6FKmtHr1aiqViux0p7X2gN+ApcDPJz3/B+Afpjq+v7/f7o9bbrnF9vT0WGDiFo1G7T/90z/t1/uOW7NmjV22bNkBea/pctttt9n777+/0WXs0jXXXGNzuVyjy5iSMcZeffXVjS5jl4aHh+21117b6DJ26d5777W33357o8vYpRtvvNGuWbOm0WXsUpiLO83M6WpxPwQcLSLzgU3AZcBbD/SH+L7P97//fT7wgQ8wMjLyR/u+8IUvAPCRj3yEdDqNyM7/8lJKqVYyLX3c1lofuBr4OfAk8ANr7eoD/TmbNm3iHe94xx+F9rhKpcJnPvMZfvGLXxzoj1ZKqYaZtj5ua+1PgJ9M1/sDbN68GcdxCIJgymNEhK1btxIEAdFow7r0lVLqgGnpmZMPPvjgLkMbwBjD73//+x0uXCqlVCtr6eB+wxvesNsRCo7jcOGFF5JMJg9SVUopNb1aOrhjsRiLFy/e5THz58+np6fnIFWklFLTr6WDu6+vj6uuumqXx1x44YWceuqpOqJEKXXIaOngdhyHiy++mDvvvJNjjz12h32ZTIbvfe97fPjDH27qQfZKKbW3Wjq4oT5q5Ic//CHPPPPMDttLpRKf//znxycATdwrpVSra+ngXrt2LR/60Ie45ZZbdhrMv/vd77jiiit48MEHMcY0oEKllDrwWjK4jTE8/fTTfPCDH2T58uW7PPYXv/gF733ve/nNb36z26GDSinVCloquK21VKtVPvvZz/Lyl7+cn//853v0uscee4yLLrqIt7/97eTz+clrqiilVMtpqamErutyww038LGPfWyvX5vL5bj11ltJpVJ87nOf0yGCSqmW1VLB/bnPfY5PfOIT+/Uey5cvJxaL8eUvf1mnwCulWlJLdJVYa7nuuuu47rrrDkgXx/Lly/mLv/gLvWCplGpJTR/cruvyb//2b3zyk5+kUqnssO/kk0/eoynvxx9//A6ta8/zuPXWW7nyyivJ5/PTUrdSSk2Xpg5uay1f/vKX+fCHP4zrujvsO+ecc/jhD3+4R8H9pS99ife9730sWbJkYnsQBHzzm9/kox/9KMVicVrqV0qp6dDUwf2///u/fPzjH99hZb+5c+dy3XXX8fWvf51sNrvH73X99ddz/fXXc9ZZZ01Mfw+CgBtvvJFbb71VR5kopVpG0wa3MYbvfve7VKvViW19fX3ccMMN/O3f/i3z58/fq/dzHIclS5bwxS9+kcnfb2mtZfny5drfrZRqGU0b3CLCW97yFhYtWgTAMcccw3/8x3/w+te/nng8vteLRokI0WiUpUuXsnz5cpYsWYKIMHfuXK688kocp2lPhVJK7aBpx8OJCK985StZtmwZd9xxB5deeimnn376Hx23t10cIsIpp5zCHXfcwb//+7+zdOlSzj//fF09UCnVMpo2uMctWbKEM844Y6fBWi6X8X1/l68PgoByuYy1dof3GBgY4JprrtHAVkq1nKbuHxARRATHcXYasDNnzuTTn/70Lt/jLW95Cy9/+ct3+t7j76vhrZRqJU0d3LsTiUTo6ura5TFtbW0kk0kNZ6XUIaOlg1sppQ5HGtxKKdViNLiVUqrFaHArpVSL0eBWSqkWo8GtlFItRoNbKaVajAa3Ukq1mJYObmvtbqe8G2N05T+l1CFlv4JbRNaLyOMi8qiIPBxu6xaRu0VkTXi/66mN+yEajTJv3ryJWZHJZJJFixbtMJuyv7+f9vb26SpBKaUOugOxyNSrrLVDk55/FFhhrf2siHw0fP6RA/A5O3XkkUdy+eWXU6vVWLBgAZ/5zGe46aabWLFiBZFIhDPOOGO6PloppRpiOlYHvBh4Zfj4ZuCXTFNwiwiLFy9m+fLlO2y/4ooruOKKK6bjI5VSquH2t4/bAv8tIo+IyJXhtgFr7WD4eAswsJ+foZRSapL9bXG/3Fq7SUT6gbtF5KnJO621VkR2+k0HYdBfCfUV/NasWbOfpUyfjRs3Mjo62tQ1Dg0NYYxp6hpLpRLr1q1jaGho9wc3iOu6TX0O8/k8pVKpqWvcsmVL0/++jI6O8sILLzT1d83ualDFfgW3tXZTeL9NRG4DlgBbRWSmtXZQRGYC26Z47TJgGUBPT4/95S9/uT+lTKvR0VE2btxIM9f47LPPkk6nGR4ebnQpUxoaGuL+++8nkUg0upQpFYvFpv45V6tVHtj+AHf88o5GlzKl9GCacyvnNvVork2bNvHII4+wdu3aRpcypV2eP2vtPt2ADNA+6fH9wAXA54GPhts/Cnxud+/V399vm9maNWvssmXLGl3GLt122232/vvvb3QZu3TNNdfYXC7X6DKmZIyxV199daPL2KXh4WF72rWnWZr4vxn3zrC33357o0/VLt144412zZo1jS5jl8Jc3Glm7k+LewC4LRyKFwW+a639mYg8BPxARN4NbADetB+foZRS6g/sc3Bba58DTt7J9mHg3P0pSiml1NRaeuakUkodjjS4lVKqxWhwK6VUi9HgVkqpFqPBrZRSLUaDWymlWowGt1JKtRgNbqWUajEa3Eop1WI0uJVSqsVocCulVIvR4FZKqRajwa2UUi1Gg1sppVqMBrdSSrUYDW6llGoxGtxKKdViNLiVUqrFaHArpVSL0eBWSqkWo8GtlFItRoNbKaVajAa3Ukq1GA1upZRqMRrcSinVYjS4lVKqxWhwK6VUi9HgVkqpFqPBrZRSLUaDWymlWsxug1tEbhKRbSKyatK2bhG5W0TWhPdd4XYRkRtEZK2IrBSRU6ezeKWUOhztSYv7m8AFf7Dto8AKa+3RwIrwOcCfAkeHtyuBrx6YMpVSrUREGl3CIW23wW2t/RWQ+4PNFwM3h49vBi6ZtP1btu43QKeIzDxQxSqlWoO1ttElHNL2tY97wFo7GD7eAgyEj2cBL0w6bmO4TSml1AGy3xcnbf2v1r3+61VErhSRh0Xk4Uqlsr9lKKXUYWNfg3vreBdIeL8t3L4JmDPpuNnhtj9irV1mrT3dWnt6KpXaxzKUUurwE93H190JXA58Nry/Y9L2q0XkVuBMYGxSl8qUgiDg9ttv38dSpt/Q0BDPPvtsU9e4atUqNmzYwNatWxtdypS2bNnCz372M5r5L+p8Pt/UP+dyuUxmMMOC2xc0upQpta9vZ1VpVVP3cz/33HNEo1FWrVq1+4MbJAiCKfftNrhF5HvAK4FeEdkIfIJ6YP9ARN4NbADeFB7+E+BCYC1QBt61JwW6rvC+9w3s/sAGSacNl1+eZmCgeWvcsGEDN96YZXS0eWtcuDDBJZf0kclkGl3KlKLRaFP/nIvFImckzuCzA59tdClTemrkKQpOoanPYzqd5l+6/4XyQLnRpUzJFXfKfbsNbmvtW6bYde5OjrXAVXtc2cTrHLZ
     },
     "metadata": {
      "needs_background": "light"
     }
    }
   ],
   "source": [
    "width, height = 8,8\n",
    "m = Board(width,height)\n",
    "m.randomize(seed=13)\n",
    "m.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "actions = { \"U\" : (0,-1), \"D\" : (0,1), \"L\" : (-1,0), \"R\" : (1,0) }\n",
    "action_idx = { a : i for i,a in enumerate(actions.keys()) }"
   ]
  },
  {
   "source": [
    "## Defining state\n",
    "\n",
    "In our new game rules, we need to keep track of energy and fatigue at each board state. Thus we will create an object `state` that will carry all required information about current problem state, including state of the board, current levels of energy and fatigue, and whether we can win the wolf while at terminal state:"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class state:\n",
    "    def __init__(self,board,energy=10,fatigue=0,init=True):\n",
    "        self.board = board\n",
    "        self.energy = energy\n",
    "        self.fatigue = fatigue\n",
    "        self.dead = False\n",
    "        if init:\n",
    "            self.board.random_start()\n",
    "        self.update()\n",
    "\n",
    "    def at(self):\n",
    "        return self.board.at()\n",
    "\n",
    "    def update(self):\n",
    "        if self.at() == Board.Cell.water:\n",
    "            self.dead = True\n",
    "            return\n",
    "        if self.at() == Board.Cell.tree:\n",
    "            self.fatigue = 0\n",
    "        if self.at() == Board.Cell.apple:\n",
    "            self.energy = 10\n",
    "\n",
    "    def move(self,a):\n",
    "        self.board.move(a)\n",
    "        self.energy -= 1\n",
    "        self.fatigue += 1\n",
    "        self.update()\n",
    "\n",
    "    def is_winning(self):\n",
    "        return self.energy > self.fatigue"
   ]
  },
  {
   "source": [
    "Let's try to solve the problem using random walk and see if we succeed:"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "metadata": {},
     "execution_count": 5
    }
   ],
   "source": [
    "def random_policy(state):\n",
    "    return random.choice(list(actions))\n",
    "\n",
    "def walk(board,policy):\n",
    "    n = 0 # number of steps\n",
    "    s = state(board)\n",
    "    while True:\n",
    "        if s.at() == Board.Cell.wolf:\n",
    "            if s.is_winning():\n",
    "                return n # success!\n",
    "            else:\n",
    "                return -n # failure!\n",
    "        if s.at() == Board.Cell.water:\n",
    "            return 0 # died\n",
    "        a = actions[policy(m)]\n",
    "        s.move(a)\n",
    "        n+=1\n",
    "\n",
    "walk(m,random_policy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Killed by wolf = 5, won: 1 times, drown: 94 times\n"
     ]
    }
   ],
   "source": [
    "def print_statistics(policy):\n",
    "    s,w,n = 0,0,0\n",
    "    for _ in range(100):\n",
    "        z = walk(m,policy)\n",
    "        if z<0:\n",
    "            w+=1\n",
    "        elif z==0:\n",
    "            n+=1\n",
    "        else:\n",
    "            s+=1\n",
    "    print(f\"Killed by wolf = {w}, won: {s} times, drown: {n} times\")\n",
    "\n",
    "print_statistics(random_policy)"
   ]
  },
  {
   "source": [
    "## Reward Function\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def reward(s):\n",
    "    r = s.energy-s.fatigue\n",
    "    if s.at()==Board.Cell.wolf:\n",
    "        return 100 if s.is_winning() else -100\n",
    "    if s.at()==Board.Cell.water:\n",
    "        return -100\n",
    "    return r"
   ]
  },
  {
   "source": [
    "## Q-Learning algorithm\n",
    "\n",
    "The actual learning algorithm stays pretty much unchanged, we just use `state` instead of just board position."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "Q = np.ones((width,height,len(actions)),dtype=np.float)*1.0/len(actions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def probs(v,eps=1e-4):\n",
    "    v = v-v.min()+eps\n",
    "    v = v/v.sum()\n",
    "    return v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      ""
     ]
    }
   ],
   "source": [
    "\n",
    "from IPython.display import clear_output\n",
    "\n",
    "lpath = []\n",
    "\n",
    "for epoch in range(10000):\n",
    "    clear_output(wait=True)\n",
    "    print(f\"Epoch = {epoch}\",end='')\n",
    "\n",
    "    # Pick initial point\n",
    "    s = state(m)\n",
    "    \n",
    "    # Start travelling\n",
    "    n=0\n",
    "    cum_reward = 0\n",
    "    while True:\n",
    "        x,y = s.board.human\n",
    "        v = probs(Q[x,y])\n",
    "        while True:\n",
    "            a = random.choices(list(actions),weights=v)[0]\n",
    "            dpos = actions[a]\n",
    "            if s.board.is_valid(s.board.move_pos(s.board.human,dpos)):\n",
    "                break \n",
    "        s.move(dpos)\n",
    "        r = reward(s)\n",
    "        if abs(r)==100: # end of game\n",
    "            print(f\" {n} steps\",end='\\r')\n",
    "            lpath.append(n)\n",
    "            break\n",
    "        alpha = np.exp(-n / 3000)\n",
    "        gamma = 0.5\n",
    "        ai = action_idx[a]\n",
    "        Q[x,y,ai] = (1 - alpha) * Q[x,y,ai] + alpha * (r + gamma * Q[x+dpos[0], y+dpos[1]].max())\n",
    "        n+=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": "<Figure size 792x432 with 1 Axes>",
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"360.646265pt\" version=\"1.1\" viewBox=\"0 0 366.6475 360.646265\" width=\"366.6475pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;white-space:pre;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 360.646265 \nL 366.6475 360.646265 \nL 366.6475 0 \nL 0 0 \nz\n\" style=\"fill:none;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 33.2875 336.76814 \nL 359.4475 336.76814 \nL 359.4475 10.60814 \nL 33.2875 10.60814 \nz\n\" style=\"fill:#ffffff;\"/>\n   </g>\n   <g clip-path=\"url(#p3155e0caf7)\">\n    <image height=\"327\" id=\"image3956b3e364\" transform=\"scale(1 -1)translate(0 -327)\" width=\"327\" x=\"33.2875\" xlink:href=\"data:image/png;base64,\niVBORw0KGgoAAAANSUhEUgAAAUcAAAFHCAYAAAAySY5rAAAABHNCSVQICAgIfAhkiAAAIABJREFUeJzsnXecVNXZx7+3zp07ZWcbvayAgqIoKKDGrokaG9iIDRMglmiEiMaoSV4VsWDDvCS2qG8sRGOLLRoNgghiEFERBKVJk4XtZfrce94/ZufK6mICzDILnC/MZ3Zm7pz73Dtzf/Oc5zznOcqZZ54punfvTkclmUzS3NxMaWlpoU3ZKg0NDei6TiAQKLQpW2XTpk1s2lSGEFqhTdkqkcgGevfe8e+i4zisWbOGPn36sHLlSioqKmhsbMR13R36HjmOQ01NDZ06ddphG9uLaDSK4ziEw+FCm7JVampqCAaD+Hy+QpuyVTZs2ADjx48XHZnly5eLhx9+uNBmfC8vvfSSeP/99wttxvcyadIk4fPVChAd9OaKc8+9Mi/HGo1Gxa9//Wsxf/58cdxxx4m1a9eKyZMni/fee2+H2q2pqRGTJ0/Oi43txZw5c8Tf//73QpvxvTz44INi+fLlhTbje7nqqquEWmiFlkjyjW3bHH/88YwZM4bFixfz85//nCVLlnDEEUcU2jTJLoReaAMkkvbgyCOP5PXXX+e8887jnnvuoXPnznlt33VdABRFQQjxH++FEKTjcZx02mtDMwxc18F1XO853TRJp1Kt9mVaFqZlbbX9nB27Grnz8u1jaeu5Ld+ztdfywZZtSnGU7Jb4/X569eqFZVn06NGDoqKivLZfW1uL4zgEg0GampooLS2loaGB0tJS6urqCIVCbN68meLiYpqbm1j0j9eZ9+RfWLFgPpANJPQZPpzNa9dSu2EDjitwBOx3zLF8MuMdMq7ARcH0+zn6J+dyxpW/JFReTll5ObFYjHA4TE1NDSUlJSiKskuKYyKRIB6P4/P5SCaTqKqKpmmkUils2yYej2OaJul0GkVR0HWdZDKJbdvEYjFM08RxHAzDQAgBgK7rJBIJ/H4/iUQC0zRxXRchBLquE4/HsW3be00IgeM4aJqG3+9vZZ8UR4lkG/n2hbXlxdbc3IxhGCQSCUKhEK7rMn/6E7x2680I18XSct4RbFgwH0eApak4isARguWzZ+FTQVMUMgIs28/Ct/9F7YavGXfXXSTDYVRVpbm5GZ/P591bllXgs7LtZDIZDMPAdV10XfcE3jRN77WcqEHWazQMg0wmg2maAGia5nnxQgjS6bQnsKqqkslkvP2l02l0XW/1Ws4DdRznO/bJmKNEso1omobjODiOg6qqOI6Druu4rutd7IZhkEqleO/RB3l36t1YKvh1Fb+uYukalqFi6Sp+TcHSFayW5326ik/X8GkKPlVgagpFxcXEmhp5aPx43JYL2ufz4TgOpml64rGrkfMEFUUhkUh4YpVIJNA0jUQiAUAqlfJELvdaPB7HdV1SqVSr0IJhGKTT6VYCu6Xo5l7LfWaapiGEQNO+m8UhxVEi2UZUNXvZ5LqzOZHMXWRCCFRVZfX8eSx44s/o6QR+Q/Vutp69+XUVy1Dx61r2NQ38mtIioFmRTDfU0bxxPbbfjxAu1/3wR54Nrut6trQHtbW1nkC1B7lztuUx5OKJW55H+CbW6Loumzdv5o033qBXr17stddedO/enR49evDoo49SWVkJ4ImdqqqoqoqiKK0+o9w2udfaCktIcZRItpFkMolhGBiGgeM4+P1+Lz7W3NyMaZo01dVSPW82btVG/LqSFUSttTDmnvfrKn5NxTI0fJqa9SQ1FZ+mYaoKqcYGNq9YTmmnztihIKs++YSmpiZ8Ph/RaJT0FoM8+eSuu+5i/vz57dI2ZL3AQCBAJpMhEAh4XeVgMOiFJRzHwbIsfD4fQgiWLl3K4MGD+elPf0pjYyNNTU00NjbS2NjIr371K/r378+8efNoamrCNE1isZjnkUajUSzLoqmpCV3XicViJJNJIOudfhspjhLJNuL3+4nFYsTjcTRNIxaLeRd0UVERiUQCJZPi0yf/nPUKNQ2/rmEb2XvLE0ct6yVqSounmO1am1r2b5+mYLbcGis3sGrBfLpVVPCvxx+nKBQkHo8TiUR2yXgjZFOuGhsbMQyDxsZGz0utr6/Htm3q6+vRNI1oNEoikWDu3LlcfPHF1NfXb7XNVCrFuHHjmDVrFslkklAo5HWtw+Ew8Xic4uJi0uk0wWDQG4Rp6xxKceygOI7Tbh7BnkRuFDOfRKNRQqEQwWCQTCZDMBikoaEB27apqanBtm3euOJn+Ixs19jfIop+Xcnem9nu9TeCmBVIvwaWLrB0NSuQWvZ5Q1MxdY26DWtZPvc9ErEobz3zDKFQiOrqamKxWN6PEbIxwZzX1R40NTVRXFxMKpWiuLjYE6qysjKampooKyvzZvvYts2HH37IunXr/mO7lZWVzJw5E8uyqK+vJ5lM4routbW1BAIBqqurMU2ThoYGotEoQJvncNeM5O7mfPrppyxevJgVK1Zw8sknM2TIkF026F5o2iPFRdM0MpmMF6vKZDLouo7jOPh8PupXrSS2djV+XUNTFFRVQRWgKCoCcAHXFbiqwEWQyQgyKmRcFU1x0QEVgaK44Kgt7wBUjURjPW4qxepPP+WQH/4IfyCAYRh5P0aAm266iR/+8IcMGzaMYDCY9/ZDoRC1tbWEw2Hq6+sxTRPDMKitraWkpISamhpCoRBNTU2oqopt2957y8vL6dmzZ6u45Pr169m8eTNCCC+Vp6ioiHQ6jeM4FBcXU1dXR0lJCY2NjYTDYW9Q59tpPCA9xw7JqFGjqK+v59BDD+Xcc8+lqamp0CZJtiA3WOC6rjdIkBsgURSFBXf9HjUZ9Uan/ZqC38iOSvt1pWXQpWXgRdPwGdnRa5+u4NM0DA1MVcGnKRiqiqGqmKqCoSkkmxrZvOILPnr1FWo3bvRy+NrrOHNpMu1BMpnEsiwymQx+v99zAPx+v5fr6Lpu9genvp733nsPgEAgwLRp05g1axazZs1i5syZzJw5k
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAFpCAYAAAC8p8I3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzdd3xUVd7H8c+Zmt5J6CC9qVQpKoqIoggIuouCoojCIkXQXUV3F0XEh10bYAFBUQFBwRXXroggIjYiKkWE0EsI6Zle7j3PH5mMsBBqkjsJ580rr2Tu3Mz9zoT8cubcc88RUkoURVGU6sNkdABFURTlzKjCrSiKUs2owq0oilLNqMKtKIpSzajCrSiKUs2owq0oilLNVFrhFkL0FUL8LoTIEkJMrqzjKIqinG9EZYzjFkKYge1AH+AA8CNwq5Rya4UfTFEU5TxTWS3uS4AsKeUuKaUfeAsYWEnHUhRFOa9UVuGuB+w/6vaB0DZFURTlHFmMOrAQYhQwCsBqtXa66KKLjIpySj6fD4fDQVpamtFRylVUVITVaiU2NtboKOXKzs4mPT0ds9lsdJRy7du3j4YNGxodo1zBYJDc3Fzq1KljdJRyOZ1OgsEgSUlJRkcpV25uLgkJCdjtdqOjlGvLli14PB5xwjullBX+AXQHPjvq9sPAw+Xtn56eLiPZjh075Lx584yOcVIrVqyQ69evNzrGSU2bNk0WFBQYHaNcuq7LcePGGR3jpPLz8+X06dONjnFS69atk++9957RMU5q7ty5cseOHUbHOKlQXTxhzaysrpIfgeZCiAuEEDbgFuD9SjqWoijKeaVSukqklEEhxDjgM8AMLJBSbqmMYymKopxvKq2PW0r5MfBxZT2+oijK+UpdOakoilLNqMKtKIpSzajCrSiKUs2owq0oilLNqMKtKIpSzajCrSiKUs3U2MK9cOHCsqs2FaXKeb1eli9fbnSM0/L222/j9/uNjlGut956K6LzGaHGFe6VK1cycOBAgsEggwYNYvHixYbkkFJy3333GXJsxVhPPfUUw4cPJzc3l4EDB7Jp0yajI53QTz/9xMCBAyksLGTo0KHMnDnT6EjHyMzMZODAgRQVFTF06FBmzZpldKSIUaMKdzAYJCsrixtuuIF+/foxatQotm3bhs/nq9IcL7zwAq1ateKWW26hZcuWzJkzp0qPX9mklBQUFBgdIyK53W42bdrEI488wk033cTFF1/Mnj170HXd6GjH0HWdPXv20LlzZwYNGsTkyZPZvHkzbrfb6GhAab7du3fTtWtXBg0axIMPPsjmzZvxeDxGR4sINapwHz58mF27dtG3b1/69u1Lly5dsNlsbN68ucoyFBQUUFRUxKJFi/D5fCxcuJCCggIKCwurLENlW716NdOnTzc6RkRavXo1HTt2JD09ndtvv52hQ4fy0Ucf4fV6jY52DJfLxRdffMGQIUO45ZZbqF+/Pq1bt2bdunVGRwNKZxhcvXo1f/rTnxgyZAiNGjWiVatWfP3110ZHiwiGTetaGerXr0/Lli0ZPXo0hw4dYvz48TRt2pROnTpVWQaPx4OmacTFxfH999+zb98+4uLi8Hg8JCcnV1mOyrJixQq2b9/Ov//9b6OjRKR+/foxZMgQfv/9d6644gruuusupk+fTkxMjNHRjhEfH8+AAQMYOXIke/fu5aGHHkJKyQMPPGB0NAASEhLo168fI0eOZM+ePTz44IMAEZPPaDWqcAP079+fNm3acOeddzJ+/HgaNGhQpcevV68eMTEx/OlPf2LZsmVcdNFFNG7cmMGDB5ORkRHRc1GXR0qJrut8+eWX7Ny5k3HjxlXJ8whPYWmqXm8Mp02bxv79+5k4cSLdunWjY8eORkc6oa5du/Kvf/2LcePGcfvtt9OoUSOjIx2je/fuzJgxg3HjxvHll1/y5ZdfGh0pYlSv34jTkJGRQY8ePYiNjaV79+6GTIo/ceJEfvjhB6ZOnYrD4WDTpk38+9//ZtCgQWRnZ1d5nnO1f/9+rrzySj7//HPuu+++KlmsYf/+/Xz33Xfce++9bNu2rVr1bbZo0YKrrrqKWbNm0aBBAyyWyGwfJScn06NHDxITE+natSvNmzc3OtIxyvKtW7eOtLS0iMhXUFCAw+EwOkbNa3FHAqvVitVqZdmyZeFtixYt4uDBgzz//PM0adKEP//5zyQkJBiY8vT89NNPvPnmm3z66adVurrOqFGjaNq0Kffccw8333wzr7/+Op07d66y458rIQRXXXUVq1ev5ocffqBXr15GR6q2oqKijI5AMBhk4cKFHDx4ELvdTt26dRk2bBhCnHiBmsqmCncVqlevHnfeeSfffvstjz76KM8++6xhP/jTsXXrVt5//30eeOCBKi3aH3zwAX369GHo0KE89NBD6LrOY489Rmpq6nH7Dho0iBtvvLHKsinnp2AwyIwZM3jzzTdxOp2MGjWKYcOGHbPP119/zSuvvHLC7+/cuTPjx4+vsDw1snA/+OCDzJ49OyKLYosWLWjRogVdu3Zl5MiRzJo1i+jo6Ih7O33kyBFmzJjBc889d8KCWZkuvfRSlixZwk033cTkyZN54oknuPnmm2nXrt1x+y5fvpypU6ee9PE+++wz0tPTKyvuSU2cOJHRo0fTuXNn4uPjDclwKgsXLuTWW2/lww8/NDrKCQkhmDVrFg8++KBhJ8VHjBjBf//7XzZs2MCzzz7LgQMHjjt30atXL/7xj3+c8Pvj4uIqNE9kVYsKkpOTQ+3atSOycJdp2bIl48ePp3///gwfPpzbb78dq9VqdCwAtm3bxrRp01i0aJEhJwZTUlKoX78+48aN4/rrr+fAgQO0adPmhH2cDz/8MJMnTz7p4xn5/yA1NZXCwsKIG8d9tHr16nHw4EGjY5RLCEHt2rXJyckxLMOrr75K9+7deffdd+nQoQM333wzmZmZx+1XVb8vNbJwVwdCCDp06MCaNWt4/fXX+de//kWbNm0YPHiwYZmOHDnC/Pnz8fv9PP/884aO5njqqacoKSnhww8/5Kuvvip3PyFERP+BBhg6dCiLFi1i3LhxRkdRzpLVamXAgAHhqTSGDx9u6P89VbgjwB133MGqVavCfcr9+/evkv8QUkrWr1/PkSNH6N+/P5MnT+ZPf/oTF198MSkpKZV+/FNJSEhg6NChRsc4ZyNGjKBjx46qcJ8jIQRSSkOKpdVqZdq0aezevZuYmBgyMjKqPMPRatxwwOpICMHVV1/N6NGjyczMpFOnTpX+1lVKSZs2bVi6dCnr1q2jXr16/N///R/XXXcddevWrdRjn2+EEKSmpqppAs5B06ZNueyyy3jzzTcNzXHBBRcYXrRBFe6IYrfbmTp1Kj/88APPP/98pR5r/fr19O7dm4cffpioqCh69uzJxo0bK/WY5yshBPPnz6/QUQXnG7PZjNVqjbipA4xSI7tKhg0bRq1atYyOcdYsFgszZsyo1GPk5eWRlpaGx+Ph5ptvZvXq1eTm5lbqMc9nqamp3H777UbHOCEhBH/729+MjnFKXbp0UVM1h9TIwn3NNdcYHSHiDRgwgAkTJuDxeMjIyODJJ5/kwIEDRseqsRITE+nbt6/RMU5ICFEtziW0adPG6AgRo0YWbuXUhBDs2LGDNWvWkJOTw8GDB7HZbEbHUhTlNKjCfR6z2Wzq3YmiVEPq5KSiKEo1owq3oihKNaMKt6IoSjWjCreiKEo1owq3oihKNXNOo0qEEHsAB6ABQSllZyFECvA20BjYA/xZSllzVspVFEUxWEW0uHtJKdtLKcuWJ5kMrJJSNgdWhW4riqIoFaQyukoGAm+Evn4DUMuTKIqiVKBzLdwS+FwIkSmEGBXaliGlLFsR9zBg/FRaiqIoNci5Xjl5mZTyoBAiHVgphNh29J1SSimEOOGsMKFCPwpKl/XZsWPHOUapPAcOHKCoqCiiM+bl5aHrekRndLlc7N69m7y8PKOjlMvv90f0a1hSUoLL5YrojIcPH47435eioiL2798f0ZNWnWzVpHMq3FLKg6HPR4QQK4BLgBwhRB0pZbYQog5wpJz
     },
     "metadata": {
      "needs_background": "light"
     }
    }
   ],
   "source": [
    "m.plot(Q)"
   ]
  },
  {
   "source": [
    "## Results\n",
    "\n",
    "Let's see if we were successful training Peter to fight the wolf!"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Killed by wolf = 1, won: 9 times, drown: 90 times\n"
     ]
    }
   ],
   "source": [
    "def qpolicy(m):\n",
    "        x,y = m.human\n",
    "        v = probs(Q[x,y])\n",
    "        a = random.choices(list(actions),weights=v)[0]\n",
    "        return a\n",
    "\n",
    "print_statistics(qpolicy)"
   ]
  },
  {
   "source": [
    "We now see much less cases of drowning, but Peter is still not always able to kill the wolf. Try to experiment and see if you can improve this result by playing with hyperparameters."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fe0987e7dd8>]"
      ]
     },
     "metadata": {},
     "execution_count": 13
    },
    {
     "output_type": "display_data",
     "data": {
      "text/plain": "<Figure size 432x288 with 1 Axes>",
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"248.518125pt\" version=\"1.1\" viewBox=\"0 0 395.093508 248.518125\" width=\"395.093508pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;white-space:pre;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 248.518125 \nL 395.093508 248.518125 \nL 395.093508 0 \nL 0 0 \nz\n\" style=\"fill:none;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 52.375 224.64 \nL 387.175 224.64 \nL 387.175 7.2 \nL 52.375 7.2 \nz\n\" style=\"fill:#ffffff;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <defs>\n       <path d=\"M 0 0 \nL 0 3.5 \n\" id=\"m95b0a9fb88\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"67.593182\" xlink:href=\"#m95b0a9fb88\" y=\"224.64\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0 -->\n      <defs>\n       <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n      </defs>\n      <g transform=\"translate(64.411932 239.238437)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"128.471997\" xlink:href=\"#m95b0a9fb88\" y=\"224.64\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 2000 -->\n      <defs>\n       <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n      </defs>\n      <g transform=\"translate(115.746997 239.238437)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"190.869141\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"189.350812\" xlink:href=\"#m95b0a9fb88\" y=\"224.64\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 4000 -->\n      <defs>\n       <path d=\"M 37.796875 64.3125 \nL 12.890625 25.390625 \nL 37.796875 25.390625 \nz\nM 35.203125 72.90625 \nL 47.609375 72.90625 \nL 47.609375 25.390625 \nL 58.015625 25.390625 \nL 58.015625 17.1875 \nL 47.6093
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAD4CAYAAADy46FuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAcD0lEQVR4nO3df3BV553f8fdHPxHCQgJkIEACrIkd4raxrcZksrOTCVmM3UzwH04Gz86azXrDtHHa7HZnEryZ1tMknkna7XrDrONdN2aDM1kTrzetaYpLKHYm05nasRw7trFNkLExkrGRkQAbJPTjfvvHeSQuQjrge4UkxOc1c9E53/Occ5/nHul+OD+kq4jAzMxsLBWT3QEzM5vaHBRmZpbLQWFmZrkcFGZmlstBYWZmuaomuwPjbd68ebF06dLJ7oaZ2UXlmWeeeScimkdbNu2CYunSpbS2tk52N8zMLiqSDoy1zKeezMwsl4PCzMxyOSjMzCyXg8LMzHI5KMzMLJeDwszMcjkozMwsl4NihOO9/Tz6XMdkd8PMbMqYdr9wV66vP/I8j734FlcuuIyrFjRMdnfMzCadjyhGePNYLwA9fYOT3BMzs6nBQWFmZrkcFGZmlstBYWZmuRwUZmaWy0FhZma5HBRmZpbLQWFmZrkcFGZmlstBYWZmuRwUZmaWy0ExhpjsDpiZTRHnDApJWyQdlvTiKMv+XFJImpfmJWmzpDZJz0u6tqjtBkn70mNDUf06SS+kdTZLUqrPkbQrtd8lqWl8hnyO8U7Ek5iZXUTO54jih8DakUVJS4A1wBtF5RuBFemxEbgvtZ0D3AVcD3wcuKvojf8+4EtF6w091yZgd0SsAHaneTMzm2DnDIqI+CXQNcqie4CvceZZmnXAg5F5EmiUtBC4AdgVEV0R0Q3sAtamZQ0R8WREBPAgcHPRtram6a1FdTMzm0AlXaOQtA7oiIjfjFi0CDhYNN+eann19lHqAPMj4lCafguYn9OfjZJaJbV2dna+3+GYmVmO9x0UkmYCfwH8x/HvzujS0caY15cj4v6IaImIlubm5onqlpnZJaGUI4rfAZYBv5H0OrAY+LWkBUAHsKSo7eJUy6svHqUO8HY6NUX6eriEvpqZWZned1BExAsRcXlELI2IpWSni66NiLeA7cBt6e6nVcCxdPpoJ7BGUlO6iL0G2JmWHZe0Kt3tdBvwaHqq7cDQ3VEbiupmZjaBzuf22IeA/wdcKald0u05zXcA+4E24L8BXwaIiC7gW8DT6fHNVCO1+UFa51XgsVT/DvD7kvYBn0nzZmY2warO1SAibj3H8qVF0wHcMUa7LcCWUeqtwNWj1I8Aq8/VPzMzu7D8m9lmZpbLQWFmZrkcFGZmlstBYWZmuRwUZmaWy0FhZma5HBRmZpbLQWFmZrkcFGZmlstBYWZmuRwUZmaWy0FhZma5HBRmZpbLQWFmZrkcFGZmlstBMYYY8xO6zcwuLQ6KEaTJ7oGZ2dRyPh+FukXSYUkvFtX+i6RXJD0v6b9LaixadqekNkl7Jd1QVF+bam2SNhXVl0l6KtV/Iqkm1WvTfFtavnS8Bm1mZufvfI4ofgisHVHbBVwdEf8c+C1wJ4CklcB64KNpne9LqpRUCdwL3AisBG5NbQG+C9wTEVcA3cDQZ3LfDnSn+j2pnZmZTbBzBkVE/BLoGlH7eUQMpNkngcVpeh2wLSJORcRrQBvw8fRoi4j9EdEHbAPWSRLwaeCRtP5W4OaibW1N048Aq1N7MzObQONxjeKPgcfS9CLgYNGy9lQbqz4XOFoUOkP1M7aVlh9L7c8iaaOkVkmtnZ2dZQ/IzMxOKysoJH0DGAB+PD7dKU1E3B8RLRHR0tzcPJldMTObdqpKXVHSHwGfBVZHDN9M2gEsKWq2ONUYo34EaJRUlY4aitsPbatdUhUwO7U3M7MJVNIRhaS1wNeAz0XEyaJF24H16Y6lZcAK4FfA08CKdIdTDdkF7+0pYJ4AbknrbwAeLdrWhjR9C/B4USCZmdkEOecRhaSHgE8B8yS1A3eR3eVUC+xK15efjIh/HRF7JD0MvER2SuqOiBhM2/kKsBOoBLZExJ70FF8Htkn6NvAs8ECqPwD8SFIb2cX09eMwXjMze5/OGRQRceso5QdGqQ21vxu4e5T6DmDHKPX9ZHdFjaz3Ap8/V//MzOzC8m9mm5lZLgeFmZnlclCYmVkuB4WZmeVyUJiZWS4HhZmZ5XJQmJlZLgeFmZnlclCYmVkuB4WZmeVyUJiZWS4HhZmZ5XJQmJlZLgeFmZnlclCMyZ+RZGYGDoqzaLI7YGY2xTgozMws1zmDQtIWSYclvVhUmyNpl6R96WtTqkvSZkltkp6XdG3ROhtS+32SNhTVr5P0Qlpns9Jnq471HGZmNrHO54jih8DaEbVNwO6IWAHsTvMANwIr0mMjcB9kb/pkn7V9PdnHnt5V9MZ/H/ClovXWnuM5zMxsAp0zKCLil0DXiPI6YGua3grcXFR/MDJPAo2SFgI3ALsioisiuoFdwNq0rCEinoyIAB4csa3RnsPMzCZQqdco5kfEoTT9FjA/TS8CDha1a0+1vHr7KPW85ziLpI2SWiW1dnZ2ljAcMzMbS9kXs9ORwAW9l/RczxER90dES0S0NDc3X8iumJldckoNirfTaSPS18Op3gEsKWq3ONXy6otHqec9h5mZTaBSg2I7MHTn0gbg0aL6benup1XAsXT6aCewRlJTuoi9BtiZlh2XtCrd7XTbiG2N9hxmZjaBqs7VQNJDwKeAeZLaye5e+g7wsKTbgQPAF1LzHcBNQBtwEvgiQER0SfoW8HRq982IGLpA/mWyO6vqgMfSg5znMDOzCXTOoIiIW8dYtHqUtgHcMcZ2tgBbRqm3AlePUj8y2nOYmdnE8m9mm5lZLgeFmZnlclCYmVkuB4WZmeVyUJiZWS4HhZmZ5XJQmJlZLgeFmZnlclCYmVkuB4WZmeVyUJiZWS4HhZmZ5XJQjCEu6EcxmZldPBwUI2Qfi2FmZkMcFGZmlstBYWZmuRwUZmaWq6ygkPRnkvZIelHSQ5JmSFom6SlJbZJ+Iqkmta1N821p+dKi7dyZ6nsl3VBUX5tqbZI2ldNXMzMrTclBIWkR8O+Aloi4GqgE1gPfBe6JiCuAbuD2tMrtQHeq35PaIWllWu+jwFrg+5IqJVUC9wI3AiuBW1NbMzObQOWeeqoC6iRVATOBQ8CngUfS8q3AzWl6XZonLV+t7BajdcC2iDgVEa8BbcDH06MtIvZHRB+wLbU1M7MJVHJQREQH8JfAG2QBcQx4BjgaEQOpWTuwKE0vAg6mdQdS+7nF9RHrjFU/i6SNkloltXZ2dpY6JDMzG0U5p56ayP6Hvwz4AFBPdupowkXE/RHREhEtzc3Nk9EFM7Npq5xTT58BXouIzojoB34KfBJoTKeiABYDHWm6A1gCkJbPBo4U10esM1bdzMwmUDlB8QawStLMdK1hNfAS8ARwS2qzAXg0TW9P86Tlj0dEpPr6dFfUMmAF8CvgaWBFuouqhuyC9/Yy+mtmZiWoOneT0UXEU5IeAX4NDADPAvcD/wvYJunbqfZAWuUB4EeS2oAusjd+ImKPpIfJQmYAuCMiBgEkfQXYSXZH1ZaI2FNqf83MrDQlBwVARNwF3DWivJ/sjqWRbXuBz4+xnbuBu0ep7wB2lNNHMzMrj38z28zMcjkozMwsl4PCzMxyOSjMzCyXg8LMzHI5KMzMLJeDwszMcjkozMwsl4PCzMxyOSjMzCyXg8LMzHI5KMYQk90BM7MpwkExgia7A2ZmU4yDwszMcjkozMwsl4PCzMxyOSjMzCxXWUEhqVHSI5JekfSypE9ImiNpl6R96WtTaitJmyW1SXpe0rVF29mQ2u+TtKGofp2kF9I6m9Nnc19QrQe6L/RTmJldVMo9ovge8L8j4irgXwAvA5uA3RGxAtid5gFuBFakx0bgPgBJc8g+TvV6so9QvWsoXFKbLxWtt7bM/pqZ2ftUclBImg38HvAAQET0RcRRYB2wNTXbCtycptcBD0bmSaBR0kLgBmB
     },
     "metadata": {
      "needs_background": "light"
     }
    }
   ],
   "source": [
    "plt.plot(lpath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ]
}