{ "metadata": { "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.0" }, "orig_nbformat": 4, "kernelspec": { "name": "python3", "display_name": "Python 3.7.0 64-bit ('3.7')" }, "interpreter": { "hash": "70b38d7a306a849643e446cd70466270a13445e5987dfa1344ef2b127438fa4d" } }, "nbformat": 4, "nbformat_minor": 2, "cells": [ { "source": [ "## CartPole Skating\n", "\n", "> **Problem**: If Peter wants to escape from the wolf, he needs to be able to move faster than him. We will see how Peter can learn to skate, in particular, to keep balance, using Q-Learning.\n", "\n", "First, let's install the gym and import required libraries:" ], "cell_type": "markdown", "metadata": {} }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Collecting gym\n", " Downloading gym-0.18.3.tar.gz (1.6 MB)\n", "\u001b[K |████████████████████████████████| 1.6 MB 2.3 MB/s \n", "\u001b[?25hRequirement already satisfied: scipy in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from gym) (1.4.1)\n", "Requirement already satisfied: numpy>=1.10.4 in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from gym) (1.19.2)\n", "Collecting pyglet<=1.5.15,>=1.4.0\n", " Downloading pyglet-1.5.15-py3-none-any.whl (1.1 MB)\n", "\u001b[K |████████████████████████████████| 1.1 MB 3.7 MB/s \n", "\u001b[?25hRequirement already satisfied: Pillow<=8.2.0 in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from gym) (7.0.0)\n", "Collecting cloudpickle<1.7.0,>=1.2.0\n", " Downloading cloudpickle-1.6.0-py3-none-any.whl (23 kB)\n", "Building wheels for collected packages: gym\n", " Building wheel for gym (setup.py) ... \u001b[?25ldone\n", "\u001b[?25h Created wheel for gym: filename=gym-0.18.3-py3-none-any.whl size=1657514 sha256=578c789ab75e603e58dd1152b2bd60d9a5adc6a057559cf8b5bdd6ee8b80abf2\n", " Stored in directory: /Users/jenlooper/Library/Caches/pip/wheels/1a/ec/6d/705d53925f481ab70fd48ec7728558745eeae14dfda3b49c99\n", "Successfully built gym\n", "Installing collected packages: pyglet, cloudpickle, gym\n", "Successfully installed cloudpickle-1.6.0 gym-0.18.3 pyglet-1.5.15\n", "\u001b[33mWARNING: You are using pip version 20.2.3; however, version 21.1.2 is available.\n", "You should consider upgrading via the '/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 -m pip install --upgrade pip' command.\u001b[0m\n" ] } ], "source": [ "import sys\n", "!{sys.executable} -m pip install gym \n", "\n", "import gym\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import random" ] }, { "source": [ "## Create a cartpole environment" ], "cell_type": "markdown", "metadata": {} }, { "source": [ "env = gym.make(\"CartPole-v1\")\n", "print(env.action_space)\n", "print(env.observation_space)\n", "print(env.action_space.sample())" ], "cell_type": "code", "metadata": {}, "execution_count": 2, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Discrete(2)\nBox(-3.4028234663852886e+38, 3.4028234663852886e+38, (4,), float32)\n1\n" ] } ] }, { "source": [ "To see how the environment works, let's run a short simulation for 100 steps." ], "cell_type": "markdown", "metadata": {} }, { "source": [ "env.reset()\n", "\n", "for i in range(100):\n", " env.render()\n", " env.step(env.action_space.sample())\n", "env.close()" ], "cell_type": "code", "metadata": {}, "execution_count": 3, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/gym/logger.py:30: UserWarning: \u001b[33mWARN: You are calling 'step()' even though this environment has already returned done = True. You should always call 'reset()' once you receive 'done = True' -- any further steps are undefined behavior.\u001b[0m\n warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))\n" ] } ] }, { "source": [ "During simulation, we need to get observations in order to decide how to act. In fact, `step` function returns us back current observations, reward function, and the `done` flag that indicates whether it makes sense to continue the simulation or not:" ], "cell_type": "markdown", "metadata": {} }, { "source": [ "env.reset()\n", "\n", "done = False\n", "while not done:\n", " env.render()\n", " obs, rew, done, info = env.step(env.action_space.sample())\n", " print(f\"{obs} -> {rew}\")\n", "env.close()" ], "cell_type": "code", "metadata": {}, "execution_count": 4, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "[-0.035025 0.21201857 -0.010404 -0.3300738 ] -> 1.0\n", "[-0.03078463 0.40728707 -0.01700547 -0.62601941] -> 1.0\n", "[-0.02263889 0.21240657 -0.02952586 -0.3387403 ] -> 1.0\n", "[-0.01839076 0.01771693 -0.03630067 -0.05551247] -> 1.0\n", "[-0.01803642 0.21334007 -0.03741092 -0.35942391] -> 1.0\n", "[-0.01376962 0.40897331 -0.0445994 -0.66366469] -> 1.0\n", "[-0.00559015 0.21449925 -0.05787269 -0.38535156] -> 1.0\n", "[-0.00130017 0.410393 -0.06557972 -0.69570532] -> 1.0\n", "[ 0.00690769 0.21623893 -0.07949383 -0.42436686] -> 1.0\n", "[ 0.01123247 0.02232776 -0.08798116 -0.15776523] -> 1.0\n", "[ 0.01167903 0.21859198 -0.09113647 -0.47685598] -> 1.0\n", "[ 0.01605087 0.02486705 -0.10067359 -0.21423159] -> 1.0\n", "[ 0.01654821 0.22127341 -0.10495822 -0.53689749] -> 1.0\n", "[ 0.02097368 0.02777162 -0.11569617 -0.27904318] -> 1.0\n", "[ 0.02152911 -0.16552613 -0.12127703 -0.02497378] -> 1.0\n", "[ 0.01821859 -0.35871897 -0.12177651 0.22711886] -> 1.0\n", "[ 0.01104421 -0.16208621 -0.11723413 -0.10135989] -> 1.0\n", "[ 0.00780248 -0.35534992 -0.11926133 0.15215788] -> 1.0\n", "[ 0.00069548 -0.15874009 -0.11621817 -0.17564179] -> 1.0\n", "[-0.00247932 0.03783674 -0.11973101 -0.50260923] -> 1.0\n", "[-0.00172258 0.23442478 -0.12978319 -0.83049704] -> 1.0\n", "[ 0.00296591 0.04129289 -0.14639313 -0.58128481] -> 1.0\n", "[ 0.00379177 0.23812991 -0.15801883 -0.9162682 ] -> 1.0\n", "[ 0.00855437 0.04545686 -0.17634419 -0.67712384] -> 1.0\n", "[ 0.00946351 0.24253346 -0.18988667 -1.01973114] -> 1.0\n", "[ 0.01431417 0.05037919 -0.21028129 -0.7921723 ] -> 1.0\n" ] } ] }, { "source": [ "We can get min and max value of those numbers:" ], "cell_type": "markdown", "metadata": {} }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "[-4.8000002e+00 -3.4028235e+38 -4.1887903e-01 -3.4028235e+38]\n[4.8000002e+00 3.4028235e+38 4.1887903e-01 3.4028235e+38]\n" ] } ], "source": [ "print(env.observation_space.low)\n", "print(env.observation_space.high)" ] }, { "source": [ "## State Discretization" ], "cell_type": "markdown", "metadata": {} }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "def discretize(x):\n", " return tuple((x/np.array([0.25, 0.25, 0.01, 0.1])).astype(np.int))" ] }, { "source": [ "Let's also explore other discretization method using bins:" ], "cell_type": "markdown", "metadata": {} }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Sample bins for interval (-5,5) with 10 bins\n [-5. -4. -3. -2. -1. 0. 1. 2. 3. 4. 5.]\n" ] } ], "source": [ "def create_bins(i,num):\n", " return np.arange(num+1)*(i[1]-i[0])/num+i[0]\n", "\n", "print(\"Sample bins for interval (-5,5) with 10 bins\\n\",create_bins((-5,5),10))\n", "\n", "ints = [(-5,5),(-2,2),(-0.5,0.5),(-2,2)] # intervals of values for each parameter\n", "nbins = [20,20,10,10] # number of bins for each parameter\n", "bins = [create_bins(ints[i],nbins[i]) for i in range(4)]\n", "\n", "def discretize_bins(x):\n", " return tuple(np.digitize(x[i],bins[i]) for i in range(4))" ] }, { "source": [ "Let's now run a short simulation and observe those discrete environment values." ], "cell_type": "markdown", "metadata": {} }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "(0, 0, -1, -3)\n(0, 0, -2, 0)\n(0, 0, -2, -3)\n(0, 1, -3, -6)\n(0, 0, -4, -3)\n(0, 1, -5, -6)\n(0, 2, -6, -9)\n(0, 1, -8, -6)\n(0, 2, -9, -9)\n(0, 2, -11, -13)\n(0, 2, -14, -10)\n(0, 1, -16, -8)\n(0, 2, -18, -11)\n(0, 3, -20, -15)\n(0, 2, -23, -12)\n" ] } ], "source": [ "env.reset()\n", "\n", "done = False\n", "while not done:\n", " #env.render()\n", " obs, rew, done, info = env.step(env.action_space.sample())\n", " #print(discretize_bins(obs))\n", " print(discretize(obs))\n", "env.close()" ] }, { "source": [ "## Q-Table Structure" ], "cell_type": "markdown", "metadata": {} }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "Q = {}\n", "actions = (0,1)\n", "\n", "def qvalues(state):\n", " return [Q.get((state,a),0) for a in actions]" ] }, { "source": [ "## Let's Start Q-Learning!" ], "cell_type": "markdown", "metadata": {} }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "# hyperparameters\n", "alpha = 0.3\n", "gamma = 0.9\n", "epsilon = 0.90" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "0: 108.0, alpha=0.3, epsilon=0.9\n" ] } ], "source": [ "def probs(v,eps=1e-4):\n", " v = v-v.min()+eps\n", " v = v/v.sum()\n", " return v\n", "\n", "Qmax = 0\n", "cum_rewards = []\n", "rewards = []\n", "for epoch in range(100000):\n", " obs = env.reset()\n", " done = False\n", " cum_reward=0\n", " # == do the simulation ==\n", " while not done:\n", " s = discretize(obs)\n", " if random.random() Qmax:\n", " Qmax = np.average(cum_rewards)\n", " Qbest = Q\n", " cum_rewards=[]" ] }, { "source": [ "## Plotting Training Progress" ], "cell_type": "markdown", "metadata": {} }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "[]" ] }, "metadata": {}, "execution_count": 20 }, { "output_type": "display_data", "data": { "text/plain": "
", "image/svg+xml": "\r\n\r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n", "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXxU9b3/8dcnCSTsa8CQgAEJIKIIBGSXTUWiYqu0Lq2o3MvV6nWhVlGrtbdasddq9dqfy9W2tr22WpdKXYu4W0VBRVBAQFACCEF2kCXk+/tjvkkm+yTMZCZn3s/HI4+c853vzPmenMl7vud7zpxjzjlERCS4UuLdABERiS0FvYhIwCnoRUQCTkEvIhJwCnoRkYBLi3cDADp37uxyc3Pj3QwRkSZl0aJFW5xzmXXVS4igz83NZeHChfFuhohIk2JmX0ZST0M3IiIBp6AXEQk4Bb2ISMAp6EVEAk5BLyIScBEFvZmtNbMlZvaxmS30ZR3NbJ6ZrfS/O/hyM7N7zWyVmX1iZoNjuQIiIlK7+vToxzvnjnfO5fv52cB851weMN/PA5wK5PmfmcD90WqsiIjU3+GcRz8VGOenHwVeB67z5X90oesfv2dm7c0syzm38XAa2pjWbd3Lj/+2mG7tMvjpaf3p3DqdbXsO8K/V31BwXBbOOf7+8XpO6NmJj77azsSjuzD5N2/yo3G9eXbxev484wTumb+S/lltOaJdBobxwBureePzIpbccjJmBsCLSzby+ooiphyXxYl9Kn7nYUnhDv7+8Xq6tk1n5tijyso/XredtBRjQHY7nHM8uaiQCf26cOvzy+jYqjnLNu7kX6u/4d5zB5HdvgWri3bz3UHZpKWm8PKnX3NMt7bc/sJybig4mpPueoNHpg+leZrxwdptzHlxOU9dOoIN2/dx+sBuPLbgK254Zglt0tO4cFQu//PqKm46rT+/eO4zlv9iMjf9fSl/W1RIn66tGdazI51apXPP/JX87PT+9OzcivfXbGXJ+h28tXJLg7fFJScexQNvrKZ3l9as2ry7rNwMUs0oLonfZbbbZKSxa19xxPVHHtWJpet3sHNfMcdmt2PJ+h306tyKL7bsAeDu7w/k6scXx6q53HJ6f7buPcgTH6zj6537YraccENzO/DB2m38YHgP1m/7luZpKbz86aZ6vcb5J/TgolG5TLrrzXov/4qJeaSacfcrn9da79YzB/DlN3v437fW8B9je/Hgm19UqfPdwdk8/eH6Gl9jcI/2PHXpSNZs2cOEX78RUftyOrTglVknktEsNaL6DWWRXI/ezNYA2wAHPOice8jMtjvn2ofV2eac62BmzwFznHNv+/L5wHXOuYWVXnMmoR4/PXr0GPLllxGd998ocmc/Xzad06EFb183ge8/+C4L1mzlvesnsm7bXqY98G5ZnYn9ujB/+eay+cqhFO73Fw1lfN8u7Nx3kONu+WdZ+do5BTW2Ifyx0vK1cwp4aelGLvnzh3Wuz7WT+3LxqJ70u+mlOuuWKv0HrUnz1BQOHCqJ+PVEksFvzxvMZY/V/T8Z7ofDj+QXZw5o0PLMbFHYKEuNIu3Rj3LObTCzLsA8M1te27KrKavyaeKcewh4CCA/Pz9h735SuO1bANZvD/0+eKiE3fsr9uLWbdtbYb6mkAfKeoDFhw5/lXd+G1lvcuvuA5TU8wYztYU8oJAXqcaufQfr/ZyiXftj0JKKIhqjd85t8L83A88Aw4BNZpYF4H+XdmkLge5hT88BNkSrwUGVO/t5LvnTong3Q0QCqM6gN7NWZtamdBo4GVgKzAWm+2rTgWf99FzgAn/2zXBgR1Man4+nlz79Ot5NEJEAimTopivwjD+AmAY85px7ycw+AJ4wsxnAV8A0X/8FYAqwCtgLXBT1VouIBETR7tgP3dQZ9M65L4CB1ZR/A0ysptwBl0WldQG0/+Ah5i7ewKijOjXaMuN4YoqI1OHLb/bWXekwJcRlipPJr15eQdGu/dxzzvGNtkyr7vC4iETdI2+viXcTqqVLIHhffrOHRV/WfqZJNJQeYd/5bf2PzotIYltZyxl38aQevXfif78OVD2fXUQklrY0whi9evRRYNV+dSBxOJf4bRSR2FHQi4gEnIZuoqAhBzsb80SY372zhjYZ2tQiyUo9+giMv/P1CvNNcRDknvkr490EEYkTBX0E1virC4qINEUK+nqq57XBouqv738Vv4WLSJOloI9QInzpaPbTS+LdBBFpghT0IiIBp6AH9h6o+7ru0R6yiecQkIgkFwU9sOCLrXXWKb0BSayHcL49cCi2CxCRpKOgr2TfwbqDNpad8UPq6otIlCnoK9l/sHFukVfbnkFjXPtCRJKHgr6eHv9gXUzv8bh19wHyb30lZq8vIslHQV9P9722it+/szZmr79174EqZbmzn+eQ7h4iIg2koAdcPUfdDxQf/gHTmobiP9uws9ryg4caZ0hJRIJHQd8Aq4sO/5IIroakX7l512G/tohIOAV9gvmwhrtcrdyUmHeuEZHEp6CvpL7DONG2uHBHteWn3/d2I7dERIJCQR8nlggXzxGRpKCgFxEJOAU98bnuTE0HY0VEok1BHwUahhGRRKagr8Sa5I0CRURqpqCvpCFn3SzbWP2XnEREEoGCnviM0Wu4R0Qai4JeRCTgFPTE9vryNS5TZ92ISCNR0MfJLf/4LN5NEJEkEXHQm1mqmX1kZs/5+Z5mtsDMVprZ42bW3Jen+/lV/vHc2DQ9etS7FpEgq0+P/kpgWdj8HcDdzrk8YBsww5fPALY553oDd/t6Ce3xD9bFuwn11uenL/L655vj3QwRaQIiCnozywEKgIf9vAETgCd9lUeBM/30VD+Pf3yiJfgpJis3l18Zsql07g8Ul/DCkq/j3QwRaQIi7dH/BrgWKL37RSdgu3Ou2M8XAtl+OhtYB+Af3+HrV2BmM81soZktLCoqamDzRUSkLnUGvZmdBmx2zi0KL66mqovgsfIC5x5yzuU75/IzMzMjamysxPvSxCIisZQWQZ1RwBlmNgXIANoS6uG3N7M032vPATb4+oVAd6DQzNKAdsDWqLc8SkpKHCVhd+lL7EEmEZH6q7NH75y73jmX45zLBc4BXnXOnQ+8Bpztq00HnvXTc/08/vFXXQKf1nLqPW+xfvu38W6GiEjMHM559NcBs8xsFaEx+Ed8+SNAJ18+C5h9eE2MrRWbdI9WEQm2SIZuyjjnXgde99NfAMOqqbMPmBaFtsVF4u57iIg0jL4ZKyIScAp6EZGAU9CLiARcvcbog+JAcQl9fvoiV0zoHe+miIjEXFL26PcVHwLg9++sjW9DREQaQVIGvYhIMlHQV6KzK0UkaBT0legKCCISNAp6EZGAU9CLiAScgr4SjdGLSNAo6CuZ95nu2iQiwZLUQb9rf3GVsi27D8ShJSIisZPUQS8ikgwU9JX8+p8r4t0EEZGoUtBXUqKjsSISMEkX9J9v2sW0+9+NdzNERBpN0gX9L19YptsHikhSSbqgFxFJNgp6EZGAU9CLiAScgl5EJOAU9CIiAaegFxEJuKQLet1YRESSTdIFvYhIslHQi4gEnIJeRCTgFPQiIgGnoBcRCbikC3oznXcjIsmlzqA3swwze9/MFpvZp2b2c1/e08wWmNlKM3vczJr78nQ/v8o/nhvbVRARkdpE0qPfD0xwzg0Ejgcmm9lw4A7gbudcHrANmOHrzwC2Oed6A3f7eglD/XkRSTZ1Br0L2e1nm/kfB0wAnvTljwJn+umpfh7/+ETTeImISNxENEZvZqlm9jGwGZgHrAa2O+eKfZVCINtPZwPrAPzjO4BO0Wy0iIhELqKgd84dcs4dD+QAw4Cjq6vmf1fXe69yJ1Yzm2lmC81sYVFRUaTtFRGReqrXWTfOue3A68BwoL2ZpfmHcoANfroQ6A7gH28HbK3mtR5yzuU75/IzMzMb1voG0CCSiCSbSM66yTSz9n66BTAJWAa8Bpztq00HnvXTc/08/vFXnXNVevQiItI40uquQhbwqJmlEvpgeMI595yZfQb81cxuBT4CHvH1HwH+ZGarCPXkz4lBu0VEJEJ1Br1z7hNgUDXlXxAar69cvg+YFpXWiYjIYUuKb8YeKnHcMvdTNmz/Nt5NERFpdJEM3TR576/Zyh/+tZbPN+2iZfPUeDdHRKRRJUWP3vmzO0t0TFhEklBSBH1FOr9SRJJLEga9iEhyUdCLiARcUgW9huhFJBklRdBb2Li8LoEgIskm0KdXOudYXbQn3s0QEYmrQPfoH3l7DZPueoPFhdvLyg6VaPxGRJJLoIP+o3WhgF+3dW9Z2avLN8erOSIicRHooBcRkSQLeg3aiEgySoqg15k2IpLMgh306sKLiAQ86D3T9W1EJIkF9jz63NnPl00/vnBdHFsiIhJfSdGjP1BcEu8miIjETVIEfRmN2YtIEgpU0O/eX8zU377D55t2xbspIiIJI1BB/69VW1i8bju/emlFvJsiIpIwAhX0dXl/7dZ4N0FEpNElVdCLiCSjQAW9jrWKiFQVqKAvpUseiIiUC2TQi4hIuUAFve4JKyJSVaCCvpRGbkREygUy6EVEpFzAgl5jNyIilQUs6EN01o2ISLlABr2IiJSrM+jNrLuZvWZmy8zsUzO70pd3NLN5ZrbS/+7gy83M7jWzVWb2iZkNjvVKlNJZNyIiVUXSoy8GfuycOxoYDlxmZv2B2cB851weMN/PA5wK5PmfmcD9UW91HXRHKRGRcnUGvXNuo3PuQz+9C1gGZANTgUd9tUeBM/30VOCPLuQ9oL2ZZUW95dXYrxuMiIhUUa8xejPLBQYBC4CuzrmNEPowALr4atlA+L37Cn1Z5deaaWYLzWxhUVFR/Vtejase/zgqryMiEiQRB72ZtQaeAq5yzu2srWo1ZVVGz51zDznn8p1z+ZmZmZE2IyI660ZEpFxEQW9mzQiF/P855572xZtKh2T8782+vBDoHvb0HGBDdJorIiL1FclZNwY8Aixzzt0V9tBcYLqfng48G1Z+gT/7Zjiwo3SIR0REGl9aBHVGAT8ElphZ6SD4DcAc4AkzmwF8BUzzj70ATAFWAXuBi6La4gho6EZEpFydQe+ce5uarxM2sZr6DrjsMNslIiJRom/GiogEXCCDXl+YEhEpF8igFxGRcgp6EZGAC2TQry7aHe8miIgkjEAG/fKvd8W7CSIiCSOQQS8iIuUU9CIiAaegFxEJOAW9iEjAKehFRAJOQS8iEnAKehGRgFPQi4gEnIJeRCTgFPQiIgGnoBcRCTgFvYhIwCnoRUQCTkEvIhJwCnoRkYBT0IuIBFxggr5w2954N0FEJCEFIujfXf0No+94Ld7NEBFJSIEI+uVf74x3E0REElYggt65eLdARCRxBSPo490AEZEEFoigFxGRmgUi6J3GbkREahSIoBcRkZo1+aDfd/AQtz6/LN7NEBFJWHUGvZn9zsw2m9nSsLKOZjbPzFb63x18uZnZvWa2ysw+MbPBsWw8wINvfBHrRYiINGmR9Oj/AEyuVDYbmO+cywPm+3mAU4E8/zMTuD86zazZ3oPFsV6EiEiTVmfQO+feBLZWKp4KPOqnHwXODCv/owt5D2hvZlnRamz1DYzpq4uINHkNHaPv6pzbCOB/d/Hl2cC6sHqFvqwKM5tpZgvNbGFRUVEDmwH/WLyhwc8VEUkG0T4Ya9WUVdvnds495JzLd87lZ2ZmNniBG3bsa/BzRUSSQUODflPpkIz/vdmXFwLdw+rlAOpyi4jEUUODfi4w3U9PB54NK7/An30zHNhROsQjIiLxkVZXBTP7CzAO6GxmhcDPgDnAE2Y2A/gKmOarvwBMAVYBe4GLYtBmERGphzqD3jl3bg0PTaymrgMuO9xGiYhI9DT5b8aKiEjtFPQiIgHXpIP+/TWVv8clIiKVNemg/8HDC+LdBBGRhNekg/7AoZJ4N0FEJOE16aAXEZG6KehFRAJOQS8iEnAKehGRgFPQi4gEnIJeRCTgFPQiInHUPC32MaygFxGJo7OH5MR8GQp6EZE4Om9Yj5gvQ0EvIhJHA7LbxXwZCnoRkTjp3aV1oyxHQS8iEidj8zIbZTkKehGROLnmlD6NshwFvYhInLRsXufdXKNCQS8iEnAKeklIfbo2zkEqaTzpjfDFIKme/vKSkJyLdwsk2sbkdY53E5KWgl6kEfzyO8dGXPfYRjivuql69OJhUX29Fs1So/p63Tu2iOrrRYuCXqSBxveNzalxd04bGPUAasq+Oyi7bPrEPuV/87F9av/7FxyXVedr5+d2qHd7Tu7ftcJ8r8xWjIvReyFaFPRS5vwTYv9V7Ibo27VNteV/nnFClbJOrZrHtC3nDO1eNn36wG4AXDDiSNbOKaj1ec1SrV7LuXh0br3bVpe1cwq45uTGOZ0vmobkdqBV86offJUDd3CP9hXmczrU3btOT0vh0nFHlQ0rRXKBsXF9u1SYP2twDtntQ8tqlpqYkZqYrZKo6pXZKqJ6t9VjeKHyP1ldLh/fO+K6o3tXHMt9+eqxVercd94gRsdhzLe095bZJp0zBnbjigm9+ckpfWt9zohenfhOWK80El3bZgBUG3CljvB1EtWpA4447Nd49rJRnDesBwtunMTim08GKgb43y4Zwc2n9WfBDRN57N+H07LS36tTq+ZMOrrm9+pPTunHdZNDPwBnDc7m/RsncvGonmV1hlbq9We2Secfl48um+/SJp3Zp/bjigm9Oe3YmvcirpjQm/wj678HEQ0K+iZkUKUeS6T+3/mDK7xxo2FafndunHI0AMNyO9ZZ/5pKYTiqd6da6885K/Sh819Tj6n28dOO6xZJMyOWF/ZV9PvOG8S8aj5cSv3homF8cOMk0lJTmHVyX9pkNAPgwpG5NK+mR/eXmcNJS03hhin9uHBkbll5Zpt0js5qW+0y+vi9mP+aOqDC3kL3ji147ZpxADzxHyPKyksD5KcFR5eVfS+/6lURv5ffvcK6Ho7OrdOrLX/1xyfy4pVjKqxruJqWX10IDuzeHjOjdXoa7VqG/s5XTepT9jpDczty8eiedG2bQUazVF798ThOCuuELLrpJB6enl82f+e0gfz7mPL/hTYZofPYB2S345Hp+fzs9GPo0iajQs/+4QuGcsvp/cvmT+rflWNz2vHFL6fwwA+GcPaQHNpkNGPWyX0Z2L3m/9ExfTI55ZjyD7+nLh1RY91oU9AniLp2Gbu0SeepS0Y26LWz2rXg5tP7c1xO+UG+W88cUOtzurWrvbc4rm8mk32P7ZJxvSJqx2XjjyqbPmdo7cNEQ47syNo5BVwwIheAD26cVGub75w2kGd+NLLsHzcSFjaaMm/WiRSE9cbyahguqs0tZxzD57edWuPjM8cexS1nlH9wDc3twItXjikbAgo3vFcn3r5uPGf5S9iWhvbLV42lZ+dWrJ1TQI9OLcvq//b8wVxzch9mjC4Psdu+cyzvXT+xwut2aZvBvFkncvNp/XllVs0fZvURvmdxZKeW9MpsXeMHWG2evDSy9/dZg7N569rxnNCramfhiHYZDO5R9QOjTXoak47uwtlDcrixoD9PXjKCn5zSl27ty/cOJh7dlQx/bCR8CLBdy2ZcOKon78yewJrbp5SVp6QYkwccgYW9kSaG7T3MOqkPr10zjldmjeXqSX3IP7JD2V7oHy4aypAj6+4gRYuCPgZKd+XrMx561/cG1vr4+zdOIiXFuOec4+vVlrVzCmjXItQTmnv5aOZ8N9RT7ndE7UFWenbDhSNzeSSsRwSw5vYpNEtNoXvHlqydU8CEflV3jR/4wZAqZT85pR9r5xSwdk5BhXBb/ovJQGg4qH9WW2af2q/KczPbpJeFeAffswt39pAcBvXoUHYlwNJeY/hxh/Drfg/s3p41txfwyqyxvHjlmBr+CiGDe7RniO9tRnMM9r/PDm3z/zl3EGvnFPC3S0YwMKcduZ1DAZ7ToTzIf3X2QNbOKajxm5Rd22Zw+YQ8zKys09AsNYUj2mUw7+qxvHRVxXW8eHRPenWuX8++8tlAE/t1oW/XNtz2ndAH8IhenXiyls5Ih5ah8ByTl1n2Plg7p4B7zjm+wvBKm/TaP6zNjO4dW9b4+FmDs8nr0pofDj+yrGzJz0/h4elDy+bzcztyWS3DiWP6hAK5Z+fyYc/s9i0qhHpNXrpqDPOuHssVE/Po2bkVvbu04cpJoW1zdFZb1s4pqDLOH2uN8/3bAMjr0pqnfjSS4275Z511Z4zuya59xfzbmF7c+c/PAUhLMYpLKp4cfnL/rvTLasuQIzsw6qjqhzLeu34i7cOC7YyB3fh80y4y0lJplpbCWYNzuP2FZTz90XpG9e7ElRP70KVNOu+s3sJx2VV3I78/tDsjj+pMj04tueOsY8nr2oY9+4tZun4nEOoZp1ioR/v2dePJateC1JSKb+7q3uxzLx/FkvU7uPGZpQBMHnAE/bPa8tnGndUeNAX4xZkDGNS9PRnNUnn3+gl0apVe655NwbFZLP96F5ecWL5n8OAPh1QYLik9KHbd5H4s/HIbV07MY8/+Yv7+8QbunDaQJxcVAnD/+YMB6N2l5g+8966fSIdWzUhPS2XvgWLunb+K8yI4YD0mrzOL123nzWvHY1T9Ww3IbsvufcW0qhRoQ3M78mzY2G8k/jRjGDu/La5Q9tx/juatlVvK5mvaOyndjOP6ZnJS/65l2y6rXQYbd+yjR8eWfLV1LwA/P+MYpo/MJXf287RJT2PX/mLyurbmjrOPA+Cta8eT3b4FKWHvlaP8h+2IXp1494tv6N+tLW9PGl/l2MLU47OZenzoGMaHN51Es1TjpLveJDuCg6nVKd1rORyl262+B9EB+h1R/72ZWDOXAN9Myc/PdwsXLqz380bcPp+NO/bV+3nXTu5L6/Q0bn72U1o0S+XpH40kNcW455WVPL9kIytvO5UHXl/Nr+eFQvrm0/pzsd8lds5x07NLuXBkLt978D227jnA9/JzSE1J4cqJeXz5zZ4Ku5TTf/c+Zw7qxstLN/HSp1+z+pdTqgRnuE8Kt3PGfe9w7rDuXDAiN6Jd4H0HD/H6is1MHlD36WSH465/rmDX/mJ+dnr14+YAS9fvoEXzVI7KbM3WPQf4dMMOxjTSFfoADhSXMH/Zpiq71KWeWlTIc59s4PcXVT0fe8XXu7jssQ956tKRZXtBySR39vMAvH/DRFYV7eaYrHZ8+NU29hwoZsqArLIQd87xwpKvmTzgiFrfy6VKShwvLo28fiLYX3yIs+7/Fz8t6M/waoaIEoWZLXLO5ddZLxZBb2aTgXuAVOBh59yc2uo3NOiLD5XwzEfr2VdcQtuMNIbmdqR5WgqpZlz+lw+57cxjSU0x9heXRHTd5/3Fh9iy+wDZ7VtQUuJYt20vR3aK7IyVSF77m90HKowJ1uTLb/bQo2PLiHYTRaJlz/5i9uwvpkuCn80j5eIW9GaWCnwOnAQUAh8A5zrnPqvpOQ0NehGRZBZp0MfiYOwwYJVz7gvn3AHgr8DUGCxHREQiEIugzwbWhc0X+rIKzGymmS00s4VFRUUxaIaIiEBsgr66geUq40POuYecc/nOufzMzMS+ToSISFMWi6AvBLqHzecAG2KwHBERiUAsgv4DIM/MeppZc+AcYG4MliMiIhGI+hemnHPFZnY58DKh0yt/55z7NNrLERGRyMTkm7HOuReAF2Lx2iIiUj+61o2ISMAlxCUQzKwI+LKBT+8MbKmzVrBonZOD1jk5HM46H+mcq/O0xYQI+sNhZgsj+WZYkGidk4PWOTk0xjpr6EZEJOAU9CIiAReEoH8o3g2IA61zctA6J4eYr3OTH6MXEZHaBaFHLyIitVDQi4gEXJMOejObbGYrzGyVmc2Od3vqw8y6m9lrZrbMzD41syt9eUczm2dmK/3vDr7czOxev66fmNngsNea7uuvNLPpYeVDzGyJf869liC3rDKzVDP7yMye8/M9zWyBb//j/hpJmFm6n1/lH88Ne43rffkKMzslrDzh3hNm1t7MnjSz5X57jwj6djazq/37eqmZ/cXMMoK2nc3sd2a22cyWhpXFfLvWtIxaOeea5A+h6+isBnoBzYHFQP94t6se7c8CBvvpNoTuytUf+BUw25fPBu7w01OAFwldBno4sMCXdwS+8L87+OkO/rH3gRH+OS8Cp8Z7vX27ZgGPAc/5+SeAc/z0A8ClfvpHwAN++hzgcT/d32/vdKCnfx+kJup7AngU+Dc/3RxoH+TtTOj+E2uAFmHb98KgbWdgLDAYWBpWFvPtWtMyam1rvP8JDuOPPAJ4OWz+euD6eLfrMNbnWUK3X1wBZPmyLGCFn36Q0C0ZS+uv8I+fCzwYVv6gL8sCloeVV6gXx/XMAeYDE4Dn/Jt4C5BWebsSujDeCD+d5utZ5W1dWi8R3xNAWx96Vqk8sNuZ8psPdfTb7TnglCBuZyCXikEf8+1a0zJq+2nKQzcR3cmqKfC7qoOABUBX59xGAP+7i69W0/rWVl5YTXm8/Qa4Fijx852A7c65Yj8f3s6ydfOP7/D16/u3iKdeQBHwez9c9bCZtSLA29k5tx64E/gK2Ehouy0i2Nu5VGNs15qWUaOmHPQR3ckq0ZlZa+Ap4Crn3M7aqlZT5hpQHjdmdhqw2Tm3KLy4mqqujseazDoT6qEOBu53zg0C9hDa3a5Jk19nP2Y8ldBwSzegFXBqNVWDtJ3rEtd1bMpB3+TvZGVmzQiF/P855572xZvMLMs/ngVs9uU1rW9t5TnVlMfTKOAMM1tL6KbxEwj18NubWekls8PbWbZu/vF2wFbq/7eIp0Kg0Dm3wM8/SSj4g7ydJwFrnHNFzrmDwNPASIK9nUs1xnataRk1aspB36TvZOWPoD8CLHPO3RX20Fyg9Mj7dEJj96XlF/ij98OBHX637WXgZDPr4HtSJxMav9wI7DKz4X5ZF4S9Vlw45653zuU453IJba9XnXPnA68BZ/tqlde59G9xtq/vfPk5/myNnkAeoQNXCfeecM59Dawzs76+aCLwGQHezoSGbIabWUvfptJ1Dux2DtMY27WmZdQsngdtonAgZAqhs1VWAzfGuz31bPtoQrtinwAf+58phMYm5wMr/e+Ovr4Bv9wf3t8AAACjSURBVPXrugTID3uti4FV/ueisPJ8YKl/zn1UOiAY5/UfR/lZN70I/QOvAv4GpPvyDD+/yj/eK+z5N/r1WkHYWSaJ+J4AjgcW+m39d0JnVwR6OwM/B5b7dv2J0JkzgdrOwF8IHYM4SKgHPqMxtmtNy6jtR5dAEBEJuKY8dCMiIhFQ0IuIBJyCXkQk4BT0IiIBp6AXEQk4Bb2ISMAp6EVEAu7/A6SijxMjKxrLAAAAAElFTkSuQmCC\n" }, "metadata": { "needs_background": "light" } } ], "source": [ "plt.plot(rewards)" ] }, { "source": [ "From this graph, it is not possible to tell anything, because due to the nature of stochastic training process the length of training sessions varies greatly. To make more sense of this graph, we can calculate **running average** over series of experiments, let's say 100. This can be done conveniently using `np.convolve`:" ], "cell_type": "markdown", "metadata": {} }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "[]" ] }, "metadata": {}, "execution_count": 22 }, { "output_type": "display_data", "data": { "text/plain": "
", "image/svg+xml": "\r\n\r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n", "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD4CAYAAAANbUbJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO2dd3gVZfbHvycdAiGUAKEZelGqkY4gICDo4rr6U3dVVKxrWdeKde2ylnXX1bWiYu8FpYmAKCol9AABAgQIBAglQALp7++PO3Mzd+70O7fk3vN5njyZeeedmXfu3HvmzHlPISEEGIZhmOgmLtwDYBiGYYIPC3uGYZgYgIU9wzBMDMDCnmEYJgZgYc8wDBMDJIR7AADQokULkZWVFe5hMAzD1CtWrVp1SAiRYaVvRAj7rKws5OTkhHsYDMMw9Qoi2mW1L5txGIZhYgAW9gzDMDEAC3uGYZgYgIU9wzBMDMDCnmEYJgZgYc8wDBMDsLBnGIaJAUyFPRGlENEKIlpHRBuJ6DGp/V0i2klEa6W/flI7EdFLRJRPROuJaECwL4JhwoUQAp/n7EFldW24h8IwhlgJqqoAMFoIUUpEiQCWEtFcads9QogvVP3PA9BV+hsE4FXpP8NEHXNz9+OeL9aj4HAZ7hnfI9zDYRhdTDV74aFUWk2U/owqnkwG8J603zIA6USUGfhQGSbyOHaqCgBw6ERlmEfCMMZYstkTUTwRrQVwEMACIcRyadNTkqnmRSJKltraAtij2L1QalMf8wYiyiGinOLi4gAugWHCB4V7AAxjEUvCXghRI4ToB6AdgIFEdAaA+wH0AHAWgGYA7pO6a33//d4EhBBvCCGyhRDZGRmW8vgwDMMwDrHljSOEKAHwE4AJQogiyVRTAeAdAAOlboUA2it2awdgnwtjZRiGYRxixRsng4jSpeUGAMYCyJPt8EREAC4EkCvtMgvAVZJXzmAAx4QQRUEZPcOEmVrpnVUYTmPFBt+v34dVu46GexiMDlY0+0wAi4loPYCV8NjsvwfwIRFtALABQAsAT0r95wDYASAfwJsA/ur6qBkmQnjn150AgFnr+OX11o/W4E+v/hbuYTA6mLpeCiHWA+iv0T5ap78AcEvgQ2OYyGfbQY+jWnmVr599eVUNVhYcwYiuPB/FRAYcQcswQeDRWRtx5YwVyNt/PNxDYRgALOyZKOPD5bvw2co95h2DjKzxnyivDtk5K6pr8I9vc1FVw9G8jD8s7Jmo4sGvc3Hvl+tt7fPBsl3YeajM1XHIE5Wh9MO/94v1mPn7Lpz97OIQnpWpL7CwZ2KSkpOVKD5RASEEHvomFxe+8mtAx5twemvNdgqhtP9l2yEAQNGxchwqrbC0z9wNRciaNtsbCcxELyzsmZhkwBMLcNZTP3pdJwMVdh2aN9Rs31tSHtBx7XCkrC5lw/8WbzfsW3yiAmUV1XhtiaffzkNluPWj1fh4xe6gjpEJHyzsmZjE6x8vAvOPv3poFgCgVVqK5vZtB04EdHynmD28znrqR5z/36UorfDMKdQKge/XF+H+rzb49T1cWoGsabOxcPMBS+cuq3A2T3Gqsiak8w3Xv5cTUw83FvZMTBNoKJRsptETrtW14Qm2+s6C3//OQ2XYXuyZqyg1mEjeVOTxKHrn1wJL5z79H/ORf9D+Q67nI/PQ9cG55h1dYsGmA5oPt2Bj1cTmNizsmZimNkDN/rf8wwCAlxZu09xeFaY895U2NeSftribjHDjPnsup1v2h+cNKNQs33EY2U/+iHm5oU8qwMKeiXpeXrQNd362VnNbgLIeW0zMNG5r9vtKTiFr2mx8u3avq8dtmZbsXe720FzNCetgpoQ4WRk6F1XA13zn5C3EKTmSl9a6wmMhO6cMC3sm6nn+h634arW5cHzw6w2ue6XY1bDNeG7+FgDA3z7Rfng5JbNJ3ZxDZXUt1u4p8a6T5ED6a/5h3dw3gc591ITY3KU83dh//YwCl11v9ZAfag0S40NyPiUs7JmoJGvabNz3hbm/vdKM8+Hy3brmGKe4bcZJSdT/yd43wXmlrMR4a6JAby5ALazJps9pqOc21Oa7/cdD4zU1e73HfPP1GnffzKzAwp6JWj7N8Y2k3V5c6tdHrZAGatZR47YMS06o0wgHPf0j8vYfR8GhMmRNm42PVuxyfNz4OGvCOU5HiKuFtcXDeQl11K/64ZSU4FwU7iguxUPfbECthZtdcPgkAGD/sdC55MqwsGdihjEvLPFrU/88F2856Oo50xsmunq8FMXr/4HjFXjz550Y9fxPAIA9R045Pq6eEAeA2RvqJhP1uqmFNdmMHS456dx8tufISaxTmJ2soH6ot0hN1u5ogZs+WIUPlu3GVhPbf/GJCnTKSAUAnKqqcXw+p7CwZ2Ia9eu8kfWhsroWk176Bb/mH7J8/GqXNdZklQYaqDeRTFqKdgLcE+VVPr7oeh9PpcpcdeSkvZq8pQ598wFgxLOLMdlmBLRbnxsAlFV4BHd1jf4xtx44gbOe+hE7ikMzN6AFC3sm5thz5KR3Wf2bH9lNPyXxvpJT2LjvuC3f7CqX7TjJicER9o00hP2O4lJMfTfHp03vYaiM3gX0Hx56lIdY01V/biWnnBeM31vieaP6YlWhbp/cvaH3vlHDwp6JOUYoEoXd+tFqn21G5gdZPOxWPCzMqDHQ9pyg9uLQe5b8aUA7W8fVemaMfmEJVhQc8WnTM/esUZlRGibZE/bqCeK1e0rwyLe5ul4+xSc8Ub2L8qxF9cpU1dTio+W7/Wz26jcTJ5yq1H9gac0XhRoW9kxMIycPk3n7151+Oej3HDmJDQ79oqtq3fbG8RX2esLQonONl2cll04zCo9qzwscV7ms2nXFVNv8L3zlV7z3+y6/ojAysqZ8rerNw4zXl2zHA19vwOc5vlp4WoPA51aM5me27GdhzzARx6SXlvqsj3h2MS54ealO7zpOf2SenznCyI7rBPVEpJ4Zx6p3jczPW61F0Cona5WotdpaIXD9eznIUb0Z6KGnWS/bcdjS/laRbeay6UXG7n06dqoKX632fWAcLtM3BVVUh35CVg0LeybieG5+HuZv3B+047dpop20TMZKgI+W5lpWWYN3fyvwaat2WbP/RFWYRe/wahu6kkADoLR4YcFWn/XiExVYsOkArn/PXPP+bt0+fLNW23//pI5ppOCw/0RnTa3Aw9/k+szJKDl4vBxfSf7tB1R+9XbnPvo+9gPu/GwdNu6re+NTH1NJhc7DbF/JqaDcDy1Y2DMRxyuLt+PG91f5tdfUCld+GOkNkxztpzy3nvfI9Ll5PutVLmv2TVTmBr0UBvM36tuyQyFb5DQSRy24VN728RpsLtLOpaPnovjYd5v82tbsPor3l+3C3z/Vji6+Q9HevplvSuqaWoH8g6XImjYby228TWzcWzfuHq0b6/ar0LiO938vwNDpi3Drx2ssny8QWNgz9YbOD8zBzR+sNu9owiYdwWIHPWHfv0O6z7rbrpcNk6xN0KrZUHgMD3/jmfB0w4PnyhnLMeCJBbrbW0spnxPsRlepsOOPLkft1kjX98rifOQfrLOV/7a9TojLkawyNULgmTmbAQCv/7xD9xzPzN2MborMnD9trYvLGNalhe5+WuUpP1qxR3MswcJU2BNRChGtIKJ1RLSRiB6T2jsS0XIi2kZEnxJRktSeLK3nS9uzgnsJTCwxL0DzTiBeF0rBeqqyRtNdsEtGI591ZWRpba0wfNW3NgZfQW1Vbl/y+m94f9kulFZUY6mNOAE9ftl2CEfKKnVt0bLg7dgiNaDznLKRIE1+rtTWCuwrOYXn5m/BJa/9ptlXbbOvqRVYmOcR3Ivy9APrXl+ywyff0dDOdQLeaJ5kh0buncY23VMDxYpmXwFgtBCiL4B+ACYQ0WAA/wTwohCiK4CjAKZK/acCOCqE6ALgRakfw0QE3wSQk0QpaEe/sASXvv67Rh/fdeXE3yuL8zHo6YW6NmUr2H1Y1dQKHC+v8nq1LNx8EFe/s9Lx+dVco3Msed7DKDLXCqcqrV+vLGxrhMDlby7z7C89kI+ZmJOcJmLLaFwXeWv3EFbSK7iJqbAXHuR3oUTpTwAYDeALqX0mgAul5cnSOqTtY8huViQmavlpy0EUn3CneMPGfcfw8iJ7ics+DKAykVogrCs85memUNvQlRO0SySPl0CSbvnbwI0FxqSXfkGfR3/wrt+hY892itI0okR+MDr55Zcoom+bNUqyLBTlB0tNLbBLykEjP59HPm9chN2p4FW+WdUKgRlLd+K5+Xn6OyjI0ckgGiws2eyJKJ6I1gI4CGABgO0ASoQQ8jtWIYC20nJbAHsAQNp+DEBzjWPeQEQ5RJRTXOxu4QQmMhFC4Op3VuKyN/w1Yidc8N+leP6HrYZ91JpwUYn1/DGnKmt8smC+9ctOvz5+9m/VqnKCVv5xV+j4jjvBzIyTF6aiIAelB7qT9ABK09fD3+Si0wNzLAljWdgfVXgiyZ+PWe6d71R28xcXbLVUhvGmDxSOBAJ44vtNeMWk/m+4sCTshRA1Qoh+ANoBGAigp1Y36b/Ws9zvTgkh3hBCZAshsjMy9EPUmehB/r1q2S9lRktJvewczwi1Nn7QxltFz0fm4V8Kl8IvV/uHw6vH0CzV19NHy/UykND8Sb0zHe/rFupJ5x83+QvFuZI/vlk+fy3ziVbZPjmD6c9bi5E1bbbmseT9lG9OAsJSKoZftvkqnP9ZuA1TZ3rcRn/NP4SsabNNJ9vdzLcTDGx54wghSgD8BGAwgHQikmcY2gGQHWULAbQHAGl7EwDWIiuYqMb7am/Qx+hBoIeRO6bd4KJAGdTJ9yVWK1gnkPS2Qzr7vSSHnNtUroLXafjSWw1SeusXf8+XCf/+xa/tsCTI//dTvu6xrnp7hV9bVY2wlD5Zzw8eAP7y1nIAwAfLjFNIq1NLRBpWvHEyiChdWm4AYCyAzQAWA7hY6jYFwLfS8ixpHdL2RSJUUQNMRFNnx3VXAMvfLq2vmVNty2nZP2WQDaBdlGOrSSlDNXn7j+OdXz0mJPXRwvHDmptr7hE1tIvnoWRUkWnBpgN4Zq41+7b8nXFiWtebV1BiZR5pywHjlAdHDQLZjLhueEdH+9nFiu9PJoCZRBQPz8PhMyHE90S0CcAnRPQkgDUAZkj9ZwB4n4jy4dHoLwvCuJl6iMvBpF4Kj57CjkOl2FfirzEfL3eWJ/3Vn5zZXZXeGYDH5FFRXYN4xQPOivBRImu61wzr6Gekj1Q9qpmUH753uya6faxE18pU1wg8Omujo7eitTZz3euR1sBYXOq9zew+bOx91ad9uuF2tzAV9kKI9QD6a7TvgMd+r24vB3CJK6NjogorZhw91HZXZaTq6Bd+QnWtwMCsZv47OpSFTic2m6uKYFTXCnR/aB7OPK2pt00vmZgVIkGzt4a7I3t/2S5NW74V1u62J+yzmjf0VpRSYpbBVK94ydnPGXsCZSu+G8GEI2iZkBHIBNYuxY/vVGUNXltSp3nLppKaCNRyZW1Pr1C3HY6drLKl2RYdc/5QCRR54nXFTnfs2E4FPQD8bjOZml49XrM6uQePOxtjqBzTWdgzISOQGBKl//oJHdOMlmdHuMW/m7VVNxYdw3u/+04SGj3fLvivvepNbmImGJduCzyKN1jojd0sqZ3Te223hKNTWNgzIcOJfVkIgQWbDvjYQ4+d0hb2WrbZUCv7ZRXVPuMwE3p6ZE2b7edi+MrifNw6uovlYwSiDQfKGoXppEDDw+qKGctDORxbqEs/ymjlt1GiDnjL23/c0kR/qBzGQpucgYlpnMi9GUt34snZmzFU4XJ47os/W95/9e7QRine9fk6n3Ut3+zz+xj7yiv91pVFU37NP+xXNtHtZ9nbV2fbLgiixU6FgB/1/E946fL+OHSiAteGyPMkGHyrk4ZZDy0XUk1Y2DPRhmyzlwXUF6sKMbxLC7Q2yC//5GxPJkK7Hiwyf/0w8CyZgaBVg9aozi3g67depkoE5nISTT9G92gVlOPeLvnmbztYig6q9MKRhtIcGIr8NWzGYaIOWdjLybnu/nxdRL/Ou4GWZm9nolrd1T/rpfvCKJjeIR+v2I1/zrPmW+8WRoF1fTVcQ5XC/sUfjdNx6GHnIREqMw4Le8YV9L7cQghvoJGydJ1snlDmG49GtD4WI+28TJUnXy3MQ6FpntFW3ze+PmKU0fLu8d392pR1g79Y5Z8iwwr/WWg9QV+o8kSysGcCZs6GInR6YI5m6t73ft+FSS8txW/5h/Dh8rqMk3IIuhJ19Gm0onYRnbOhCFnTZuO6mSv9hItSTt00srPfHMRelc9+kt1K4xo8MLEnLh/YPuDjuMnoHi2Dclx1MRjA15/eaaUxW8Le0Rnsw8KeCZj7vlwPAPhcQwvK2++pCrXzcJmpa5q60He0otbW5XmFHzcfRO5e3wee0uX0tSXbsXiLb8Ku3YoH7PIHxmDj4+NxaXZggjopIQ4jutpPTtiiUbJ5J4cESyAmJ/gLe6UHlds1hLXQ8y5zGxb2TMDILmk/aFSRypVqdC7bcSTkbpCRipFZQZ2Qy+wzi1MYfFulpSAxPk63Lm2wCaY1wqkLqxmZGs4BynkRvYLnbtKicfAekkpY2DOuoZViYIOkqS42KPWmh7KIRTRhJOzV+ffNkqZZqVyVZrH8nbKEoLqwuRXcKkqjRbA07OYabyNK000gZSz1yGru643EZhwmIjlUWoGsabPx3bp9OF5ehSe/3+Td1qJRku5+aSkJttMl9Htcv6B1faZWCNTUCgx5ZqFfmcRyVU1X2fU0EBIs2vHH9apzuxwaAamUlVRVh+5txWmJQquo8+4EWrrRKizsGVtskbT36XPz8MycPLy1tK56U28DL47z+7aJWjPOZzcOsdW/tLwaldW1KDpW7p3vkLFSaMMuWsLkvDNa+7UpTUiRVkk0GLni9dIvh8JOr4Rz4zARyQGpCtDeklPe/N1yHdbzemfins/XadaF/WHj/rDZkoON3QIpLy3K934Wai2y3IWShWkpviYYLcVeK9mX+s2rbXqDgMcSyXRt1UizPdiafbhgYc/oIoTw8xzZXHTcuyxPmsn/5+Xux+erCjXrwhYcPomhnVtonqf/4z/gjk/WaG6rDxxxULSi1yPzAfhPPBpVTDJiYu86TT1O9fDR0uzj4wjjT/eNllULuQYabolGhLgoWMDozTEEazJYD9bsmbDT+9EfcI6qJux36+oKM/+oKshcWmGcKEqtccocPVmFb2zmHXGbu8d1c7yvmxPJLR16ZijNLuoHtJawv/Ss9pg6vJNPmzrRl16qXz3s9jdjapDz6Dx/SV/N9mCbG9VzW2yzZ8JGaUU1sqbNRmlFtd9k0sET+vnU1bnLlfbnsT1bRrQZZ4jOW4cVurdu7No4lmwtNu+kgZHASIj33fbjnWdjcKfmGNixGWZMyfa2b1NFMyfpZH/Uw+lbiR7BNKe0SkvGsC6ee948Vd+xIBh8fP1g/H1snXLB3jhM2DjjH/N1tyl/fyO66gvI8qoa9Hh4nnd9cKfmPtkcI4U2kp91IIXJu7RshC9vHurWkByhHL16cjVBdW2dM+ps1crUCEoTHQAkxYfXLuM0VYEVFtw50rusNnsFkxFdW6Brq8b429iu3jZOl8BEPOkNPRqRlqDUCg5atsN9jwqnvHbFAKx8cKz3XSM+gB9cw6QEDOgQmjqiehjJq8Nllbhy8GnedaVwaZWmn3HULH97sLGS5O2/l/tVTLWE0qQYaHyAnYLh708d5F1u38wzAc6aPVNvsPK6LSBw48hOpv1CxYiuGchonOy1z8bZ+CXcf14Pv7Zwuyoanb/kZBUenNTT9jG3mAR0BZvINfoBtymKyDx0fi9Hx/j8xqF47YoBIXuzYGHP4EhZJY7rlPpzyk9b/CNmX9Dw0gkWPTPTDLfLNm55HsHOJNmNIzujkyLaNBIwG32lg0T46RpRtI0tRuO6QSA1i43o0lLb5RKw7hlz1zj/bJl2ad0kBRPOMC5k4yamwp6I2hPRYiLaTEQbiehvUvujRLSXiNZKfxMV+9xPRPlEtIWIxgfzApjAGfDEAvR59Afb+2nlapf52ydrfdZnLN2JvSXuFsD+6e5RutvMfrPqH7WVH/n401uhhzQZG2la51drjMvfNUryCOnbbJQ1/EPfNn5tTtIoOMXK/OzI7vYTtv2osNeridbAP8CaZl8N4C4hRE8AgwHcQkTye8uLQoh+0t8cAJC2XQbgdAATAPyPiOw57DJhxWjiVcncXP/EZ3ocOO5+3pQsA+3a6oSrnOjKimb/+pXZmHfH2dYGF2TsCt24OELB9Em2NNLxp3t895VvMWYxBdcN74ibRna2NTYtRvdoaclmn5aSiB1Pe/VM3DyqMz5Q2MXVPOTAnBUtmAp7IUSREGK1tHwCwGYAbQ12mQzgEyFEhRBiJ4B8AAPdGCzjLsdOVaGi2j88f3An37woe46c9EbORhp6KRrMZL0s2+VJSLtm02BUiLLD97cNd/2Y6jmVoV1a4Pf7R/t4rsjLKYnaouOh83uhUbI13U7P5XFsz1Z4++qz0L6ptfKFSpv39+v3GZp/AvG6knnnmrP82sxKTQLASw4nk93Cls2eiLIA9AcgV564lYjWE9HbRCTXMmsLYI9it0IYPxyYMNH3sR9w0f9+865nTZuNmb8V+PUb8exiDHp6YQhHZh09bwyt6M9hXeoeYuq6n+GeYLVLexfruA7q2AyAtsDKbNLAR0C2TW+AgumTsPyBsbqBaLLw7WpgGwf0I3RrpNw0vTVKBpqx58gpNG2o7zfvxl0epfE5vXlVXbyC3ptxqs2IZLexLOyJqBGALwHcIYQ4DuBVAJ0B9ANQBOAFuavG7n6PWiK6gYhyiCinuNhZIAnjnG2Sp8XGfb6+1f+YtTEcw3FMok7gT8Mk/4nEBIXLjSy/5PS/ekLg+hHBjeIMBKMso3aQg67s5P9q0iARt47u6tP2x/4enU42iZlNsOoVs1Gn4bBD4+QEpFp8s3CKlmKgDEB79A+na+53tgXtP5hYEvZElAiPoP9QCPEVAAghDgghaoQQtQDeRJ2pphCAslROOwB+sfBCiDeEENlCiOyMjPB+CLFIcWnwco87wUwL1EPrrXxsz5Z4WMMdTlkFSv7BJkuZD/Vs9r3bafvPy/t/fpO9jJdu4pYlSb72QLM9pjf0zCPEe4W9cX+9eRzZlVcvvYYR8fFkrNkH8Q1umuSSm9E4GZec2c5nW9v0Bq6nk7CLFW8cAjADwGYhxL8U7UqfoT8CyJWWZwG4jIiSiagjgK4AVrg3ZMYNjL54Rn7z/YMUPETkzJ6qtc9bU87yKxABAGWVdUFC8l7HpZJwejKgfVPjzI+h9E5R888/9XHlOHKErdF9//624XjtigGGx5Ft8LIZx+zhcUZbbfdYWaN/aFJPPGLRh13O0NmuaQOkN0zE+X0yNR/EWt/fyf38vY6ccOPZnbDj6YlIS0nEMxf19tlmN/VEMLAygmEArgQwWuVm+SwRbSCi9QDOAfB3ABBCbATwGYBNAOYBuEUIEfzaXowtWjXWj5w0ErqntzH2X7fCoxf0woWqH1gcEZo5yFGSoVP3VEuDG9WtpWK7578c6aun2evVVd15qMyvrbVBNGowOPO0puadLBAvmbeMhP0ZbZtY9gk/Q/qO9G9vPL42TbQfpLXSOFKTE3CtxehUeXK5b7t0EBFe/vMAnJXVzKdPzkNj0UfjTe2JC8+wdA4ziMj7oFMXjFGnrAgHVrxxlgohSAjRR+lmKYS4UgjRW2r/gxCiSLHPU0KIzkKI7kKIucG9BMYJRknJ9IT920t3ouRk4MFXU4ZmITXZ16ZeUV3ryMPFzmv5PEWNXHm/RtI4GupMniXreJ3IKGMH/jKog+WxuIHepdsVK3KQUdMAE4LJz4pBnZpj+QNjfNIua6FV/xUAaix+D/oqJnDJGySnzZJ7Ruk+uPXMRaMc+PDrYbVaWDAJ/wiYoCGEwKK8A5rBT0a/Jz0l5PHvN+H79UXaG21ARLhYZdOsrK7VtPEqPWiCid5Dw8yMfbKi7qX15lGB+5e7gd1H5l3juuH9qQP9NGG7KD25jHLuyOgJdavZLpVxFrKicEqnQLiTNMLPumQmA/yTzIUDFvZRRnVNLW7/eA027TuOV5dsx7Xv5uAJRZ1YGSNPCbVbYjDo38H3Fb9FoyTU1Aq/4tjn9vQtsBEIsouhFnoPuETJU2WgjiBUCqxQa29W71Oyib04MT4OI7oGrsUeVgVcDeroeVC/8mdtW79aB5FNHVZ81gHf5HVyLd+vdSKJzfLPTOrtb6IKdDJX77rDBQv7KGPHoTLMWrcPE1/6Bc/O2wIAmPn7LszLLUKZorjI+sJjeocIC/FxhFoh8CeVxq/8kVqN7NXjuhH6idi0BOel2e3RvFEyCqZPwmc6XjdWA4jcpGD6JM+CjixSm8OSwmRCaJqahILpkzCpT6amSadG9dok55dXpmA2QvnduELK6qlnSjMzmb94aT+seHCM3z5P/7G34wC2SX1Cl/fGCizsowy97/RNH6zGtK82eNeX79RPNzw3N3BTjV2ICEJ4XrevHprl0y5z7bCO+G3aaEfHv/+8HprauSwYSeOXcG4v/bcK2QunV6Z54I+ebdpNOiiCrNQvbZP7u+NtYkYPgyIu3Vt5Jm1lU9fAjs1QXeM70LvGdcODE3viAo2cPEr+fWk/AHXzLYDHx/+tq7LxmI6Pu555RyYpIQ4tVU4L8XGEPw/q4JPzvz7Dwj7KMHrz3H2kruqU0WTo6t0luttG92ipuy0QjpRVorSiGodKK5CSWKctKzWyuDhCG40i2B9ep58LRaZVWophGmOtj23fMf3EbfKErpXJxFJFXninWSO1JpCV9/rZi/Xty+eFKLOiUVKyW0d3wZc3D8V9E3rgzauy8f7UgX5BU8kJ8bj+7E5+DgK3j/EN3jq/TybuHtcN94yvy/NDRBjbq5WuKa2Rg8/9pMkDor7Bwj7qMJD2CsFkZQJN+xDByQkjuzJ+u3afz+u9cmJN71Vcfv0HgDeuPFP3HHYn6YwmCmWBVGthMvGEwnw2pJP2hLPaD1vtpy3nAFIH68go507BDLIAAB5KSURBVGDUIwpVJoi2Gg9imfg48rqKnturFZIT4i1PxGarXEwT4uNw6+iufh5dRlg1ZXXKqJv0VR5f+WCRuXxgB812La4ZlmWpXzBhYR9lGP2wjygKYzv10Q5iWVAvyrwvPpq9wcWdJgVR6b1yt0pLMfxstCbjurbUN0skeAOH7H0getegFkbqdXm/sxSTzMojpSrSQ6ifx6EoaD2sS3NcMeg0844K1MJezx1YmWbgkxsG2x8crJcenHVrnX1emcvmNI0gvWcu6o1bzrGWMtrouxQqWNhHGUZf6T1H6swSTos5Oy2IbQel0FAKLiOZNfOagbh9TFdN+/hzF/fBkM7NbQs9I/kgexPp+efroTeERJN6r17ThuLzUNqse2TWCZN+qijRUCj2fxrQznbFJav+9ACw4oEx+PLmoX4ZWa1iteyk8jNVmpMCVXL0ooVDSejKzjAhwaq72M/bIjf5nFJoLMqrq3hl5GqY1SIVd56rnYUxo7EnmMaugmv0WT5zUW9cMyzLtjlMT76ZuW2SV9YLRRvhg6mD8P6yAiQn1D10/tC3DVo1Tsalbyzz9gsWk/pkYvb6Ikdup2qFw+j+tkxLQcsAIpSdpOJQfm5OzZf5T52HgsMnDatjhQrW7KOI4+VVun7GMq8t2Q4AeOfXghCMKHDi4wj92ns0VacR57JA1hImsneIli+6kYxMSYzXDL03Q8scAAAX9PH1QFGfW09YDe/aAq9fme3XrpzIDmbuffnYTu6NOuulUVR3oITClKVFQnxcRAh6gDX7qMJKacHpc/NcqSQUKpQTl04LM8sapNbut47u6peq13u+IAiIjhrVtQZ0SEc7VcK1MT1bIS0lAcclTx5ZMFo1JyhTAARzmkWeS3fyWQWrxqwWgeamkYeqVaqxvsCaPeMKcnpbtyk8esorrALNJWXFnKFMtbxi5+HATqg5Bv+27q3T/K6tSYNEzL59hHf913zPWL5b55ctXJMmivsRTJlaG4Bm73TeyAlOFYVogoV9PWb5jsPImjYbBRoZGENF89QkbHlyQtBCw1ftOqpYc/aDlQWs2e99zcPn+nhjHCo1rrdqh/sm9ECvzDS0VmR6bCNNJv+hbxvNB5GWgDqh8Nm3SqVBYfhAkeW1k3kBdVCV1UnUYBMsxSXcsBkngjlwvBylFdW64ePfrPVoeb9uP2RYfDuYpCTGIzkh3kcMd2jW0CeAyy2cygLZVm8mkNRZH7cXlzo7oQY3j+rsnR8474zWmHBGa0zuV1etM2+/f6IsrdE6sWuv31NiOd+MXWQvIidmEqXb6gMTe0SMbXvtI+PCPYSgwMI+gpHrvnpzoahIkn5oVdXB09zMGCj5fSsF6ZmnNdUV9p0znD+UQq33bZVKN7rNq1f4B35pmVq0nk25e+1nT3QaQGeFJy48A+2aNsCo7vYjq5U2+xvOjux5pLQGHlHZKk07TXJ9gIV9PaZK0ozCWWJQjvRUCiYj7w+tdAdK1JWfOrZIDdjobJQmwYhQBsLIV9gzMw3T5c/Upcfb6J7BSXEBeIq7PDjJWjUpNS0b1x/BeU73lvjX//WNuORmdmCbfT1mzgZPwrJXFm8P2xjkPDZKsbS5SF8jNpuUO3bKtziK0uUwmP7iWgQrD5AW8gNySKfm6Bugq6kavaId4SZbSkx3ncVqVOGEiHDRgHY+8Qz1DRb29Ri5apS6xJ8dPphqnkTMCsrJxC0G5g8zd7uemZ5Iw2uHeQRAIAU1ZN95py8GoawbKrwTnYpGDWFvZ/Lwrauy8anD9AKhxEnAE2MfFvZRQEJ8nKWEXDJZ02Z7l4cHmCNexurP1UzwPj7Zk6L2ppGd0LRhok+6Y7s4qWmrxKzoh1WsJOHScmFs1rBu/PIEq53Sh2N7tcIgh+kFooV7xnc3TL0cS7Cwr8dcdlZ7AJ6MiP/7KT9o51ELXPm8SqxaWJSavVYmQNkdr2VaCtY8Mg7dFT9Uu5GgRcfKAQB7j+qnKjYiOdH5K/tfR3X2ZuDsoBM1q6QulqDug1SmIHj9yjPx0fWDcPc4a1kWGQ+3nNMF8+44O9zDiAh4grYekxAvuxQCz/+wNWjnaZPu682hLNwtY2RP3/T4eCzZUoybP1ztEwGqFU2qDqEH6kxEdmNwZBfQds2MJ4X1cKLZt01vgN5tm+DeCT0AeEwpfdqbF7/wPgT1iogTMLSzO29hkUboQqtiGxb2UUKz1CQcKXMvCEhJlSr4peRkFR6c2NNH6zZS7BsmJXiTWCknaLX20cqEKAfb2I24TEmMk87jzCbsRNj/qqqkNdag2pUdIiXgyE2i74oiG9NvMxG1J6LFRLSZiDYS0d+k9mZEtICItkn/m0rtREQvEVE+Ea0nosiquhtFyHKxvKomaIIe0M59f/3ZnXzyjJt5yshavFwrFAC+X+9f/vBsjcLX8gSeXWEve3uo3TmtEsoJWjPClcgrmLSV8gEp6xcwwcOKZl8N4C4hxGoiagxgFREtAHA1gIVCiOlENA3ANAD3ATgPQFfpbxCAV6X/jENOVlajYZL/rZJF39Nz8oJ6/gwL/tC5e40LmDeTik8r2bTPP0BIyzPj6Yt64+VF+bYLrjx6wem4emgWWjusAevWBK0bRGNul0m9M9HsuiQM6Rzbk8ihwvTbLIQoEkKslpZPANgMoC2AyQBmSt1mArhQWp4M4D3hYRmAdCKqv5EILiGEcJxqttcj83WOGciIrJNoISpJq/jGvRO64zmD2qhWldXOGY3w4qX9bGvaSQlx6NbKuSdGUnz99amuDxARhnZpEfL4iVjF1q+HiLIA9AewHEArIUQR4HkgAJAjUNoC2KPYrVBqUx/rBiLKIaKc4uLILaThFh3vn4O7PlsX8HFeX7IdP0gTpBVVdQWRz8pyVmbQCu2aNsDFOrVPZZR1YGW6t2qMS7L9PXdkjjtI6gUA/7msH2ZM8c/h7jaJCaETQt55BZ6tZIKEZWFPRI0AfAngDiGEUYIO7fxN6gYh3hBCZAshsjMygpOkKdL4yqSwiBHHpACqZ+bm4Yb3V/kdzzc7pDWmDDGvGdqjdWPExZGhhg5om1+ClcJ2cr+2GNPTnYlP7eN7gtQymzjz4nECK7dMsLEk7IkoER5B/6EQ4iup+YBsnpH+y/XjCgEo1bl2AKwl4Y5S7AQ86fHb9kM+6+8v2+V7DgensFLPU37FNnvV1hL2ai8eM+6T3BXDzX8u64+C6ZMcT+w6Qf74QlnQg4ktrHjjEIAZADYLIf6l2DQLwBRpeQqAbxXtV0leOYMBHJPNPbHKzsOB55tX1/h8+JvcgI95pgXTz+Yi/5c4rYhE2TWwqSKcv1xhZtIiSxVs9OeB1qNDo41mqZ5JcHWa5Wcu6o2+7cz99BnGDCveOMMAXAlgAxGtldoeADAdwGdENBXAbgCXSNvmAJgIIB/ASQDXuDriesjyHUcCPsb+4+V+2n2gNE4211wzNTxZtLR42TVQqZdWmKRevnNcd9z+8RrveiS5Ooaai/q3hRACF/b3nd66fGAHXB7DD0HGPUyFvRBiKfTjH8Zo9BcAbglwXFGFVlSomoMnypHRKNlrLlHb4P+7cBsOnnA3lXGDJHNvk/SG/vllxvVq7dcmW3mUJqvKamPNvqzCd4JWy6MnVoiLI8PJbIYJlNhVpUKIme03d+8xDHxqIT7PKfS2qQtnuCXo7SYH03IXvW10F782cqDZK/OZF0yf5GeqYhjGPfjXFQIqdDTcrGmzkTVtNj5cvhsAsKKgztwTjBzkfx/bDRdKpfCmnWdtMnScRri/VoBPqvSWoJxkNRP2IzSiZRmGCQ6cGycEmAm9j1d4hH2ipNl+tnIPdh1xPqk7Y0o2ps7M8WufOqIjGibG465x3ZCabH7rVz98LtIteqQkxMd5I2QfkiaP9R5yMpzHnGFCBwt7FymvqsHGfcf9wvorqqzViJWLNt/75fqAxqH0QR/ZLQNLtnqC1uLIo5VbEfRA4Pngza5blvWRlJaAYaIV/pW5yKOzNuJPr/6GXZKr5Wc5e5B/sNRUw5UJhqL77jVneZfdqmlqFbO5CiLCfRN6YNatw0M0IoaJXVizd5FNkk96yckqnNYcuPeL9UhOiMP1Izp5++wrOaVbdNvNHCEfXz8YLdOSfY7ptPC2Xc48rSlW7TqKszqalxS8eVTnEIyIYRgW9i6yvtCT+fFwWYU3VUBFda2PZp9g4F7oZhpbrUyCdo7fVueBZIUGUoUnKy6noeaFS/oikc1GTAzCwj4IrCw4iiGd6hKDKSdodxSXoWXjFM2UwMEOlQ+VEUf2l49EYf8nk4RuDBOtsIoTBI6dqsI9X9RluFSmDZCXH9JId/DubwX4dOVuW+d6aFJPy31DVQDjkQtOx9ndMnweeAzDhBfW7IPAR8t9BbZSs1+ytRiz1u7TzQh535cbbJ1rytAsPDl7s6W+ZgUwWqelYP9xT5FurefCm1dlW8rJ37FFKt67dqClMTEMExpY2IcApQviO78WAACSXIoWVdcm/UPfNo6PpXwWaAn7c12qp8owTOhhM04AlFVU47n5eag0CZoqq/Qv0lHpkj1bra2/eGk/x8dSeu6E2k2TYZjgwsI+AK6bmYNXFm/HN2uNi5L8ss3dbJVGaEWlWk0wdk4PZQFx14bEMEwEwMI+AH7fcRgAsGZ3ScjPTQTT6lEyP/x9JF68tK9pv39ccDoW3TUSDZPiMS1CCokwDOMObLN3gd0B5LExY94dIzDh37/4tV8x6DTLKXE7tkhFxxaputu/u3U45m/cj8T4OHTKaIRNj09wPF6GYSIT1uxdoE2TBkGrt9o8VTv75Ucr7LloGtG7XRPcPb67a8djGCbyYGHvAkO7NMftn6wx7+gAoarV3q99OoDgFfNmGCY6YWHvAmUVNZi9PjhldlMSfatJrd0T+vkBhmHqPyzsXeDRWRuDduyGiealA7+8eUjQzs8wTHTAwl6DaV+uxyWv/Wa5f3UQTSpWSvWVW8yXzzBM7MLeOBp8snKP4facgiO4+LXfvetxBITThB6MEoYMw0QXpmojEb1NRAeJKFfR9igR7SWitdLfRMW2+4kon4i2ENH4YA08nCgFPRCe8nqt0uoEfLdWjUJ+foZh6hdWNPt3AbwM4D1V+4tCiOeVDUTUC8BlAE4H0AbAj0TUTQhhrVRThPPIt7nomZnm1x7kzMSa3DWuzlXSzaInDMNEJ6bCXgjxMxFlWTzeZACfCCEqAOwkonwAAwH8brxbZLLrcBlOa14XjPTe77s0+wXTZq9HUUl5yM/JMEz9JZAJ2luJaL1k5pErbLcFoDR4F0ptfhDRDUSUQ0Q5xcXFAQwjeIx87iesL4xMV8fOLfUjYhmGYdQ4FfavAugMoB+AIgAvSO1a9gRNtVcI8YYQIlsIkZ2RkaHVJSLYfeSkX1v/Duk+6/dOCH30aeHRUz7rvds24XquDMPo4kjYCyEOCCFqhBC1AN6Ex1QDeDR5ZcKWdgD2BTbEyEOd+OyLnEJHx9kcQA6aU5W+0yDf3TYc93HyMoZhdHAk7IkoU7H6RwCyp84sAJcRUTIRdQTQFcCKwIYYXuS87ifKq3T77DjkLBFag6S6gKl5d4zwLmc09njaZDVvqLuv3IdhGMYKphO0RPQxgFEAWhBRIYB/ABhFRP3gMdEUALgRAIQQG4noMwCbAFQDuKW+e+I8Oz8Pk/pk4vI3l7l63LvHdfNZ79G6zstn5YNjTfcf2rm5q+NhGCa6seKNc7lG8wyD/k8BeCqQQUUSuw6fRL/Hf0DJSX3N3gm3ju7q1/bhdYOQf7DUrz37tKbI2XUUo3u0xKK8gwCAZAtpFBiGYWQ4XYIF3Bb0rdNSNNuHdWmBKUOz/Npfv/JMNE9Nwp3n1r0NWK0+xTAMA3C6BC+HSyuwcd9xnN0t+J5BN43s5F3u2z4dI7q0MOzfvFEyVj18rk9bYhw/pxmGsQ4Le4m/vLUceftPIPcxdzI8/HjnSGQ2ScGO4jJc8PJSn23tm9VNvH57yzBHx09MYGHPMIx1WGJIbDlwAgDw1OzNrhyvY4tUpCYnoHe7Jt422bumVxv/lAt2SWFhzzCMDVizB5C3/7g3v82+klPGnS2ilRxt0V2jcOxUFZqmJjk+7tpHzsX+4+WWUh8zDMPIsLAH8MmKugwPp6qC5ykaF0cBCXoASG+YhPSGgR2DYZjYI+aF/UPfbMAHy+qKd6/YeSSMo2EYhgkOMW8LUAp6hmGYaCXmhT3DMEwsENPCvromsNqt7107EL/ce45f+9y/jdDozTAMEz5i2mb/yKyNAe2vF4ClVc2KYRgmnMS0Zj9nQ5HjfXu3rfOfb8kZKBmGiXBiWrMPpHbsjKuzvctL7jkHOw6VYtJLS/GPC3r59f3qr0NRXROGQrUMwzASMS3sA6Fl47pkZg2S4nF6myYomD5Js++ADk012xmGYUJFTJtxGIZhYoWYFvbCoR1nbM+WLo+EYRgmuMS0sHfCiK4t8MaV2eYdGYZhIoiYFvZ6ev3401vp7pPVPBVxGknOGIZhIpmYFvYnyqs12+dvPKC7zxCu/cowTD0k5oR9Ta3A/mPleOTbXL9tZlWqLj6zHSb2zgzW0BiGYYJGzLle/nNeHt74eYfmtkbJxkW8OygqTDEMw9QnTDV7InqbiA4SUa6irRkRLSCibdL/plI7EdFLRJRPROuJaEAwB2+XlQVHdAU9ACSZFATp3yHd7SExDMOEBCtmnHcBTFC1TQOwUAjRFcBCaR0AzgPQVfq7AcCr7gzTHS557XfD7ftKyr3Lc24fgXeuPguXD+zgbRtuUhicYRgmUjEV9kKInwGoK3pMBjBTWp4J4EJF+3vCwzIA6URUb4zcKwrqLrNXmzSc06MlhnWpm5AlYi8chmHqJ04naFsJIYoAQPovRxm1BbBH0a9QavODiG4gohwiyikuLnY4DPdY8cAYzfZmXAKQYZgowG1vHC3VV9OdXQjxhhAiWwiRnZFh7AUTCL9tP4SPlu9GZbVx7vqWaSma7a2baLczDMPUJ5wK+wOyeUb6f1BqLwTQXtGvHYB9zocXOH9+czke+HoDKi0UKpHTFr91VV2EbAanL2YYJgpwKuxnAZgiLU8B8K2i/SrJK2cwgGOyuSfcHCmt1N3WOSMVABAvRcY2Ta0z3TROSQzuwBiGYUKAFdfLjwH8DqA7ERUS0VQA0wGcS0TbAJwrrQPAHAA7AOQDeBPAX4MyagekJOlfavfWjQEAXVs2AgA0aeAv4FulsYbPMEz9xTSoSghxuc4mvxlN4UkjeUuggwoGRsVD+rbz+M8/ceEZmNyvLbpIQl9GL089wzBMfSGq0yUcKq3wLg+dvki33/UjOgEAUhLjMbwr+9IzDBN9RLWwv+Kt5Zb6cRZLhmGinagW9nn7T4R7CAzDMBFBVAt7K2x8bHy4h8AwDBN0olbYV1vwq7+wXxukJsdc4k+GYWKQqBX2XR6ca9qndzvOYskwTGwQtcLeCuVVNeEeAsMwTEiIaWE/4YzW4R4CwzBMSIhpYS/046wYhmGiipgS9lufPM9nPS2FJ2cZhokNokra1dQKfLJyN/4vu71Pe3JCHH6+9xwkJcThgYk90CApAWN6tNRNa8wwDBNtRJWw/2LVHjz4dS5KTlZ52+4/rwduHNnZu37D2Z21dmUYholqosqMU1rh8a5Zs7sEAHDjyE4+gp5hGCZWiSphXyUFUv24+QAAYN2eknAOh2EYJmKIKmE/fW6ez3r+wbIwjYRhGCayiCphr0aZ4phhGCaWiWphzzAMw3iIamGf89DYcA+BYRgmIohqYd+iEdeNZRiGAaJY2PeQiogzDMMwUSzs371mYLiHwDAMEzEEFEFLRAUATgCoAVAthMgmomYAPgWQBaAAwP8JIY4GNkxz5HTF7Zs1wPSL+qB1E06FwDAMI+OGZn+OEKKfECJbWp8GYKEQoiuAhdJ60Bn+z0UAgPQGSRjWpUUoTskwDFNvCIYZZzKAmdLyTAAXBuEcfhwqrQQAbNh7LBSnYxiGqVcEKuwFgB+IaBUR3SC1tRJCFAGA9L9lgOewxcTeXJCEYRhGTaBZL4cJIfYRUUsAC4goz3QPCenhcAMAdOjQIcBh1PHipf1cOxbDMEy0EJBmL4TYJ/0/COBrAAMBHCCiTACQ/h/U2fcNIUS2ECI7IyMjkGEAAAZ08BQPT06ID/hYDMMw0YZjYU9EqUTUWF4GMA5ALoBZAKZI3aYA+DbQQVqhY4tGaM3FSBiGYTQJxIzTCsDXRCQf5yMhxDwiWgngMyKaCmA3gEsCH6YxczYU4cvVhcE+DcMwTL3FsbAXQuwA0Fej/TCAMYEMyi5//XB1KE/HMAxT74jaCFqGYRimDhb2DMMwMUC9F/alFdXhHgLDMEzEU++F/Wcr94R7CAzDMBFPvRf2S7YWe5ffuirboCfDMEzsEmgEbdhp3igJAPDtLcPQt316mEfDMAwTmdRrzf5kZTW+Wr0XANC+WcMwj4ZhGCZyqdfC/pFvN3qXGyXX+5cUhmGYoFGvhf1pCm0+KaFeXwrDMExQqdfq8G1juiI1OQF7S06FeygMwzARTb0W9gBw7fCO4R4CwzBMxMO2D4ZhmBiAhT3DMEwMwMKeYRgmBmBhzzAMEwOwsGcYhokBWNgzDMPEACzsGYZhYgAW9gzDMDEACSHCPQYQUTGAXQ53bwHgkIvDqQ/wNccGfM2xQSDXfJoQIsNKx4gQ9oFARDlCiJhKZM/XHBvwNccGobpmNuMwDMPEACzsGYZhYoBoEPZvhHsAYYCvOTbga44NQnLN9d5mzzAMw5gTDZo9wzAMYwILe4ZhmBigXgt7IppARFuIKJ+IpoV7PHYgovZEtJiINhPRRiL6m9TejIgWENE26X9TqZ2I6CXpWtcT0QDFsaZI/bcR0RRF+5lEtEHa5yUiotBfqT9EFE9Ea4joe2m9IxEtl8b/KRElSe3J0nq+tD1LcYz7pfYtRDRe0R5x3wkiSieiL4goT7rfQ6L9PhPR36XvdS4RfUxEKdF2n4nobSI6SES5irag31e9c5gihKiXfwDiAWwH0AlAEoB1AHqFe1w2xp8JYIC03BjAVgC9ADwLYJrUPg3AP6XliQDmAiAAgwEsl9qbAdgh/W8qLTeVtq0AMETaZy6A88J93dK47gTwEYDvpfXPAFwmLb8G4GZp+a8AXpOWLwPwqbTcS7rfyQA6St+D+Ej9TgCYCeA6aTkJQHo032cAbQHsBNBAcX+vjrb7DOBsAAMA5Cragn5f9c5hOt5w/xAC+KCHAJivWL8fwP3hHlcA1/MtgHMBbAGQKbVlAtgiLb8O4HJF/y3S9ssBvK5of11qywSQp2j36RfG62wHYCGA0QC+l77IhwAkqO8rgPkAhkjLCVI/Ut9ruV8kficApEmCj1TtUXuf4RH2eyQBliDd5/HReJ8BZMFX2Af9vuqdw+yvPptx5C+UTKHUVu+QXlv7A1gOoJUQoggApP8tpW5612vUXqjRHm7+DeBeALXSenMAJUKIamldOU7vtUnbj0n97X4W4aQTgGIA70imq7eIKBVRfJ+FEHsBPA9gN4AieO7bKkT3fZYJxX3VO4ch9VnYa9kl650fKRE1AvAlgDuEEMeNumq0CQftYYOIzgdwUAixStms0VWYbKs31wyPpjoAwKtCiP4AyuB59daj3l+zZEOeDI/ppQ2AVADnaXSNpvtsRtivsT4L+0IA7RXr7QDsC9NYHEFEifAI+g+FEF9JzQeIKFPangngoNSud71G7e002sPJMAB/IKICAJ/AY8r5N4B0IkqQ+ijH6b02aXsTAEdg/7MIJ4UACoUQy6X1L+AR/tF8n8cC2CmEKBZCVAH4CsBQRPd9lgnFfdU7hyH1WdivBNBVmuFPgmdiZ1aYx2QZaWZ9BoDNQoh/KTbNAiDPyE+Bx5Yvt18lzeoPBnBMeoWbD2AcETWVNKpx8NgziwCcIKLB0rmuUhwrLAgh7hdCtBNCZMFzvxYJIf4CYDGAi6Vu6muWP4uLpf5Car9M8uLoCKArPJNZEfedEELsB7CHiLpLTWMAbEIU32d4zDeDiaihNCb5mqP2PisIxX3VO4cx4ZzIcWFyZCI8XizbATwY7vHYHPtweF7L1gNYK/1NhMdWuRDANul/M6k/AXhFutYNALIVx7oWQL70d42iPRtArrTPy1BNEob5+kehzhunEzw/4nwAnwNIltpTpPV8aXsnxf4PSte1BQrvk0j8TgDoByBHutffwON1EdX3GcBjAPKkcb0Pj0dNVN1nAB/DMydRBY8mPjUU91XvHGZ/nC6BYRgmBqjPZhyGYRjGIizsGYZhYgAW9gzDMDEAC3uGYZgYgIU9wzBMDMDCnmEYJgZgYc8wDBMD/D9pwksMstgtRgAAAABJRU5ErkJggg==\n" }, "metadata": { "needs_background": "light" } } ], "source": [ "def running_average(x,window):\n", " return np.convolve(x,np.ones(window)/window,mode='valid')\n", "\n", "plt.plot(running_average(rewards,100))" ] }, { "source": [ "## Varying Hyperparameters and Seeing the Result in Action\n", "\n", "Now it would be interesting to actually see how the trained model behaves. Let's run the simulation, and we will be following the same action selection strategy as during training: sampling according to the probability distribution in Q-Table: " ], "cell_type": "markdown", "metadata": {} }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": [ "obs = env.reset()\n", "done = False\n", "while not done:\n", " s = discretize(obs)\n", " env.render()\n", " v = probs(np.array(qvalues(s)))\n", " a = random.choices(actions,weights=v)[0]\n", " obs,_,done,_ = env.step(a)\n", "env.close()" ] }, { "source": [ "\n", "## Saving result to an animated GIF\n", "\n", "If you want to impress your friends, you may want to send them the animated GIF picture of the balancing pole. To do this, we can invoke `env.render` to produce an image frame, and then save those to animated GIF using PIL library:" ], "cell_type": "markdown", "metadata": {} }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "360\n" ] } ], "source": [ "from PIL import Image\n", "obs = env.reset()\n", "done = False\n", "i=0\n", "ims = []\n", "while not done:\n", " s = discretize(obs)\n", " img=env.render(mode='rgb_array')\n", " ims.append(Image.fromarray(img))\n", " v = probs(np.array([Qbest.get((s,a),0) for a in actions]))\n", " a = random.choices(actions,weights=v)[0]\n", " obs,_,done,_ = env.step(a)\n", " i+=1\n", "env.close()\n", "ims[0].save('images/cartpole-balance.gif',save_all=True,append_images=ims[1::2],loop=0,duration=5)\n", "print(i)" ] } ] }