msb-public
/
ML-For-Beginners
mirror of https://github.com/microsoft/ML-For-Beginners.git
1
0
Code Issues Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '08b59dab18'
${ noResults }
ML-For-Beginners/2-Regression/2-Data/01-numpy-and-pandas.ipynb

2729 lines
95 KiB
Raw Blame History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exploring Data with Python\n",
"\n",
"A significant part of a a data scientist's role is to explore, analyze, and visualize data. There are many tools and programming languages that they can use to do this. One of the most popular approaches is to use Jupyter notebooks (like this one) and Python.\n",
"\n",
"Python is a flexible programming language that's used in a wide range of scenarios, from web applications to device programming. It's extremely popular in the data-science and machine-learning communities because of the many packages it supports for data analysis and visualization.\n",
"\n",
"In this notebook, we'll explore some of these packages and apply basic techniques to analyze data. This isn't intended to be a comprehensive Python programming exercise or even a deep dive into data analysis. Rather, it's intended as a crash course in some of the common ways in which data scientists can use Python to work with data.\n",
"\n",
"> **Note**: If you've never used the Jupyter Notebooks environment before, there are a few things of which you should be aware:\n",
"> \n",
"> - Notebooks are made up of *cells*. Some cells (like this one) contain *markdown* text, while others (like the one following this one) contain code.\n",
"> - You can run each code cell by using the **► Run** button. The **► Run** button shows up when you hover over the cell.\n",
"> - The output from each code cell is displayed immediately below the cell.\n",
"> - Even though you can run the code cells individually, some variables the code uses are global to the notebook. That means that you should run all of the code cells <u>**in order**</u>. There might be dependencies between code cells, so if you skip a cell, subsequent cells might not run correctly.\n",
"> \n",
"\n",
"\n",
"## Exploring data arrays with NumPy\n",
"\n",
"Let's start by looking at some simple data.\n",
"\n",
"Suppose a college professor takes a sample of student grades from a class to analyze.\n",
"\n",
"Run the code in the following cell by selecting the **&#9658; Run** button to see the data."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[50, 50, 47, 97, 49, 3, 53, 42, 26, 74, 82, 62, 37, 15, 70, 27, 36, 35, 48, 52, 63, 64]\n"
]
}
],
"source": [
"data = [50,50,47,97,49,3,53,42,26,74,82,62,37,15,70,27,36,35,48,52,63,64]\n",
"print(data)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The data has been loaded into a Python **list** structure, which is a good data type for general data manipulation, but it's not optimized for numeric analysis. For that, we're going to use the **NumPy** package, which includes specific data types and functions for working with *Num*bers in *Py*thon.\n",
"\n",
"Run the following cell to load the data into a NumPy **array**."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[50 50 47 97 49 3 53 42 26 74 82 62 37 15 70 27 36 35 48 52 63 64]\n"
]
}
],
"source": [
"import numpy as np\n",
"\n",
"grades = np.array(data)\n",
"print(grades)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Just in case you're wondering about the differences between a **list** and a NumPy **array**, let's compare how these data types behave when we use them in an expression that multiplies them by two."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'list'> x 2: [50, 50, 47, 97, 49, 3, 53, 42, 26, 74, 82, 62, 37, 15, 70, 27, 36, 35, 48, 52, 63, 64, 50, 50, 47, 97, 49, 3, 53, 42, 26, 74, 82, 62, 37, 15, 70, 27, 36, 35, 48, 52, 63, 64]\n",
"---\n",
"<class 'numpy.ndarray'> x 2: [100 100 94 194 98 6 106 84 52 148 164 124 74 30 140 54 72 70\n",
" 96 104 126 128]\n"
]
}
],
"source": [
"print (type(data),'x 2:', data * 2)\n",
"print('---')\n",
"print (type(grades),'x 2:', grades * 2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that multiplying a list by two creates a new list of twice the length with the original sequence of list elements repeated. Multiplying a NumPy array, on the other hand, performs an element-wise calculation in which the array behaves like a *vector*, so we end up with an array of the same size in which each element has been multiplied by two.\n",
"\n",
"The key takeaway from this is that NumPy arrays are specifically designed to support mathematical operations on numeric data, which makes them more useful for data analysis than a generic list.\n",
"\n",
"You might have spotted that the class type for the preceding NumPy array is **numpy.ndarray**. The **nd** indicates that this is a structure that can consist of multiple *dimensions*. (It can have *n* dimensions.) Our specific instance has a single dimension of student grades.\n",
"\n",
"Run the following cell to view the **shape** of the array."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(22,)"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"grades.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The shape confirms that this array has only one dimension, which contains 22 elements. (There are 22 grades in the original list.) You can access the individual elements in the array by their zero-based ordinal position. Let's get the first element (the one in position 0)."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"np.int64(50)"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"grades[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that you know your way around a NumPy array, it's time to perform some analysis of the grades data.\n",
"\n",
"You can apply aggregations across the elements in the array, so let's find the simple average grade (in other words, the *mean* grade value)."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"np.float64(49.18181818181818)"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"grades.mean()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So the mean grade is just around 50, more or less in the middle of the possible range from 0 to 100.\n",
"\n",
"Let's add a second set of data for the same students. This time, we'll record the typical number of hours per week they devoted to studying."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[10. , 11.5 , 9. , 16. , 9.25, 1. , 11.5 , 9. , 8.5 ,\n",
" 14.5 , 15.5 , 13.75, 9. , 8. , 15.5 , 8. , 9. , 6. ,\n",
" 10. , 12. , 12.5 , 12. ],\n",
" [50. , 50. , 47. , 97. , 49. , 3. , 53. , 42. , 26. ,\n",
" 74. , 82. , 62. , 37. , 15. , 70. , 27. , 36. , 35. ,\n",
" 48. , 52. , 63. , 64. ]])"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Define an array of study hours\n",
"study_hours = [10.0,11.5,9.0,16.0,9.25,1.0,11.5,9.0,8.5,14.5,15.5,\n",
" 13.75,9.0,8.0,15.5,8.0,9.0,6.0,10.0,12.0,12.5,12.0]\n",
"\n",
"# Create a 2D array (an array of arrays)\n",
"student_data = np.array([study_hours, grades])\n",
"\n",
"# display the array\n",
"student_data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now the data consists of a two-dimensional array; an array of arrays. Let's look at its shape."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(2, 22)"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Show shape of 2D array\n",
"student_data.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The **student_data** array contains two elements, each of which is an array containing 22 elements.\n",
"\n",
"To navigate this structure, you need to specify the position of each element in the hierarchy. So to find the first value in the first array (which contains the study hours data), you can use the following code."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"np.float64(10.0)"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Show the first element of the first element\n",
"student_data[0][0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now you have a multidimensional array containing both the student's study time and grade information, which you can use to compare study time to a student's grade."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Average study hours: 10.52\n",
"Average grade: 49.18\n"
]
}
],
"source": [
"# Get the mean value of each sub-array\n",
"avg_study = student_data[0].mean()\n",
"avg_grade = student_data[1].mean()\n",
"\n",
"print('Average study hours: {:.2f}\\nAverage grade: {:.2f}'.format(avg_study, avg_grade))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exploring tabular data with Pandas\n",
"\n",
"NumPy provides a lot of the functionality and tools you need to work with numbers, such as arrays of numeric values. However, when you start to deal with two-dimensional tables of data, the **Pandas** package offers a more convenient structure to work with: the **DataFrame**.\n",
"\n",
"Run the following cell to import the Pandas library and create a DataFrame with three columns. The first column is a list of student names, and the second and third columns are the NumPy arrays containing the study time and grade data."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Name</th>\n",
" <th>StudyHours</th>\n",
" <th>Grade</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Dan</td>\n",
" <td>10.00</td>\n",
" <td>50.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Joann</td>\n",
" <td>11.50</td>\n",
" <td>50.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Pedro</td>\n",
" <td>9.00</td>\n",
" <td>47.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Rosie</td>\n",
" <td>16.00</td>\n",
" <td>97.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Ethan</td>\n",
" <td>9.25</td>\n",
" <td>49.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Vicky</td>\n",
" <td>1.00</td>\n",
" <td>3.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Frederic</td>\n",
" <td>11.50</td>\n",
" <td>53.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Jimmie</td>\n",
" <td>9.00</td>\n",
" <td>42.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Rhonda</td>\n",
" <td>8.50</td>\n",
" <td>26.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Giovanni</td>\n",
" <td>14.50</td>\n",
" <td>74.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>Francesca</td>\n",
" <td>15.50</td>\n",
" <td>82.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>Rajab</td>\n",
" <td>13.75</td>\n",
" <td>62.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>Naiyana</td>\n",
" <td>9.00</td>\n",
" <td>37.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>Kian</td>\n",
" <td>8.00</td>\n",
" <td>15.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>Jenny</td>\n",
" <td>15.50</td>\n",
" <td>70.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>Jakeem</td>\n",
" <td>8.00</td>\n",
" <td>27.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>Helena</td>\n",
" <td>9.00</td>\n",
" <td>36.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>Ismat</td>\n",
" <td>6.00</td>\n",
" <td>35.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>Anila</td>\n",
" <td>10.00</td>\n",
" <td>48.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>Skye</td>\n",
" <td>12.00</td>\n",
" <td>52.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>Daniel</td>\n",
" <td>12.50</td>\n",
" <td>63.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>Aisha</td>\n",
" <td>12.00</td>\n",
" <td>64.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Name StudyHours Grade\n",
"0 Dan 10.00 50.0\n",
"1 Joann 11.50 50.0\n",
"2 Pedro 9.00 47.0\n",
"3 Rosie 16.00 97.0\n",
"4 Ethan 9.25 49.0\n",
"5 Vicky 1.00 3.0\n",
"6 Frederic 11.50 53.0\n",
"7 Jimmie 9.00 42.0\n",
"8 Rhonda 8.50 26.0\n",
"9 Giovanni 14.50 74.0\n",
"10 Francesca 15.50 82.0\n",
"11 Rajab 13.75 62.0\n",
"12 Naiyana 9.00 37.0\n",
"13 Kian 8.00 15.0\n",
"14 Jenny 15.50 70.0\n",
"15 Jakeem 8.00 27.0\n",
"16 Helena 9.00 36.0\n",
"17 Ismat 6.00 35.0\n",
"18 Anila 10.00 48.0\n",
"19 Skye 12.00 52.0\n",
"20 Daniel 12.50 63.0\n",
"21 Aisha 12.00 64.0"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"\n",
"df_students = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', \n",
" 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny',\n",
" 'Jakeem','Helena','Ismat','Anila','Skye','Daniel','Aisha'],\n",
" 'StudyHours':student_data[0],\n",
" 'Grade':student_data[1]})\n",
"\n",
"df_students "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that in addition to the columns you specified, the DataFrame includes an *index* to uniquely identify each row. We could've specified the index explicitly and assigned any kind of appropriate value (for example, an email address). However, because we didn't specify an index, one has been created with a unique integer value for each row.\n",
"\n",
"### Finding and filtering data in a DataFrame\n",
"\n",
"You can use the DataFrame's **loc** method to retrieve data for a specific index value, like this."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"Name Vicky\n",
"StudyHours 1.0\n",
"Grade 3.0\n",
"Name: 5, dtype: object"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Get the data for index value 5\n",
"df_students.loc[5]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can also get the data at a range of index values, like this:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Name</th>\n",
" <th>StudyHours</th>\n",
" <th>Grade</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Dan</td>\n",
" <td>10.00</td>\n",
" <td>50.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Joann</td>\n",
" <td>11.50</td>\n",
" <td>50.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Pedro</td>\n",
" <td>9.00</td>\n",
" <td>47.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Rosie</td>\n",
" <td>16.00</td>\n",
" <td>97.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Ethan</td>\n",
" <td>9.25</td>\n",
" <td>49.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Vicky</td>\n",
" <td>1.00</td>\n",
" <td>3.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Name StudyHours Grade\n",
"0 Dan 10.00 50.0\n",
"1 Joann 11.50 50.0\n",
"2 Pedro 9.00 47.0\n",
"3 Rosie 16.00 97.0\n",
"4 Ethan 9.25 49.0\n",
"5 Vicky 1.00 3.0"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Get the rows with index values from 0 to 5\n",
"df_students.loc[0:5]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In addition to being able to use the **loc** method to find rows based on the index, you can use the **iloc** method to find rows based on their ordinal position in the DataFrame (regardless of the index):"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Name</th>\n",
" <th>StudyHours</th>\n",
" <th>Grade</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Dan</td>\n",
" <td>10.00</td>\n",
" <td>50.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Joann</td>\n",
" <td>11.50</td>\n",
" <td>50.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Pedro</td>\n",
" <td>9.00</td>\n",
" <td>47.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Rosie</td>\n",
" <td>16.00</td>\n",
" <td>97.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Ethan</td>\n",
" <td>9.25</td>\n",
" <td>49.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Name StudyHours Grade\n",
"0 Dan 10.00 50.0\n",
"1 Joann 11.50 50.0\n",
"2 Pedro 9.00 47.0\n",
"3 Rosie 16.00 97.0\n",
"4 Ethan 9.25 49.0"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Get data in the first five rows\n",
"df_students.iloc[0:5]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Look carefully at the `iloc[0:5]` results and compare them to the `loc[0:5]` results you obtained previously. Can you spot the difference?\n",
"\n",
"The **loc** method returned rows with index *label* in the list of values from *0* to *5*, which includes *0*, *1*, *2*, *3*, *4*, and *5* (six rows). However, the **iloc** method returns the rows in the *positions* included in the range 0 to 5. Since integer ranges don't include the upper-bound value, this includes positions *0*, *1*, *2*, *3*, and *4* (five rows).\n",
"\n",
"**iloc** identifies data values in a DataFrame by *position*, which extends beyond rows to columns. So, for example, you can use it to find the values for the columns in positions 1 and 2 in row 0, like this:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"StudyHours 10.0\n",
"Grade 50.0\n",
"Name: 0, dtype: object"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_students.iloc[0,[1,2]]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's return to the **loc** method and see how it works with columns. Remember that you use **loc** to locate data items based on index values rather than positions. In the absence of an explicit index column, the rows in our DataFrame are indexed as integer values, but the columns are identified by name:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"np.float64(50.0)"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_students.loc[0,'Grade']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here's another useful trick. You can use the **loc** method to find indexed rows based on a filtering expression that references named columns other than the index, like this:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Name</th>\n",
" <th>StudyHours</th>\n",
" <th>Grade</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>Aisha</td>\n",
" <td>12.0</td>\n",
" <td>64.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Name StudyHours Grade\n",
"21 Aisha 12.0 64.0"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_students.loc[df_students['Name']=='Aisha']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Actually, you don't need to explicitly use the **loc** method to do this. You can simply apply a DataFrame filtering expression, like this:"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Name</th>\n",
" <th>StudyHours</th>\n",
" <th>Grade</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>Aisha</td>\n",
" <td>12.0</td>\n",
" <td>64.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Name StudyHours Grade\n",
"21 Aisha 12.0 64.0"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_students[df_students['Name']=='Aisha']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And for good measure, you can achieve the same results by using the DataFrame's **query** method, like this:"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Name</th>\n",
" <th>StudyHours</th>\n",
" <th>Grade</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>Aisha</td>\n",
" <td>12.0</td>\n",
" <td>64.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Name StudyHours Grade\n",
"21 Aisha 12.0 64.0"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_students.query('Name==\"Aisha\"')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The three previous examples underline a confusing truth about working with Pandas. Often, there are multiple ways to achieve the same results. Another example of this is the way you refer to a DataFrame column name. You can specify the column name as a named index value (as in the `df_students['Name']` examples we've seen so far), or you can use the column as a property of the DataFrame, like this:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Name</th>\n",
" <th>StudyHours</th>\n",
" <th>Grade</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>Aisha</td>\n",
" <td>12.0</td>\n",
" <td>64.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Name StudyHours Grade\n",
"21 Aisha 12.0 64.0"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_students[df_students.Name == 'Aisha']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Loading a DataFrame from a file\n",
"\n",
"We constructed the DataFrame from some existing arrays. However, in many real-world scenarios, data is loaded from sources such as files. Let's replace the student grades DataFrame with the contents of a text file."
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Name</th>\n",
" <th>StudyHours</th>\n",
" <th>Grade</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Dan</td>\n",
" <td>10.00</td>\n",
" <td>50.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Joann</td>\n",
" <td>11.50</td>\n",
" <td>50.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Pedro</td>\n",
" <td>9.00</td>\n",
" <td>47.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Rosie</td>\n",
" <td>16.00</td>\n",
" <td>97.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Ethan</td>\n",
" <td>9.25</td>\n",
" <td>49.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Name StudyHours Grade\n",
"0 Dan 10.00 50.0\n",
"1 Joann 11.50 50.0\n",
"2 Pedro 9.00 47.0\n",
"3 Rosie 16.00 97.0\n",
"4 Ethan 9.25 49.0"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"\n",
"df_students = pd.read_csv('grades.csv',delimiter=',',header='infer')\n",
"df_students.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The DataFrame's **read_csv** method is used to load data from text files. As you can see in the example code, you can specify options such as the column delimiter and which row (if any) contains column headers. (In this case, the delimiter is a comma and the first row contains the column names. These are the default settings, so we could've omitted the parameters.)\n",
"\n",
"\n",
"### Handling missing values\n",
"\n",
"One of the most common issues data scientists need to deal with is incomplete or missing data. So how would we know that the DataFrame contains missing values? You can use the **isnull** method to identify which individual values are null, like this:"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Name</th>\n",
" <th>StudyHours</th>\n",
" <th>Grade</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>22</th>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23</th>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Name StudyHours Grade\n",
"0 False False False\n",
"1 False False False\n",
"2 False False False\n",
"3 False False False\n",
"4 False False False\n",
"5 False False False\n",
"6 False False False\n",
"7 False False False\n",
"8 False False False\n",
"9 False False False\n",
"10 False False False\n",
"11 False False False\n",
"12 False False False\n",
"13 False False False\n",
"14 False False False\n",
"15 False False False\n",
"16 False False False\n",
"17 False False False\n",
"18 False False False\n",
"19 False False False\n",
"20 False False False\n",
"21 False False False\n",
"22 False False True\n",
"23 False True True"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_students.isnull()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Of course, with a larger DataFrame, it would be inefficient to review all of the rows and columns individually, so we can get the sum of missing values for each column like this:"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Name 0\n",
"StudyHours 1\n",
"Grade 2\n",
"dtype: int64"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_students.isnull().sum()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So now we know that there's one missing **StudyHours** value and two missing **Grade** values.\n",
"\n",
"To see them in context, we can filter the DataFrame to include only rows where any of the columns (axis 1 of the DataFrame) are null."
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Name</th>\n",
" <th>StudyHours</th>\n",
" <th>Grade</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>22</th>\n",
" <td>Bill</td>\n",
" <td>8.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23</th>\n",
" <td>Ted</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Name StudyHours Grade\n",
"22 Bill 8.0 NaN\n",
"23 Ted NaN NaN"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_students[df_students.isnull().any(axis=1)]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When the DataFrame is retrieved, the missing numeric values show up as **NaN** (*not a number*).\n",
"\n",
"So now that we've found the null values, what can we do about them?\n",
"\n",
"One common approach is to *impute* replacement values. For example, if the number of study hours is missing, we could just assume that the student studied for an average amount of time and replace the missing value with the mean study hours. To do this, we can use the **fillna** method, like this:"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Name</th>\n",
" <th>StudyHours</th>\n",
" <th>Grade</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Dan</td>\n",
" <td>10.000000</td>\n",
" <td>50.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Joann</td>\n",
" <td>11.500000</td>\n",
" <td>50.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Pedro</td>\n",
" <td>9.000000</td>\n",
" <td>47.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Rosie</td>\n",
" <td>16.000000</td>\n",
" <td>97.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Ethan</td>\n",
" <td>9.250000</td>\n",
" <td>49.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Vicky</td>\n",
" <td>1.000000</td>\n",
" <td>3.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Frederic</td>\n",
" <td>11.500000</td>\n",
" <td>53.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Jimmie</td>\n",
" <td>9.000000</td>\n",
" <td>42.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Rhonda</td>\n",
" <td>8.500000</td>\n",
" <td>26.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Giovanni</td>\n",
" <td>14.500000</td>\n",
" <td>74.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>Francesca</td>\n",
" <td>15.500000</td>\n",
" <td>82.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>Rajab</td>\n",
" <td>13.750000</td>\n",
" <td>62.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>Naiyana</td>\n",
" <td>9.000000</td>\n",
" <td>37.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>Kian</td>\n",
" <td>8.000000</td>\n",
" <td>15.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>Jenny</td>\n",
" <td>15.500000</td>\n",
" <td>70.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>Jakeem</td>\n",
" <td>8.000000</td>\n",
" <td>27.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>Helena</td>\n",
" <td>9.000000</td>\n",
" <td>36.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>Ismat</td>\n",
" <td>6.000000</td>\n",
" <td>35.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>Anila</td>\n",
" <td>10.000000</td>\n",
" <td>48.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>Skye</td>\n",
" <td>12.000000</td>\n",
" <td>52.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>Daniel</td>\n",
" <td>12.500000</td>\n",
" <td>63.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>Aisha</td>\n",
" <td>12.000000</td>\n",
" <td>64.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>22</th>\n",
" <td>Bill</td>\n",
" <td>8.000000</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23</th>\n",
" <td>Ted</td>\n",
" <td>10.413043</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Name StudyHours Grade\n",
"0 Dan 10.000000 50.0\n",
"1 Joann 11.500000 50.0\n",
"2 Pedro 9.000000 47.0\n",
"3 Rosie 16.000000 97.0\n",
"4 Ethan 9.250000 49.0\n",
"5 Vicky 1.000000 3.0\n",
"6 Frederic 11.500000 53.0\n",
"7 Jimmie 9.000000 42.0\n",
"8 Rhonda 8.500000 26.0\n",
"9 Giovanni 14.500000 74.0\n",
"10 Francesca 15.500000 82.0\n",
"11 Rajab 13.750000 62.0\n",
"12 Naiyana 9.000000 37.0\n",
"13 Kian 8.000000 15.0\n",
"14 Jenny 15.500000 70.0\n",
"15 Jakeem 8.000000 27.0\n",
"16 Helena 9.000000 36.0\n",
"17 Ismat 6.000000 35.0\n",
"18 Anila 10.000000 48.0\n",
"19 Skye 12.000000 52.0\n",
"20 Daniel 12.500000 63.0\n",
"21 Aisha 12.000000 64.0\n",
"22 Bill 8.000000 NaN\n",
"23 Ted 10.413043 NaN"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_students.StudyHours = df_students.StudyHours.fillna(df_students.StudyHours.mean())\n",
"df_students"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Alternatively, it might be important to ensure that you only use data you know to be absolutely correct. In this case, you can drop rows or columns that contain null values by using the **dropna** method. For example, we'll remove rows (axis 0 of the DataFrame) where any of the columns contain null values:"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Name</th>\n",
" <th>StudyHours</th>\n",
" <th>Grade</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Dan</td>\n",
" <td>10.00</td>\n",
" <td>50.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Joann</td>\n",
" <td>11.50</td>\n",
" <td>50.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Pedro</td>\n",
" <td>9.00</td>\n",
" <td>47.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Rosie</td>\n",
" <td>16.00</td>\n",
" <td>97.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Ethan</td>\n",
" <td>9.25</td>\n",
" <td>49.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Vicky</td>\n",
" <td>1.00</td>\n",
" <td>3.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Frederic</td>\n",
" <td>11.50</td>\n",
" <td>53.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Jimmie</td>\n",
" <td>9.00</td>\n",
" <td>42.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Rhonda</td>\n",
" <td>8.50</td>\n",
" <td>26.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Giovanni</td>\n",
" <td>14.50</td>\n",
" <td>74.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>Francesca</td>\n",
" <td>15.50</td>\n",
" <td>82.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>Rajab</td>\n",
" <td>13.75</td>\n",
" <td>62.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>Naiyana</td>\n",
" <td>9.00</td>\n",
" <td>37.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>Kian</td>\n",
" <td>8.00</td>\n",
" <td>15.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>Jenny</td>\n",
" <td>15.50</td>\n",
" <td>70.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>Jakeem</td>\n",
" <td>8.00</td>\n",
" <td>27.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>Helena</td>\n",
" <td>9.00</td>\n",
" <td>36.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>Ismat</td>\n",
" <td>6.00</td>\n",
" <td>35.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>Anila</td>\n",
" <td>10.00</td>\n",
" <td>48.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>Skye</td>\n",
" <td>12.00</td>\n",
" <td>52.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>Daniel</td>\n",
" <td>12.50</td>\n",
" <td>63.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>Aisha</td>\n",
" <td>12.00</td>\n",
" <td>64.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Name StudyHours Grade\n",
"0 Dan 10.00 50.0\n",
"1 Joann 11.50 50.0\n",
"2 Pedro 9.00 47.0\n",
"3 Rosie 16.00 97.0\n",
"4 Ethan 9.25 49.0\n",
"5 Vicky 1.00 3.0\n",
"6 Frederic 11.50 53.0\n",
"7 Jimmie 9.00 42.0\n",
"8 Rhonda 8.50 26.0\n",
"9 Giovanni 14.50 74.0\n",
"10 Francesca 15.50 82.0\n",
"11 Rajab 13.75 62.0\n",
"12 Naiyana 9.00 37.0\n",
"13 Kian 8.00 15.0\n",
"14 Jenny 15.50 70.0\n",
"15 Jakeem 8.00 27.0\n",
"16 Helena 9.00 36.0\n",
"17 Ismat 6.00 35.0\n",
"18 Anila 10.00 48.0\n",
"19 Skye 12.00 52.0\n",
"20 Daniel 12.50 63.0\n",
"21 Aisha 12.00 64.0"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_students = df_students.dropna(axis=0, how='any')\n",
"df_students"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Explore data in the DataFrame\n",
"\n",
"Now that we've cleaned up the missing values, we're ready to explore the data in the DataFrame. Let's start by comparing the mean study hours and grades."
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Average weekly study hours: 10.52\n",
"Average grade: 49.18\n"
]
}
],
"source": [
"# Get the mean study hours using to column name as an index\n",
"mean_study = df_students['StudyHours'].mean()\n",
"\n",
"# Get the mean grade using the column name as a property (just to make the point!)\n",
"mean_grade = df_students.Grade.mean()\n",
"\n",
"# Print the mean study hours and mean grade\n",
"print('Average weekly study hours: {:.2f}\\nAverage grade: {:.2f}'.format(mean_study, mean_grade))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"OK, let's filter the DataFrame to find only the students who studied for more than the average amount of time."
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Name</th>\n",
" <th>StudyHours</th>\n",
" <th>Grade</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Joann</td>\n",
" <td>11.50</td>\n",
" <td>50.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Rosie</td>\n",
" <td>16.00</td>\n",
" <td>97.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Frederic</td>\n",
" <td>11.50</td>\n",
" <td>53.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Giovanni</td>\n",
" <td>14.50</td>\n",
" <td>74.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>Francesca</td>\n",
" <td>15.50</td>\n",
" <td>82.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>Rajab</td>\n",
" <td>13.75</td>\n",
" <td>62.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>Jenny</td>\n",
" <td>15.50</td>\n",
" <td>70.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>Skye</td>\n",
" <td>12.00</td>\n",
" <td>52.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>Daniel</td>\n",
" <td>12.50</td>\n",
" <td>63.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>Aisha</td>\n",
" <td>12.00</td>\n",
" <td>64.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Name StudyHours Grade\n",
"1 Joann 11.50 50.0\n",
"3 Rosie 16.00 97.0\n",
"6 Frederic 11.50 53.0\n",
"9 Giovanni 14.50 74.0\n",
"10 Francesca 15.50 82.0\n",
"11 Rajab 13.75 62.0\n",
"14 Jenny 15.50 70.0\n",
"19 Skye 12.00 52.0\n",
"20 Daniel 12.50 63.0\n",
"21 Aisha 12.00 64.0"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Get students who studied for the mean or more hours\n",
"df_students[df_students.StudyHours > mean_study]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that the filtered result is itself a DataFrame, so you can work with its columns just like any other DataFrame.\n",
"\n",
"For example, let's find the average grade for students who undertook more than the average amount of study time."
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"np.float64(66.7)"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# What was their mean grade?\n",
"df_students[df_students.StudyHours > mean_study].Grade.mean()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's assume that the passing grade for the course is 60.\n",
"\n",
"We can use that information to add a new column to the DataFrame that indicates whether or not each student passed.\n",
"\n",
"First, we'll create a Pandas **Series** containing the pass/fail indicator (True or False), and then we'll concatenate that series as a new column (axis 1) in the DataFrame."
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Name</th>\n",
" <th>StudyHours</th>\n",
" <th>Grade</th>\n",
" <th>Pass</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Dan</td>\n",
" <td>10.00</td>\n",
" <td>50.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Joann</td>\n",
" <td>11.50</td>\n",
" <td>50.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Pedro</td>\n",
" <td>9.00</td>\n",
" <td>47.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Rosie</td>\n",
" <td>16.00</td>\n",
" <td>97.0</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Ethan</td>\n",
" <td>9.25</td>\n",
" <td>49.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Vicky</td>\n",
" <td>1.00</td>\n",
" <td>3.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Frederic</td>\n",
" <td>11.50</td>\n",
" <td>53.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Jimmie</td>\n",
" <td>9.00</td>\n",
" <td>42.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Rhonda</td>\n",
" <td>8.50</td>\n",
" <td>26.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Giovanni</td>\n",
" <td>14.50</td>\n",
" <td>74.0</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>Francesca</td>\n",
" <td>15.50</td>\n",
" <td>82.0</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>Rajab</td>\n",
" <td>13.75</td>\n",
" <td>62.0</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>Naiyana</td>\n",
" <td>9.00</td>\n",
" <td>37.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>Kian</td>\n",
" <td>8.00</td>\n",
" <td>15.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>Jenny</td>\n",
" <td>15.50</td>\n",
" <td>70.0</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>Jakeem</td>\n",
" <td>8.00</td>\n",
" <td>27.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>Helena</td>\n",
" <td>9.00</td>\n",
" <td>36.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>Ismat</td>\n",
" <td>6.00</td>\n",
" <td>35.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>Anila</td>\n",
" <td>10.00</td>\n",
" <td>48.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>Skye</td>\n",
" <td>12.00</td>\n",
" <td>52.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>Daniel</td>\n",
" <td>12.50</td>\n",
" <td>63.0</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>Aisha</td>\n",
" <td>12.00</td>\n",
" <td>64.0</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Name StudyHours Grade Pass\n",
"0 Dan 10.00 50.0 False\n",
"1 Joann 11.50 50.0 False\n",
"2 Pedro 9.00 47.0 False\n",
"3 Rosie 16.00 97.0 True\n",
"4 Ethan 9.25 49.0 False\n",
"5 Vicky 1.00 3.0 False\n",
"6 Frederic 11.50 53.0 False\n",
"7 Jimmie 9.00 42.0 False\n",
"8 Rhonda 8.50 26.0 False\n",
"9 Giovanni 14.50 74.0 True\n",
"10 Francesca 15.50 82.0 True\n",
"11 Rajab 13.75 62.0 True\n",
"12 Naiyana 9.00 37.0 False\n",
"13 Kian 8.00 15.0 False\n",
"14 Jenny 15.50 70.0 True\n",
"15 Jakeem 8.00 27.0 False\n",
"16 Helena 9.00 36.0 False\n",
"17 Ismat 6.00 35.0 False\n",
"18 Anila 10.00 48.0 False\n",
"19 Skye 12.00 52.0 False\n",
"20 Daniel 12.50 63.0 True\n",
"21 Aisha 12.00 64.0 True"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"passes = pd.Series(df_students['Grade'] >= 60)\n",
"df_students = pd.concat([df_students, passes.rename(\"Pass\")], axis=1)\n",
"\n",
"df_students"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"DataFrames are designed for tabular data, and you can use them to perform many of the same kinds of data-analytics operations you can do in a relational database, such as grouping and aggregating tables of data.\n",
"\n",
"For example, you can use the **groupby** method to group the student data into groups based on the **Pass** column you added previously and to count the number of names in each group. In other words, you can determine how many students passed and failed."
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Pass\n",
"False 15\n",
"True 7\n",
"Name: Name, dtype: int64\n"
]
}
],
"source": [
"print(df_students.groupby(df_students.Pass).Name.count())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can aggregate multiple fields in a group using any available aggregation function. For example, you can find the mean study time and grade for the groups of students who passed and failed the course."
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" StudyHours Grade\n",
"Pass \n",
"False 8.783333 38.000000\n",
"True 14.250000 73.142857\n"
]
}
],
"source": [
"print(df_students.groupby(df_students.Pass)[['StudyHours', 'Grade']].mean())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"DataFrames are amazingly versatile and make it easy to manipulate data. Many DataFrame operations return a new copy of the DataFrame, so if you want to modify a DataFrame but keep the existing variable, you need to assign the result of the operation to the existing variable. For example, the following code sorts the student data into descending order by Grade and assigns the resulting sorted DataFrame to the original **df_students** variable."
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Name</th>\n",
" <th>StudyHours</th>\n",
" <th>Grade</th>\n",
" <th>Pass</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Rosie</td>\n",
" <td>16.00</td>\n",
" <td>97.0</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>Francesca</td>\n",
" <td>15.50</td>\n",
" <td>82.0</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Giovanni</td>\n",
" <td>14.50</td>\n",
" <td>74.0</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>Jenny</td>\n",
" <td>15.50</td>\n",
" <td>70.0</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>Aisha</td>\n",
" <td>12.00</td>\n",
" <td>64.0</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>Daniel</td>\n",
" <td>12.50</td>\n",
" <td>63.0</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>Rajab</td>\n",
" <td>13.75</td>\n",
" <td>62.0</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Frederic</td>\n",
" <td>11.50</td>\n",
" <td>53.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>Skye</td>\n",
" <td>12.00</td>\n",
" <td>52.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Dan</td>\n",
" <td>10.00</td>\n",
" <td>50.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Joann</td>\n",
" <td>11.50</td>\n",
" <td>50.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Ethan</td>\n",
" <td>9.25</td>\n",
" <td>49.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>Anila</td>\n",
" <td>10.00</td>\n",
" <td>48.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Pedro</td>\n",
" <td>9.00</td>\n",
" <td>47.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Jimmie</td>\n",
" <td>9.00</td>\n",
" <td>42.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>Naiyana</td>\n",
" <td>9.00</td>\n",
" <td>37.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>Helena</td>\n",
" <td>9.00</td>\n",
" <td>36.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>Ismat</td>\n",
" <td>6.00</td>\n",
" <td>35.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>Jakeem</td>\n",
" <td>8.00</td>\n",
" <td>27.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Rhonda</td>\n",
" <td>8.50</td>\n",
" <td>26.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>Kian</td>\n",
" <td>8.00</td>\n",
" <td>15.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Vicky</td>\n",
" <td>1.00</td>\n",
" <td>3.0</td>\n",
" <td>False</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Name StudyHours Grade Pass\n",
"3 Rosie 16.00 97.0 True\n",
"10 Francesca 15.50 82.0 True\n",
"9 Giovanni 14.50 74.0 True\n",
"14 Jenny 15.50 70.0 True\n",
"21 Aisha 12.00 64.0 True\n",
"20 Daniel 12.50 63.0 True\n",
"11 Rajab 13.75 62.0 True\n",
"6 Frederic 11.50 53.0 False\n",
"19 Skye 12.00 52.0 False\n",
"0 Dan 10.00 50.0 False\n",
"1 Joann 11.50 50.0 False\n",
"4 Ethan 9.25 49.0 False\n",
"18 Anila 10.00 48.0 False\n",
"2 Pedro 9.00 47.0 False\n",
"7 Jimmie 9.00 42.0 False\n",
"12 Naiyana 9.00 37.0 False\n",
"16 Helena 9.00 36.0 False\n",
"17 Ismat 6.00 35.0 False\n",
"15 Jakeem 8.00 27.0 False\n",
"8 Rhonda 8.50 26.0 False\n",
"13 Kian 8.00 15.0 False\n",
"5 Vicky 1.00 3.0 False"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Create a DataFrame with the data sorted by Grade (descending)\n",
"df_students = df_students.sort_values('Grade', ascending=False)\n",
"\n",
"# Show the DataFrame\n",
"df_students"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"NumPy and DataFrames are the workhorses of data science in Python. They provide us ways to load, explore, and analyze tabular data. As we will learn in subsequent modules, even advanced analysis methods typically rely on NumPy and Pandas for these important roles.\n",
"\n",
"In our next workbook, we'll take a look at how create graphs and explore your data in more interesting ways."
]
}
],
"metadata": {
"kernel_info": {
"name": "conda-env-azureml_py38-py"
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.5"
},
"nteract": {
"version": "nteract-front-end@1.0.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Reference in new issue View Git Blame Copy Permalink