msb-public
/
Data-Science-For-Beginners
mirror of https://github.com/microsoft/Data-Science-For-Beginners.git
1
0
Code Issues Projects Releases Wiki Activity

Delete 2-Working-With-Data/07-python/notebook.ipynb

Browse Source
pull/724/head
wangamber8-collab 2 weeks ago committed by GitHub
parent 6df2b1a2ad
commit ebf274d3c4
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 0 additions and 390 deletions
Show Stats Download Patch File Download Diff File

390
2-Working-With-Data/07-python/notebook.ipynb
Unescape View File

@ -1,390 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Basic Pandas Examples\n",
"\n",
"This notebook will walk you through some very basic Pandas concepts. We will start with importing typical data science libraries:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Series\n",
"\n",
"Series is like a list or 1D-array, but with index. All operations are index-aligned."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"a = pd.Series(range(1,10))\n",
"b = pd.Series([\"I\",\"like\",\"to\",\"use\",\"Python\",\"and\",\"Pandas\",\"very\",\"much\"],index=range(0,9))\n",
"print(a,b)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"One of the frequent usage of series is **time series**. In time series, index has a special structure - typically a range of dates or datetimes. We can create such an index with `pd.date_range`.\n",
"\n",
"Suppose we have a series that shows the amount of product bought every day, and we know that every sunday we also need to take one item for ourselves. Here is how to model that using series:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start_date = \"Jan 1, 2020\"\n",
"end_date = \"Dec 31, 2020\"\n",
"idx = pd.date_range(start_date,end_date)\n",
"print(f\"Length of index is {len(idx)}\")\n",
"items_sold = pd.Series(np.random.randint(25,50,size=len(idx)),index=idx)\n",
"items_sold.plot(figsize=(10,3))\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"additional_items = pd.Series(10,index=pd.date_range(start_date,end_date,freq=\"W\"))\n",
"print(f\"Additional items (10 item each week):\\n{additional_items}\")\n",
"total_items = items_sold+additional_items\n",
"print(f\"Total items (sum of two series):\\n{total_items}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As you can see, we are having problems here, because in the weekly series non-mentioned days are considered to be missing (`NaN`), and adding `NaN` to a number gives us `NaN`. In order to get correct result, we need to specify `fill_value` when adding series:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"total_items = items_sold.add(additional_items,fill_value=0)\n",
"print(total_items)\n",
"total_items.plot(figsize=(10,3))\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"monthly = total_items.resample(\"1M\").mean()\n",
"ax = monthly.plot(kind='bar',figsize=(10,3))\n",
"ax.set_xticklabels([x.strftime(\"%b-%Y\") for x in monthly.index], rotation=45)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## DataFrame\n",
"\n",
"A dataframe is essentially a collection of series with the same index. We can combine several series together into a dataframe. Given `a` and `b` series defined above:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df = pd.DataFrame([a,b])\n",
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also use Series as columns, and specify column names using dictionary:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df = pd.DataFrame({ 'A' : a, 'B' : b })\n",
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The same result can be achieved by transposing (and then renaming columns, to match the previous example):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pd.DataFrame([a,b]).T.rename(columns={ 0 : 'A', 1 : 'B' })"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Selecting columns** from DataFrame can be done like this:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(f\"Column A (series):\\n{df['A']}\")\n",
"print(f\"Columns B and A (DataFrame):\\n{df[['B','A']]}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Selecting rows** based on filter expression:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df[df['A']<5]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The way it works is that expression `df['A']<5` returns a boolean series, which indicates whether expression is `True` or `False` for each elements of the series. When series is used as an index, it returns subset of rows in the DataFrame. Thus it is not possible to use arbitrary Python boolean expression, for example, writing `df[df['A']>5 and df['A']<7]` would be wrong. Instead, you should use special `&` operation on boolean series:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df[(df['A']>5) & (df['A']<7)]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Creating new computable columns**. We can easily create new computable columns for our DataFrame by using intuitive expressions. The code below calculates divergence of A from its mean value."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df['DivA'] = df['A']-df['A'].mean()\n",
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What actually happens is we are computing a series, and then assigning this series to the left-hand-side, creating another column."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# WRONG: df['ADescr'] = \"Low\" if df['A'] < 5 else \"Hi\"\n",
"df['LenB'] = len(df['B']) # Wrong result"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df['LenB'] = df['B'].apply(lambda x: len(x))\n",
"# or\n",
"df['LenB'] = df['B'].apply(len)\n",
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Selecting rows based on numbers** can be done using `iloc` construct. For example, to select first 5 rows from the DataFrame:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df.iloc[:5]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Grouping** is often used to get a result similar to *pivot tables* in Excel. Suppose that we want to compute mean value of column `A` for each given number of `LenB`. Then we can group our DataFrame by `LenB`, and call `mean`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df.groupby(by='LenB').mean()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If we need to compute mean and the number of elements in the group, then we can use more complex `aggregate` function:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df.groupby(by='LenB') \\\n",
" .aggregate({ 'DivA' : len, 'A' : lambda x: x.mean() }) \\\n",
" .rename(columns={ 'DivA' : 'Count', 'A' : 'Mean'})"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Printing and Plotting\n",
"\n",
"Data Scientist often has to explore the data, thus it is important to be able to visualize it. When DataFrame is big, many times we want just to make sure we are doing everything correctly by printing out the first few rows. This can be done by calling `df.head()`. If you are running it from Jupyter Notebook, it will print out the DataFrame in a nice tabular form."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"We have also seen the usage of `plot` function to visualize some columns. While `plot` is very useful for many tasks, and supports many different graph types via `kind=` parameter, you can always use raw `matplotlib` library to plot something more complex. We will cover data visualization in detail in separate course lessons.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df['A'].plot()\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df['A'].plot(kind='bar')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"This overview covers most important concepts of Pandas, however, the library is very rich, and there is no limit to what you can do with it! Let's now apply this knowledge for solving specific problem."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
}
],
"metadata": {
"interpreter": {
"hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
},
"kernelspec": {
"display_name": "Python 3.8.8 64-bit (conda)",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.8"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Write Preview
Loading…
Cancel
Save