Jasmine Greenaway
3 years ago
committed by
GitHub
parent
7826170f20
commit
6aa3bac052
@ -1,92 +0,0 @@
|
||||
# Working with Data: Python and the Pandas Library
|
||||
|
||||
While databases offer very efficient ways to store data and query them using query languages, the most flexible way of data processing is writing your own program to manipulate data. While in many cases doing database query will be more effective way, in some cases you might need some more complex data processing, which cannot be easily done using SQL.
|
||||
|
||||
Data processing can be programmed in any programming language, but there are certain languages that are higher level with respect to working with data. Data scientists typically prefer one of the following languages:
|
||||
|
||||
* **[Python](https://www.python.org/)**, a general-purpose programming language, which is often considered one of the best options for beginners due to its simplicity. Python has a lot of additional libraries that can help you solve many practical problems, such as extracting your data from ZIP archive, or converting picture to grayscale. In addition to data science, Python is also often used for web development.
|
||||
* **[R](https://www.r-project.org/)** is a traditional toolbox developed with statistical data processing in mind. It also contains large repository of libraries (CRAN), making it a good choice for data processing. However, R is not a general-purpose programming language, and is rarely used outside of data science domain.
|
||||
* **[Julia](https://julialang.org/)** is another language developed specifically for data science. It is intended to give better performance than Python, making it a great tool for scientific experimentation.
|
||||
|
||||
In this lesson, we will focus on using Python for simple data processing. We will assume basic familiarity with the language. If you want deeper tour of Python, you can refer to one of the following resources:
|
||||
|
||||
* [Learn Python in a Fun Way with Turtle Graphics and Fractals](https://github.com/shwars/pycourse) - GitHub-based quick intro course into Python Programming
|
||||
* [Take your First Steps with Python](https://docs.microsoft.com/en-us/learn/paths/python-first-steps/?WT.mc_id=acad-31812-dmitryso) Learning Path on [Microsoft Learn](http://learn.microsoft.com/?WT.mc_id=acad-31812-dmitryso)
|
||||
|
||||
Data can come in many forms. In this lesson, we will consider three forms of data - **tabular data**, **text** and **images**.
|
||||
|
||||
We will focus on a few examples of data processing, instead of giving you full overview of all related libraries. This would allow you to get the main idea of what's possible, and leave you with understanding on where to find solutions to your problems when you need them.
|
||||
|
||||
> **Most useful advice**. When you need to perform certain operation on data that you do not know how to do, try searching for it in the internet. [Stackoverflow](https://stackoverflow.com/) usually contains a lot of useful code sample in Python for many typical tasks.
|
||||
|
||||
## Pre-Lecture Quiz
|
||||
|
||||
[Pre-lecture quiz]()
|
||||
|
||||
## Tabular Data and Dataframes
|
||||
|
||||
You have already met tabular data when we talked about relational databases. When you have a lot of data, and it is contained in many different linked tables, it definitely makes sense to use SQL for working with it. However, there are many cases when we have a table of data, and we need to gain some **understanding** or **insights** about this data, such as the distribution, correlation between values, etc. In data science, there are a lot of cases when we need to perform some transformations of the original data, followed by visualization. Both those steps can be easily done using Python.
|
||||
|
||||
There are two most useful libraries in Python that can help you deal with tabular data:
|
||||
* **[Pandas](https://pandas.pydata.org/)** allows you to manipulate so-called **Dataframes**, which are analogous to relational tables. You can have named columns, and perform different operations on row, columns and dataframes in general.
|
||||
* **[Numpy](https://numpy.org/)** is a library for working with **tensors**, i.e. multi-dimensional **arrays**. Array has values of the same underlying type, and it is simpler than dataframe, but it offers more mathematical operations, and creates less overhead.
|
||||
|
||||
There are also a couple of other libraries you should know about:
|
||||
* **[Matplotlib](https://matplotlib.org/)** is a library used for data visualization and plotting graphs
|
||||
* **[SciPy](https://www.scipy.org/)** is a library with some additional scientific functions. We have already come accross this library when talking about probability and statistics
|
||||
|
||||
Here is a piece of code that you would typically use to import those libraries in the beginning of your Python program:
|
||||
```python
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import matplotlib.pyplot as plt
|
||||
from scipy import ... # you need to specify exact sub-packages that you need
|
||||
```
|
||||
|
||||
Pandas is centered around the following basic concepts:
|
||||
|
||||
**Series** is a sequence of values, similar to a list or numpy array. The main difference is that series also has and **index**, and when we operate on series (eg., add them), the index is taken into account. Index can be as simple as integer row number (it is the index used by default when creating a series from list or array), or it can have a complex structure, such as date interval.
|
||||
|
||||
> **Note**: There is some introductory Pandas code in the accompanying notebook [`notebook.ipynb`](notebook.ipynb). We only outline some the examples here, and you are definitely welcome to check out the full notebook.
|
||||
|
||||
Consider an example: we want to analyze sales of our ice-cream spot. Let's generate a series of sales numbers (number of items sold each day) for some time period:
|
||||
|
||||
```python
|
||||
tart_date = "Jan 1, 2020"
|
||||
end_date = "Mar 31, 2020"
|
||||
idx = pd.date_range(start_date,end_date)
|
||||
print(f"Length of index is {len(idx)}")
|
||||
items_sold = pd.Series(np.random.randint(25,50,size=len(idx)),index=idx)
|
||||
items_sold.plot()
|
||||
```
|
||||
|
||||
|
||||
Now suppose that each week we are organizing a party for friends, and we take additional 10 packs of ice-cream for a party. We can create another series, indexed by week, to demonstrate that:
|
||||
```python
|
||||
additional_items = pd.Series(10,index=pd.date_range(start_date,end_date,freq="W"))
|
||||
```
|
||||
When we add two series together, we get total number:
|
||||
```python
|
||||
total_items = items_sold.add(additional_items,fill_value=0)
|
||||
total_items.plot()
|
||||
```
|
||||
|
||||
## 🚀 Challenge
|
||||
|
||||
First problem we will focus on is modelling of epidemic spread of COVID-19. In order to do that, we will use the data on the number of infected individuals in different countries, provided by the [Center for Systems Science and Engineering](https://systems.jhu.edu/) (CSSE) at [Johns Hopkins University](https://jhu.edu/). Dataset is available in [this GitHub Repository](https://github.com/CSSEGISandData/COVID-19).
|
||||
|
||||
Since we want to demonstrate how to deal with data, we invite you to open [`notebook-pandas.ipynb`](notebook-pandas.ipynb) and read it from top to bottom. You can also execute cells, and do some challenges that we have leaf for you along the way.
|
||||
|
||||
|
||||
|
||||
## Post-Lecture Quiz
|
||||
|
||||
[Post-lecture quiz]()
|
||||
|
||||
## Review & Self Study
|
||||
|
||||
* [Wes McKinney. Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython](https://www.amazon.com/gp/product/1491957662)
|
||||
|
||||
## Assignment
|
||||
|
||||
[Assignment Title](assignment.md)
|
||||
|
Before Width: | Height: | Size: 35 KiB |
|
Before Width: | Height: | Size: 75 KiB |
File diff suppressed because one or more lines are too long
Loading…
Reference in new issue