uv012
4 days ago
parent
0d9af55c68
commit
bf84b17f93
@ -0,0 +1,17 @@
|
||||
# Introduction to Data Science
|
||||
|
||||
|
||||
> Photo by <a href="https://unsplash.com/@dawson2406?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Stephen Dawson</a> on <a href="https://unsplash.com/s/photos/data?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>
|
||||
|
||||
In these lessons, you will discover how Data Science is defined and learn about ethical considerations that must be considered by a data scientist. You will also learn how data is defined and learn a bit about statistics and probability, the core academic domains of Data Science.
|
||||
|
||||
### Topics
|
||||
|
||||
1. [Defining Data Science](01-defining-data-science/README.md)
|
||||
2. [Data Science Ethics](02-ethics/README.md)
|
||||
3. [Defining Data](03-defining-data/README.md)
|
||||
4. [Introduction to Statistics and Probability](04-stats-and-probability/README.md)
|
||||
|
||||
### Credits
|
||||
|
||||
These lessons were written with ❤️ by [Nitya Narasimhan](https://twitter.com/nitya) and [Dmitry Soshnikov](https://twitter.com/shwars).
|
||||
@ -0,0 +1,165 @@
|
||||
# Defining Data Science
|
||||
|
||||
|  ](../../sketchnotes/01-Definitions.png) |
|
||||
| :----------------------------------------------------------------------------------------------------: |
|
||||
| Defining Data Science - _Sketchnote by [@nitya](https://twitter.com/nitya)_ |
|
||||
|
||||
---
|
||||
|
||||
[](https://youtu.be/beZ7Mb_oz9I)
|
||||
|
||||
## [Pre-lecture quiz](https://ff-quizzes.netlify.app/en/ds/quiz/0)
|
||||
|
||||
## What is Data?
|
||||
In our everyday life, we are constantly surrounded by data. The text you are reading now is data. The list of phone numbers of your friends in your smartphone is data, as well as the current time displayed on your watch. As human beings, we naturally operate with data by counting the money we have or by writing letters to our friends.
|
||||
|
||||
However, data became much more critical with the creation of computers. The primary role of computers is to perform computations, but they need data to operate on. Thus, we need to understand how computers store and process data.
|
||||
|
||||
With the emergence of the Internet, the role of computers as data handling devices increased. If you think about it, we now use computers more and more for data processing and communication, rather than actual computations. When we write an e-mail to a friend or search for some information on the Internet - we are essentially creating, storing, transmitting, and manipulating data.
|
||||
> Can you remember the last time you have used computers to actually compute something?
|
||||
|
||||
## What is Data Science?
|
||||
|
||||
In [Wikipedia](https://en.wikipedia.org/wiki/Data_science), **Data Science** is defined as *a scientific field that uses scientific methods to extract knowledge and insights from structured and unstructured data, and apply knowledge and actionable insights from data across a broad range of application domains*.
|
||||
|
||||
This definition highlights the following important aspects of data science:
|
||||
|
||||
* The main goal of data science is to **extract knowledge** from data, in other words - to **understand** data, find some hidden relationships and build a **model**.
|
||||
* Data science uses **scientific methods**, such as probability and statistics. In fact, when the term *data science* was first introduced, some people argued that data science was just a new fancy name for statistics. Nowadays it has become evident that the field is much broader.
|
||||
* Obtained knowledge should be applied to produce some **actionable insights**, i.e. practical insights that you can apply to real business situations.
|
||||
* We should be able to operate on both **structured** and **unstructured** data. We will come back to discuss different types of data later in the course.
|
||||
* **Application domain** is an important concept, and data scientists often need at least some degree of expertise in the problem domain, for example: finance, medicine, marketing, etc.
|
||||
|
||||
> Another important aspect of Data Science is that it studies how data can be gathered, stored and operated upon using computers. While statistics gives us mathematical foundations, data science applies mathematical concepts to actually draw insights from data.
|
||||
|
||||
One of the ways (attributed to [Jim Gray](https://en.wikipedia.org/wiki/Jim_Gray_(computer_scientist))) to look at the data science is to consider it to be a separate paradigm of science:
|
||||
* **Empirical**, in which we rely mostly on observations and results of experiments
|
||||
* **Theoretical**, where new concepts emerge from existing scientific knowledge
|
||||
* **Computational**, where we discover new principles based on some computational experiments
|
||||
* **Data-Driven**, based on discovering relationships and patterns in the data
|
||||
|
||||
## Other Related Fields
|
||||
|
||||
Since data is pervasive, data science itself is also a broad field, touching many other disciplines.
|
||||
|
||||
<dl>
|
||||
<dt>Databases</dt>
|
||||
<dd>
|
||||
A critical consideration is <b>how to store</b> the data, i.e. how to structure it in a way that allows faster processing. There are different types of databases that store structured and unstructured data, which <a href="../../2-Working-With-Data/README.md">we will consider in our course</a>.
|
||||
</dd>
|
||||
<dt>Big Data</dt>
|
||||
<dd>
|
||||
Often we need to store and process very large quantities of data with a relatively simple structure. There are special approaches and tools to store that data in a distributed manner on a computer cluster, and process it efficiently.
|
||||
</dd>
|
||||
<dt>Machine Learning</dt>
|
||||
<dd>
|
||||
One way to understand data is to <b>build a model</b> that will be able to predict a desired outcome. Developing models from data is called <b>machine learning</b>. You may want to have a look at our <a href="https://aka.ms/ml-beginners">Machine Learning for Beginners</a> Curriculum to learn more about it.
|
||||
</dd>
|
||||
<dt>Artificial Intelligence</dt>
|
||||
<dd>
|
||||
An area of machine learning known as artificial intelligence (AI) also relies on data, and it involves building high complexity models that mimic human thought processes. AI methods often allow us to turn unstructured data (e.g. natural language) into structured insights.
|
||||
</dd>
|
||||
<dt>Visualization</dt>
|
||||
<dd>
|
||||
Vast amounts of data are incomprehensible for a human being, but once we create useful visualizations using that data, we can make more sense of the data, and draw some conclusions. Thus, it is important to know many ways to visualize information - something that we will cover in <a href="../../3-Data-Visualization/README.md">Section 3</a> of our course. Related fields also include <b>Infographics</b>, and <b>Human-Computer Interaction</b> in general.
|
||||
</dd>
|
||||
</dl>
|
||||
|
||||
## Types of Data
|
||||
|
||||
As we have already mentioned, data is everywhere. We just need to capture it in the right way! It is useful to distinguish between **structured** and **unstructured** data. The former is typically represented in some well-structured form, often as a table or number of tables, while the latter is just a collection of files. Sometimes we can also talk about **semi-structured** data, that have some sort of a structure that may vary greatly.
|
||||
|
||||
| Structured | Semi-structured | Unstructured |
|
||||
| ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | --------------------------------------- |
|
||||
| List of people with their phone numbers | Wikipedia pages with links | Text of Encyclopedia Britannica |
|
||||
| Temperature in all rooms of a building at every minute for the last 20 years | Collection of scientific papers in JSON format with authors, data of publication, and abstract | File share with corporate documents |
|
||||
| Data for age and gender of all people entering the building | Internet pages | Raw video feed from surveillance camera |
|
||||
|
||||
## Where to get Data
|
||||
|
||||
There are many possible sources of data, and it will be impossible to list all of them! However, let's mention some of the typical places where you can get data:
|
||||
|
||||
* **Structured**
|
||||
- **Internet of Things** (IoT), including data from different sensors, such as temperature or pressure sensors, provides a lot of useful data. For example, if an office building is equipped with IoT sensors, we can automatically control heating and lighting in order to minimize costs.
|
||||
- **Surveys** that we ask users to complete after a purchase, or after visiting a web site.
|
||||
- **Analysis of behavior** can, for example, help us understand how deeply a user goes into a site, and what is the typical reason for leaving the site.
|
||||
* **Unstructured**
|
||||
- **Texts** can be a rich source of insights, such as an overall **sentiment score**, or extracting keywords and semantic meaning.
|
||||
- **Images** or **Video**. A video from a surveillance camera can be used to estimate traffic on the road, and inform people about potential traffic jams.
|
||||
- Web server **Logs** can be used to understand which pages of our site are most often visited, and for how long.
|
||||
* Semi-structured
|
||||
- **Social Network** graphs can be great sources of data about user personalities and potential effectiveness in spreading information around.
|
||||
- When we have a bunch of photographs from a party, we can try to extract **Group Dynamics** data by building a graph of people taking pictures with each other.
|
||||
|
||||
By knowing different possible sources of data, you can try to think about different scenarios where data science techniques can be applied to know the situation better, and to improve business processes.
|
||||
|
||||
## What you can do with Data
|
||||
|
||||
In Data Science, we focus on the following steps of data journey:
|
||||
|
||||
<dl>
|
||||
<dt>1) Data Acquisition</dt>
|
||||
<dd>
|
||||
The first step is to collect the data. While in many cases it can be a straightforward process, like data coming to a database from a web application, sometimes we need to use special techniques. For example, data from IoT sensors can be overwhelming, and it is a good practice to use buffering endpoints such as IoT Hub to collect all the data before further processing.
|
||||
</dd>
|
||||
<dt>2) Data Storage</dt>
|
||||
<dd>
|
||||
Storing data can be challenging, especially if we are talking about big data. When deciding how to store data, it makes sense to anticipate the way you would like to query the data in the future. There are several ways data can be stored:
|
||||
<ul>
|
||||
<li>A relational database stores a collection of tables, and uses a special language called SQL to query them. Typically, tables are organized into different groups called schemas. In many cases we need to convert the data from original form to fit the schema.</li>
|
||||
<li><a href="https://en.wikipedia.org/wiki/NoSQL">A NoSQL</a> database, such as <a href="https://azure.microsoft.com/services/cosmos-db/?WT.mc_id=academic-77958-bethanycheum">CosmosDB</a>, does not enforce schemas on data, and allows storing more complex data, for example, hierarchical JSON documents or graphs. However, NoSQL databases do not have the rich querying capabilities of SQL, and cannot enforce referential integrity, i.e. rules on how the data is structured in tables and governing the relationships between tables.</li>
|
||||
<li><a href="https://en.wikipedia.org/wiki/Data_lake">Data Lake</a> storage is used for large collections of data in raw, unstructured form. Data lakes are often used with big data, where all data cannot fit on one machine, and has to be stored and processed by a cluster of servers. <a href="https://en.wikipedia.org/wiki/Apache_Parquet">Parquet</a> is the data format that is often used in conjunction with big data.</li>
|
||||
</ul>
|
||||
</dd>
|
||||
<dt>3) Data Processing</dt>
|
||||
<dd>
|
||||
This is the most exciting part of the data journey, which involves converting the data from its original form into a form that can be used for visualization/model training. When dealing with unstructured data such as text or images, we may need to use some AI techniques to extract <b>features</b> from the data, thus converting it to structured form.
|
||||
</dd>
|
||||
<dt>4) Visualization / Human Insights</dt>
|
||||
<dd>
|
||||
Oftentimes, in order to understand the data, we need to visualize it. Having many different visualization techniques in our toolbox, we can find the right view to make an insight. Often, a data scientist needs to "play with data", visualizing it many times and looking for some relationships. Also, we may use statistical techniques to test a hypotheses or prove a correlation between different pieces of data.
|
||||
</dd>
|
||||
<dt>5) Training a predictive model</dt>
|
||||
<dd>
|
||||
Because the ultimate goal of data science is to be able to make decisions based on data, we may want to use the techniques of <a href="http://github.com/microsoft/ml-for-beginners">Machine Learning</a> to build a predictive model. We can then use this to make predictions using new data sets with similar structures.
|
||||
</dd>
|
||||
</dl>
|
||||
|
||||
Of course, depending on the actual data, some steps might be missing (e.g., when we already have the data in the database, or when we do not need model training), or some steps might be repeated several times (such as data processing).
|
||||
|
||||
## Digitalization and Digital Transformation
|
||||
|
||||
In the last decade, many businesses started to understand the importance of data when making business decisions. To apply data science principles to running a business, one first needs to collect some data, i.e. translate business processes into digital form. This is known as **digitalization**. Applying data science techniques to this data to guide decisions can lead to significant increases in productivity (or even business pivot), called **digital transformation**.
|
||||
|
||||
Let's consider an example. Suppose we have a data science course (like this one) which we deliver online to students, and we want to use data science to improve it. How can we do it?
|
||||
|
||||
We can start by asking "What can be digitized?" The simplest way would be to measure the time it takes each student to complete each module, and to measure the obtained knowledge by giving a multiple-choice test at the end of each module. By averaging time-to-complete across all students, we can find out which modules cause the most difficulties for students, and work on simplifying them.
|
||||
|
||||
> You may argue that this approach is not ideal, because modules can be of different lengths. It is probably more fair to divide the time by the length of the module (in number of characters), and compare those values instead.
|
||||
|
||||
When we start analyzing results of multiple-choice tests, we can try to determine which concepts that students have difficulty understanding, and use that information to improve the content. To do that, we need to design tests in such a way that each question maps to a certain concept or chunk of knowledge.
|
||||
|
||||
If we want to get even more complicated, we can plot the time taken for each module against the age category of students. We might find out that for some age categories it takes an inappropriately long time to complete the module, or that students drop out before completing it. This can help us provide age recommendations for the module, and minimize people's dissatisfaction from wrong expectations.
|
||||
|
||||
## 🚀 Challenge
|
||||
|
||||
In this challenge, we will try to find concepts relevant to the field of Data Science by looking at texts. We will take a Wikipedia article on Data Science, download and process the text, and then build a word cloud like this one:
|
||||
|
||||
|
||||
|
||||
Visit [`notebook.ipynb`](./notebook.ipynb ':ignore') to read through the code. You can also run the code, and see how it performs all data transformations in real time.
|
||||
|
||||
> If you do not know how to run code in a Jupyter Notebook, have a look at [this article](https://soshnikov.com/education/how-to-execute-notebooks-from-github/).
|
||||
|
||||
|
||||
|
||||
## [Post-lecture quiz](https://ff-quizzes.netlify.app/en/ds/quiz/1)
|
||||
|
||||
## Assignments
|
||||
|
||||
* **Task 1**: Modify the code above to find out related concepts for the fields of **Big Data** and **Machine Learning**
|
||||
* **Task 2**: [Think About Data Science Scenarios](assignment.md)
|
||||
|
||||
## Credits
|
||||
|
||||
This lesson has been authored with ♥️ by [Dmitry Soshnikov](http://soshnikov.com)
|
||||
@ -0,0 +1,31 @@
|
||||
# Assignment: Data Science Scenarios
|
||||
|
||||
In this first assignment, we ask you to think about some real-life process or problem in different problem domains, and how you can improve it using the Data Science process. Think about the following:
|
||||
|
||||
1. Which data can you collect?
|
||||
1. How would you collect it?
|
||||
1. How would you store the data? How large the data is likely to be?
|
||||
1. Which insights you might be able to get from this data? Which decisions we would be able to take based on the data?
|
||||
|
||||
Try to think about 3 different problems/processes and describe each of the points above for each problem domain.
|
||||
|
||||
Here are some of the problem domains and problems that can get you started thinking:
|
||||
|
||||
1. How can you use data to improve education process for children in schools?
|
||||
1. How can you use data to control vaccination during the pandemic?
|
||||
1. How can you use data to make sure you are being productive at work?
|
||||
## Instructions
|
||||
|
||||
Fill in the following table (substitute suggested problem domains for your own ones if needed):
|
||||
|
||||
| Problem Domain | Problem | Which data to collect | How to store the data | Which insights/decisions we can make |
|
||||
|----------------|---------|-----------------------|-----------------------|--------------------------------------|
|
||||
| Education | | | | |
|
||||
| Vaccination | | | | |
|
||||
| Productivity | | | | |
|
||||
|
||||
## Rubric
|
||||
|
||||
Exemplary | Adequate | Needs Improvement
|
||||
--- | --- | -- |
|
||||
One was able to identify reasonable data sources, ways of storing data and possible decisions/insights for all problem domains | Some of the aspects of the solution are not detailed, data storage is not discussed, at least 2 problem domains are described | Only parts of the data solution are described, only one problem domain is considered.
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -0,0 +1,6 @@
|
||||
{
|
||||
"cells": [],
|
||||
"metadata": {},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
|
After Width: | Height: | Size: 236 KiB |
|
After Width: | Height: | Size: 103 KiB |
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -0,0 +1,33 @@
|
||||
# Assignment: Data Science Scenarios
|
||||
|
||||
In this first assignment, we ask you to think about some real-life process or problem in different problem domains, and how you can improve it using the Data Science process. Think about the following:
|
||||
|
||||
1. Which data can you collect?
|
||||
1. How would you collect it?
|
||||
1. How would you store the data? How large the data is likely to be?
|
||||
1. Which insights you might be able to get from this data? Which decisions we would be able to take based on the data?
|
||||
|
||||
Try to think about 3 different problems/processes and describe each of the points above for each problem domain.
|
||||
|
||||
Here are some of the problem domains and problems that can get you started thinking:
|
||||
|
||||
1. How can you use data to improve education process for children in schools?
|
||||
1. How can you use data to control vaccination during the pandemic?
|
||||
1. How can you use data to make sure you are being productive at work?
|
||||
## Instructions
|
||||
|
||||
Fill in the following table (substitute suggested problem domains for your own ones if needed):
|
||||
|
||||
| Problem Domain | Problem | Which data to collect | How to store the data | Which insights/decisions we can make |
|
||||
|----------------|---------|-----------------------|-----------------------|--------------------------------------|
|
||||
| Education | In university, we typically have low attendance to lectures, and we have the hypothesis that students who attend lectures on average to better during exams. We want to stimulate attendance and test the hypothesis. | We can track attendance through pictures taken by the security camera in class, or by tracking bluetooth/wifi addresses of student mobile phones in class. Exam data is already available in the university database. | In case we track security camera images - we need to store a few (5-10) photographs during class (unstructured data), and then use AI to identify faces of students (convert data to structured form). | We can compute average attendance data for each student, and see if there is any correlation with exam grades. We will talk more about correlation in [probability and statistics](../../04-stats-and-probability/README.md) section. In order to stimulate student attendance, we can publish the weekly attendance rating on school portal, and draw prizes among those with highest attendance. |
|
||||
| Vaccination | | | | |
|
||||
| Productivity | | | | |
|
||||
|
||||
> *We provide just one answer as an example, so that you can get an idea of what is expected in this assignment.*
|
||||
|
||||
## Rubric
|
||||
|
||||
Exemplary | Adequate | Needs Improvement
|
||||
--- | --- | -- |
|
||||
One was able to identify reasonable data sources, ways of storing data and possible decisions/insights for all problem domains | Some of the aspects of the solution are not detailed, data storage is not discussed, at least 2 problem domains are described | Only parts of the data solution are described, only one problem domain is considered.
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -0,0 +1,568 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Introduction to Probability and Statistics\n",
|
||||
"In this notebook, we will play around with some of the concepts we have previously discussed. Many concepts from probability and statistics are well-represented in major libraries for data processing in Python, such as `numpy` and `pandas`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import pandas as pd\n",
|
||||
"import random\n",
|
||||
"import matplotlib.pyplot as plt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Random Variables and Distributions\n",
|
||||
"Let's start with drawing a sample of 30 values from a uniform distribution from 0 to 9. We will also compute mean and variance."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"sample = [ random.randint(0,10) for _ in range(30) ]\n",
|
||||
"print(f\"Sample: {sample}\")\n",
|
||||
"print(f\"Mean = {np.mean(sample)}\")\n",
|
||||
"print(f\"Variance = {np.var(sample)}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"To visually estimate how many different values are there in the sample, we can plot the **histogram**:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"plt.hist(sample)\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Analyzing Real Data\n",
|
||||
"\n",
|
||||
"Mean and variance are very important when analyzing real-world data. Let's load the data about baseball players from [SOCR MLB Height/Weight Data](http://wiki.stat.ucla.edu/socr/index.php/SOCR_Data_MLB_HeightsWeights)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df = pd.read_csv(\"../../data/SOCR_MLB.tsv\",sep='\\t', header=None, names=['Name','Team','Role','Weight','Height','Age'])\n",
|
||||
"df\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> We are using a package called [**Pandas**](https://pandas.pydata.org/) here for data analysis. We will talk more about Pandas and working with data in Python later in this course.\n",
|
||||
"\n",
|
||||
"Let's compute average values for age, height and weight:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df[['Age','Height','Weight']].mean()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now let's focus on height, and compute standard deviation and variance: "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(list(df['Height'])[:20])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"mean = df['Height'].mean()\n",
|
||||
"var = df['Height'].var()\n",
|
||||
"std = df['Height'].std()\n",
|
||||
"print(f\"Mean = {mean}\\nVariance = {var}\\nStandard Deviation = {std}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In addition to mean, it makes sense to look at the median value and quartiles. They can be visualized using a **box plot**:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"plt.figure(figsize=(10,2))\n",
|
||||
"plt.boxplot(df['Height'].ffill(), vert=False, showmeans=True)\n",
|
||||
"plt.grid(color='gray', linestyle='dotted')\n",
|
||||
"plt.tight_layout()\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can also make box plots of subsets of our dataset, for example, grouped by player role."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df.boxplot(column='Height', by='Role', figsize=(10,8))\n",
|
||||
"plt.xticks(rotation='vertical')\n",
|
||||
"plt.tight_layout()\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Note**: This diagram suggests, that on average, the heights of first basemen are higher than heights of second basemen. Later we will learn how we can test this hypothesis more formally, and how to demonstrate that our data is statistically significant to show that. \n",
|
||||
"\n",
|
||||
"Age, height and weight are all continuous random variables. What do you think their distribution is? A good way to find out is to plot the histogram of values: "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df['Weight'].hist(bins=15, figsize=(10,6))\n",
|
||||
"plt.suptitle('Weight distribution of MLB Players')\n",
|
||||
"plt.xlabel('Weight')\n",
|
||||
"plt.ylabel('Count')\n",
|
||||
"plt.tight_layout()\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Normal Distribution\n",
|
||||
"\n",
|
||||
"Let's create an artificial sample of weights that follows a normal distribution with the same mean and variance as our real data:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"generated = np.random.normal(mean, std, 1000)\n",
|
||||
"generated[:20]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"plt.figure(figsize=(10,6))\n",
|
||||
"plt.hist(generated, bins=15)\n",
|
||||
"plt.tight_layout()\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"plt.figure(figsize=(10,6))\n",
|
||||
"plt.hist(np.random.normal(0,1,50000), bins=300)\n",
|
||||
"plt.tight_layout()\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Since most values in real life are normally distributed, we should not use a uniform random number generator to generate sample data. Here is what happens if we try to generate weights with a uniform distribution (generated by `np.random.rand`):"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"wrong_sample = np.random.rand(1000)*2*std+mean-std\n",
|
||||
"plt.figure(figsize=(10,6))\n",
|
||||
"plt.hist(wrong_sample)\n",
|
||||
"plt.tight_layout()\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Confidence Intervals\n",
|
||||
"\n",
|
||||
"Let's now calculate confidence intervals for the weights and heights of baseball players. We will use the code [from this stackoverflow discussion](https://stackoverflow.com/questions/15033511/compute-a-confidence-interval-from-sample-data):"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import scipy.stats\n",
|
||||
"\n",
|
||||
"def mean_confidence_interval(data, confidence=0.95):\n",
|
||||
" a = 1.0 * np.array(data)\n",
|
||||
" n = len(a)\n",
|
||||
" m, se = np.mean(a), scipy.stats.sem(a)\n",
|
||||
" h = se * scipy.stats.t.ppf((1 + confidence) / 2., n-1)\n",
|
||||
" return m, h\n",
|
||||
"\n",
|
||||
"for p in [0.85, 0.9, 0.95]:\n",
|
||||
" m, h = mean_confidence_interval(df['Weight'].fillna(method='pad'),p)\n",
|
||||
" print(f\"p={p:.2f}, mean = {m:.2f} ± {h:.2f}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Hypothesis Testing\n",
|
||||
"\n",
|
||||
"Let's explore different roles in our baseball players dataset:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df.groupby('Role').agg({ 'Weight' : 'mean', 'Height' : 'mean', 'Age' : 'count'}).rename(columns={ 'Age' : 'Count'})"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's test the hypothesis that First Basemen are taller than Second Basemen. The simplest way to do this is to test the confidence intervals:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"for p in [0.85,0.9,0.95]:\n",
|
||||
" m1, h1 = mean_confidence_interval(df.loc[df['Role']=='First_Baseman',['Height']],p)\n",
|
||||
" m2, h2 = mean_confidence_interval(df.loc[df['Role']=='Second_Baseman',['Height']],p)\n",
|
||||
" print(f'Conf={p:.2f}, 1st basemen height: {m1-h1[0]:.2f}..{m1+h1[0]:.2f}, 2nd basemen height: {m2-h2[0]:.2f}..{m2+h2[0]:.2f}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can see that the intervals do not overlap.\n",
|
||||
"\n",
|
||||
"A statistically more correct way to prove the hypothesis is to use a **Student t-test**:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from scipy.stats import ttest_ind\n",
|
||||
"\n",
|
||||
"tval, pval = ttest_ind(df.loc[df['Role']=='First_Baseman',['Height']], df.loc[df['Role']=='Second_Baseman',['Height']],equal_var=False)\n",
|
||||
"print(f\"T-value = {tval[0]:.2f}\\nP-value: {pval[0]}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The two values returned by the `ttest_ind` function are:\n",
|
||||
"* p-value can be considered as the probability of two distributions having the same mean. In our case, it is very low, meaning that there is strong evidence supporting that first basemen are taller.\n",
|
||||
"* t-value is the intermediate value of normalized mean difference that is used in the t-test, and it is compared against a threshold value for a given confidence value."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Simulating a Normal Distribution with the Central Limit Theorem\n",
|
||||
"\n",
|
||||
"The pseudo-random generator in Python is designed to give us a uniform distribution. If we want to create a generator for normal distribution, we can use the central limit theorem. To get a normally distributed value we will just compute a mean of a uniform-generated sample."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def normal_random(sample_size=100):\n",
|
||||
" sample = [random.uniform(0,1) for _ in range(sample_size) ]\n",
|
||||
" return sum(sample)/sample_size\n",
|
||||
"\n",
|
||||
"sample = [normal_random() for _ in range(100)]\n",
|
||||
"plt.figure(figsize=(10,6))\n",
|
||||
"plt.hist(sample)\n",
|
||||
"plt.tight_layout()\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Correlation and Evil Baseball Corp\n",
|
||||
"\n",
|
||||
"Correlation allows us to find relations between data sequences. In our toy example, let's pretend there is an evil baseball corporation that pays its players according to their height - the taller the player is, the more money he/she gets. Suppose there is a base salary of $1000, and an additional bonus from $0 to $100, depending on height. We will take the real players from MLB, and compute their imaginary salaries:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"heights = df['Height'].fillna(method='pad')\n",
|
||||
"salaries = 1000+(heights-heights.min())/(heights.max()-heights.mean())*100\n",
|
||||
"print(list(zip(heights, salaries))[:10])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's now compute covariance and correlation of those sequences. `np.cov` will give us a so-called **covariance matrix**, which is an extension of covariance to multiple variables. The element $M_{ij}$ of the covariance matrix $M$ is a correlation between input variables $X_i$ and $X_j$, and diagonal values $M_{ii}$ is the variance of $X_{i}$. Similarly, `np.corrcoef` will give us the **correlation matrix**."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(f\"Covariance matrix:\\n{np.cov(heights, salaries)}\")\n",
|
||||
"print(f\"Covariance = {np.cov(heights, salaries)[0,1]}\")\n",
|
||||
"print(f\"Correlation = {np.corrcoef(heights, salaries)[0,1]}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"A correlation equal to 1 means that there is a strong **linear relation** between two variables. We can visually see the linear relation by plotting one value against the other:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"plt.figure(figsize=(10,6))\n",
|
||||
"plt.scatter(heights,salaries)\n",
|
||||
"plt.tight_layout()\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's see what happens if the relation is not linear. Suppose that our corporation decided to hide the obvious linear dependency between heights and salaries, and introduced some non-linearity into the formula, such as `sin`:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"salaries = 1000+np.sin((heights-heights.min())/(heights.max()-heights.mean()))*100\n",
|
||||
"print(f\"Correlation = {np.corrcoef(heights, salaries)[0,1]}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In this case, the correlation is slightly smaller, but it is still quite high. Now, to make the relation even less obvious, we might want to add some extra randomness by adding some random variable to the salary. Let's see what happens:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"salaries = 1000+np.sin((heights-heights.min())/(heights.max()-heights.mean()))*100+np.random.random(size=len(heights))*20-10\n",
|
||||
"print(f\"Correlation = {np.corrcoef(heights, salaries)[0,1]}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"plt.figure(figsize=(10,6))\n",
|
||||
"plt.scatter(heights, salaries)\n",
|
||||
"plt.tight_layout()\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> Can you guess why the dots line up into vertical lines like this?\n",
|
||||
"\n",
|
||||
"We have observed the correlation between an artificially engineered concept like salary and the observed variable *height*. Let's also see if the two observed variables, such as height and weight, correlate too:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"np.corrcoef(df['Height'].ffill(),df['Weight'])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Unfortunately, we did not get any results - only some strange `nan` values. This is due to the fact that some of the values in our series are undefined, represented as `nan`, which causes the result of the operation to be undefined as well. By looking at the matrix we can see that `Weight` is the problematic column, because self-correlation between `Height` values has been computed.\n",
|
||||
"\n",
|
||||
"> This example shows the importance of **data preparation** and **cleaning**. Without proper data we cannot compute anything.\n",
|
||||
"\n",
|
||||
"Let's use `fillna` method to fill the missing values, and compute the correlation: "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"np.corrcoef(df['Height'].fillna(method='pad'), df['Weight'])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"There is indeed a correlation, but not such a strong one as in our artificial example. Indeed, if we look at the scatter plot of one value against the other, the relation would be much less obvious:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"plt.figure(figsize=(10,6))\n",
|
||||
"plt.scatter(df['Weight'],df['Height'])\n",
|
||||
"plt.xlabel('Weight')\n",
|
||||
"plt.ylabel('Height')\n",
|
||||
"plt.tight_layout()\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Conclusion\n",
|
||||
"\n",
|
||||
"In this notebook we have learnt how to perform basic operations on data to compute statistical functions. We now know how to use a sound apparatus of math and statistics in order to prove some hypotheses, and how to compute confidence intervals for arbitrary variables given a data sample. "
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,87 @@
|
||||
"""
|
||||
Hello World - Data Science Style!
|
||||
|
||||
This is your very first data science program. It introduces you to the basic
|
||||
concepts of working with data in Python.
|
||||
|
||||
What you'll learn:
|
||||
- How to create a simple dataset
|
||||
- How to display data
|
||||
- How to work with Python lists and dictionaries
|
||||
- Basic data manipulation
|
||||
|
||||
Prerequisites: Just Python installed on your computer!
|
||||
"""
|
||||
|
||||
# Let's start with the classic "Hello, World!" but with a data science twist
|
||||
print("=" * 50)
|
||||
print("Hello, World of Data Science!")
|
||||
print("=" * 50)
|
||||
print()
|
||||
|
||||
# In data science, we work with data. Let's create our first simple dataset.
|
||||
# We'll use a list to store information about students and their test scores.
|
||||
|
||||
# A list is a collection of items in Python, written with square brackets []
|
||||
students = ["Alice", "Bob", "Charlie", "Diana", "Eve"]
|
||||
scores = [85, 92, 78, 95, 88]
|
||||
|
||||
print("Our Dataset:")
|
||||
print("-" * 50)
|
||||
print("Students:", students)
|
||||
print("Scores:", scores)
|
||||
print()
|
||||
|
||||
# Now let's do something useful with this data!
|
||||
# We can find basic statistics about the scores
|
||||
|
||||
# Find the highest score
|
||||
highest_score = max(scores)
|
||||
print(f"📊 Highest score: {highest_score}")
|
||||
|
||||
# Find the lowest score
|
||||
lowest_score = min(scores)
|
||||
print(f"📊 Lowest score: {lowest_score}")
|
||||
|
||||
# Calculate the average score
|
||||
# sum() adds all numbers together, len() tells us how many items we have
|
||||
average_score = sum(scores) / len(scores)
|
||||
print(f"📊 Average score: {average_score:.2f}") # .2f means show 2 decimal places
|
||||
print()
|
||||
|
||||
# Let's find who got the highest score
|
||||
# We use index() to find where the highest_score is in our list
|
||||
top_student_index = scores.index(highest_score)
|
||||
top_student = students[top_student_index]
|
||||
print(f"🏆 Top student: {top_student} with a score of {highest_score}")
|
||||
print()
|
||||
|
||||
# Now let's organize this data in a more structured way
|
||||
# We'll use a dictionary - it pairs keys (student names) with values (scores)
|
||||
print("Student Scores (organized as key-value pairs):")
|
||||
print("-" * 50)
|
||||
|
||||
# Create a dictionary by pairing students with their scores
|
||||
student_scores = {}
|
||||
for i in range(len(students)):
|
||||
student_scores[students[i]] = scores[i]
|
||||
|
||||
# Display each student and their score
|
||||
for student, score in student_scores.items():
|
||||
# Add a special marker for the top student
|
||||
marker = "⭐" if student == top_student else " "
|
||||
print(f"{marker} {student}: {score} points")
|
||||
|
||||
print()
|
||||
print("=" * 50)
|
||||
print("Congratulations! You've completed your first data science program!")
|
||||
print("=" * 50)
|
||||
|
||||
# What did we just do?
|
||||
# 1. Created a simple dataset (student names and scores)
|
||||
# 2. Performed basic analysis (max, min, average)
|
||||
# 3. Found insights (who is the top student)
|
||||
# 4. Organized the data in a useful structure (dictionary)
|
||||
#
|
||||
# These are the fundamental building blocks of data science!
|
||||
# Next, you'll learn to work with real datasets using powerful libraries.
|
||||
Loading…
Reference in new issue