You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
575 lines
18 KiB
575 lines
18 KiB
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Intro to Probability and Statistics\n",
|
|
"For dis notebook, we go dey play wit some of di concepts we don talk before. Plenti ideas from probability and statistics dey well represent for major libraries for data processing for Python, like `numpy` and `pandas`.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import numpy as np\n",
|
|
"import pandas as pd\n",
|
|
"import random\n",
|
|
"import matplotlib.pyplot as plt"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Random Variables and Distributions\n",
|
|
"Mek we start by draw sample of 30 values from one uniform distribution from 0 reach 9. We go also calculate mean and variance.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"sample = [ random.randint(0,10) for _ in range(30) ]\n",
|
|
"print(f\"Sample: {sample}\")\n",
|
|
"print(f\"Mean = {np.mean(sample)}\")\n",
|
|
"print(f\"Variance = {np.var(sample)}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"To visually estimate how many different values den dey for di sample, we fit plot di **histogram**:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"plt.hist(sample)\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Analyzing Real Data\n",
|
|
"\n",
|
|
"Mean and variance na very important tin wen you dey analyze real-world data. Make we load di data about baseball players from [SOCR MLB Height/Weight Data](http://wiki.stat.ucla.edu/socr/index.php/SOCR_Data_MLB_HeightsWeights)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"df = pd.read_csv(\"../../data/SOCR_MLB.tsv\",sep='\\t', header=None, names=['Name','Team','Role','Weight','Height','Age'])\n",
|
|
"df\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"> We dey use one package wey dem dey call [**Pandas**](https://pandas.pydata.org/) here for data analysis. We go talk more about Pandas and how to waka with data for Python later for this course.\n",
|
|
"\n",
|
|
"Make we calculate average values for age, height and weight:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"df[['Age','Height','Weight']].mean()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Now mek we focus on height, and calculate standard deviation and variance: \n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(list(df['Height'])[:20])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"mean = df['Height'].mean()\n",
|
|
"var = df['Height'].var()\n",
|
|
"std = df['Height'].std()\n",
|
|
"print(f\"Mean = {mean}\\nVariance = {var}\\nStandard Deviation = {std}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Besides di mean, e make sense to look di median value and di quartiles. Dem fit show am with **box plot**:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"plt.figure(figsize=(10,2))\n",
|
|
"plt.boxplot(df['Height'].ffill(), orientation='horizontal', showmeans=True)\n",
|
|
"plt.grid(color='gray', linestyle='dotted')\n",
|
|
"plt.tight_layout()\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"We fit also make box plots for small small part dem of our dataset, like how e be when we group am by player role.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"df.boxplot(column='Height', by='Role', figsize=(10,8))\n",
|
|
"plt.xticks(rotation='vertical')\n",
|
|
"plt.tight_layout()\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"> **Note**: Dis diagram mean say, on top finish, di height of first basemen pass di height of second basemen. Later, we go learn how we fit test dis assumption well-well, and how to show say our data get statistical meaning to prove am. \n",
|
|
"\n",
|
|
"Age, height and weight na all continuous random variables. Wetin you think be dia distribution? One beta way we fit find am be to draw the histogram of di values: \n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"df['Weight'].hist(bins=15, figsize=(10,6))\n",
|
|
"plt.suptitle('Weight distribution of MLB Players')\n",
|
|
"plt.xlabel('Weight')\n",
|
|
"plt.ylabel('Count')\n",
|
|
"plt.tight_layout()\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Normal Distribution\n",
|
|
"\n",
|
|
"Make we create one artificial sample of weights wey follow normal distribution wey get the same mean and variance as the real data wey we get:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"generated = np.random.normal(mean, std, 1000)\n",
|
|
"generated[:20]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"plt.figure(figsize=(10,6))\n",
|
|
"plt.hist(generated, bins=15)\n",
|
|
"plt.tight_layout()\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"plt.figure(figsize=(10,6))\n",
|
|
"plt.hist(np.random.normal(0,1,50000), bins=300)\n",
|
|
"plt.tight_layout()\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Since most values for real life dey normally distributed, we no suppose use uniform random number generator to generate sample data. Na wetin go happen if we try generate weights with uniform distribution (wey `np.random.rand` generate) be dis:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"wrong_sample = np.random.rand(1000)*2*std+mean-std\n",
|
|
"plt.figure(figsize=(10,6))\n",
|
|
"plt.hist(wrong_sample)\n",
|
|
"plt.tight_layout()\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Confidence Intervals\n",
|
|
"\n",
|
|
"Make we calculate confidence intervals for weights and heights of baseball players now. We go use de code [from this stackoverflow discussion](https://stackoverflow.com/questions/15033511/compute-a-confidence-interval-from-sample-data):\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import scipy.stats\n",
|
|
"\n",
|
|
"def mean_confidence_interval(data, confidence=0.95):\n",
|
|
" a = 1.0 * np.array(data)\n",
|
|
" n = len(a)\n",
|
|
" m, se = np.mean(a), scipy.stats.sem(a)\n",
|
|
" h = se * scipy.stats.t.ppf((1 + confidence) / 2., n-1)\n",
|
|
" return m, h\n",
|
|
"\n",
|
|
"for p in [0.85, 0.9, 0.95]:\n",
|
|
" m, h = mean_confidence_interval(df['Weight'].ffill(),p)\n",
|
|
" print(f\"p={p:.2f}, mean = {m:.2f} ± {h:.2f}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Hypothesis Testing\n",
|
|
"\n",
|
|
"Mek we explore different roles for our baseball players dataset:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"df.groupby('Role').agg({ 'Weight' : 'mean', 'Height' : 'mean', 'Age' : 'count'}).rename(columns={ 'Age' : 'Count'})"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Make we test di tok say First Basemen dey taller pass Second Basemen. Di easiest way to do am na to test di confidence intervals:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"for p in [0.85,0.9,0.95]:\n",
|
|
" m1, h1 = mean_confidence_interval(df.loc[df['Role']=='First_Baseman',['Height']],p)\n",
|
|
" m2, h2 = mean_confidence_interval(df.loc[df['Role']=='Second_Baseman',['Height']],p)\n",
|
|
" print(f'Conf={p:.2f}, 1st basemen height: {m1-h1[0]:.2f}..{m1+h1[0]:.2f}, 2nd basemen height: {m2-h2[0]:.2f}..{m2+h2[0]:.2f}')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"We fit see sey di intervals no dey overlap.\n",
|
|
"\n",
|
|
"One statistically beta wei to prove di hypothesis na to use **Student t-test**:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from scipy.stats import ttest_ind\n",
|
|
"\n",
|
|
"tval, pval = ttest_ind(df.loc[df['Role']=='First_Baseman',['Height']], df.loc[df['Role']=='Second_Baseman',['Height']],equal_var=False)\n",
|
|
"print(f\"T-value = {tval[0]:.2f}\\nP-value: {pval[0]}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Di two value wey di `ttest_ind` function return na:\n",
|
|
"* p-value fit be di chance say two distribution get di same mean. For our case, e low sotay, which mean say e get strong evidence wey support say di first basemen dem taller.\n",
|
|
"* t-value na di level wey mean difference don normalize and na im dem dey use for di t-test, and dem dey compare am against one threshold value wey dem get for one confidence value.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Simulate Normal Distribution Wit Central Limit Theorem\n",
|
|
"\n",
|
|
"Di pseudo-random generator for Python na to give us uniform distribution. If we wan create generator wey dey give normal distribution, we fit use central limit theorem. To get normal distribution value, we go just calculate mean of uniform-generated sample.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def normal_random(sample_size=100):\n",
|
|
" sample = [random.uniform(0,1) for _ in range(sample_size) ]\n",
|
|
" return sum(sample)/sample_size\n",
|
|
"\n",
|
|
"sample = [normal_random() for _ in range(100)]\n",
|
|
"plt.figure(figsize=(10,6))\n",
|
|
"plt.hist(sample)\n",
|
|
"plt.tight_layout()\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Correlation and Evil Baseball Corp\n",
|
|
"\n",
|
|
"Correlation dey help us find relation dem between data sequence dem. For our small example, mek we pretend say one evil baseball company dey wey dey pay im players based on dia height - di taller di player be, di more money wey im go get. Make we suppose say di base salary na $1000, plus one bonus from $0 reach $100, wey depend on height. We go use real players from MLB, then calculate dem imaginary salary dem:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"heights = df['Height'].ffill()\n",
|
|
"salaries = 1000+(heights-heights.min())/(heights.max()-heights.mean())*100\n",
|
|
"print(list(zip(heights, salaries))[:10])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Make we now calculate covariance and correlation of those sequences. `np.cov` go give us one kain **covariance matrix**, wey be extension of covariance to many variables dem. The element $M_{ij}$ for the covariance matrix $M$ na correlation between input variables $X_i$ and $X_j$, and diagonal values $M_{ii}$ na the variance of $X_{i}$. E same way, `np.corrcoef` go give us the **correlation matrix**.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(f\"Covariance matrix:\\n{np.cov(heights, salaries)}\")\n",
|
|
"print(f\"Covariance = {np.cov(heights, salaries)[0,1]}\")\n",
|
|
"print(f\"Correlation = {np.corrcoef(heights, salaries)[0,1]}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Correlation wey equal to 1 mean say two variables get strong **linear relation**. We fit see the linear relation gidigba if we plot one value against the oda:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"plt.figure(figsize=(10,6))\n",
|
|
"plt.scatter(heights,salaries)\n",
|
|
"plt.tight_layout()\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Make we see wetin go happen if relation no be linear. Suppose say our company decide to hide the obvious linear dependency between heights and salaries, and put some non-linearity inside the formula, like `sin`:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"salaries = 1000+np.sin((heights-heights.min())/(heights.max()-heights.mean()))*100\n",
|
|
"print(f\"Correlation = {np.corrcoef(heights, salaries)[0,1]}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"For dis case, di correlation small small, but e still high well well. Now, if we wan make di relation no too show, we fit add extra randomness by putting some random variable for di salary. Make we see wetin go happen:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"salaries = 1000+np.sin((heights-heights.min())/(heights.max()-heights.mean()))*100+np.random.random(size=len(heights))*20-10\n",
|
|
"print(f\"Correlation = {np.corrcoef(heights, salaries)[0,1]}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"plt.figure(figsize=(10,6))\n",
|
|
"plt.scatter(heights, salaries)\n",
|
|
"plt.tight_layout()\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"> You fit gess why di dots dem line up into vertical line like dis?\n",
|
|
"\n",
|
|
"We don observe di connection wey dey between one artificially engineer concept like salary and di observed variable *height*. Make we try see if di two observed variables dem, like height and weight, dey relate too:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"np.corrcoef(df['Height'].ffill(),df['Weight'])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Unfortunenitly, we no get any results - na only some strange `nan` values. Dis na becos some of di values for our series no get meaning, dem represent as `nan`, wey dey cause di result of di operation to no get meaning too. By lookin di matrix we fit see say `Weight` na di problem column, becos self-correlation between `Height` values don compute.\n",
|
|
"\n",
|
|
"> Dis example dey show di importance of **data preparation** and **cleaning**. Without correct data we no fit compute anything.\n",
|
|
"\n",
|
|
"Make we use `fillna` method to fill di missing values, and compute di correlation: \n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"np.corrcoef(df['Height'].ffill(), df['Weight'])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"E get correlation for true, but e no strong like for our fake example. True true, if we check scatter plot wey show one value con face the oda, the relation no go too clear like dat:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"plt.figure(figsize=(10,6))\n",
|
|
"plt.scatter(df['Weight'],df['Height'])\n",
|
|
"plt.xlabel('Weight')\n",
|
|
"plt.ylabel('Height')\n",
|
|
"plt.tight_layout()\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Konklushon\n",
|
|
"\n",
|
|
"For dis notebook, we don learn how to do basic operations for data to calculate statistical functions dem. Now we sabi how to use beta math and statistics tools to take prove some hypotheses, plus how to calculate confidence intervals for any kind variables based on data sample. \n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n\n<!-- CO-OP TRANSLATOR DISCLAIMER START -->\n**Disclaimer**:\nDis document don translate wit AI translation service [Co-op Translator](https://github.com/Azure/co-op-translator). Even tho we dey try make am correct, abeg make you know say automated translation fit get errors or mistakes. Di original document for dia own language na im be di correct source. For important info, make person wey sabi human translation do am. We no go responsible for any misunderstanding or wrong understanding wey fit happen because of dis translation.\n<!-- CO-OP TRANSLATOR DISCLAIMER END -->\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"interpreter": {
|
|
"hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.6"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 4
|
|
} |