msb-public
/
Data-Science-For-Beginners
mirror of https://github.com/microsoft/Data-Science-For-Beginners.git
1
0
Code Issues Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'bf84b17f93'
${ noResults }
Data-Science-For-Beginners/1-Introduction/04-stats-and-probability/.ipynb_checkpoints/Challenge2-checkpoint.ipynb

592 lines
17 KiB
Raw Blame History Escape

This file contains ambiguous Unicode characters!

This file contains ambiguous Unicode characters that may be confused with others in your current locale. If your use case is intentional and legitimate, you can safely ignore this warning. Use the Escape button to highlight these characters.

{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"id": "fb546a97-c009-46af-91bb-99b5962302db",
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "778e4798-ab37-4e4b-9844-fc8f44a31f34",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Adam_Donachie</th>\n",
" <th>BAL</th>\n",
" <th>Catcher</th>\n",
" <th>74</th>\n",
" <th>180</th>\n",
" <th>22.99</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Paul_Bako</td>\n",
" <td>BAL</td>\n",
" <td>Catcher</td>\n",
" <td>74</td>\n",
" <td>215.0</td>\n",
" <td>34.69</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Ramon_Hernandez</td>\n",
" <td>BAL</td>\n",
" <td>Catcher</td>\n",
" <td>72</td>\n",
" <td>210.0</td>\n",
" <td>30.78</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Kevin_Millar</td>\n",
" <td>BAL</td>\n",
" <td>First_Baseman</td>\n",
" <td>72</td>\n",
" <td>210.0</td>\n",
" <td>35.43</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Chris_Gomez</td>\n",
" <td>BAL</td>\n",
" <td>First_Baseman</td>\n",
" <td>73</td>\n",
" <td>188.0</td>\n",
" <td>35.71</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Brian_Roberts</td>\n",
" <td>BAL</td>\n",
" <td>Second_Baseman</td>\n",
" <td>69</td>\n",
" <td>176.0</td>\n",
" <td>29.39</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1028</th>\n",
" <td>Brad_Thompson</td>\n",
" <td>STL</td>\n",
" <td>Relief_Pitcher</td>\n",
" <td>73</td>\n",
" <td>190.0</td>\n",
" <td>25.08</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1029</th>\n",
" <td>Tyler_Johnson</td>\n",
" <td>STL</td>\n",
" <td>Relief_Pitcher</td>\n",
" <td>74</td>\n",
" <td>180.0</td>\n",
" <td>25.73</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1030</th>\n",
" <td>Chris_Narveson</td>\n",
" <td>STL</td>\n",
" <td>Relief_Pitcher</td>\n",
" <td>75</td>\n",
" <td>205.0</td>\n",
" <td>25.19</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1031</th>\n",
" <td>Randy_Keisler</td>\n",
" <td>STL</td>\n",
" <td>Relief_Pitcher</td>\n",
" <td>75</td>\n",
" <td>190.0</td>\n",
" <td>31.01</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1032</th>\n",
" <td>Josh_Kinney</td>\n",
" <td>STL</td>\n",
" <td>Relief_Pitcher</td>\n",
" <td>73</td>\n",
" <td>195.0</td>\n",
" <td>27.92</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>1033 rows × 6 columns</p>\n",
"</div>"
],
"text/plain": [
" Adam_Donachie BAL Catcher 74 180 22.99\n",
"0 Paul_Bako BAL Catcher 74 215.0 34.69\n",
"1 Ramon_Hernandez BAL Catcher 72 210.0 30.78\n",
"2 Kevin_Millar BAL First_Baseman 72 210.0 35.43\n",
"3 Chris_Gomez BAL First_Baseman 73 188.0 35.71\n",
"4 Brian_Roberts BAL Second_Baseman 69 176.0 29.39\n",
"... ... ... ... .. ... ...\n",
"1028 Brad_Thompson STL Relief_Pitcher 73 190.0 25.08\n",
"1029 Tyler_Johnson STL Relief_Pitcher 74 180.0 25.73\n",
"1030 Chris_Narveson STL Relief_Pitcher 75 205.0 25.19\n",
"1031 Randy_Keisler STL Relief_Pitcher 75 190.0 31.01\n",
"1032 Josh_Kinney STL Relief_Pitcher 73 195.0 27.92\n",
"\n",
"[1033 rows x 6 columns]"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df = pd.read_table('../../data/SOCR_MLB.tsv')\n",
"df"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "b4f7df78-72e7-423c-8884-88ec32f5df89",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Name</th>\n",
" <th>Team</th>\n",
" <th>Role</th>\n",
" <th>Weight</th>\n",
" <th>Height</th>\n",
" <th>Age</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Paul_Bako</td>\n",
" <td>BAL</td>\n",
" <td>Catcher</td>\n",
" <td>74</td>\n",
" <td>215.0</td>\n",
" <td>34.69</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Ramon_Hernandez</td>\n",
" <td>BAL</td>\n",
" <td>Catcher</td>\n",
" <td>72</td>\n",
" <td>210.0</td>\n",
" <td>30.78</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Kevin_Millar</td>\n",
" <td>BAL</td>\n",
" <td>First_Baseman</td>\n",
" <td>72</td>\n",
" <td>210.0</td>\n",
" <td>35.43</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Chris_Gomez</td>\n",
" <td>BAL</td>\n",
" <td>First_Baseman</td>\n",
" <td>73</td>\n",
" <td>188.0</td>\n",
" <td>35.71</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Brian_Roberts</td>\n",
" <td>BAL</td>\n",
" <td>Second_Baseman</td>\n",
" <td>69</td>\n",
" <td>176.0</td>\n",
" <td>29.39</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1028</th>\n",
" <td>Brad_Thompson</td>\n",
" <td>STL</td>\n",
" <td>Relief_Pitcher</td>\n",
" <td>73</td>\n",
" <td>190.0</td>\n",
" <td>25.08</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1029</th>\n",
" <td>Tyler_Johnson</td>\n",
" <td>STL</td>\n",
" <td>Relief_Pitcher</td>\n",
" <td>74</td>\n",
" <td>180.0</td>\n",
" <td>25.73</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1030</th>\n",
" <td>Chris_Narveson</td>\n",
" <td>STL</td>\n",
" <td>Relief_Pitcher</td>\n",
" <td>75</td>\n",
" <td>205.0</td>\n",
" <td>25.19</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1031</th>\n",
" <td>Randy_Keisler</td>\n",
" <td>STL</td>\n",
" <td>Relief_Pitcher</td>\n",
" <td>75</td>\n",
" <td>190.0</td>\n",
" <td>31.01</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1032</th>\n",
" <td>Josh_Kinney</td>\n",
" <td>STL</td>\n",
" <td>Relief_Pitcher</td>\n",
" <td>73</td>\n",
" <td>195.0</td>\n",
" <td>27.92</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>1033 rows × 6 columns</p>\n",
"</div>"
],
"text/plain": [
" Name Team Role Weight Height Age\n",
"0 Paul_Bako BAL Catcher 74 215.0 34.69\n",
"1 Ramon_Hernandez BAL Catcher 72 210.0 30.78\n",
"2 Kevin_Millar BAL First_Baseman 72 210.0 35.43\n",
"3 Chris_Gomez BAL First_Baseman 73 188.0 35.71\n",
"4 Brian_Roberts BAL Second_Baseman 69 176.0 29.39\n",
"... ... ... ... ... ... ...\n",
"1028 Brad_Thompson STL Relief_Pitcher 73 190.0 25.08\n",
"1029 Tyler_Johnson STL Relief_Pitcher 74 180.0 25.73\n",
"1030 Chris_Narveson STL Relief_Pitcher 75 205.0 25.19\n",
"1031 Randy_Keisler STL Relief_Pitcher 75 190.0 31.01\n",
"1032 Josh_Kinney STL Relief_Pitcher 73 195.0 27.92\n",
"\n",
"[1033 rows x 6 columns]"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.columns = ['Name', 'Team', 'Role', 'Weight', 'Height', 'Age']\n",
"df"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "5028c04b-71b3-4632-92f8-cb405fd34722",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import scipy.stats"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "b60fb4ab-f0fb-489f-b431-075b53bbb687",
"metadata": {},
"outputs": [],
"source": [
"def mean_confidence_interval(data, confidence=0.95):\n",
" a = 1.0 * data\n",
" n = len(a)\n",
" m, se = np.mean(a), scipy.stats.sem(a)\n",
" h = se * scipy.stats.t.ppf((1+confidence)/2, n-1)\n",
" return m, h"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "17da783d-f98e-40d3-be32-bcc29ce3cd0d",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Mean_Height</th>\n",
" <th>Count</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Role</th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>Catcher</th>\n",
" <td>204.653333</td>\n",
" <td>75</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Designated_Hitter</th>\n",
" <td>220.888889</td>\n",
" <td>18</td>\n",
" </tr>\n",
" <tr>\n",
" <th>First_Baseman</th>\n",
" <td>213.109091</td>\n",
" <td>55</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Outfielder</th>\n",
" <td>199.113402</td>\n",
" <td>194</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Relief_Pitcher</th>\n",
" <td>203.517460</td>\n",
" <td>315</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Second_Baseman</th>\n",
" <td>184.344828</td>\n",
" <td>58</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Shortstop</th>\n",
" <td>182.923077</td>\n",
" <td>52</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Starting_Pitcher</th>\n",
" <td>205.163636</td>\n",
" <td>221</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Third_Baseman</th>\n",
" <td>200.955556</td>\n",
" <td>45</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Mean_Height Count\n",
"Role \n",
"Catcher 204.653333 75\n",
"Designated_Hitter 220.888889 18\n",
"First_Baseman 213.109091 55\n",
"Outfielder 199.113402 194\n",
"Relief_Pitcher 203.517460 315\n",
"Second_Baseman 184.344828 58\n",
"Shortstop 182.923077 52\n",
"Starting_Pitcher 205.163636 221\n",
"Third_Baseman 200.955556 45"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.groupby('Role').agg(\n",
" Mean_Height=('Height', 'mean'),\n",
" Count=('Age', 'count')\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "e4fb81c4-069c-4aa7-bd07-5a4e3f450fb9",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"p: 0.85, 1st_Baseman_Height: 209.36..216.86, 3rd_Baseman_Height: 196.99..204.92\n",
"p: 0.9, 1st_Baseman_Height: 208.82..217.40, 3rd_Baseman_Height: 196.41..205.50\n",
"p: 0.95, 1st_Baseman_Height: 207.97..218.25, 3rd_Baseman_Height: 195.51..206.41\n"
]
}
],
"source": [
"for p in [0.85, 0.9, 0.95]:\n",
" m1, h1 = mean_confidence_interval(df.loc[df['Role'] == 'First_Baseman', ['Height']], p)\n",
" m2, h2 = mean_confidence_interval(df.loc[df['Role'] == 'Third_Baseman', ['Height']], p)\n",
" print(f\"p: {p}, 1st_Baseman_Height: {m1-h1[0]:.2f}..{m1+h1[0]:.2f}, 3rd_Baseman_Height: {m2-h2[0]:.2f}..{m2+h2[0]:.2f}\")"
]
},
{
"cell_type": "markdown",
"id": "09b8fbc9-59e7-480e-b9ed-9b2dbe93247d",
"metadata": {},
"source": [
"# To test the hypothesis that first basemen are taller than third basemen"
]
},
{
"cell_type": "markdown",
"id": "e5f74ec7-415f-4a0e-941c-a7b1e6b900a4",
"metadata": {},
"source": [
"## We'll do the Student's t-test"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "53194c25-9470-4d73-ab6c-19dfe8b75da9",
"metadata": {},
"outputs": [],
"source": [
"from scipy.stats import ttest_ind"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "deaaaeb2-79d9-43e1-bf34-22b5befac82c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"t_val: 3.26, p-value: 0.00\n"
]
}
],
"source": [
"tval, pval = ttest_ind(df.loc[df['Role'] == 'First_Baseman', 'Height'], df.loc[df['Role'] == 'Third_Baseman', 'Height'], equal_var=False)\n",
"print(f\"t_val: {tval:.2f}, p-value: {pval:.2f}\")"
]
},
{
"cell_type": "markdown",
"id": "69b6a712-21bc-4f52-8a30-4ff4d5fdecf7",
"metadata": {},
"source": [
"> Since p-value is zero, we failed to reject the Alternate Hypothesis (H1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "754515cc-894f-4dfa-8a97-c29383c005d3",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.4"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Reference in new issue View Git Blame Copy Permalink