{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Probability and Statistics\n",
    "在本筆記本中，我們將實作一些之前討論過的概念。許多來自機率與統計的概念，在 Python 的主要資料處理庫中都有很好的呈現，例如 `numpy` 和 `pandas`。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import random\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 隨機變量與分佈\n",
    "讓我們從 0 到 9 的均勻分佈中抽取 30 個值的樣本。我們還會計算平均值和變異數。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample = [ random.randint(0,10) for _ in range(30) ]\n",
    "print(f\"Sample: {sample}\")\n",
    "print(f\"Mean = {np.mean(sample)}\")\n",
    "print(f\"Variance = {np.var(sample)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "為了視覺上估計樣本中有多少不同的值，我們可以繪製 **直方圖**：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.hist(sample)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 分析實際數據\n",
    "\n",
    "平均值和變異數在分析現實數據時非常重要。讓我們從 [SOCR MLB Height/Weight Data](http://wiki.stat.ucla.edu/socr/index.php/SOCR_Data_MLB_HeightsWeights) 載入有關棒球選手的數據。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"../../data/SOCR_MLB.tsv\",sep='\\t', header=None, names=['Name','Team','Role','Weight','Height','Age'])\n",
    "df\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 我們這裡使用一個名為 [**Pandas**](https://pandas.pydata.org/) 的套件進行資料分析。我們會在本課程後面更詳細地討論 Pandas 以及如何在 Python 中處理資料。\n",
    "\n",
    "讓我們計算年齡、身高和體重的平均值：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[['Age','Height','Weight']].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "現在讓我們集中於身高，並計算標準差與方差：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(list(df['Height'])[:20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean = df['Height'].mean()\n",
    "var = df['Height'].var()\n",
    "std = df['Height'].std()\n",
    "print(f\"Mean = {mean}\\nVariance = {var}\\nStandard Deviation = {std}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除平均值外，觀察中位數和四分位數亦是合理的。它們可以用**箱形圖**來視覺化：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,2))\n",
    "plt.boxplot(df['Height'].ffill(), vert=False, showmeans=True)\n",
    "plt.grid(color='gray', linestyle='dotted')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們也可以製作數據集子集的箱型圖，例如按球員角色分組。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.boxplot(column='Height', by='Role', figsize=(10,8))\n",
    "plt.xticks(rotation='vertical')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **注意**：這張圖表顯示，平均而言，一壘手的身高比二壘手的身高高。稍後我們會學習如何更正式地檢驗這個假設，以及如何證明我們的數據在統計上具有顯著性。  \n",
    "\n",
    "年齡、身高和體重都是連續隨機變數。你覺得它們的分布是怎樣的？一個好方法是繪製值的直方圖：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Weight'].hist(bins=15, figsize=(10,6))\n",
    "plt.suptitle('Weight distribution of MLB Players')\n",
    "plt.xlabel('Weight')\n",
    "plt.ylabel('Count')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 常態分佈\n",
    "\n",
    "讓我們建立一個人工的重量樣本，該樣本遵循與我們真實資料相同的平均值和變異數的常態分佈：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "generated = np.random.normal(mean, std, 1000)\n",
    "generated[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,6))\n",
    "plt.hist(generated, bins=15)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,6))\n",
    "plt.hist(np.random.normal(0,1,50000), bins=300)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由於現實生活中大多數數值呈常態分佈，我們不應使用均勻隨機數生成器來產生樣本數據。以下是如果嘗試使用均勻分佈（由 `np.random.rand` 生成）來生成體重時會發生的情況：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wrong_sample = np.random.rand(1000)*2*std+mean-std\n",
    "plt.figure(figsize=(10,6))\n",
    "plt.hist(wrong_sample)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 信心水準區間\n",
    "\n",
    "現在讓我們計算棒球選手的體重和身高的信心水準區間。我們將使用[這個 stackoverflow 討論](https://stackoverflow.com/questions/15033511/compute-a-confidence-interval-from-sample-data)中的程式碼：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scipy.stats\n",
    "\n",
    "def mean_confidence_interval(data, confidence=0.95):\n",
    "    a = 1.0 * np.array(data)\n",
    "    n = len(a)\n",
    "    m, se = np.mean(a), scipy.stats.sem(a)\n",
    "    h = se * scipy.stats.t.ppf((1 + confidence) / 2., n-1)\n",
    "    return m, h\n",
    "\n",
    "for p in [0.85, 0.9, 0.95]:\n",
    "    m, h = mean_confidence_interval(df['Weight'].fillna(method='pad'),p)\n",
    "    print(f\"p={p:.2f}, mean = {m:.2f} ± {h:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 假設檢定\n",
    "\n",
    "讓我們探索棒球球員數據集中不同的角色：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.groupby('Role').agg({ 'Weight' : 'mean', 'Height' : 'mean', 'Age' : 'count'}).rename(columns={ 'Age' : 'Count'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "讓我們測試一下「一壘手比二壘手身高較高」的假設。最簡單的方法是檢驗信賴區間：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for p in [0.85,0.9,0.95]:\n",
    "    m1, h1 = mean_confidence_interval(df.loc[df['Role']=='First_Baseman',['Height']],p)\n",
    "    m2, h2 = mean_confidence_interval(df.loc[df['Role']=='Second_Baseman',['Height']],p)\n",
    "    print(f'Conf={p:.2f}, 1st basemen height: {m1-h1[0]:.2f}..{m1+h1[0]:.2f}, 2nd basemen height: {m2-h2[0]:.2f}..{m2+h2[0]:.2f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們可以看到區間沒有重疊。\n",
    "\n",
    "一個在統計上更正確的方法來驗證假設是使用 **Student t 檢定**：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import ttest_ind\n",
    "\n",
    "tval, pval = ttest_ind(df.loc[df['Role']=='First_Baseman',['Height']], df.loc[df['Role']=='Second_Baseman',['Height']],equal_var=False)\n",
    "print(f\"T-value = {tval[0]:.2f}\\nP-value: {pval[0]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`ttest_ind` 函數返回的兩個值是： \n",
    "* p 值可以被視為兩個分布具有相同平均值的概率。在我們的例子中，它非常低，這意味著有強有力的證據支持一壘手較高。 \n",
    "* t 值是用於 t 檢驗的標準化平均差異的中間值，並且它會與給定置信值的閾值進行比較。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用中心極限定理模擬常態分佈\n",
    "\n",
    "Python 中的偽隨機產生器是設計用來給我們均勻分佈。如果我們想要創建一個常態分佈的產生器，我們可以使用中心極限定理。要獲得一個常態分佈的值，我們只需計算一組均勻分佈樣本的平均值。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def normal_random(sample_size=100):\n",
    "    sample = [random.uniform(0,1) for _ in range(sample_size) ]\n",
    "    return sum(sample)/sample_size\n",
    "\n",
    "sample = [normal_random() for _ in range(100)]\n",
    "plt.figure(figsize=(10,6))\n",
    "plt.hist(sample)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 相關性與邪惡棒球公司\n",
    "\n",
    "相關性讓我們能夠找出資料序列之間的關係。在我們的範例中，假設有一間邪惡的棒球公司，根據球員的身高支付薪水——球員越高，拿到的錢就越多。假設基本薪資是1000美元，並且根據身高額外獲得0到100美元的獎金。我們將以真實的MLB球員為例，計算他們的假想薪水：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "heights = df['Height'].fillna(method='pad')\n",
    "salaries = 1000+(heights-heights.min())/(heights.max()-heights.mean())*100\n",
    "print(list(zip(heights, salaries))[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "現在讓我們計算這些序列的共變異數和相關係數。`np.cov` 會給我們一個所謂的 **共變異數矩陣**，這是共變異數對多個變量的擴展。共變異數矩陣 $M$ 的元素 $M_{ij}$ 是輸入變量 $X_i$ 和 $X_j$ 之間的共變異數，而對角線上的值 $M_{ii}$ 是 $X_i$ 的變異數。同樣地，`np.corrcoef` 會給我們 **相關係數矩陣**。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Covariance matrix:\\n{np.cov(heights, salaries)}\")\n",
    "print(f\"Covariance = {np.cov(heights, salaries)[0,1]}\")\n",
    "print(f\"Correlation = {np.corrcoef(heights, salaries)[0,1]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "相關係數等於 1 意味著兩個變量之間存在強烈的**線性關係**。我們可以通過將一個值繪製對另一個值來直觀地看到這種線性關係：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,6))\n",
    "plt.scatter(heights,salaries)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "讓我們看看如果關係不是線性的會發生什麼。假設我們的公司決定隱藏身高和薪水之間明顯的線性依賴關係，並在公式中引入一些非線性，例如 `sin`：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "salaries = 1000+np.sin((heights-heights.min())/(heights.max()-heights.mean()))*100\n",
    "print(f\"Correlation = {np.corrcoef(heights, salaries)[0,1]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在這種情況下，相關性略微減小，但仍然相當高。現在，為了使關係變得不那麼明顯，我們可能想通過給薪水增加一些隨機變量來加入一些額外的隨機性。讓我們看看會發生什麼：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "salaries = 1000+np.sin((heights-heights.min())/(heights.max()-heights.mean()))*100+np.random.random(size=len(heights))*20-10\n",
    "print(f\"Correlation = {np.corrcoef(heights, salaries)[0,1]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,6))\n",
    "plt.scatter(heights, salaries)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 你能猜到為什麼這些點會排列成像這樣的垂直線嗎？\n",
    "\n",
    "我們已經觀察到像薪水這樣的人為設計概念與觀察到的變量*身高*之間的關聯。接下來讓我們看看兩個觀察到的變量，例如身高和體重，是否也相關：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.corrcoef(df['Height'].ffill(),df['Weight'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "不幸地，我們沒有得到任何結果——只有一些奇怪的 `nan` 值。這是由於我們序列中有些值是未定義的，用 `nan` 表示，這導致運算結果也未定義。透過觀察矩陣，我們可以看到 `Weight` 是有問題的欄位，因為已經計算了 `Height` 值之間的自相關。\n",
    "\n",
    "> 這個例子顯示了**資料準備**和**清理**的重要性。沒有適當的資料，我們無法計算任何東西。\n",
    "\n",
    "讓我們使用 `fillna` 方法來填補遺失值，並計算相關性：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.corrcoef(df['Height'].fillna(method='pad'), df['Weight'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "確實存在相關性，但不像我們的人工範例中那麼強烈。事實上，如果我們查看一個數值與另一個數值的散點圖，關係會少得多明顯：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,6))\n",
    "plt.scatter(df['Weight'],df['Height'])\n",
    "plt.xlabel('Weight')\n",
    "plt.ylabel('Height')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "在本筆記本中，我們已經學會了如何對數據執行基本操作以計算統計函數。我們現在知道如何使用完善的數學和統計工具來驗證一些假設，以及如何根據數據樣本計算任意變量的置信區間。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n\n<!-- CO-OP TRANSLATOR DISCLAIMER START -->\n**免責聲明**：  \n本文件由 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。雖然我們致力於確保準確性，但請注意，自動翻譯可能包含錯誤或不準確之處。原始語言版本文件應視為權威來源。對於重要資訊，建議採用專業人工翻譯。對於因使用本翻譯而產生的任何誤解或誤譯，我們概不負責。\n<!-- CO-OP TRANSLATOR DISCLAIMER END -->\n"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "coopTranslator": {
   "original_hash": "0f899e3c5019f948e7c787b22f3b2304",
   "translation_date": "2026-01-16T10:09:45+00:00",
   "source_file": "1-Introduction/04-stats-and-probability/notebook.ipynb",
   "language_code": "hk"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}