{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 概率与统计简介\n",
    "在本笔记本中，我们将练习一些之前讨论过的概念。许多概率与统计的概念在用于数据处理的主要Python库中都有良好的体现，比如 `numpy` 和 `pandas`。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import random\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 随机变量和分布\n",
    "让我们从 0 到 9 的均匀分布中抽取一个包含 30 个值的样本。我们还将计算均值和方差。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample = [ random.randint(0,10) for _ in range(30) ]\n",
    "print(f\"Sample: {sample}\")\n",
    "print(f\"Mean = {np.mean(sample)}\")\n",
    "print(f\"Variance = {np.var(sample)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了直观估计样本中有多少不同的值，我们可以绘制**直方图**：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.hist(sample)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 分析真实数据\n",
    "\n",
    "均值和方差在分析现实世界数据时非常重要。让我们加载关于棒球运动员的数据，来自 [SOCR MLB Height/Weight Data](http://wiki.stat.ucla.edu/socr/index.php/SOCR_Data_MLB_HeightsWeights)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"../../data/SOCR_MLB.tsv\",sep='\\t', header=None, names=['Name','Team','Role','Weight','Height','Age'])\n",
    "df\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 我们这里使用了一个名为[**Pandas**](https://pandas.pydata.org/)的软件包来进行数据分析。我们将在本课程后面讨论更多关于Pandas和在Python中处理数据的内容。\n",
    "\n",
    "让我们计算年龄、身高和体重的平均值：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[['Age','Height','Weight']].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在让我们关注身高，并计算标准差和方差：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(list(df['Height'])[:20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean = df['Height'].mean()\n",
    "var = df['Height'].var()\n",
    "std = df['Height'].std()\n",
    "print(f\"Mean = {mean}\\nVariance = {var}\\nStandard Deviation = {std}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除了平均值外，查看中位数和四分位数也是有意义的。它们可以通过**箱线图**来可视化：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,2))\n",
    "plt.boxplot(df['Height'].ffill(), vert=False, showmeans=True)\n",
    "plt.grid(color='gray', linestyle='dotted')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们还可以制作数据集子集的箱线图，例如按球员角色分组。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.boxplot(column='Height', by='Role', figsize=(10,8))\n",
    "plt.xticks(rotation='vertical')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **注意**：该图表表明，平均来看，一垒手的身高高于二垒手的身高。稍后我们将学习如何更正式地检验这个假设，以及如何证明我们的数据在统计上有显著性以支持该结论。\n",
    "\n",
    "年龄、身高和体重都是连续随机变量。你认为它们的分布是什么样的？发现答案的一个好方法是绘制数值的直方图：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Weight'].hist(bins=15, figsize=(10,6))\n",
    "plt.suptitle('Weight distribution of MLB Players')\n",
    "plt.xlabel('Weight')\n",
    "plt.ylabel('Count')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 正态分布\n",
    "\n",
    "让我们创建一个人工权重样本，该样本服从与我们的真实数据相同均值和方差的正态分布：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "generated = np.random.normal(mean, std, 1000)\n",
    "generated[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,6))\n",
    "plt.hist(generated, bins=15)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,6))\n",
    "plt.hist(np.random.normal(0,1,50000), bins=300)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由于现实生活中的大多数数值服从正态分布，因此我们不应该使用均匀随机数生成器来生成样本数据。以下是尝试使用均匀分布（由 `np.random.rand` 生成）生成体重时的情况：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wrong_sample = np.random.rand(1000)*2*std+mean-std\n",
    "plt.figure(figsize=(10,6))\n",
    "plt.hist(wrong_sample)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 置信区间\n",
    "\n",
    "现在让我们计算棒球运动员的体重和身高的置信区间。我们将使用[这个stackoverflow讨论](https://stackoverflow.com/questions/15033511/compute-a-confidence-interval-from-sample-data)中的代码：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scipy.stats\n",
    "\n",
    "def mean_confidence_interval(data, confidence=0.95):\n",
    "    a = 1.0 * np.array(data)\n",
    "    n = len(a)\n",
    "    m, se = np.mean(a), scipy.stats.sem(a)\n",
    "    h = se * scipy.stats.t.ppf((1 + confidence) / 2., n-1)\n",
    "    return m, h\n",
    "\n",
    "for p in [0.85, 0.9, 0.95]:\n",
    "    m, h = mean_confidence_interval(df['Weight'].fillna(method='pad'),p)\n",
    "    print(f\"p={p:.2f}, mean = {m:.2f} ± {h:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 假设检验\n",
    "\n",
    "让我们探索棒球运动员数据集中不同的角色：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.groupby('Role').agg({ 'Weight' : 'mean', 'Height' : 'mean', 'Age' : 'count'}).rename(columns={ 'Age' : 'Count'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "让我们来检验一项假设：一垒手比二垒手更高。最简单的方法是检验置信区间：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for p in [0.85,0.9,0.95]:\n",
    "    m1, h1 = mean_confidence_interval(df.loc[df['Role']=='First_Baseman',['Height']],p)\n",
    "    m2, h2 = mean_confidence_interval(df.loc[df['Role']=='Second_Baseman',['Height']],p)\n",
    "    print(f'Conf={p:.2f}, 1st basemen height: {m1-h1[0]:.2f}..{m1+h1[0]:.2f}, 2nd basemen height: {m2-h2[0]:.2f}..{m2+h2[0]:.2f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以看到这些区间没有重叠。\n",
    "\n",
    "一种统计上更正确的验证假设的方法是使用**Student t检验**：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import ttest_ind\n",
    "\n",
    "tval, pval = ttest_ind(df.loc[df['Role']=='First_Baseman',['Height']], df.loc[df['Role']=='Second_Baseman',['Height']],equal_var=False)\n",
    "print(f\"T-value = {tval[0]:.2f}\\nP-value: {pval[0]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`ttest_ind` 函数返回的两个值是：\n",
    "* p 值可以被视为两个分布具有相同均值的概率。在我们的例子中，它非常低，这意味着有强有力的证据支持一垒手更高。\n",
    "* t 值是用于 t 检验的标准化均值差的中间值，并且它会与给定置信度的阈值进行比较。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用中心极限定理模拟正态分布\n",
    "\n",
    "Python 中的伪随机生成器设计为产生均匀分布。如果我们想创建一个正态分布的生成器，可以使用中心极限定理。为了获得一个正态分布的值，我们只需计算一个均匀生成样本的平均值。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def normal_random(sample_size=100):\n",
    "    sample = [random.uniform(0,1) for _ in range(sample_size) ]\n",
    "    return sum(sample)/sample_size\n",
    "\n",
    "sample = [normal_random() for _ in range(100)]\n",
    "plt.figure(figsize=(10,6))\n",
    "plt.hist(sample)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 相关性与邪恶棒球公司\n",
    "\n",
    "相关性使我们能够发现数据序列之间的关系。在我们的玩具示例中，假设有一家邪恶的棒球公司根据球员的身高支付薪水——球员越高，拿到的钱越多。假设有一个基本工资为1000美元，并根据身高额外获得0到100美元的奖金。我们将采用MLB的真实球员数据，计算他们的虚拟薪水：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "heights = df['Height'].fillna(method='pad')\n",
    "salaries = 1000+(heights-heights.min())/(heights.max()-heights.mean())*100\n",
    "print(list(zip(heights, salaries))[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在让我们计算这些序列的协方差和相关系数。`np.cov` 将给我们所谓的**协方差矩阵**，这是协方差在多变量上的扩展。协方差矩阵 $M$ 的元素 $M_{ij}$ 是输入变量 $X_i$ 和 $X_j$ 之间的协方差，主对角线上的值 $M_{ii}$ 是 $X_i$ 的方差。类似地，`np.corrcoef` 会给我们**相关矩阵**。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Covariance matrix:\\n{np.cov(heights, salaries)}\")\n",
    "print(f\"Covariance = {np.cov(heights, salaries)[0,1]}\")\n",
    "print(f\"Correlation = {np.corrcoef(heights, salaries)[0,1]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "相关系数等于1表示两个变量之间存在强烈的**线性关系**。我们可以通过将一个值与另一个值绘制在图上直观地看到线性关系：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,6))\n",
    "plt.scatter(heights,salaries)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "让我们看看如果关系不是线性的会发生什么。假设我们的公司决定隐藏身高和薪水之间明显的线性依赖关系，并在公式中引入一些非线性，比如 `sin`：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "salaries = 1000+np.sin((heights-heights.min())/(heights.max()-heights.mean()))*100\n",
    "print(f\"Correlation = {np.corrcoef(heights, salaries)[0,1]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在这种情况下，相关性略小，但仍然相当高。现在，为了让关系更不明显，我们可能想通过向工资中添加一些随机变量来增加额外的随机性。让我们看看会发生什么：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "salaries = 1000+np.sin((heights-heights.min())/(heights.max()-heights.mean()))*100+np.random.random(size=len(heights))*20-10\n",
    "print(f\"Correlation = {np.corrcoef(heights, salaries)[0,1]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,6))\n",
    "plt.scatter(heights, salaries)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 你能猜到为什么这些点会排列成这样的垂直线吗？\n",
    "\n",
    "我们已经观察到像工资这样的人为设计的概念与观察变量*身高*之间的相关性。让我们也看看两个观察变量，例如身高和体重，是否也相关：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.corrcoef(df['Height'].ffill(),df['Weight'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "不幸的是，我们没有得到任何结果——只有一些奇怪的 `nan` 值。这是因为我们的序列中有些值未定义，表示为 `nan`，这导致运算结果也未定义。通过查看矩阵我们可以看到，`Weight` 是有问题的列，因为计算了 `Height` 值的自相关。\n",
    "\n",
    "> 这个例子展示了**数据准备**和**清理**的重要性。没有适当的数据，我们无法计算任何东西。\n",
    "\n",
    "让我们使用 `fillna` 方法填充缺失值，并计算相关性：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.corrcoef(df['Height'].fillna(method='pad'), df['Weight'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "确实存在相关性，但没有我们人工示例中那么强烈。实际上，如果我们查看一个值与另一个值的散点图，关系会明显不那么明显：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,6))\n",
    "plt.scatter(df['Weight'],df['Height'])\n",
    "plt.xlabel('Weight')\n",
    "plt.ylabel('Height')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "在本笔记本中，我们学习了如何对数据执行基本操作以计算统计函数。我们现在知道如何使用完善的数学和统计工具来验证某些假设，以及如何根据数据样本计算任意变量的置信区间。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n\n<!-- CO-OP TRANSLATOR DISCLAIMER START -->\n**免责声明**：  \n本文件由AI翻译服务[Co-op Translator](https://github.com/Azure/co-op-translator)翻译。我们尽力确保译文的准确性，但请注意自动翻译可能存在错误或不准确之处。原始文件的母语版本应被视为权威来源。对于关键信息，建议采用专业人工翻译。因使用本翻译而产生的任何误解或误释，我们概不负责。\n<!-- CO-OP TRANSLATOR DISCLAIMER END -->\n"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "coopTranslator": {
   "original_hash": "0f899e3c5019f948e7c787b22f3b2304",
   "translation_date": "2026-01-16T09:17:01+00:00",
   "source_file": "1-Introduction/04-stats-and-probability/notebook.ipynb",
   "language_code": "zh"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}