Data-Science-For-Beginners/translations/zh-CN/1-Introduction/01-defining-data-science/notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 挑战：分析有关数据科学的文本\n",
    "\n",
    "在此示例中，让我们做一个涵盖传统数据科学过程所有步骤的简单练习。你不必编写任何代码，可以直接点击下面的单元格来执行并观察结果。作为一个挑战，鼓励你尝试使用不同的数据运行此代码。\n",
    "\n",
    "## 目标\n",
    "\n",
    "在本课中，我们一直在讨论与数据科学相关的不同概念。让我们通过做一些<strong>文本挖掘</strong>来尝试发现更多相关概念。我们将从一段关于数据科学的文本开始，从中提取关键词，然后尝试对结果进行可视化。\n",
    "\n",
    "作为文本，我将使用维基百科上关于数据科学的页面：\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://en.wikipedia.org/wiki/Data_science'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第一步：获取数据\n",
    "\n",
    "每个数据科学流程的第一步是获取数据。我们将使用 `requests` 库来完成这一步：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "# Define a custom header.\n",
    "headers = {\n",
    "    'User-Agent': 'DataScienceChallenge/1.0 (myemail@gmail.com)'\n",
    "}\n",
    "\n",
    "# Pass the headers into the get request\n",
    "response = requests.get(url, headers=headers)\n",
    "\n",
    "if response.status_code == 200:\n",
    "    text = response.content.decode('utf-8')\n",
    "    print(text[:1000])\n",
    "else:\n",
    "    print(f\"Error: {response.status_code}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第2步：转换数据\n",
    "\n",
    "接下来的步骤是将数据转换成适合处理的形式。在我们的例子中，我们已经下载了页面的HTML源代码，需要将其转换为纯文本。\n",
    "\n",
    "有很多方法可以做到这一点。我们将使用 [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/)，这是一个流行的Python库，用于解析HTML。BeautifulSoup允许我们定位特定的HTML元素，因此我们可以专注于维基百科的主要文章内容，减少一些导航菜单、侧边栏、页脚和其他无关内容（尽管可能仍会残留一些模板文本）。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先，我们需要安装用于HTML解析的BeautifulSoup库：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install beautifulsoup4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Parse the HTML content\n",
    "soup = BeautifulSoup(text, 'html.parser')\n",
    "\n",
    "# Extract only the main article content from Wikipedia\n",
    "# Wikipedia uses 'mw-parser-output' class for the main article content\n",
    "content = soup.find('div', class_='mw-parser-output')\n",
    "\n",
    "def clean_wikipedia_content(content_node):\n",
    "    \"\"\"Remove common non-article elements from a Wikipedia content node.\"\"\"\n",
    "    # Strip jump links, navboxes, reference lists/superscripts, edit sections, TOC, sidebars, etc.\n",
    "    selectors = [\n",
    "        '.mw-jump-link',\n",
    "        '.navbox',\n",
    "        '.reflist',\n",
    "        'sup.reference',\n",
    "        '.mw-editsection',\n",
    "        '.hatnote',\n",
    "        '.metadata',\n",
    "        '.infobox',\n",
    "        '#toc',\n",
    "        '.toc',\n",
    "        '.sidebar',\n",
    "    ]\n",
    "    for selector in selectors:\n",
    "        for el in content_node.select(selector):\n",
    "            el.decompose()\n",
    "\n",
    "if content:\n",
    "    # Clean the content node to better approximate article text only.\n",
    "    clean_wikipedia_content(content)\n",
    "    text = content.get_text(separator=' ', strip=True)\n",
    "    print(text[:1000])\n",
    "else:\n",
    "    print(\"Could not find main content. Using full page text.\")\n",
    "    text = soup.get_text(separator=' ', strip=True)\n",
    "    print(text[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第3步：获取洞察\n",
    "\n",
    "最重要的步骤是将我们的数据转换成可以从中提取洞察的某种形式。在我们的案例中，我们想要从文本中提取关键词，看看哪些关键词更有意义。\n",
    "\n",
    "我们将使用名为[RAKE](https://github.com/aneesha/RAKE)的Python库来提取关键词。首先，如果没有安装该库，让我们先安装它：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install nlp_rake"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "主要功能来自 `Rake` 对象，我们可以使用一些参数对其进行自定义。在我们的例子中，我们将关键字的最小长度设置为5个字符，关键字在文档中的最小出现频率设置为3，关键字中最多包含的词数设置为2。欢迎自由尝试其他值并观察结果。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nlp_rake\n",
    "extractor = nlp_rake.Rake(max_words=2,min_freq=3,min_chars=5)\n",
    "res = extractor.apply(text)\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "我们获得了一组术语及其相关的重要程度。如你所见，最相关的学科，如机器学习和大数据，排在列表的前列。\n",
    "\n",
    "## 第四步：可视化结果\n",
    "\n",
    "人们能够最好地通过可视化来理解数据。因此通常有必要将数据可视化，以便获取一些洞见。我们可以使用 Python 中的 `matplotlib` 库来绘制关键词及其相关性的简单分布：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def plot(pair_list):\n",
    "    k,v = zip(*pair_list)\n",
    "    plt.bar(range(len(k)),v)\n",
    "    plt.xticks(range(len(k)),k,rotation='vertical')\n",
    "    plt.show()\n",
    "\n",
    "plot(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但是，还有一种更好的方式来可视化词频——使用<strong>词云</strong>。我们需要安装另一个库来从我们的关键词列表绘制词云。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!{sys.executable} -m pip install wordcloud"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`WordCloud` 对象负责接收原始文本或预先计算好的词语及其频率列表，并返回一张图片，然后可以使用 `matplotlib` 显示该图片：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from wordcloud import WordCloud\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "wc = WordCloud(background_color='white',width=800,height=600)\n",
    "plt.figure(figsize=(15,7))\n",
    "plt.imshow(wc.generate_from_frequencies({ k:v for k,v in res }))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们也可以将原始文本传递给 `WordCloud` - 让我们看看是否能够得到类似的结果：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15,7))\n",
    "plt.imshow(wc.generate(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wc.generate(text).to_file('images/ds_wordcloud.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你可以看到词云现在看起来更加引人注目，但它也包含了很多噪音（例如无关的词语如 `Retrieved on`）。此外，我们得到的由两个单词组成的关键词更少了，例如 <em>数据科学家</em> 或 <em>计算机科学</em>。这是因为 RAKE 算法在从文本中选择优质关键词方面做得更好。这个例子说明了数据预处理和清理的重要性，因为最终的清晰图像将帮助我们做出更好的决策。\n",
    "\n",
    "在这个练习中，我们通过一个简单的过程从维基百科文本中提取了一些含义，以关键词和词云的形式展示。这个例子相当简单，但很好地展示了数据科学家在处理数据时会采取的典型步骤，从数据获取到可视化。\n",
    "\n",
    "在我们的课程中，我们将详细讨论所有这些步骤。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n\n<!-- CO-OP TRANSLATOR DISCLAIMER START -->\n**免责声明**：\n本文件由 AI 翻译服务 [Co-op Translator](https://github.com/Azure/co-op-translator) 翻译完成。尽管我们力求准确，但请注意，自动翻译可能包含错误或不准确之处。原始语言版文件应视为权威来源。对于重要信息，建议使用专业人工翻译。我们对因使用本翻译而产生的任何误解或误释不承担责任。\n<!-- CO-OP TRANSLATOR DISCLAIMER END -->\n"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "c28e7b6bf4e5b397b8288a85bf0a94ea8d3585ce2b01919feb195678ec71581b"
  },
  "kernelspec": {
   "display_name": "Python 3.8.11 64-bit ('base': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}