Data-Science-For-Beginners/translations/zh-MO/1-Introduction/01-defining-data-science/notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 挑戰：分析關於數據科學的文本\n",
    "\n",
    "在這個例子中，讓我們做一個涵蓋傳統數據科學流程所有步驟的簡單練習。你不需要撰寫任何程式碼，只需點擊下方的單元格來執行它們並觀察結果。作為挑戰，你可以嘗試用不同的數據來執行這段程式碼。\n",
    "\n",
    "## 目標\n",
    "\n",
    "在這堂課中，我們討論了與數據科學相關的不同概念。讓我們透過進行一些<strong>文本挖掘</strong>來嘗試發掘更多相關概念。我們將從一篇關於數據科學的文本開始，從中提取關鍵詞，然後嘗試將結果視覺化。\n",
    "\n",
    "作為文本，我將使用維基百科上關於數據科學的頁面：\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://en.wikipedia.org/wiki/Data_science'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第一步：獲取資料\n",
    "\n",
    "每個資料科學流程的第一步是獲取資料。我們將使用 `requests` 庫來完成這個步驟：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "# Define a custom header.\n",
    "headers = {\n",
    "    'User-Agent': 'DataScienceChallenge/1.0 (myemail@gmail.com)'\n",
    "}\n",
    "\n",
    "# Pass the headers into the get request\n",
    "response = requests.get(url, headers=headers)\n",
    "\n",
    "if response.status_code == 200:\n",
    "    text = response.content.decode('utf-8')\n",
    "    print(text[:1000])\n",
    "else:\n",
    "    print(f\"Error: {response.status_code}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第 2 步：轉換數據\n",
    "\n",
    "下一步是將數據轉換成適合處理的形式。在我們的情況中，我們已經從頁面下載了 HTML 原始碼，我們需要將其轉換為純文本。\n",
    "\n",
    "這可以通過多種方式完成。我們將使用 [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/)，一個流行的 Python HTML 解析庫。BeautifulSoup 允許我們定位特定的 HTML 元素，這樣可以專注於維基百科的主要文章內容，並減少一些導航菜單、側邊欄、頁腳及其他無關內容（雖然有些標準文本仍可能存在）。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先，我哋需要安裝 BeautifulSoup 函式庫用嚟解析 HTML：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install beautifulsoup4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Parse the HTML content\n",
    "soup = BeautifulSoup(text, 'html.parser')\n",
    "\n",
    "# Extract only the main article content from Wikipedia\n",
    "# Wikipedia uses 'mw-parser-output' class for the main article content\n",
    "content = soup.find('div', class_='mw-parser-output')\n",
    "\n",
    "def clean_wikipedia_content(content_node):\n",
    "    \"\"\"Remove common non-article elements from a Wikipedia content node.\"\"\"\n",
    "    # Strip jump links, navboxes, reference lists/superscripts, edit sections, TOC, sidebars, etc.\n",
    "    selectors = [\n",
    "        '.mw-jump-link',\n",
    "        '.navbox',\n",
    "        '.reflist',\n",
    "        'sup.reference',\n",
    "        '.mw-editsection',\n",
    "        '.hatnote',\n",
    "        '.metadata',\n",
    "        '.infobox',\n",
    "        '#toc',\n",
    "        '.toc',\n",
    "        '.sidebar',\n",
    "    ]\n",
    "    for selector in selectors:\n",
    "        for el in content_node.select(selector):\n",
    "            el.decompose()\n",
    "\n",
    "if content:\n",
    "    # Clean the content node to better approximate article text only.\n",
    "    clean_wikipedia_content(content)\n",
    "    text = content.get_text(separator=' ', strip=True)\n",
    "    print(text[:1000])\n",
    "else:\n",
    "    print(\"Could not find main content. Using full page text.\")\n",
    "    text = soup.get_text(separator=' ', strip=True)\n",
    "    print(text[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第 3 步：獲取洞察\n",
    "\n",
    "最重要的一步是將我們的資料轉換成一種可以從中提取洞察的形式。在我們的例子中，我們希望從文本中提取關鍵字，並查看哪些關鍵字更有意義。\n",
    "\n",
    "我們將使用名為 [RAKE](https://github.com/aneesha/RAKE) 的 Python 函式庫來提取關鍵字。首先，讓我們安裝這個函式庫（如果尚未安裝的話）： \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install nlp_rake"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "主要功能可從 `Rake` 物件獲取，我們可以使用一些參數來自訂。在我們的例子中，我們會將關鍵字的最小長度設定為 5 個字元，關鍵字在文件中出現的最小頻率設定為 3，以及關鍵字中最大詞數設定為 2。歡迎隨意更改其他數值並觀察結果。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nlp_rake\n",
    "extractor = nlp_rake.Rake(max_words=2,min_freq=3,min_chars=5)\n",
    "res = extractor.apply(text)\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "我們取得了一個術語列表以及相關的重要程度。如你所見，最相關的學科，如機器學習和大數據，都出現在列表的頂部位置。\n",
    "\n",
    "## 第四步：結果視覺化\n",
    "\n",
    "人們可以最有效地以視覺形式詮釋數據。因此，經常有意義將數據視覺化以提取一些見解。我們可以使用 Python 的 `matplotlib` 庫來繪製關鍵字及其相關性的簡單分佈圖：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def plot(pair_list):\n",
    "    k,v = zip(*pair_list)\n",
    "    plt.bar(range(len(k)),v)\n",
    "    plt.xticks(range(len(k)),k,rotation='vertical')\n",
    "    plt.show()\n",
    "\n",
    "plot(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "不過，還有一種更佳的方法來視覺化詞頻——使用 <strong>詞雲</strong>。我們需要安裝另一個庫來從關鍵字列表繪製詞雲。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!{sys.executable} -m pip install wordcloud"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`WordCloud` 物件負責接收原始文字或預先計算好的詞語及其出現頻率清單，並返回一張影像，然後可以使用 `matplotlib` 顯示該影像：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from wordcloud import WordCloud\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "wc = WordCloud(background_color='white',width=800,height=600)\n",
    "plt.figure(figsize=(15,7))\n",
    "plt.imshow(wc.generate_from_frequencies({ k:v for k,v in res }))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們也可以將原始文字傳入 `WordCloud` — 讓我們看看是否能獲得類似的結果：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15,7))\n",
    "plt.imshow(wc.generate(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wc.generate(text).to_file('images/ds_wordcloud.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你可以見到那個文字雲現時看起來更令人印象深刻，但它同時包含了很多雜訊（例如無關的詞語如 `Retrieved on`）。而且，我們得到的由兩個詞組成的關鍵詞更少，例如 *data scientist*，或 *computer science*。這是因為 RAKE 演算法在從文本中選擇好關鍵詞方面做得更好。這個例子說明了資料預處理和清理的重要性，因為最終清晰的圖像將讓我們做出更好的決策。\n",
    "\n",
    "在這個練習中，我們經歷了一個從維基百科文本中提取部分含義的簡單過程，以關鍵詞和文字雲的形式呈現。這個例子相當簡單，但它很好地展示了資料科學家在處理資料時通常會採取的所有典型步驟，從資料獲取開始，到視覺化結束。\n",
    "\n",
    "在我們的課程中，我們將詳細討論所有這些步驟。 \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n\n<!-- CO-OP TRANSLATOR DISCLAIMER START -->\n**免責聲明**：\n本文件使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。雖然我們力求準確，但請注意，自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於重要資訊，建議尋求專業人工翻譯。我們不對因使用本翻譯而引起的任何誤解或曲解承擔責任。\n<!-- CO-OP TRANSLATOR DISCLAIMER END -->\n"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "c28e7b6bf4e5b397b8288a85bf0a94ea8d3585ce2b01919feb195678ec71581b"
  },
  "kernelspec": {
   "display_name": "Python 3.8.11 64-bit ('base': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}