{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 挑戰：分析關於數據科學的文本\n",
    "\n",
    "在這個例子中，我們做一個涵蓋傳統數據科學流程所有步驟的簡單練習。你不需要寫任何代碼，只需點擊下面的單元格執行它們並觀察結果。作為挑戰，鼓勵你使用不同的數據嘗試這段代碼。\n",
    "\n",
    "## 目標\n",
    "\n",
    "在本課程中，我們討論了與數據科學相關的不同概念。讓我們透過做一些<strong>文本挖掘</strong>來嘗試發現更多相關概念。我們將從一段關於數據科學的文本開始，從中提取關鍵字，然後嘗試將結果視覺化。\n",
    "\n",
    "我將使用維基百科上關於數據科學的頁面作為文本：\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://en.wikipedia.org/wiki/Data_science'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第一步：取得數據\n",
    "\n",
    "每個數據科學流程的第一步都是取得數據。我們將使用 `requests` 函式庫來完成這個步驟：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "# Define a custom header.\n",
    "headers = {\n",
    "    'User-Agent': 'DataScienceChallenge/1.0 (myemail@gmail.com)'\n",
    "}\n",
    "\n",
    "# Pass the headers into the get request\n",
    "response = requests.get(url, headers=headers)\n",
    "\n",
    "if response.status_code == 200:\n",
    "    text = response.content.decode('utf-8')\n",
    "    print(text[:1000])\n",
    "else:\n",
    "    print(f\"Error: {response.status_code}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第 2 步：轉換數據\n",
    "\n",
    "下一步是將數據轉換成適合處理的形式。在我們的例子中，我們已經從網頁下載了 HTML 原始碼，現在需要將它轉換成純文本。\n",
    "\n",
    "這可以用多種方法完成。我們將使用 [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/)，一個流行的 Python 解析 HTML 的庫。BeautifulSoup 讓我們能夠定位特定的 HTML 元素，因此我們可以專注於維基百科的主要文章內容，並減少一些導航菜單、側邊欄、頁腳和其他無關的內容（雖然仍可能留有一些樣板文字）。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先，我們需要安裝 BeautifulSoup 庫以進行 HTML 解析：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install beautifulsoup4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Parse the HTML content\n",
    "soup = BeautifulSoup(text, 'html.parser')\n",
    "\n",
    "# Extract only the main article content from Wikipedia\n",
    "# Wikipedia uses 'mw-parser-output' class for the main article content\n",
    "content = soup.find('div', class_='mw-parser-output')\n",
    "\n",
    "def clean_wikipedia_content(content_node):\n",
    "    \"\"\"Remove common non-article elements from a Wikipedia content node.\"\"\"\n",
    "    # Strip jump links, navboxes, reference lists/superscripts, edit sections, TOC, sidebars, etc.\n",
    "    selectors = [\n",
    "        '.mw-jump-link',\n",
    "        '.navbox',\n",
    "        '.reflist',\n",
    "        'sup.reference',\n",
    "        '.mw-editsection',\n",
    "        '.hatnote',\n",
    "        '.metadata',\n",
    "        '.infobox',\n",
    "        '#toc',\n",
    "        '.toc',\n",
    "        '.sidebar',\n",
    "    ]\n",
    "    for selector in selectors:\n",
    "        for el in content_node.select(selector):\n",
    "            el.decompose()\n",
    "\n",
    "if content:\n",
    "    # Clean the content node to better approximate article text only.\n",
    "    clean_wikipedia_content(content)\n",
    "    text = content.get_text(separator=' ', strip=True)\n",
    "    print(text[:1000])\n",
    "else:\n",
    "    print(\"Could not find main content. Using full page text.\")\n",
    "    text = soup.get_text(separator=' ', strip=True)\n",
    "    print(text[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第 3 步：獲取見解\n",
    "\n",
    "最重要的一步是將我們的數據轉換成某種格式，從中提取見解。在我們的案例中，我們想從文本中提取關鍵詞，並了解哪些關鍵詞更有意義。\n",
    "\n",
    "我們將使用名為 [RAKE](https://github.com/aneesha/RAKE) 的 Python 函式庫進行關鍵詞提取。首先，讓我們安裝此函式庫，以防尚未安裝： \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install nlp_rake"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "主要功能由 `Rake` 物件提供，我們可以使用一些參數來自訂。以我們的例子，我們會設定關鍵字的最小長度為5個字元，文件中關鍵字的最小出現頻率為3，以及關鍵字中最多的字數為2。隨意嘗試其他值並觀察結果。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nlp_rake\n",
    "extractor = nlp_rake.Rake(max_words=2,min_freq=3,min_chars=5)\n",
    "res = extractor.apply(text)\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "我們獲得了一個詞彙列表以及相關的重要程度。如你所見，最相關的學科，如機器學習和大數據，在列表中位於頂部位置。\n",
    "\n",
    "## 步驟4：結果視覺化\n",
    "\n",
    "人們能夠以視覺形式最佳解讀數據。因此，通常將數據視覺化以獲取一些洞見是合理的。我們可以使用 Python 中的 `matplotlib` 庫繪製關鍵詞及其相關性的簡單分佈圖：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def plot(pair_list):\n",
    "    k,v = zip(*pair_list)\n",
    "    plt.bar(range(len(k)),v)\n",
    "    plt.xticks(range(len(k)),k,rotation='vertical')\n",
    "    plt.show()\n",
    "\n",
    "plot(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "不過，有一個更好的方法去視覺化字詞頻率 —— 使用 <strong>詞雲</strong>。我們需要安裝另一個庫，來從我們的關鍵字列表繪製詞雲。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!{sys.executable} -m pip install wordcloud"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`WordCloud` 物件負責接收原始文字或預先計算好的字詞及其頻率列表，並回傳一張圖片，該圖片隨後可以使用 `matplotlib` 顯示：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from wordcloud import WordCloud\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "wc = WordCloud(background_color='white',width=800,height=600)\n",
    "plt.figure(figsize=(15,7))\n",
    "plt.imshow(wc.generate_from_frequencies({ k:v for k,v in res }))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們也可以將原始文字傳給 `WordCloud` - 讓我們看看是否能獲得相似的結果：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15,7))\n",
    "plt.imshow(wc.generate(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wc.generate(text).to_file('images/ds_wordcloud.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你可以看到那個詞雲現在看起來更有氣勢，但它也包含了很多噪音（例如無關的詞如 `Retrieved on`）。此外，我們獲得的由兩個字組成的關鍵字較少，比如 *data scientist* 或 *computer science*。這是因為 RAKE 演算法在從文本中選擇好的關鍵字方面效果更佳。這個例子說明了數據預處理和清理的重要性，因為最後清晰的圖像將允許我們做出更好的決策。\n",
    "\n",
    "在這個練習中，我們通過一個簡單的流程從維基百科文本中提取一些意義，以關鍵字和詞雲的形式呈現。這個例子相當簡單，但它很好地展示了數據科學家在處理數據時通常會採取的所有步驟，從數據獲取一直到視覺化。\n",
    "\n",
    "在我們的課程中，我們會詳細討論所有這些步驟。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n\n<!-- CO-OP TRANSLATOR DISCLAIMER START -->\n**免責聲明**：\n本文件由 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 翻譯而成。雖然我們致力於確保準確性，但請注意，機器自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於重要資訊，建議進行專業人工翻譯。我們不對因使用本翻譯而產生的任何誤解或誤釋承擔責任。\n<!-- CO-OP TRANSLATOR DISCLAIMER END -->\n"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "c28e7b6bf4e5b397b8288a85bf0a94ea8d3585ce2b01919feb195678ec71581b"
  },
  "kernelspec": {
   "display_name": "Python 3.8.11 64-bit ('base': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}