{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 挑戰：分析關於資料科學的文本\n",
    "\n",
    "在這個範例中，我們來做一個涵蓋傳統資料科學流程所有步驟的簡單練習。你不必寫任何程式碼，只需點擊下面的儲存格執行它們並觀察結果。作為挑戰，鼓勵你使用不同的資料來嘗試這段程式碼。\n",
    "\n",
    "## 目標\n",
    "\n",
    "在本課程中，我們已經討論了與資料科學相關的不同概念。讓我們嘗試透過做一些<strong>文本探勘</strong>來發掘更多相關概念。我們將從一段關於資料科學的文本開始，從中萃取關鍵詞，然後嘗試將結果視覺化。\n",
    "\n",
    "作為文本，我將使用維基百科關於資料科學的頁面：\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://en.wikipedia.org/wiki/Data_science'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第一步：取得資料\n",
    "\n",
    "每個資料科學流程的第一步都是取得資料。我們將使用 `requests` 函式庫來做到這點：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "# Define a custom header.\n",
    "headers = {\n",
    "    'User-Agent': 'DataScienceChallenge/1.0 (myemail@gmail.com)'\n",
    "}\n",
    "\n",
    "# Pass the headers into the get request\n",
    "response = requests.get(url, headers=headers)\n",
    "\n",
    "if response.status_code == 200:\n",
    "    text = response.content.decode('utf-8')\n",
    "    print(text[:1000])\n",
    "else:\n",
    "    print(f\"Error: {response.status_code}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 步驟 2：轉換資料\n",
    "\n",
    "下一步是將資料轉換成適合處理的形式。以我們的情境來說，我們已經從網頁下載了 HTML 原始碼，而我們需要將它轉換成純文字。\n",
    "\n",
    "有許多方法可以做到這件事。我們將使用 [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/)，一個流行的 Python HTML 解析函式庫。BeautifulSoup 讓我們可以針對特定的 HTML 元素處理，因此我們能專注於維基百科的主要文章內容，並減少一些導覽選單、側欄、頁尾以及其他無關內容（雖然某些樣板文字仍可能殘留）。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先，我們需要安裝用於 HTML 解析的 BeautifulSoup 函式庫：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install beautifulsoup4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Parse the HTML content\n",
    "soup = BeautifulSoup(text, 'html.parser')\n",
    "\n",
    "# Extract only the main article content from Wikipedia\n",
    "# Wikipedia uses 'mw-parser-output' class for the main article content\n",
    "content = soup.find('div', class_='mw-parser-output')\n",
    "\n",
    "def clean_wikipedia_content(content_node):\n",
    "    \"\"\"Remove common non-article elements from a Wikipedia content node.\"\"\"\n",
    "    # Strip jump links, navboxes, reference lists/superscripts, edit sections, TOC, sidebars, etc.\n",
    "    selectors = [\n",
    "        '.mw-jump-link',\n",
    "        '.navbox',\n",
    "        '.reflist',\n",
    "        'sup.reference',\n",
    "        '.mw-editsection',\n",
    "        '.hatnote',\n",
    "        '.metadata',\n",
    "        '.infobox',\n",
    "        '#toc',\n",
    "        '.toc',\n",
    "        '.sidebar',\n",
    "    ]\n",
    "    for selector in selectors:\n",
    "        for el in content_node.select(selector):\n",
    "            el.decompose()\n",
    "\n",
    "if content:\n",
    "    # Clean the content node to better approximate article text only.\n",
    "    clean_wikipedia_content(content)\n",
    "    text = content.get_text(separator=' ', strip=True)\n",
    "    print(text[:1000])\n",
    "else:\n",
    "    print(\"Could not find main content. Using full page text.\")\n",
    "    text = soup.get_text(separator=' ', strip=True)\n",
    "    print(text[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第三步：獲取洞察\n",
    "\n",
    "最重要的步驟是將資料轉換成我們能夠從中獲取洞察的某種形式。在我們的案例中，我們想從文本中擷取關鍵字，並查看哪些關鍵字更有意義。\n",
    "\n",
    "我們將使用一個名為 [RAKE](https://github.com/aneesha/RAKE) 的 Python 函式庫來進行關鍵字擷取。首先，讓我們安裝這個函式庫，以防尚未安裝： \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install nlp_rake"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "主要功能來自 `Rake` 物件，我們可以使用一些參數來自訂。就我們來說，我們會設定關鍵字的最小長度為 5 個字元，關鍵字在文件中的最小出現次數為 3，關鍵字中最大詞數為 2。歡迎自由嘗試其他數值並觀察結果。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nlp_rake\n",
    "extractor = nlp_rake.Rake(max_words=2,min_freq=3,min_chars=5)\n",
    "res = extractor.apply(text)\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "我們獲得了一組詞條及其相關的重要程度。如你所見，一些最相關的領域，例如機器學習和大數據，位於清單的前列。\n",
    "\n",
    "## 第四步：結果視覺化\n",
    "\n",
    "人們通常能更好地解讀視覺化的資料。因此，將資料視覺化以便獲得一些洞見常常是有意義的。我們可以使用 Python 的 `matplotlib` 函式庫來繪製關鍵詞與其相關度的簡單分佈圖：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def plot(pair_list):\n",
    "    k,v = zip(*pair_list)\n",
    "    plt.bar(range(len(k)),v)\n",
    "    plt.xticks(range(len(k)),k,rotation='vertical')\n",
    "    plt.show()\n",
    "\n",
    "plot(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "不過，有一種更好的方式來視覺化詞頻——使用 **文字雲（Word Cloud）**。我們需要安裝另一個函式庫來從我們的關鍵字列表繪製文字雲。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!{sys.executable} -m pip install wordcloud"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`WordCloud` 物件負責接收原始文本或預先計算的詞彙頻率清單，並回傳一張圖像，該圖像可以用 `matplotlib` 進行顯示：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from wordcloud import WordCloud\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "wc = WordCloud(background_color='white',width=800,height=600)\n",
    "plt.figure(figsize=(15,7))\n",
    "plt.imshow(wc.generate_from_frequencies({ k:v for k,v in res }))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們也可以將原始文字傳入 `WordCloud` — 讓我們看看是否能得到類似的結果：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15,7))\n",
    "plt.imshow(wc.generate(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wc.generate(text).to_file('images/ds_wordcloud.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你現在可以看到詞雲看起來更令人印象深刻，但它也包含了很多噪音（例如不相關的詞如 `Retrieved on`）。此外，我們得到的由兩個詞組成的關鍵字較少，例如 *data scientist* 或 *computer science*。這是因為 RAKE 演算法在從文本中選取好的關鍵字時表現更佳。這個例子說明了資料預處理和清理的重要性，因為最後清晰的圖像將讓我們做出更好的決策。\n",
    "\n",
    "在這個練習中，我們經歷了一個簡單的過程，從維基百科文本中提取一些意義，以關鍵字和詞雲的形式呈現。這個例子相當簡單，但它很好地展現了資料科學家在處理數據時會採取的所有典型步驟，從資料獲取到視覺化。\n",
    "\n",
    "在我們的課程中，我們將詳細討論所有這些步驟。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n\n<!-- CO-OP TRANSLATOR DISCLAIMER START -->\n**免責聲明**：\n此文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。雖然我們努力追求準確性，但請注意自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應視為權威來源。對於關鍵資訊，建議採用專業人工翻譯。我們不對因使用此翻譯所產生的任何誤解或誤譯承擔責任。\n<!-- CO-OP TRANSLATOR DISCLAIMER END -->\n"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "c28e7b6bf4e5b397b8288a85bf0a94ea8d3585ce2b01919feb195678ec71581b"
  },
  "kernelspec": {
   "display_name": "Python 3.8.11 64-bit ('base': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}