Data-Science-For-Beginners/translations/pcm/1-Introduction/01-defining-data-science/notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenge: Analyzing Text about Data Science\n",
    "\n",
    "For dis example, make we do one simple exercise wey go cover all di steps of one traditional data science process. You no need write any code, you fit just click di cells wey dey below to run dem and see di result. As challenge, you dey encouraged to try dis code wit different data. \n",
    "\n",
    "## Goal\n",
    "\n",
    "For dis lesson, we don dey talk about different concepts wey relate to Data Science. Make we try find more concepts wey connect by doing some **text mining**. We go start wit one text about Data Science, extract keywords from am, and den try to show di result.\n",
    "\n",
    "As text, I go use di page wey dey for Data Science from Wikipedia:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://en.wikipedia.org/wiki/Data_science'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Getting the Data\n",
    "\n",
    "First step for every data science process na to get the data. We go use `requests` library do am:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "# Define a custom header.\n",
    "headers = {\n",
    "    'User-Agent': 'DataScienceChallenge/1.0 (myemail@gmail.com)'\n",
    "}\n",
    "\n",
    "# Pass the headers into the get request\n",
    "response = requests.get(url, headers=headers)\n",
    "\n",
    "if response.status_code == 200:\n",
    "    text = response.content.decode('utf-8')\n",
    "    print(text[:1000])\n",
    "else:\n",
    "    print(f\"Error: {response.status_code}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Transfom di Data\n",
    "\n",
    "Di next step na to change di data to the form wey fit make processing easy. For our case, we don download HTML source code from di page, an we need change am into plain text.\n",
    "\n",
    "Plenty ways dey we fit do dis. We go use [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/), one correct Python library wey dem dey use parse HTML. BeautifulSoup dey allow us to select correct HTML elements, so we fit focus on main article content from Wikipedia and reduce some navigation menus, sidebars, footers, and other content wey no too important (though some boilerplate text fit still dey).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fes, we need to install the BeautifulSoup library for HTML parsing:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install beautifulsoup4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Parse the HTML content\n",
    "soup = BeautifulSoup(text, 'html.parser')\n",
    "\n",
    "# Extract only the main article content from Wikipedia\n",
    "# Wikipedia uses 'mw-parser-output' class for the main article content\n",
    "content = soup.find('div', class_='mw-parser-output')\n",
    "\n",
    "def clean_wikipedia_content(content_node):\n",
    "    \"\"\"Remove common non-article elements from a Wikipedia content node.\"\"\"\n",
    "    # Strip jump links, navboxes, reference lists/superscripts, edit sections, TOC, sidebars, etc.\n",
    "    selectors = [\n",
    "        '.mw-jump-link',\n",
    "        '.navbox',\n",
    "        '.reflist',\n",
    "        'sup.reference',\n",
    "        '.mw-editsection',\n",
    "        '.hatnote',\n",
    "        '.metadata',\n",
    "        '.infobox',\n",
    "        '#toc',\n",
    "        '.toc',\n",
    "        '.sidebar',\n",
    "    ]\n",
    "    for selector in selectors:\n",
    "        for el in content_node.select(selector):\n",
    "            el.decompose()\n",
    "\n",
    "if content:\n",
    "    # Clean the content node to better approximate article text only.\n",
    "    clean_wikipedia_content(content)\n",
    "    text = content.get_text(separator=' ', strip=True)\n",
    "    print(text[:1000])\n",
    "else:\n",
    "    print(\"Could not find main content. Using full page text.\")\n",
    "    text = soup.get_text(separator=' ', strip=True)\n",
    "    print(text[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Get Insights\n",
    "\n",
    "Di most important step na to change our data into one kain form wey we fit use draw insight. For our case, we wan comot keywords from di text, and see which keywords dey more meaningful.\n",
    "\n",
    "We go use Python library wey dem dey call [RAKE](https://github.com/aneesha/RAKE) for keyword extraction. First, make we install dis library in case e never dey: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install nlp_rake"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Di main wok sabi dey available from `Rake` object, we fit customize am wit some parameters. For our case, we go set di minimum length of keyword to 5 characters, minimum frequency of keyword for di document to 3, and maximum number of words wey fit dey for keyword - to 2. Abeg feel free to play wit oda values and see di result.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nlp_rake\n",
    "extractor = nlp_rake.Rake(max_words=2,min_freq=3,min_chars=5)\n",
    "res = extractor.apply(text)\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "We don get list of terms plus di level of importance wey dey join dem. As you fit see, di most important disciplines, like machine learning and big data, dey for di top part of di list.\n",
    "\n",
    "## Step 4: Visualize di Result\n",
    "\n",
    "People fit understand di data beta wen na for visual form. So e dey make sense to visualize di data make we fit gather insight. We fit use `matplotlib` library for Python to plot simple distribution of di keywords plus their relevance:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def plot(pair_list):\n",
    "    k,v = zip(*pair_list)\n",
    "    plt.bar(range(len(k)),v)\n",
    "    plt.xticks(range(len(k)),k,rotation='vertical')\n",
    "    plt.show()\n",
    "\n",
    "plot(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bot, beta waya dey to show word frequencies - na **Word Cloud**. We go need install anoda library to fit plot word cloud from our keyword list.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!{sys.executable} -m pip install wordcloud"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`WordCloud` object dey responsible for to accept either original text, or pre-computed list of words with dia frequencies, and e go return image, we fit use `matplotlib` show:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from wordcloud import WordCloud\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "wc = WordCloud(background_color='white',width=800,height=600)\n",
    "plt.figure(figsize=(15,7))\n",
    "plt.imshow(wc.generate_from_frequencies({ k:v for k,v in res }))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We fit also chook di original text inside `WordCloud` - make we see if we fit get di same result:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15,7))\n",
    "plt.imshow(wc.generate(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wc.generate(text).to_file('images/ds_wordcloud.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You fit see sey dat word cloud don dey more impressive now, but e still get plenti noise (like unrelated words like `Retrieved on`). Plus, we get less keywords wey get two words, like *data scientist*, or *computer science*. Na becos RAKE algorithm dey do better work to select beta keywords from text. Dis example show how data pre-processing and cleaning sabi important, because if picture clear for end, e go make us fit take better decisions.\n",
    "\n",
    "For dis exercise we don go through simple process to extract some sense from Wikipedia text, for form of keywords and word cloud. Dis example easy well, but e show all normal steps wey data scientist go take when e dey work with data, start from data acquisition, reach visualization.\n",
    "\n",
    "For our course, we go talk about all dis steps well well. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n\n<!-- CO-OP TRANSLATOR DISCLAIMER START -->\n**Disclaimer**:\nDis document don translate wit AI translation service [Co-op Translator](https://github.com/Azure/co-op-translator). Even tho we dey try make am correct, abeg make you know say automated translation fit get errors or mistakes. Di original document for dia own language na im be di correct source. For important info, make person wey sabi human translation do am. We no go responsible for any misunderstanding or wrong understanding wey fit happen because of dis translation.\n<!-- CO-OP TRANSLATOR DISCLAIMER END -->\n"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "c28e7b6bf4e5b397b8288a85bf0a94ea8d3585ce2b01919feb195678ec71581b"
  },
  "kernelspec": {
   "display_name": "Python 3.8.11 64-bit ('base': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}