msb-public
/
Data-Science-For-Beginners
mirror of https://github.com/microsoft/Data-Science-For-Beginners.git
1
0
Code Issues Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'copilot/fix-relevant-content-extraction'
${ noResults }
Data-Science-For-Beginners/translations/sw/2-Working-With-Data/08-data-preparation/notebook.ipynb

4241 lines
122 KiB
Raw Permalink Blame History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "rQ8UhzFpgRra"
},
"source": [
"# Maandalizi ya Data\n",
"\n",
"[Chanzo cha Notebook asili kutoka *Data Science: Introduction to Machine Learning for Data Science Python and Machine Learning Studio na Lee Stott*](https://github.com/leestott/intro-Datascience/blob/master/Course%20Materials/4-Cleaning_and_Manipulating-Reference.ipynb)\n",
"\n",
"## Kuchunguza taarifa za `DataFrame`\n",
"\n",
"> **Lengo la kujifunza:** Mwisho wa sehemu hii ndogo, unapaswa kuwa na uelewa wa jinsi ya kupata taarifa za jumla kuhusu data iliyohifadhiwa kwenye pandas DataFrames.\n",
"\n",
"Baada ya kupakia data yako kwenye pandas, kuna uwezekano mkubwa kuwa itakuwa katika `DataFrame`. Hata hivyo, ikiwa seti ya data katika `DataFrame` yako ina safu 60,000 na nguzo 400, utaanzaje kupata hisia ya unachofanya kazi nacho? Kwa bahati nzuri, pandas inatoa zana rahisi za kuangalia haraka taarifa za jumla kuhusu `DataFrame` pamoja na safu chache za mwanzo na za mwisho.\n",
"\n",
"Ili kuchunguza utendaji huu, tutapakia maktaba ya Python scikit-learn na kutumia seti ya data maarufu ambayo kila mwanasayansi wa data ameiona mara nyingi: seti ya data ya *Iris* ya mwana-bailojia wa Uingereza Ronald Fisher iliyotumika katika karatasi yake ya mwaka 1936 \"Matumizi ya vipimo vingi katika matatizo ya taxonomia\":\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true,
"id": "hB1RofhdgRrp",
"trusted": false
},
"outputs": [],
"source": [
"import pandas as pd\n",
"from sklearn.datasets import load_iris\n",
"\n",
"iris = load_iris()\n",
"iris_df = pd.DataFrame(data=iris['data'], columns=iris['feature_names'])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AGA0A_Y8hMdz"
},
"source": [
"### `DataFrame.shape`\n",
"Tumepakia Dataset ya Iris kwenye kigezo `iris_df`. Kabla ya kuanza kuchambua data, itakuwa muhimu kujua idadi ya vipengele tulivyo navyo na ukubwa wa jumla wa dataset. Ni muhimu kuangalia kiasi cha data tunachoshughulikia.\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "LOe5jQohhulf",
"outputId": "fb0577ac-3b4a-4623-cb41-20e1b264b3e9"
},
"outputs": [
{
"data": {
"text/plain": [
"(150, 4)"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris_df.shape"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "smE7AGzOhxk2"
},
"source": [
"Kwa hivyo, tunashughulika na safu 150 na nguzo 4 za data. Kila safu inawakilisha kipengele kimoja cha data, na kila nguzo inawakilisha sifa moja inayohusiana na fremu ya data. Kwa hivyo kimsingi, kuna vipengele 150 vya data vyenye sifa 4 kila moja.\n",
"\n",
"`shape` hapa ni sifa ya fremu ya data na si kazi, ndiyo sababu haimalizwi na jozi ya mabano.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d3AZKs0PinGP"
},
"source": [
"### `DataFrame.columns`\n",
"Sasa tuingie kwenye safu 4 za data. Kila moja inawakilisha nini hasa? Sifa ya `columns` itatupa majina ya safu katika dataframe.\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "YPGh_ziji-CY",
"outputId": "74e7a43a-77cc-4c80-da56-7f50767c37a0"
},
"outputs": [
{
"data": {
"text/plain": [
"Index(['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)',\n",
" 'petal width (cm)'],\n",
" dtype='object')"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris_df.columns"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TsobcU_VjCC_"
},
"source": [
"Kama tunavyoona, kuna safu nne (4). Sifa ya `columns` inatuambia majina ya safu na kimsingi hakuna kingine. Sifa hii inakuwa muhimu tunapotaka kutambua vipengele ambavyo seti ya data ina.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2UTlvkjmgRrs"
},
"source": [
"### `DataFrame.info`\n",
"Kiasi cha data (kinachotolewa na sifa ya `shape`) na majina ya vipengele au safu (vinavyotolewa na sifa ya `columns`) vinatupa taarifa fulani kuhusu seti ya data. Sasa, tungependa kuchunguza zaidi seti ya data. Kazi ya `DataFrame.info()` ni muhimu sana kwa hili.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "dHHRyG0_gRrt",
"outputId": "d8fb0c40-4f18-4e19-da48-c8db77d1d3a5",
"trusted": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 150 entries, 0 to 149\n",
"Data columns (total 4 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 sepal length (cm) 150 non-null float64\n",
" 1 sepal width (cm) 150 non-null float64\n",
" 2 petal length (cm) 150 non-null float64\n",
" 3 petal width (cm) 150 non-null float64\n",
"dtypes: float64(4)\n",
"memory usage: 4.8 KB\n"
]
}
],
"source": [
"iris_df.info()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1XgVMpvigRru"
},
"source": [
"Kutoka hapa, tunaweza kufanya uchunguzi kadhaa:\n",
"1. Aina ya Data ya kila safu: Katika seti hii ya data, data yote imehifadhiwa kama nambari za desimali za 64-bit.\n",
"2. Idadi ya Thamani Zisizo Null: Kushughulikia thamani za null ni hatua muhimu katika maandalizi ya data. Hili litatatuliwa baadaye kwenye daftari.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "IYlyxbpWFEF4"
},
"source": [
"### DataFrame.describe()\n",
"Tuseme tuna data nyingi za namba kwenye seti yetu ya data. Mahesabu ya takwimu za upande mmoja kama wastani, mediani, robo n.k. yanaweza kufanywa kwa kila safu moja moja. Kazi ya `DataFrame.describe()` inatupatia muhtasari wa takwimu za safu za namba kwenye seti ya data.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 297
},
"id": "tWV-CMstFIRA",
"outputId": "4fc49941-bc13-4b0c-a412-cb39e7d3f289"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sepal length (cm)</th>\n",
" <th>sepal width (cm)</th>\n",
" <th>petal length (cm)</th>\n",
" <th>petal width (cm)</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>count</th>\n",
" <td>150.000000</td>\n",
" <td>150.000000</td>\n",
" <td>150.000000</td>\n",
" <td>150.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>mean</th>\n",
" <td>5.843333</td>\n",
" <td>3.057333</td>\n",
" <td>3.758000</td>\n",
" <td>1.199333</td>\n",
" </tr>\n",
" <tr>\n",
" <th>std</th>\n",
" <td>0.828066</td>\n",
" <td>0.435866</td>\n",
" <td>1.765298</td>\n",
" <td>0.762238</td>\n",
" </tr>\n",
" <tr>\n",
" <th>min</th>\n",
" <td>4.300000</td>\n",
" <td>2.000000</td>\n",
" <td>1.000000</td>\n",
" <td>0.100000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25%</th>\n",
" <td>5.100000</td>\n",
" <td>2.800000</td>\n",
" <td>1.600000</td>\n",
" <td>0.300000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>50%</th>\n",
" <td>5.800000</td>\n",
" <td>3.000000</td>\n",
" <td>4.350000</td>\n",
" <td>1.300000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>75%</th>\n",
" <td>6.400000</td>\n",
" <td>3.300000</td>\n",
" <td>5.100000</td>\n",
" <td>1.800000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>max</th>\n",
" <td>7.900000</td>\n",
" <td>4.400000</td>\n",
" <td>6.900000</td>\n",
" <td>2.500000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sepal length (cm) sepal width (cm) petal length (cm) petal width (cm)\n",
"count 150.000000 150.000000 150.000000 150.000000\n",
"mean 5.843333 3.057333 3.758000 1.199333\n",
"std 0.828066 0.435866 1.765298 0.762238\n",
"min 4.300000 2.000000 1.000000 0.100000\n",
"25% 5.100000 2.800000 1.600000 0.300000\n",
"50% 5.800000 3.000000 4.350000 1.300000\n",
"75% 6.400000 3.300000 5.100000 1.800000\n",
"max 7.900000 4.400000 6.900000 2.500000"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris_df.describe()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zjjtW5hPGMuM"
},
"source": [
"Matokeo hapo juu yanaonyesha jumla ya idadi ya alama za data, wastani, upotofu wa kawaida, kiwango cha chini, robo ya chini (25%), mediani (50%), robo ya juu (75%) na thamani ya juu ya kila safu.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-lviAu99gRrv"
},
"source": [
"### `DataFrame.head`\n",
"Kwa kutumia kazi na sifa zote zilizotajwa hapo juu, tumepata muhtasari wa juu wa seti ya data. Tunajua idadi ya pointi za data zilizopo, idadi ya vipengele vilivyopo, aina ya data ya kila kipengele, na idadi ya thamani zisizo tupu kwa kila kipengele.\n",
"\n",
"Sasa ni wakati wa kuangalia data yenyewe. Hebu tuone jinsi safu chache za mwanzo (pointi chache za mwanzo za data) za `DataFrame` yetu zinavyoonekana:\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "DZMJZh0OgRrw",
"outputId": "d9393ee5-c106-4797-f815-218f17160e00",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sepal length (cm)</th>\n",
" <th>sepal width (cm)</th>\n",
" <th>petal length (cm)</th>\n",
" <th>petal width (cm)</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>5.1</td>\n",
" <td>3.5</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>4.9</td>\n",
" <td>3.0</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>4.7</td>\n",
" <td>3.2</td>\n",
" <td>1.3</td>\n",
" <td>0.2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>4.6</td>\n",
" <td>3.1</td>\n",
" <td>1.5</td>\n",
" <td>0.2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>5.0</td>\n",
" <td>3.6</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sepal length (cm) sepal width (cm) petal length (cm) petal width (cm)\n",
"0 5.1 3.5 1.4 0.2\n",
"1 4.9 3.0 1.4 0.2\n",
"2 4.7 3.2 1.3 0.2\n",
"3 4.6 3.1 1.5 0.2\n",
"4 5.0 3.6 1.4 0.2"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris_df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EBHEimZuEFQK"
},
"source": [
"Kama matokeo hapa, tunaweza kuona maingizo matano (5) ya seti ya data. Tukitazama kwenye faharasa upande wa kushoto, tunagundua kwamba hizi ni safu tano za kwanza.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oj7GkrTdgRry"
},
"source": [
"### Zoezi:\n",
"\n",
"Kutoka kwa mfano uliotolewa hapo juu, ni wazi kwamba, kwa chaguo-msingi, `DataFrame.head` inarudisha safu tano za kwanza za `DataFrame`. Katika seli ya msimbo hapa chini, unaweza kugundua njia ya kuonyesha zaidi ya safu tano?\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": true,
"id": "EKRmRFFegRrz",
"trusted": false
},
"outputs": [],
"source": [
"# Hint: Consult the documentation by using iris_df.head?"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BJ_cpZqNgRr1"
},
"source": [
"### `DataFrame.tail`\n",
"Njia nyingine ya kuangalia data inaweza kuwa kutoka mwisho (badala ya mwanzo). Kinyume cha `DataFrame.head` ni `DataFrame.tail`, ambayo inarudisha safu tano za mwisho za `DataFrame`:\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 0
},
"id": "heanjfGWgRr2",
"outputId": "6ae09a21-fe09-4110-b0d7-1a1fbf34d7f3",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sepal length (cm)</th>\n",
" <th>sepal width (cm)</th>\n",
" <th>petal length (cm)</th>\n",
" <th>petal width (cm)</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>145</th>\n",
" <td>6.7</td>\n",
" <td>3.0</td>\n",
" <td>5.2</td>\n",
" <td>2.3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>146</th>\n",
" <td>6.3</td>\n",
" <td>2.5</td>\n",
" <td>5.0</td>\n",
" <td>1.9</td>\n",
" </tr>\n",
" <tr>\n",
" <th>147</th>\n",
" <td>6.5</td>\n",
" <td>3.0</td>\n",
" <td>5.2</td>\n",
" <td>2.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>148</th>\n",
" <td>6.2</td>\n",
" <td>3.4</td>\n",
" <td>5.4</td>\n",
" <td>2.3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>149</th>\n",
" <td>5.9</td>\n",
" <td>3.0</td>\n",
" <td>5.1</td>\n",
" <td>1.8</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sepal length (cm) sepal width (cm) petal length (cm) petal width (cm)\n",
"145 6.7 3.0 5.2 2.3\n",
"146 6.3 2.5 5.0 1.9\n",
"147 6.5 3.0 5.2 2.0\n",
"148 6.2 3.4 5.4 2.3\n",
"149 5.9 3.0 5.1 1.8"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris_df.tail()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "31kBWfyLgRr3"
},
"source": [
"Kwa vitendo, ni muhimu kuwa na uwezo wa kuchunguza kwa urahisi safu chache za mwanzo au safu chache za mwisho za `DataFrame`, hasa unapokuwa unatafuta thamani zisizo za kawaida katika seti za data zilizo na mpangilio.\n",
"\n",
"Kazi zote na sifa zilizoonyeshwa hapo juu kwa msaada wa mifano ya msimbo, zinatusaidia kupata mwonekano na hisia ya data.\n",
"\n",
"> **Mafunzo:** Hata kwa kuangalia tu metadata kuhusu taarifa ndani ya DataFrame au thamani chache za mwanzo na mwisho, unaweza kupata wazo la haraka kuhusu ukubwa, umbo, na maudhui ya data unayoshughulikia.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TvurZyLSDxq_"
},
"source": [
"### Data Inayokosekana\n",
"Hebu tuangalie data inayokosekana. Data inayokosekana hutokea pale ambapo hakuna thamani iliyohifadhiwa katika baadhi ya safu.\n",
"\n",
"Hebu tuchukue mfano: tuseme mtu anajali sana kuhusu uzito wake na hatoi taarifa ya uzito wake kwenye dodoso. Basi, thamani ya uzito kwa mtu huyo itakosekana.\n",
"\n",
"Mara nyingi, katika seti za data za ulimwengu halisi, thamani zinazokosekana hutokea.\n",
"\n",
"**Jinsi Pandas Inavyoshughulikia Data Inayokosekana**\n",
"\n",
"Pandas hushughulikia thamani zinazokosekana kwa njia mbili. Ya kwanza umeiona katika sehemu za awali: `NaN`, au Not a Number. Hii ni thamani maalum ambayo ni sehemu ya maelezo ya IEEE floating-point na hutumika tu kuonyesha thamani za nambari zinazokosekana.\n",
"\n",
"Kwa thamani zinazokosekana ambazo si za nambari za desimali, pandas hutumia kitu cha Python `None`. Ingawa inaweza kuonekana kuwa ni jambo linalochanganya kwamba utakutana na aina mbili tofauti za thamani zinazosema kimsingi jambo lile lile, kuna sababu za kimaandishi za programu kwa chaguo hili la muundo, na kwa vitendo, njia hii inaiwezesha pandas kutoa suluhisho bora kwa hali nyingi. Pamoja na hayo, `None` na `NaN` zote zina vizuizi ambavyo unapaswa kuzingatia kuhusu jinsi zinavyoweza kutumika.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lOHqUlZFgRr5"
},
"source": [
"### `None`: data isiyokuwepo isiyo ya float\n",
"Kwa sababu `None` inatoka Python, haiwezi kutumika katika arrays za NumPy na pandas ambazo hazina aina ya data `'object'`. Kumbuka, arrays za NumPy (na miundo ya data katika pandas) zinaweza kuwa na aina moja tu ya data. Hii ndiyo inawapa nguvu kubwa kwa kazi za data na hesabu za kiwango kikubwa, lakini pia inapunguza uwezo wao wa kubadilika. Arrays kama hizi zinapaswa kubadilishwa hadi \"kiwango cha chini cha kawaida,\" aina ya data ambayo itajumuisha kila kitu kilichomo kwenye array. Wakati `None` ipo kwenye array, inamaanisha unafanya kazi na vitu vya Python.\n",
"\n",
"Ili kuona hili likifanya kazi, fikiria array ifuatayo (angalia `dtype` yake):\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "QIoNdY4ngRr7",
"outputId": "92779f18-62f4-4a03-eca2-e9a101604336",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"array([2, None, 6, 8], dtype=object)"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import numpy as np\n",
"\n",
"example1 = np.array([2, None, 6, 8])\n",
"example1"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pdlgPNbhgRr7"
},
"source": [
"Uhalisia wa aina za data zilizopandishwa daraja huleta athari mbili. Kwanza, operesheni zitafanyika katika kiwango cha msimbo wa Python unaotafsiriwa badala ya msimbo wa NumPy uliosimbwa. Kimsingi, hii inamaanisha kuwa operesheni yoyote inayohusisha `Series` au `DataFrames` zenye `None` ndani yake itakuwa polepole. Ingawa huenda usione athari hii ya utendaji, kwa seti kubwa za data inaweza kuwa tatizo.\n",
"\n",
"Athari ya pili inatokana na ya kwanza. Kwa sababu `None` kimsingi huirudisha `Series` au `DataFrame` katika ulimwengu wa Python ya kawaida, kutumia hesabu za NumPy/pandas kama `sum()` au `min()` kwenye arrays zenye thamani ya ``None`` kwa kawaida kutasababisha kosa:\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 292
},
"id": "gWbx-KB9gRr8",
"outputId": "ecba710a-22ec-41d5-a39c-11f67e645b50",
"trusted": false
},
"outputs": [
{
"ename": "TypeError",
"evalue": "ignored",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-10-ce9901ad18bd>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mexample1\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m/usr/local/lib/python3.7/dist-packages/numpy/core/_methods.py\u001b[0m in \u001b[0;36m_sum\u001b[0;34m(a, axis, dtype, out, keepdims, initial, where)\u001b[0m\n\u001b[1;32m 45\u001b[0m def _sum(a, axis=None, dtype=None, out=None, keepdims=False,\n\u001b[1;32m 46\u001b[0m initial=_NoValue, where=True):\n\u001b[0;32m---> 47\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mumr_sum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkeepdims\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minitial\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwhere\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 48\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 49\u001b[0m def _prod(a, axis=None, dtype=None, out=None, keepdims=False,\n",
"\u001b[0;31mTypeError\u001b[0m: unsupported operand type(s) for +: 'int' and 'NoneType'"
]
}
],
"source": [
"example1.sum()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LcEwO8UogRr9"
},
"source": [
"**Mambo ya msingi**: Kuongeza (na operesheni nyingine) kati ya nambari nzima na thamani za `None` haina maana, jambo ambalo linaweza kupunguza kile unachoweza kufanya na seti za data zinazozijumuisha.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pWvVHvETgRr9"
},
"source": [
"### `NaN`: thamani za float zilizokosekana\n",
"\n",
"Tofauti na `None`, NumPy (na hivyo pia pandas) inasaidia `NaN` kwa ajili ya operesheni zake za haraka, za vektori, na ufuncs. Habari mbaya ni kwamba hesabu yoyote inayofanywa kwenye `NaN` daima hutoa `NaN`. Kwa mfano:\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "rcFYfMG9gRr9",
"outputId": "699e81b7-5c11-4b46-df1d-06071768690f",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"nan"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.nan + 1"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "BW3zQD2-gRr-",
"outputId": "4525b6c4-495d-4f7b-a979-efce1dae9bd0",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"nan"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.nan * 0"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fU5IPRcCgRr-"
},
"source": [
"Habari njema: mkusanyiko unaoendeshwa kwenye safu zenye `NaN` ndani yake hauleti makosa. Habari mbaya: matokeo si ya msaada kwa kiwango sawa:\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "LCInVgSSgRr_",
"outputId": "fa06495a-0930-4867-87c5-6023031ea8b5",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"(nan, nan, nan)"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example2 = np.array([2, np.nan, 6, 8]) \n",
"example2.sum(), example2.min(), example2.max()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nhlnNJT7gRr_"
},
"source": [
"### Mazoezi:\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": true,
"id": "yan3QRaOgRr_",
"trusted": false
},
"outputs": [],
"source": [
"# What happens if you add np.nan and None together?\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_iDvIRC8gRsA"
},
"source": [
"Kumbuka: `NaN` ni kwa ajili ya thamani za nambari za desimali zinazokosekana tu; hakuna sawa na `NaN` kwa nambari kamili, maandishi, au Boolean.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kj6EKdsAgRsA"
},
"source": [
"### `NaN` na `None`: Thamani za sifuri katika pandas\n",
"\n",
"Ingawa `NaN` na `None` zinaweza kuonyesha tabia tofauti kidogo, pandas imeundwa kushughulikia zote kwa njia inayofanana. Ili kuelewa tunachomaanisha, fikiria `Series` ya nambari za mzima:\n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Nji-KGdNgRsA",
"outputId": "36aa14d2-8efa-4bfd-c0ed-682991288822",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"0 1\n",
"1 2\n",
"2 3\n",
"dtype: int64"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"int_series = pd.Series([1, 2, 3], dtype=int)\n",
"int_series"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WklCzqb8gRsB"
},
"source": [
"### Mazoezi:\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": true,
"id": "Cy-gqX5-gRsB",
"trusted": false
},
"outputs": [],
"source": [
"# Now set an element of int_series equal to None.\n",
"# How does that element show up in the Series?\n",
"# What is the dtype of the Series?\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WjMQwltNgRsB"
},
"source": [
"Katika mchakato wa kubadilisha aina za data ili kuanzisha usawa wa data katika `Series` na `DataFrame`s, pandas itabadilisha kwa hiari thamani zilizokosekana kati ya `None` na `NaN`. Kwa sababu ya kipengele hiki cha muundo, inaweza kuwa muhimu kufikiria `None` na `NaN` kama ladha mbili tofauti za \"null\" katika pandas. Kwa kweli, baadhi ya mbinu za msingi ambazo utatumia kushughulikia thamani zilizokosekana katika pandas zinaakisi wazo hili katika majina yao:\n",
"\n",
"- `isnull()`: Hutengeneza maski ya Boolean inayoonyesha thamani zilizokosekana\n",
"- `notnull()`: Kinyume cha `isnull()`\n",
"- `dropna()`: Hurejesha toleo lililochujwa la data\n",
"- `fillna()`: Hurejesha nakala ya data yenye thamani zilizokosekana kujazwa au kuhesabiwa\n",
"\n",
"Hizi ni mbinu muhimu za kuzifahamu na kuzoea, kwa hivyo hebu tuzipitie kila moja kwa undani zaidi.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Yh5ifd9FgRsB"
},
"source": [
"### Kugundua thamani za null\n",
"\n",
"Sasa kwa kuwa tumeelewa umuhimu wa thamani zinazokosekana, tunahitaji kuzitambua kwenye seti yetu ya data kabla ya kuzishughulikia. \n",
"Zote `isnull()` na `notnull()` ni mbinu zako za msingi za kugundua data ya null. Zote zinarudisha maski za Boolean juu ya data yako.\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": true,
"id": "e-vFp5lvgRsC",
"trusted": false
},
"outputs": [],
"source": [
"example3 = pd.Series([0, np.nan, '', None])"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "1XdaJJ7PgRsC",
"outputId": "92fc363a-1874-471f-846d-f4f9ce1f51d0",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"0 False\n",
"1 True\n",
"2 False\n",
"3 True\n",
"dtype: bool"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example3.isnull()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PaSZ0SQygRsC"
},
"source": [
"Angalia kwa makini matokeo. Je, kuna chochote kinachokushangaza? Ingawa `0` ni null ya hesabu, bado ni nambari kamili nzuri na pandas inaitendea hivyo. `''` ni kidogo zaidi ya hila. Ingawa tulitumia katika Sehemu ya 1 kuwakilisha thamani ya mnyororo tupu, bado ni kitu cha mnyororo na si uwakilishi wa null kulingana na pandas.\n",
"\n",
"Sasa, hebu tugeuze hili na tutumie mbinu hizi kwa namna ambayo ni karibu na jinsi utakavyotumia kwa vitendo. Unaweza kutumia vinyago vya Boolean moja kwa moja kama ``Series`` au ``DataFrame`` index, ambayo inaweza kuwa muhimu unapojaribu kufanya kazi na thamani zilizokosekana (au zilizopo) pekee.\n",
"\n",
"Ikiwa tunataka jumla ya idadi ya thamani zilizokosekana, tunaweza kufanya jumla juu ya kinyago kinachozalishwa na mbinu ya `isnull()`.\n"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "JCcQVoPkHDUv",
"outputId": "001daa72-54f8-4bd5-842a-4df627a79d4d"
},
"outputs": [
{
"data": {
"text/plain": [
"2"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example3.isnull().sum()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PlBqEo3mgRsC"
},
"source": [
"### Mazoezi:\n"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"collapsed": true,
"id": "ggDVf5uygRsD",
"trusted": false
},
"outputs": [],
"source": [
"# Try running example3[example3.notnull()].\n",
"# Before you do so, what do you expect to see?\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "D_jWN7mHgRsD"
},
"source": [
"**Ujumbe muhimu**: Njia zote za `isnull()` na `notnull()` hutoa matokeo yanayofanana unapotumia katika DataFrames: zinaonyesha matokeo na faharasa ya matokeo hayo, ambayo yatakusaidia sana unaposhughulika na data yako.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BvnoojWsgRr4"
},
"source": [
"### Kushughulikia Data Inayokosekana\n",
"\n",
"> **Lengo la kujifunza:** Mwisho wa sehemu hii ndogo, unapaswa kujua jinsi na wakati wa kubadilisha au kuondoa thamani za null kutoka kwa DataFrames.\n",
"\n",
"Mifano ya Kujifunza Mashine haiwezi kushughulikia data inayokosekana yenyewe. Kwa hivyo, kabla ya kupitisha data kwenye mfano, tunahitaji kushughulikia thamani hizi zinazokosekana.\n",
"\n",
"Jinsi data inayokosekana inavyoshughulikiwa huleta maamuzi yenye athari ndogo, inaweza kuathiri uchambuzi wako wa mwisho na matokeo halisi ya ulimwengu.\n",
"\n",
"Kuna njia mbili kuu za kushughulikia data inayokosekana:\n",
"\n",
"1. Kuondoa safu inayojumuisha thamani inayokosekana\n",
"2. Kubadilisha thamani inayokosekana na thamani nyingine\n",
"\n",
"Tutajadili njia hizi zote mbili na faida na hasara zake kwa undani.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3VaYC1TvgRsD"
},
"source": [
"### Kuondoa Thamani Zisizo na Maana\n",
"\n",
"Kiasi cha data tunachopitisha kwa modeli yetu kina athari ya moja kwa moja kwenye utendaji wake. Kuondoa thamani zisizo na maana (null values) kunamaanisha tunapunguza idadi ya vipengele vya data, na hivyo kupunguza ukubwa wa seti ya data. Kwa hivyo, inashauriwa kuondoa safu zenye thamani zisizo na maana pale ambapo seti ya data ni kubwa sana.\n",
"\n",
"Mfano mwingine unaweza kuwa kwamba safu fulani au safu wima ina idadi kubwa ya thamani zisizo na maana. Katika hali hiyo, zinaweza kuondolewa kwa sababu hazitachangia sana kwenye uchambuzi wetu kwani data nyingi zinakosekana kwa safu/safu wima hiyo.\n",
"\n",
"Zaidi ya kutambua thamani zisizo na maana, pandas inatoa njia rahisi ya kuondoa thamani zisizo na maana kutoka kwa `Series` na `DataFrame`s. Ili kuona hili likifanya kazi, hebu turudi kwenye `example3`. Kazi ya `DataFrame.dropna()` husaidia kuondoa safu zenye thamani zisizo na maana.\n"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "7uIvS097gRsD",
"outputId": "c13fc117-4ca1-4145-a0aa-42ac89e6e218",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"0 0\n",
"2 \n",
"dtype: object"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example3 = example3.dropna()\n",
"example3"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hil2cr64gRsD"
},
"source": [
"Kumbuka kwamba hii inapaswa kuonekana kama matokeo yako kutoka `example3[example3.notnull()]`. Tofauti hapa ni kwamba, badala ya kuorodhesha tu thamani zilizofichwa, `dropna` imeondoa zile thamani zilizokosekana kutoka kwa `Series` `example3`.\n",
"\n",
"Kwa sababu DataFrames zina vipimo viwili, zinatoa chaguo zaidi za kuondoa data.\n"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "an-l74sPgRsE",
"outputId": "340876a0-63ad-40f6-bd54-6240cdae50ab",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>6.0</td>\n",
" <td>9</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 1.0 NaN 7\n",
"1 2.0 5.0 8\n",
"2 NaN 6.0 9"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4 = pd.DataFrame([[1, np.nan, 7], \n",
" [2, 5, 8], \n",
" [np.nan, 6, 9]])\n",
"example4"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "66wwdHZrgRsE"
},
"source": [
"(Je, uliona kwamba pandas ilibadilisha aina ya safu mbili kuwa nambari za desimali ili kuzingatia `NaN`?)\n",
"\n",
"Huwezi kuondoa thamani moja kutoka kwa `DataFrame`, kwa hivyo lazima uondoe safu nzima au safu wima. Kulingana na unachofanya, unaweza kutaka kufanya moja au nyingine, na hivyo pandas inakupa chaguo kwa zote mbili. Kwa sababu katika sayansi ya data, safu wima kwa kawaida huwakilisha vigezo na safu huwakilisha uchunguzi, kuna uwezekano mkubwa wa kuondoa safu za data; mpangilio wa chaguo-msingi wa `dropna()` ni kuondoa safu zote zinazojumuisha thamani zozote tupu:\n"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 80
},
"id": "jAVU24RXgRsE",
"outputId": "0b5e5aee-7187-4d3f-b583-a44136ae5f80",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"1 2.0 5.0 8"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4.dropna()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TrQRBuTDgRsE"
},
"source": [
"Ikiwa ni lazima, unaweza kuondoa thamani za NA kutoka kwa safu. Tumia `axis=1` kufanya hivyo:\n"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "GrBhxu9GgRsE",
"outputId": "ff4001f3-2e61-4509-d60e-0093d1068437",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>8</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>9</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 2\n",
"0 7\n",
"1 8\n",
"2 9"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4.dropna(axis='columns')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KWXiKTfMgRsF"
},
"source": [
"Kumbuka kwamba hii inaweza kuondoa data nyingi ambayo unaweza kutaka kuhifadhi, hasa katika seti ndogo za data. Je, itakuwaje ikiwa unataka tu kuondoa safu au nguzo ambazo zina thamani kadhaa au hata zote zikiwa tupu? Unaweza kubainisha mipangilio hiyo katika `dropna` kwa kutumia vigezo vya `how` na `thresh`.\n",
"\n",
"Kwa chaguo-msingi, `how='any'` (ikiwa ungependa kuangalia mwenyewe au kuona vigezo vingine ambavyo njia hii ina, endesha `example4.dropna?` katika seli ya msimbo). Vinginevyo, unaweza kubainisha `how='all'` ili kuondoa tu safu au nguzo ambazo zina thamani zote zikiwa tupu. Hebu tuongeze mfano wetu wa `DataFrame` ili kuona hili likifanya kazi katika zoezi lijalo.\n"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "Bcf_JWTsgRsF",
"outputId": "72e0b1b8-52fa-4923-98ce-b6fbed6e44b1",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>7</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>6.0</td>\n",
" <td>9</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"0 1.0 NaN 7 NaN\n",
"1 2.0 5.0 8 NaN\n",
"2 NaN 6.0 9 NaN"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4[3] = np.nan\n",
"example4"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pNZer7q9JPNC"
},
"source": [
"> Mambo muhimu ya kuzingatia: \n",
"1. Kuondoa thamani za null ni wazo zuri tu ikiwa seti ya data ni kubwa vya kutosha. \n",
"2. Safu nzima au nguzo zinaweza kuondolewa ikiwa zina data nyingi iliyokosekana. \n",
"3. Njia ya `DataFrame.dropna(axis=)` husaidia kuondoa thamani za null. Kipengele cha `axis` kinaonyesha ikiwa safu zinapaswa kuondolewa au nguzo. \n",
"4. Kipengele cha `how` pia kinaweza kutumika. Kwa chaguo-msingi kimewekwa kuwa `any`. Kwa hivyo, kinaondoa tu safu/nguzo ambazo zina thamani yoyote ya null. Kinaweza kuwekwa kuwa `all` ili kubainisha kwamba tutaondoa tu safu/nguzo ambapo thamani zote ni null. \n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oXXSfQFHgRsF"
},
"source": [
"### Mazoezi:\n"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"collapsed": true,
"id": "ExUwQRxpgRsF",
"trusted": false
},
"outputs": [],
"source": [
"# How might you go about dropping just column 3?\n",
"# Hint: remember that you will need to supply both the axis parameter and the how parameter.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "38kwAihWgRsG"
},
"source": [
"Kigezo cha `thresh` kinakupa udhibiti wa kina zaidi: unaweka idadi ya thamani *zisizo-null* ambazo safu au safu wima inahitaji kuwa nazo ili kuhifadhiwa:\n"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 80
},
"id": "M9dCNMaagRsG",
"outputId": "8093713a-54d2-4e54-c73f-4eea315cb6f2",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"1 2.0 5.0 8 NaN"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4.dropna(axis='rows', thresh=3)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fmSFnzZegRsG"
},
"source": [
"Hapa, safu ya kwanza na ya mwisho zimeondolewa, kwa sababu zina maadili mawili tu yasiyo tupu.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mCcxLGyUgRsG"
},
"source": [
"### Kujaza Thamani Zilizokosekana\n",
"\n",
"Wakati mwingine ina mantiki kujaza thamani zilizokosekana na zile ambazo zinaweza kuwa sahihi. Kuna mbinu kadhaa za kujaza thamani za null. Ya kwanza ni kutumia Ujuzi wa Eneo (maarifa ya somo ambalo dataset inahusu) ili kukadiria kwa namna fulani thamani zilizokosekana.\n",
"\n",
"Unaweza kutumia `isnull` kufanya hili moja kwa moja, lakini hilo linaweza kuwa kazi ngumu, hasa ikiwa una thamani nyingi za kujaza. Kwa sababu hili ni jukumu la kawaida katika sayansi ya data, pandas inatoa `fillna`, ambayo inarudisha nakala ya `Series` au `DataFrame` na thamani zilizokosekana kubadilishwa na moja unayochagua. Hebu tuunde mfano mwingine wa `Series` ili kuona jinsi hii inavyofanya kazi kwa vitendo.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CE8S7louLezV"
},
"source": [
"### Data ya Kategoria (Isiyo ya Nambari)\n",
"Kwanza, tuzingatie data isiyo ya nambari. Katika seti za data, tunayo safu zenye data ya kategoria. Mfano: Jinsia, Kweli au Siyo Kweli, n.k.\n",
"\n",
"Katika hali nyingi, tunabadilisha thamani zilizokosekana kwa kutumia `mode` ya safu husika. Kwa mfano, tuseme tuna alama 100 za data ambapo 90 zimesema Kweli, 8 zimesema Siyo Kweli, na 2 hazijajazwa. Basi, tunaweza kujaza zile 2 kwa Kweli, tukizingatia safu nzima.\n",
"\n",
"Tena, hapa tunaweza kutumia maarifa ya uwanja husika. Hebu tuchukue mfano wa kujaza kwa kutumia mode.\n"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "MY5faq4yLdpQ",
"outputId": "19ab472e-1eed-4de8-f8a7-db2a3af3cb1a"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" <td>None</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>5</td>\n",
" <td>6</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>7</td>\n",
" <td>8</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>9</td>\n",
" <td>10</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 1 2 True\n",
"1 3 4 None\n",
"2 5 6 False\n",
"3 7 8 True\n",
"4 9 10 True"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_mode = pd.DataFrame([[1,2,\"True\"],\n",
" [3,4,None],\n",
" [5,6,\"False\"],\n",
" [7,8,\"True\"],\n",
" [9,10,\"True\"]])\n",
"\n",
"fill_with_mode"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MLAoMQOfNPlA"
},
"source": [
"Sasa, hebu kwanza tupate modi kabla ya kujaza thamani ya `None` na modi.\n"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "WKy-9Y2tN5jv",
"outputId": "8da9fa16-e08c-447e-dea1-d4b1db2feebf"
},
"outputs": [
{
"data": {
"text/plain": [
"True 3\n",
"False 1\n",
"Name: 2, dtype: int64"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_mode[2].value_counts()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6iNz_zG_OKrx"
},
"source": [
"Kwa hivyo, tutabadilisha None na True\n"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"id": "TxPKteRvNPOs"
},
"outputs": [],
"source": [
"fill_with_mode[2].fillna('True',inplace=True)"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "tvas7c9_OPWE",
"outputId": "ec3c8e44-d644-475e-9e22-c65101965850"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>5</td>\n",
" <td>6</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>7</td>\n",
" <td>8</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>9</td>\n",
" <td>10</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 1 2 True\n",
"1 3 4 True\n",
"2 5 6 False\n",
"3 7 8 True\n",
"4 9 10 True"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_mode"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SktitLxxOR16"
},
"source": [
"Kama tunavyoweza kuona, thamani ya null imebadilishwa. Bila shaka, tungeweza kuandika chochote badala ya `'True'` na kingesubstituliwa.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "heYe1I0dOmQ_"
},
"source": [
"### Data ya Nambari\n",
"Sasa, tukija kwenye data ya nambari. Hapa, tuna njia mbili za kawaida za kubadilisha thamani zilizopotea:\n",
"\n",
"1. Badilisha na Median ya safu\n",
"2. Badilisha na Mean ya safu\n",
"\n",
"Tunabadilisha na Median, endapo data ina mwelekeo na ina outliers. Hii ni kwa sababu median haiguswi sana na outliers.\n",
"\n",
"Wakati data imesawazishwa, tunaweza kutumia mean, kwani katika hali hiyo, mean na median zitakuwa karibu sana.\n",
"\n",
"Kwanza, hebu tuchukue safu ambayo imegawanyika kawaida na tujaze thamani iliyopotea kwa mean ya safu.\n"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "09HM_2feOj5Y",
"outputId": "7e309013-9acb-411c-9b06-4de795bbeeff"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>-2.0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>-1.0</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1.0</td>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2.0</td>\n",
" <td>8</td>\n",
" <td>9</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 -2.0 0 1\n",
"1 -1.0 2 3\n",
"2 NaN 4 5\n",
"3 1.0 6 7\n",
"4 2.0 8 9"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_mean = pd.DataFrame([[-2,0,1],\n",
" [-1,2,3],\n",
" [np.nan,4,5],\n",
" [1,6,7],\n",
" [2,8,9]])\n",
"\n",
"fill_with_mean"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ka7-wNfzSxbx"
},
"source": [
"Wastani wa safu ni\n"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "XYtYEf5BSxFL",
"outputId": "68a78d18-f0e5-4a9a-a959-2c3676a57c70"
},
"outputs": [
{
"data": {
"text/plain": [
"0.0"
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.mean(fill_with_mean[0])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oBSRGxKRS39K"
},
"source": [
"Kujaza kwa wastani\n"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "FzncQLmuS5jh",
"outputId": "00f74fff-01f4-4024-c261-796f50f01d2e"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>-2.0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>-1.0</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>0.0</td>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1.0</td>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2.0</td>\n",
" <td>8</td>\n",
" <td>9</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 -2.0 0 1\n",
"1 -1.0 2 3\n",
"2 0.0 4 5\n",
"3 1.0 6 7\n",
"4 2.0 8 9"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_mean[0].fillna(np.mean(fill_with_mean[0]),inplace=True)\n",
"fill_with_mean"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CwpVFCrPTC5z"
},
"source": [
"Kama tunavyoona, thamani iliyokosekana imebadilishwa na wastani wake.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jIvF13a1i00Z"
},
"source": [
"Sasa hebu tujaribu dataframe nyingine, na wakati huu tutabadilisha thamani za None na wastani wa safu.\n"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "DA59Bqo3jBYZ",
"outputId": "85dae6ec-7394-4c36-fda0-e04769ec4a32"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>-2</td>\n",
" <td>0.0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>-1</td>\n",
" <td>2.0</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>0</td>\n",
" <td>NaN</td>\n",
" <td>5</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1</td>\n",
" <td>6.0</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2</td>\n",
" <td>8.0</td>\n",
" <td>9</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 -2 0.0 1\n",
"1 -1 2.0 3\n",
"2 0 NaN 5\n",
"3 1 6.0 7\n",
"4 2 8.0 9"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_median = pd.DataFrame([[-2,0,1],\n",
" [-1,2,3],\n",
" [0,np.nan,5],\n",
" [1,6,7],\n",
" [2,8,9]])\n",
"\n",
"fill_with_median"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mM1GpXYmjHnc"
},
"source": [
"Kati ya safu ya pili ni\n"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "uiDy5v3xjHHX",
"outputId": "564b6b74-2004-4486-90d4-b39330a64b88"
},
"outputs": [
{
"data": {
"text/plain": [
"4.0"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_median[1].median()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "z9PLF75Jj_1s"
},
"source": [
"Kujaza kwa mediani\n"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "lFKbOxCMkBbg",
"outputId": "a8bd18fb-2765-47d4-e5fe-e965f57ed1f4"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>-2</td>\n",
" <td>0.0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>-1</td>\n",
" <td>2.0</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>0</td>\n",
" <td>4.0</td>\n",
" <td>5</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1</td>\n",
" <td>6.0</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2</td>\n",
" <td>8.0</td>\n",
" <td>9</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 -2 0.0 1\n",
"1 -1 2.0 3\n",
"2 0 4.0 5\n",
"3 1 6.0 7\n",
"4 2 8.0 9"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_median[1].fillna(fill_with_median[1].median(),inplace=True)\n",
"fill_with_median"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8JtQ53GSkKWC"
},
"source": [
"Kama tunavyoona, thamani ya NaN imebadilishwa na wastani wa safu.\n"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "0ybtWLDdgRsG",
"outputId": "b8c238ef-6024-4ee2-be2b-aa1f0fcac61d",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"a 1.0\n",
"b NaN\n",
"c 2.0\n",
"d NaN\n",
"e 3.0\n",
"dtype: float64"
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example5 = pd.Series([1, np.nan, 2, None, 3], index=list('abcde'))\n",
"example5"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yrsigxRggRsH"
},
"source": [
"Unaweza kujaza nafasi zote tupu kwa thamani moja, kama `0`:\n"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "KXMIPsQdgRsH",
"outputId": "aeedfa0a-a421-4c2f-cb0d-183ce8f0c91d",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"a 1.0\n",
"b 0.0\n",
"c 2.0\n",
"d 0.0\n",
"e 3.0\n",
"dtype: float64"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example5.fillna(0)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RRlI5f_hkfKe"
},
"source": [
"> Mambo muhimu ya kuzingatia:\n",
"1. Kujaza thamani zilizokosekana kunapaswa kufanyika pale ambapo kuna data kidogo au kuna mkakati wa kujaza data iliyokosekana.\n",
"2. Maarifa ya uwanja yanaweza kutumika kujaza thamani zilizokosekana kwa kuzikadiria.\n",
"3. Kwa data ya Kategoria, mara nyingi, thamani zilizokosekana hubadilishwa na hali ya kawaida ya safu husika.\n",
"4. Kwa data ya nambari, thamani zilizokosekana kwa kawaida hujazwa kwa wastani (kwa seti za data zilizonormalishwa) au kwa median ya safu.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FI9MmqFJgRsH"
},
"source": [
"### Mazoezi:\n"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {
"collapsed": true,
"id": "af-ezpXdgRsH",
"trusted": false
},
"outputs": [],
"source": [
"# What happens if you try to fill null values with a string, like ''?\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kq3hw1kLgRsI"
},
"source": [
"Unaweza **kujaza mbele** thamani za null, ambayo ni kutumia thamani halali ya mwisho kujaza null:\n"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "vO3BuNrggRsI",
"outputId": "e2bc591b-0b48-4e88-ee65-754f2737c196",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"a 1.0\n",
"b 1.0\n",
"c 2.0\n",
"d 2.0\n",
"e 3.0\n",
"dtype: float64"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example5.fillna(method='ffill')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nDXeYuHzgRsI"
},
"source": [
"Unaweza pia **kujaza nyuma** ili kusambaza thamani halali inayofuata kurudi nyuma kujaza null:\n"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "4M5onHcEgRsI",
"outputId": "8f32b185-40dd-4a9f-bd85-54d6b6a414fe",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"a 1.0\n",
"b 2.0\n",
"c 2.0\n",
"d 3.0\n",
"e 3.0\n",
"dtype: float64"
]
},
"execution_count": 42,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example5.fillna(method='bfill')"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"id": "MbBzTom5gRsI"
},
"source": [
"Kama unavyoweza kudhani, hii inafanya kazi vivyo hivyo na DataFrames, lakini pia unaweza kubainisha `axis` ambayo itajazwa thamani za null:\n"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "aRpIvo4ZgRsI",
"outputId": "905a980a-a808-4eca-d0ba-224bd7d85955",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>7</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>6.0</td>\n",
" <td>9</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"0 1.0 NaN 7 NaN\n",
"1 2.0 5.0 8 NaN\n",
"2 NaN 6.0 9 NaN"
]
},
"execution_count": 43,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "VM1qtACAgRsI",
"outputId": "71f2ad28-9b4e-4ff4-f5c3-e731eb489ade",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>7.0</td>\n",
" <td>7.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8.0</td>\n",
" <td>8.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>6.0</td>\n",
" <td>9.0</td>\n",
" <td>9.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"0 1.0 1.0 7.0 7.0\n",
"1 2.0 5.0 8.0 8.0\n",
"2 NaN 6.0 9.0 9.0"
]
},
"execution_count": 44,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4.fillna(method='ffill', axis=1)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZeMc-I1EgRsI"
},
"source": [
"Kumbuka kwamba wakati thamani ya awali haipatikani kwa kujaza mbele, thamani ya null inabaki.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eeAoOU0RgRsJ"
},
"source": [
"### Mazoezi:\n"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {
"collapsed": true,
"id": "e8S-CjW8gRsJ",
"trusted": false
},
"outputs": [],
"source": [
"# What output does example4.fillna(method='bfill', axis=1) produce?\n",
"# What about example4.fillna(method='ffill') or example4.fillna(method='bfill')?\n",
"# Can you think of a longer code snippet to write that can fill all of the null values in example4?\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YHgy0lIrgRsJ"
},
"source": [
"Unaweza kuwa mbunifu kuhusu jinsi unavyotumia `fillna`. Kwa mfano, hebu tuangalie tena `example4`, lakini wakati huu tujaze thamani zilizokosekana kwa wastani wa thamani zote katika `DataFrame`:\n"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "OtYVErEygRsJ",
"outputId": "708b1e67-45ca-44bf-a5ee-8b2de09ece73",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>5.5</td>\n",
" <td>7</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>1.5</td>\n",
" <td>6.0</td>\n",
" <td>9</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"0 1.0 5.5 7 NaN\n",
"1 2.0 5.0 8 NaN\n",
"2 1.5 6.0 9 NaN"
]
},
"execution_count": 46,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4.fillna(example4.mean())"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zpMvCkLSgRsJ"
},
"source": [
"Kumbuka kwamba safu ya 3 bado haina thamani: mwelekeo wa chaguo-msingi ni kujaza thamani kwa kila mstari.\n",
"\n",
"> **Mafunzo:** Kuna njia nyingi za kushughulikia thamani zinazokosekana katika seti zako za data. Mkakati maalum unaotumia (kuondoa, kubadilisha, au hata jinsi unavyobadilisha) unapaswa kuamuliwa na maelezo ya data hiyo. Utapata uelewa bora wa jinsi ya kushughulikia thamani zinazokosekana kadri unavyoshughulikia na kuingiliana na seti za data.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bauDnESIl9FH"
},
"source": [
"### Usimbaji wa Data ya Kategoria\n",
"\n",
"Mifano ya kujifunza kwa mashine hushughulikia tu namba na aina yoyote ya data ya namba. Haiwezi kutofautisha kati ya Ndiyo na Hapana, lakini inaweza kutofautisha kati ya 0 na 1. Kwa hivyo, baada ya kujaza thamani zilizokosekana, tunahitaji kusimba data ya kategoria kwa namna ya namba ili mfano uweze kuelewa.\n",
"\n",
"Usimbaji unaweza kufanywa kwa njia mbili. Tutazijadili hapa chini.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uDq9SxB7mu5i"
},
"source": [
"**KODI YA LEBO**\n",
"\n",
"Kodi ya lebo ni mchakato wa kubadilisha kila kategoria kuwa namba. Kwa mfano, tuseme tuna seti ya data ya abiria wa ndege na kuna safu inayojumuisha daraja lao miongoni mwa haya ['daraja la biashara', 'daraja la uchumi', 'daraja la kwanza']. Ikiwa kodi ya lebo itafanywa kwenye hii, itabadilishwa kuwa [0,1,2]. Hebu tuone mfano kupitia msimbo. Kwa kuwa tutakuwa tunajifunza `scikit-learn` katika daftari zijazo, hatutaitumia hapa.\n"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 235
},
"id": "1vGz7uZyoWHL",
"outputId": "9e252855-d193-4103-a54d-028ea7787b34"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>ID</th>\n",
" <th>class</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>10</td>\n",
" <td>business class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>20</td>\n",
" <td>first class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>30</td>\n",
" <td>economy class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>40</td>\n",
" <td>economy class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>50</td>\n",
" <td>economy class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>60</td>\n",
" <td>business class</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" ID class\n",
"0 10 business class\n",
"1 20 first class\n",
"2 30 economy class\n",
"3 40 economy class\n",
"4 50 economy class\n",
"5 60 business class"
]
},
"execution_count": 47,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"label = pd.DataFrame([\n",
" [10,'business class'],\n",
" [20,'first class'],\n",
" [30, 'economy class'],\n",
" [40, 'economy class'],\n",
" [50, 'economy class'],\n",
" [60, 'business class']\n",
"],columns=['ID','class'])\n",
"label"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "IDHnkwTYov-h"
},
"source": [
"Ili kufanya usimbaji wa lebo kwenye safu ya kwanza, tunapaswa kwanza kuelezea ramani kutoka kila darasa hadi nambari, kabla ya kubadilisha.\n"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 235
},
"id": "ZC5URJG3o1ES",
"outputId": "aab0f1e7-e0f3-4c14-8459-9f9168c85437"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>ID</th>\n",
" <th>class</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>10</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>20</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>30</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>40</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>50</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>60</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" ID class\n",
"0 10 0\n",
"1 20 2\n",
"2 30 1\n",
"3 40 1\n",
"4 50 1\n",
"5 60 0"
]
},
"execution_count": 48,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class_labels = {'business class':0,'economy class':1,'first class':2}\n",
"label['class'] = label['class'].replace(class_labels)\n",
"label"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ftnF-TyapOPt"
},
"source": [
"Kama tunavyoona, matokeo yanalingana na tulivyotarajia yatokee. Kwa hivyo, tunatumia label encoding lini? Label encoding hutumika katika mojawapo au zote za hali zifuatazo:\n",
"1. Wakati idadi ya makundi ni kubwa\n",
"2. Wakati makundi yako katika mpangilio.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eQPAPVwsqWT7"
},
"source": [
"**ONE HOT ENCODING**\n",
"\n",
"Aina nyingine ya usimbaji ni One Hot Encoding. Katika aina hii ya usimbaji, kila kategoria ya safu huongezwa kama safu tofauti, na kila data itapewa 0 au 1 kulingana na kama inajumuisha kategoria hiyo. Kwa hivyo, ikiwa kuna kategoria n tofauti, safu n zitaongezwa kwenye dataframe.\n",
"\n",
"Kwa mfano, hebu tuchukue mfano ule ule wa darasa la ndege. Kategoria zilikuwa: ['business class', 'economy class', 'first class']. Kwa hivyo, tukifanya one hot encoding, safu tatu zifuatazo zitaongezwa kwenye dataset: ['class_business class', 'class_economy class', 'class_first class'].\n"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 235
},
"id": "ZM0eVh0ArKUL",
"outputId": "83238a76-b3a5-418d-c0b6-605b02b6891b"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>ID</th>\n",
" <th>class</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>10</td>\n",
" <td>business class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>20</td>\n",
" <td>first class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>30</td>\n",
" <td>economy class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>40</td>\n",
" <td>economy class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>50</td>\n",
" <td>economy class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>60</td>\n",
" <td>business class</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" ID class\n",
"0 10 business class\n",
"1 20 first class\n",
"2 30 economy class\n",
"3 40 economy class\n",
"4 50 economy class\n",
"5 60 business class"
]
},
"execution_count": 49,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"one_hot = pd.DataFrame([\n",
" [10,'business class'],\n",
" [20,'first class'],\n",
" [30, 'economy class'],\n",
" [40, 'economy class'],\n",
" [50, 'economy class'],\n",
" [60, 'business class']\n",
"],columns=['ID','class'])\n",
"one_hot"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aVnZ7paDrWmb"
},
"source": [
"Tufanye usimbaji wa moja kwa moja kwenye safu ya kwanza\n"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {
"id": "RUPxf7egrYKr"
},
"outputs": [],
"source": [
"one_hot_data = pd.get_dummies(one_hot,columns=['class'])"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 235
},
"id": "TM37pHsFr4ge",
"outputId": "7be15f53-79b2-447a-979c-822658339a9e"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>ID</th>\n",
" <th>class_business class</th>\n",
" <th>class_economy class</th>\n",
" <th>class_first class</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>10</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>20</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>30</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>40</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>50</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>60</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" ID class_business class class_economy class class_first class\n",
"0 10 1 0 0\n",
"1 20 0 0 1\n",
"2 30 0 1 0\n",
"3 40 0 1 0\n",
"4 50 0 1 0\n",
"5 60 1 0 0"
]
},
"execution_count": 51,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"one_hot_data"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_zXRLOjXujdA"
},
"source": [
"Kila safu iliyosimbwa kwa moto mmoja ina 0 au 1, ambayo inaonyesha ikiwa jamii hiyo ipo kwa data hiyo.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bDnC4NQOu0qr"
},
"source": [
"Tunatumia one hot encoding lini? One hot encoding hutumika katika mojawapo au zote za hali zifuatazo:\n",
"\n",
"1. Wakati idadi ya makundi na ukubwa wa seti ya data ni ndogo.\n",
"2. Wakati makundi hayafuati mpangilio maalum.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XnUmci_4uvyu"
},
"source": [
"> Mambo Muhimu:\n",
"1. Usimbaji hufanywa kubadilisha data isiyo ya nambari kuwa data ya nambari.\n",
"2. Kuna aina mbili za usimbaji: Usimbaji wa Lebo na Usimbaji wa One Hot, zote zinaweza kufanywa kulingana na mahitaji ya seti ya data.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "K8UXOJYRgRsJ"
},
"source": [
"## Kuondoa data zinazojirudia\n",
"\n",
"> **Lengo la kujifunza:** Kufikia mwisho wa sehemu hii ndogo, unapaswa kuwa na ujuzi wa kutambua na kuondoa thamani zinazojirudia kutoka kwa DataFrames.\n",
"\n",
"Mbali na data inayokosekana, mara nyingi utakutana na data zinazojirudia katika seti za data za ulimwengu halisi. Kwa bahati nzuri, pandas inatoa njia rahisi ya kugundua na kuondoa maingizo yanayojirudia.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qrEG-Wa0gRsJ"
},
"source": [
"### Kutambua nakala: `duplicated`\n",
"\n",
"Unaweza kutambua kwa urahisi thamani zinazojirudia ukitumia njia ya `duplicated` katika pandas, ambayo inarudisha mask ya Boolean inayoonyesha ikiwa kiingilio katika `DataFrame` ni nakala ya kilichotangulia. Hebu tuunde mfano mwingine wa `DataFrame` ili kuona hili likifanya kazi.\n"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "ZLu6FEnZgRsJ",
"outputId": "376512d1-d842-4db1-aea3-71052aeeecaf",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>letters</th>\n",
" <th>numbers</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>B</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>A</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>B</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>B</td>\n",
" <td>3</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" letters numbers\n",
"0 A 1\n",
"1 B 2\n",
"2 A 1\n",
"3 B 3\n",
"4 B 3"
]
},
"execution_count": 52,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example6 = pd.DataFrame({'letters': ['A','B'] * 2 + ['B'],\n",
" 'numbers': [1, 2, 1, 3, 3]})\n",
"example6"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "cIduB5oBgRsK",
"outputId": "3da27b3d-4d69-4e1d-bb52-0af21bae87f2",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"0 False\n",
"1 False\n",
"2 True\n",
"3 False\n",
"4 True\n",
"dtype: bool"
]
},
"execution_count": 53,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example6.duplicated()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0eDRJD4SgRsK"
},
"source": [
"### Kuondoa nakala: `drop_duplicates`\n",
"`drop_duplicates` inarudisha tu nakala ya data ambayo thamani zote za `duplicated` ni `False`:\n"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "w_YPpqIqgRsK",
"outputId": "ac66bd2f-8671-4744-87f5-8b8d96553dea",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>letters</th>\n",
" <th>numbers</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>B</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>B</td>\n",
" <td>3</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" letters numbers\n",
"0 A 1\n",
"1 B 2\n",
"3 B 3"
]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example6.drop_duplicates()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "69AqoCZAgRsK"
},
"source": [
"Zote `duplicated` na `drop_duplicates` kwa chaguo-msingi huzingatia safu zote lakini unaweza kubainisha kwamba zichunguze tu sehemu ya safu katika `DataFrame` yako:\n"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 111
},
"id": "BILjDs67gRsK",
"outputId": "ef6dcc08-db8b-4352-c44e-5aa9e2bec0d3",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>letters</th>\n",
" <th>numbers</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>B</td>\n",
" <td>2</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" letters numbers\n",
"0 A 1\n",
"1 B 2"
]
},
"execution_count": 55,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example6.drop_duplicates(['letters'])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GvX4og1EgRsL"
},
"source": [
"> **Muktasari:** Kuondoa data zinazojirudia ni sehemu muhimu ya karibu kila mradi wa sayansi ya data. Data zinazojirudia zinaweza kubadilisha matokeo ya uchambuzi wako na kukupa matokeo yasiyo sahihi!\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Ukaguzi wa Ubora wa Data ya Ulimwengu Halisi\n",
"\n",
"> **Lengo la kujifunza:** Kufikia mwisho wa sehemu hii, unapaswa kuwa na ujuzi wa kugundua na kurekebisha changamoto za kawaida za ubora wa data ya ulimwengu halisi, ikiwa ni pamoja na thamani za kategoria zisizo thabiti, thamani za nambari zisizo za kawaida (outliers), na entiti zinazojirudia zenye tofauti.\n",
"\n",
"Ingawa thamani zinazokosekana na nakala halisi ni changamoto za kawaida, seti za data za ulimwengu halisi mara nyingi zina matatizo ya hila zaidi:\n",
"\n",
"1. **Thamani za kategoria zisizo thabiti**: Kategoria moja kuandikwa kwa njia tofauti (mfano, \"USA\", \"U.S.A\", \"United States\")\n",
"2. **Thamani za nambari zisizo za kawaida**: Outliers kali zinazoashiria makosa ya kuingiza data (mfano, umri = 999)\n",
"3. **Safu zinazokaribia kujirudia**: Rekodi zinazowakilisha entiti moja lakini zikiwa na tofauti ndogo\n",
"\n",
"Hebu tuchunguze mbinu za kugundua na kushughulikia changamoto hizi.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Kuunda Seti ya Data \"Chafu\" ya Mfano\n",
"\n",
"Kwanza, hebu tuunde seti ya data ya mfano inayojumuisha aina za changamoto tunazokutana nazo mara kwa mara katika data halisi:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"\n",
"# Create a sample dataset with quality issues\n",
"dirty_data = pd.DataFrame({\n",
" 'customer_id': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12],\n",
" 'name': ['John Smith', 'Jane Doe', 'John Smith', 'Bob Johnson', \n",
" 'Alice Williams', 'Charlie Brown', 'John Smith', 'Eva Martinez',\n",
" 'Bob Johnson', 'Diana Prince', 'Frank Castle', 'Alice Williams'],\n",
" 'age': [25, 32, 25, 45, 28, 199, 25, 31, 45, 27, -5, 28],\n",
" 'country': ['USA', 'UK', 'U.S.A', 'Canada', 'USA', 'United Kingdom',\n",
" 'United States', 'Mexico', 'canada', 'USA', 'UK', 'usa'],\n",
" 'purchase_amount': [100.50, 250.00, 105.00, 320.00, 180.00, 90.00,\n",
" 102.00, 275.00, 325.00, 195.00, 410.00, 185.00]\n",
"})\n",
"\n",
"print(\"Sample 'Dirty' Dataset:\")\n",
"print(dirty_data)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1. Kugundua Thamani Zisizoendana za Kategoria\n",
"\n",
"Angalia safu ya `country` ina uwakilishi tofauti kwa nchi zile zile. Hebu tutambue kutokubaliana huku:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Check unique values in the country column\n",
"print(\"Unique country values:\")\n",
"print(dirty_data['country'].unique())\n",
"print(f\"\\nTotal unique values: {dirty_data['country'].nunique()}\")\n",
"\n",
"# Count occurrences of each variation\n",
"print(\"\\nValue counts:\")\n",
"print(dirty_data['country'].value_counts())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Kuweka Thamani za Kategoria katika Muundo wa Kawaida\n",
"\n",
"Tunaweza kuunda ramani ili kuweka thamani hizi katika muundo wa kawaida. Njia rahisi ni kubadilisha kuwa herufi ndogo na kuunda kamusi ya ramani:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a standardization mapping\n",
"country_mapping = {\n",
" 'usa': 'USA',\n",
" 'u.s.a': 'USA',\n",
" 'united states': 'USA',\n",
" 'uk': 'UK',\n",
" 'united kingdom': 'UK',\n",
" 'canada': 'Canada',\n",
" 'mexico': 'Mexico'\n",
"}\n",
"\n",
"# Standardize the country column\n",
"dirty_data['country_clean'] = dirty_data['country'].str.lower().map(country_mapping)\n",
"\n",
"print(\"Before standardization:\")\n",
"print(dirty_data['country'].value_counts())\n",
"print(\"\\nAfter standardization:\")\n",
"print(dirty_data[['country_clean']].value_counts())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Njia Mbadala: Kutumia Ulinganishaji wa Maneno Usio Sahihi**\n",
"\n",
"Kwa hali ngumu zaidi, tunaweza kutumia ulinganishaji wa maneno usio sahihi kwa kutumia maktaba ya `rapidfuzz` ili kugundua maneno yanayofanana kiotomatiki:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"try:\n",
" from rapidfuzz import process, fuzz\n",
"except ImportError:\n",
" print(\"rapidfuzz is not installed. Please install it with 'pip install rapidfuzz' to use fuzzy matching.\")\n",
" process = None\n",
" fuzz = None\n",
"\n",
"# Get unique countries\n",
"unique_countries = dirty_data['country'].unique()\n",
"\n",
"# For each country, find similar matches\n",
"if process is not None and fuzz is not None:\n",
" print(\"Finding similar country names (similarity > 70%):\")\n",
" for country in unique_countries:\n",
" matches = process.extract(country, unique_countries, scorer=fuzz.ratio, limit=3)\n",
" # Filter matches with similarity > 70 and not identical\n",
" similar = [m for m in matches if m[1] > 70 and m[0] != country]\n",
" if similar:\n",
" print(f\"\\n'{country}' is similar to:\")\n",
" for match, score, _ in similar:\n",
" print(f\" - '{match}' (similarity: {score}%)\")\n",
"else:\n",
" print(\"Skipping fuzzy matching because rapidfuzz is not available.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2. Kugundua Thamani za Nambari Zisizo za Kawaida (Outliers)\n",
"\n",
"Tukiangalia safu ya `age`, tunaona baadhi ya thamani zinazotia shaka kama 199 na -5. Hebu tutumie mbinu za takwimu kugundua outliers hizi.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Display basic statistics\n",
"print(\"Age column statistics:\")\n",
"print(dirty_data['age'].describe())\n",
"\n",
"# Identify impossible values using domain knowledge\n",
"print(\"\\nRows with impossible age values (< 0 or > 120):\")\n",
"impossible_ages = dirty_data[(dirty_data['age'] < 0) | (dirty_data['age'] > 120)]\n",
"print(impossible_ages[['customer_id', 'name', 'age']])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Kutumia Njia ya IQR (Interquartile Range)\n",
"\n",
"Njia ya IQR ni mbinu thabiti ya takwimu kwa kugundua thamani zisizo za kawaida ambayo haiguswi sana na thamani za kupita kiasi:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Calculate IQR for age (excluding impossible values)\n",
"valid_ages = dirty_data[(dirty_data['age'] >= 0) & (dirty_data['age'] <= 120)]['age']\n",
"\n",
"Q1 = valid_ages.quantile(0.25)\n",
"Q3 = valid_ages.quantile(0.75)\n",
"IQR = Q3 - Q1\n",
"\n",
"# Define outlier bounds\n",
"lower_bound = Q1 - 1.5 * IQR\n",
"upper_bound = Q3 + 1.5 * IQR\n",
"\n",
"print(f\"IQR-based outlier bounds for age: [{lower_bound:.2f}, {upper_bound:.2f}]\")\n",
"\n",
"# Identify outliers\n",
"age_outliers = dirty_data[(dirty_data['age'] < lower_bound) | (dirty_data['age'] > upper_bound)]\n",
"print(f\"\\nRows with age outliers:\")\n",
"print(age_outliers[['customer_id', 'name', 'age']])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Kutumia Njia ya Z-Score\n",
"\n",
"Njia ya Z-score inatambua vipimo vya nje kwa kuzingatia tofauti za kawaida kutoka kwa wastani:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"try:\n",
" from scipy import stats\n",
"except ImportError:\n",
" print(\"scipy is required for Z-score calculation. Please install it with 'pip install scipy' and rerun this cell.\")\n",
"else:\n",
" # Calculate Z-scores for age, handling NaN values\n",
" age_nonan = dirty_data['age'].dropna()\n",
" zscores = np.abs(stats.zscore(age_nonan))\n",
" dirty_data['age_zscore'] = np.nan\n",
" dirty_data.loc[age_nonan.index, 'age_zscore'] = zscores\n",
"\n",
" # Typically, Z-score > 3 indicates an outlier\n",
" print(\"Rows with age Z-score > 3:\")\n",
" zscore_outliers = dirty_data[dirty_data['age_zscore'] > 3]\n",
" print(zscore_outliers[['customer_id', 'name', 'age', 'age_zscore']])\n",
"\n",
" # Clean up the temporary column\n",
" dirty_data = dirty_data.drop('age_zscore', axis=1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Kushughulikia Thamani Zisizo za Kawaida\n",
"\n",
"Baada ya kugunduliwa, thamani zisizo za kawaida zinaweza kushughulikiwa kwa njia kadhaa:\n",
"1. **Kuondoa**: Futa safu zenye thamani zisizo za kawaida (ikiwa ni makosa)\n",
"2. **Kuweka Kikomo**: Badilisha na thamani za mipaka\n",
"3. **Badilisha na NaN**: Zitendee kama data iliyokosekana na tumia mbinu za kujaza\n",
"4. **Kuhifadhi**: Ikiwa ni thamani halali za kipekee\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a cleaned version by replacing impossible ages with NaN\n",
"dirty_data['age_clean'] = dirty_data['age'].apply(\n",
" lambda x: np.nan if (x < 0 or x > 120) else x\n",
")\n",
"\n",
"print(\"Age column before and after cleaning:\")\n",
"print(dirty_data[['customer_id', 'name', 'age', 'age_clean']])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 3. Kugundua Safu Zinazokaribiana Kuwa Nakala\n",
"\n",
"Angalia kwamba seti yetu ya data ina maingizo mengi kwa \"John Smith\" yenye thamani zinazotofautiana kidogo. Hebu tutambue nakala zinazoweza kutokea kulingana na mfanano wa majina.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# First, let's look at exact name matches (ignoring extra whitespace)\n",
"dirty_data['name_normalized'] = dirty_data['name'].str.strip().str.lower()\n",
"\n",
"print(\"Checking for duplicate names:\")\n",
"duplicate_names = dirty_data[dirty_data.duplicated(['name_normalized'], keep=False)]\n",
"print(duplicate_names.sort_values('name_normalized')[['customer_id', 'name', 'age', 'country']])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Kupata Nakala Karibu Zinazofanana kwa Kutumia Ulinganishaji wa Kifuzzy\n",
"\n",
"Kwa kugundua nakala zinazofanana kwa njia ya hali ya juu zaidi, tunaweza kutumia ulinganishaji wa kifuzzy ili kupata majina yanayofanana:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"try:\n",
" from rapidfuzz import process, fuzz\n",
"\n",
" # Function to find potential duplicates\n",
" def find_near_duplicates(df, column, threshold=90):\n",
" \"\"\"\n",
" Find near-duplicate entries in a column using fuzzy matching.\n",
" \n",
" Parameters:\n",
" - df: DataFrame\n",
" - column: Column name to check for duplicates\n",
" - threshold: Similarity threshold (0-100)\n",
" \n",
" Returns: List of potential duplicate groups\n",
" \"\"\"\n",
" values = df[column].unique()\n",
" duplicate_groups = []\n",
" checked = set()\n",
" \n",
" for value in values:\n",
" if value in checked:\n",
" continue\n",
" \n",
" # Find similar values\n",
" matches = process.extract(value, values, scorer=fuzz.ratio, limit=len(values))\n",
" similar = [m[0] for m in matches if m[1] >= threshold]\n",
" \n",
" if len(similar) > 1:\n",
" duplicate_groups.append(similar)\n",
" checked.update(similar)\n",
" \n",
" return duplicate_groups\n",
"\n",
" # Find near-duplicate names\n",
" duplicate_groups = find_near_duplicates(dirty_data, 'name', threshold=90)\n",
"\n",
" print(\"Potential duplicate groups:\")\n",
" for i, group in enumerate(duplicate_groups, 1):\n",
" print(f\"\\nGroup {i}:\")\n",
" for name in group:\n",
" matching_rows = dirty_data[dirty_data['name'] == name]\n",
" print(f\" '{name}': {len(matching_rows)} occurrence(s)\")\n",
" for _, row in matching_rows.iterrows():\n",
" print(f\" - Customer {row['customer_id']}: age={row['age']}, country={row['country']}\")\n",
"except ImportError:\n",
" print(\"rapidfuzz is not installed. Skipping fuzzy matching for near-duplicates.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Kushughulikia Nakala Zilizojirudia\n",
"\n",
"Baada ya kuzitambua, unahitaji kuamua jinsi ya kushughulikia nakala zilizojirudia:\n",
"1. **Hifadhi tukio la kwanza**: Tumia `drop_duplicates(keep='first')`\n",
"2. **Hifadhi tukio la mwisho**: Tumia `drop_duplicates(keep='last')`\n",
"3. **Kusanya taarifa**: Changanya taarifa kutoka kwa safu zilizojirudia\n",
"4. **Ukaguzi wa mikono**: Weka alama kwa ukaguzi wa binadamu\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Example: Remove duplicates based on normalized name, keeping first occurrence\n",
"cleaned_data = dirty_data.drop_duplicates(subset=['name_normalized'], keep='first')\n",
"\n",
"print(f\"Original dataset: {len(dirty_data)} rows\")\n",
"print(f\"After removing name duplicates: {len(cleaned_data)} rows\")\n",
"print(f\"Removed: {len(dirty_data) - len(cleaned_data)} duplicate rows\")\n",
"\n",
"print(\"\\nCleaned dataset:\")\n",
"print(cleaned_data[['customer_id', 'name', 'age', 'country_clean']])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Muhtasari: Mchakato Kamili wa Kusafisha Data\n",
"\n",
"Hebu tuunganishe yote pamoja katika mchakato wa kusafisha data kwa ukamilifu:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def clean_dataset(df):\n",
" \"\"\"\n",
" Comprehensive data cleaning function.\n",
" \"\"\"\n",
" # Create a copy to avoid modifying the original\n",
" cleaned = df.copy()\n",
" \n",
" # 1. Standardize categorical values (country)\n",
" country_mapping = {\n",
" 'usa': 'USA', 'u.s.a': 'USA', 'united states': 'USA',\n",
" 'uk': 'UK', 'united kingdom': 'UK',\n",
" 'canada': 'Canada', 'mexico': 'Mexico'\n",
" }\n",
" cleaned['country'] = cleaned['country'].str.lower().map(country_mapping)\n",
" \n",
" # 2. Clean abnormal age values\n",
" cleaned['age'] = cleaned['age'].apply(\n",
" lambda x: np.nan if (x < 0 or x > 120) else x\n",
" )\n",
" \n",
" # 3. Remove near-duplicate names (normalize whitespace)\n",
" cleaned['name'] = cleaned['name'].str.strip()\n",
" cleaned = cleaned.drop_duplicates(subset=['name'], keep='first')\n",
" \n",
" return cleaned\n",
"\n",
"# Apply the cleaning pipeline\n",
"final_cleaned_data = clean_dataset(dirty_data)\n",
"\n",
"print(\"Before cleaning:\")\n",
"print(f\" Rows: {len(dirty_data)}\")\n",
"print(f\" Unique countries: {dirty_data['country'].nunique()}\")\n",
"print(f\" Invalid ages: {((dirty_data['age'] < 0) | (dirty_data['age'] > 120)).sum()}\")\n",
"\n",
"print(\"\\nAfter cleaning:\")\n",
"print(f\" Rows: {len(final_cleaned_data)}\")\n",
"print(f\" Unique countries: {final_cleaned_data['country'].nunique()}\")\n",
"print(f\" Invalid ages: {((final_cleaned_data['age'] < 0) | (final_cleaned_data['age'] > 120)).sum()}\")\n",
"\n",
"print(\"\\nCleaned dataset:\")\n",
"print(final_cleaned_data[['customer_id', 'name', 'age', 'country', 'purchase_amount']])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 🎯 Zoezi la Changamoto\n",
"\n",
"Sasa ni zamu yako! Hapa chini kuna safu mpya ya data yenye masuala kadhaa ya ubora. Je, unaweza:\n",
"\n",
"1. Kutambua masuala yote katika safu hii\n",
"2. Kuandika msimbo wa kusafisha kila tatizo\n",
"3. Kuongeza safu iliyosafishwa kwenye seti ya data\n",
"\n",
"Hii hapa data yenye matatizo:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# New problematic row\n",
"new_row = pd.DataFrame({\n",
" 'customer_id': [13],\n",
" 'name': [' Diana Prince '], # Extra whitespace\n",
" 'age': [250], # Impossible age\n",
" 'country': ['U.S.A.'], # Inconsistent format\n",
" 'purchase_amount': [150.00]\n",
"})\n",
"\n",
"print(\"New row to clean:\")\n",
"print(new_row)\n",
"\n",
"# TODO: Your code here to clean this row\n",
"# Hints:\n",
"# 1. Strip whitespace from the name\n",
"# 2. Check if the name is a duplicate (Diana Prince already exists)\n",
"# 3. Handle the impossible age value\n",
"# 4. Standardize the country name\n",
"\n",
"# Example solution (uncomment and modify as needed):\n",
"# new_row_cleaned = new_row.copy()\n",
"# new_row_cleaned['name'] = new_row_cleaned['name'].str.strip()\n",
"# new_row_cleaned['age'] = np.nan # Invalid age\n",
"# new_row_cleaned['country'] = 'USA' # Standardized\n",
"# print(\"\\nCleaned row:\")\n",
"# print(new_row_cleaned)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Mambo Muhimu\n",
"\n",
"1. **Makundi yasiyo thabiti** ni jambo la kawaida katika data halisi. Daima hakikisha unakagua thamani za kipekee na kuzistandardisha kwa kutumia ramani au kulinganisha kwa ukaribu.\n",
"\n",
"2. **Vipimo vya mbali (Outliers)** vinaweza kuathiri uchambuzi wako kwa kiasi kikubwa. Tumia maarifa ya uwanja pamoja na mbinu za takwimu (IQR, Z-score) ili kuvitambua.\n",
"\n",
"3. **Karibu nakala rudufu (Near-duplicates)** ni ngumu zaidi kutambua kuliko nakala rudufu halisi. Fikiria kutumia kulinganisha kwa ukaribu na kusawazisha data (kuandika kwa herufi ndogo, kuondoa nafasi zisizo za lazima) ili kuvitambua.\n",
"\n",
"4. **Usafi wa data ni wa hatua kwa hatua**. Huenda ukahitaji kutumia mbinu mbalimbali na kupitia matokeo kabla ya kukamilisha seti yako ya data iliyosafishwa.\n",
"\n",
"5. **Rekodi maamuzi yako**. Weka kumbukumbu ya hatua za usafi ulizotumia na sababu zake, kwani hili ni muhimu kwa kurudia na uwazi.\n",
"\n",
"> **Mazoea Bora:** Daima weka nakala ya data yako \"chafu\" ya awali. Usibadilishe faili zako za chanzo - tengeneza matoleo yaliyosafishwa na majina ya faili yaliyo wazi kama `data_cleaned.csv`.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n---\n\n**Kanusho**: \nHati hii imetafsiriwa kwa kutumia huduma ya tafsiri ya AI [Co-op Translator](https://github.com/Azure/co-op-translator). Ingawa tunajitahidi kuhakikisha usahihi, tafadhali fahamu kuwa tafsiri za kiotomatiki zinaweza kuwa na makosa au kutokuwa sahihi. Hati ya asili katika lugha yake ya awali inapaswa kuzingatiwa kama chanzo cha mamlaka. Kwa taarifa muhimu, tafsiri ya kitaalamu ya binadamu inapendekezwa. Hatutawajibika kwa kutoelewana au tafsiri zisizo sahihi zinazotokana na matumizi ya tafsiri hii.\n"
]
}
],
"metadata": {
"anaconda-cloud": {},
"colab": {
"name": "notebook.ipynb",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.4"
},
"coopTranslator": {
"original_hash": "6301339d1c9a301b00639c635dc9b731",
"translation_date": "2025-10-03T20:49:12+00:00",
"source_file": "2-Working-With-Data/08-data-preparation/notebook.ipynb",
"language_code": "sw"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Reference in new issue View Git Blame Copy Permalink