msb-public
/
Data-Science-For-Beginners
mirror of https://github.com/microsoft/Data-Science-For-Beginners.git
1
0
Code Issues Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'copilot/fix-serialize-javascript-vulnerability'
${ noResults }
Data-Science-For-Beginners/translations/id/2-Working-With-Data/08-data-preparation/notebook.ipynb

4241 lines
123 KiB
Raw Permalink Blame History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "rQ8UhzFpgRra"
},
"source": [
"# Persiapan Data\n",
"\n",
"[Sumber Notebook asli dari *Data Science: Introduction to Machine Learning for Data Science Python and Machine Learning Studio oleh Lee Stott*](https://github.com/leestott/intro-Datascience/blob/master/Course%20Materials/4-Cleaning_and_Manipulating-Reference.ipynb)\n",
"\n",
"## Mengeksplorasi informasi `DataFrame`\n",
"\n",
"> **Tujuan pembelajaran:** Pada akhir subbagian ini, Anda diharapkan nyaman dalam menemukan informasi umum tentang data yang disimpan dalam pandas DataFrames.\n",
"\n",
"Setelah Anda memuat data ke dalam pandas, kemungkinan besar data tersebut akan berada dalam bentuk `DataFrame`. Namun, jika data dalam `DataFrame` Anda memiliki 60.000 baris dan 400 kolom, bagaimana Anda mulai memahami apa yang sedang Anda kerjakan? Untungnya, pandas menyediakan beberapa alat yang praktis untuk dengan cepat melihat informasi umum tentang sebuah `DataFrame` selain beberapa baris pertama dan terakhir.\n",
"\n",
"Untuk mengeksplorasi fungsi ini, kita akan mengimpor pustaka Python scikit-learn dan menggunakan dataset ikonik yang telah dilihat oleh setiap ilmuwan data ratusan kali: dataset *Iris* milik ahli biologi Inggris Ronald Fisher yang digunakan dalam makalahnya tahun 1936 \"The use of multiple measurements in taxonomic problems\":\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true,
"id": "hB1RofhdgRrp",
"trusted": false
},
"outputs": [],
"source": [
"import pandas as pd\n",
"from sklearn.datasets import load_iris\n",
"\n",
"iris = load_iris()\n",
"iris_df = pd.DataFrame(data=iris['data'], columns=iris['feature_names'])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AGA0A_Y8hMdz"
},
"source": [
"### `DataFrame.shape`\n",
"Kami telah memuat Dataset Iris ke dalam variabel `iris_df`. Sebelum mendalami data, akan sangat berguna untuk mengetahui jumlah data yang kita miliki dan ukuran keseluruhan dataset. Penting untuk melihat volume data yang sedang kita hadapi.\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "LOe5jQohhulf",
"outputId": "fb0577ac-3b4a-4623-cb41-20e1b264b3e9"
},
"outputs": [
{
"data": {
"text/plain": [
"(150, 4)"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris_df.shape"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "smE7AGzOhxk2"
},
"source": [
"Jadi, kita sedang menangani 150 baris dan 4 kolom data. Setiap baris mewakili satu titik data dan setiap kolom mewakili satu fitur yang terkait dengan kerangka data. Jadi pada dasarnya, ada 150 titik data yang masing-masing memiliki 4 fitur.\n",
"\n",
"`shape` di sini adalah atribut dari kerangka data dan bukan sebuah fungsi, itulah sebabnya tidak diakhiri dengan sepasang tanda kurung.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d3AZKs0PinGP"
},
"source": [
"### `DataFrame.columns`\n",
"Sekarang mari kita bahas 4 kolom data. Apa sebenarnya yang masing-masing kolom wakili? Atribut `columns` akan memberikan kita nama-nama kolom dalam dataframe.\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "YPGh_ziji-CY",
"outputId": "74e7a43a-77cc-4c80-da56-7f50767c37a0"
},
"outputs": [
{
"data": {
"text/plain": [
"Index(['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)',\n",
" 'petal width (cm)'],\n",
" dtype='object')"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris_df.columns"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TsobcU_VjCC_"
},
"source": [
"Seperti yang dapat kita lihat, ada empat(4) kolom. Atribut `columns` memberi tahu kita nama kolom dan pada dasarnya tidak ada yang lain. Atribut ini menjadi penting ketika kita ingin mengidentifikasi fitur yang dimiliki oleh sebuah dataset.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2UTlvkjmgRrs"
},
"source": [
"### `DataFrame.info`\n",
"Jumlah data (diberikan oleh atribut `shape`) dan nama fitur atau kolom (diberikan oleh atribut `columns`) memberikan informasi tentang dataset. Sekarang, kita ingin menggali lebih dalam ke dalam dataset. Fungsi `DataFrame.info()` sangat berguna untuk ini.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "dHHRyG0_gRrt",
"outputId": "d8fb0c40-4f18-4e19-da48-c8db77d1d3a5",
"trusted": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 150 entries, 0 to 149\n",
"Data columns (total 4 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 sepal length (cm) 150 non-null float64\n",
" 1 sepal width (cm) 150 non-null float64\n",
" 2 petal length (cm) 150 non-null float64\n",
" 3 petal width (cm) 150 non-null float64\n",
"dtypes: float64(4)\n",
"memory usage: 4.8 KB\n"
]
}
],
"source": [
"iris_df.info()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1XgVMpvigRru"
},
"source": [
"Dari sini, kita dapat membuat beberapa pengamatan:\n",
"1. Tipe Data dari setiap kolom: Dalam dataset ini, semua data disimpan sebagai angka floating-point 64-bit.\n",
"2. Jumlah nilai Non-Null: Menangani nilai null adalah langkah penting dalam persiapan data. Hal ini akan ditangani nanti di notebook.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "IYlyxbpWFEF4"
},
"source": [
"### DataFrame.describe()\n",
"Misalkan kita memiliki banyak data numerik dalam dataset kita. Perhitungan statistik univariat seperti rata-rata, median, kuartil, dll. dapat dilakukan pada masing-masing kolom secara individual. Fungsi `DataFrame.describe()` memberikan ringkasan statistik dari kolom-kolom numerik dalam sebuah dataset.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 297
},
"id": "tWV-CMstFIRA",
"outputId": "4fc49941-bc13-4b0c-a412-cb39e7d3f289"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sepal length (cm)</th>\n",
" <th>sepal width (cm)</th>\n",
" <th>petal length (cm)</th>\n",
" <th>petal width (cm)</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>count</th>\n",
" <td>150.000000</td>\n",
" <td>150.000000</td>\n",
" <td>150.000000</td>\n",
" <td>150.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>mean</th>\n",
" <td>5.843333</td>\n",
" <td>3.057333</td>\n",
" <td>3.758000</td>\n",
" <td>1.199333</td>\n",
" </tr>\n",
" <tr>\n",
" <th>std</th>\n",
" <td>0.828066</td>\n",
" <td>0.435866</td>\n",
" <td>1.765298</td>\n",
" <td>0.762238</td>\n",
" </tr>\n",
" <tr>\n",
" <th>min</th>\n",
" <td>4.300000</td>\n",
" <td>2.000000</td>\n",
" <td>1.000000</td>\n",
" <td>0.100000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25%</th>\n",
" <td>5.100000</td>\n",
" <td>2.800000</td>\n",
" <td>1.600000</td>\n",
" <td>0.300000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>50%</th>\n",
" <td>5.800000</td>\n",
" <td>3.000000</td>\n",
" <td>4.350000</td>\n",
" <td>1.300000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>75%</th>\n",
" <td>6.400000</td>\n",
" <td>3.300000</td>\n",
" <td>5.100000</td>\n",
" <td>1.800000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>max</th>\n",
" <td>7.900000</td>\n",
" <td>4.400000</td>\n",
" <td>6.900000</td>\n",
" <td>2.500000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sepal length (cm) sepal width (cm) petal length (cm) petal width (cm)\n",
"count 150.000000 150.000000 150.000000 150.000000\n",
"mean 5.843333 3.057333 3.758000 1.199333\n",
"std 0.828066 0.435866 1.765298 0.762238\n",
"min 4.300000 2.000000 1.000000 0.100000\n",
"25% 5.100000 2.800000 1.600000 0.300000\n",
"50% 5.800000 3.000000 4.350000 1.300000\n",
"75% 6.400000 3.300000 5.100000 1.800000\n",
"max 7.900000 4.400000 6.900000 2.500000"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris_df.describe()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zjjtW5hPGMuM"
},
"source": [
"Output di atas menunjukkan jumlah total titik data, rata-rata, standar deviasi, nilai minimum, kuartil bawah (25%), median (50%), kuartil atas (75%), dan nilai maksimum dari setiap kolom.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-lviAu99gRrv"
},
"source": [
"### `DataFrame.head`\n",
"Dengan semua fungsi dan atribut di atas, kita telah mendapatkan gambaran umum tentang dataset. Kita tahu berapa banyak data yang ada, berapa banyak fitur yang ada, tipe data dari setiap fitur, dan jumlah nilai non-null untuk setiap fitur.\n",
"\n",
"Sekarang saatnya melihat datanya sendiri. Mari kita lihat seperti apa beberapa baris pertama (beberapa data pertama) dari `DataFrame` kita:\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "DZMJZh0OgRrw",
"outputId": "d9393ee5-c106-4797-f815-218f17160e00",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sepal length (cm)</th>\n",
" <th>sepal width (cm)</th>\n",
" <th>petal length (cm)</th>\n",
" <th>petal width (cm)</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>5.1</td>\n",
" <td>3.5</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>4.9</td>\n",
" <td>3.0</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>4.7</td>\n",
" <td>3.2</td>\n",
" <td>1.3</td>\n",
" <td>0.2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>4.6</td>\n",
" <td>3.1</td>\n",
" <td>1.5</td>\n",
" <td>0.2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>5.0</td>\n",
" <td>3.6</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sepal length (cm) sepal width (cm) petal length (cm) petal width (cm)\n",
"0 5.1 3.5 1.4 0.2\n",
"1 4.9 3.0 1.4 0.2\n",
"2 4.7 3.2 1.3 0.2\n",
"3 4.6 3.1 1.5 0.2\n",
"4 5.0 3.6 1.4 0.2"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris_df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EBHEimZuEFQK"
},
"source": [
"Sebagai output di sini, kita dapat melihat lima (5) entri dari dataset. Jika kita melihat indeks di sebelah kiri, kita menemukan bahwa ini adalah lima baris pertama.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oj7GkrTdgRry"
},
"source": [
"### Latihan:\n",
"\n",
"Dari contoh yang diberikan di atas, jelas bahwa secara default, `DataFrame.head` mengembalikan lima baris pertama dari sebuah `DataFrame`. Pada sel kode di bawah ini, bisakah Anda menemukan cara untuk menampilkan lebih dari lima baris?\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": true,
"id": "EKRmRFFegRrz",
"trusted": false
},
"outputs": [],
"source": [
"# Hint: Consult the documentation by using iris_df.head?"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BJ_cpZqNgRr1"
},
"source": [
"### `DataFrame.tail`\n",
"Cara lain untuk melihat data adalah dari akhir (bukan dari awal). Kebalikan dari `DataFrame.head` adalah `DataFrame.tail`, yang mengembalikan lima baris terakhir dari sebuah `DataFrame`:\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 0
},
"id": "heanjfGWgRr2",
"outputId": "6ae09a21-fe09-4110-b0d7-1a1fbf34d7f3",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sepal length (cm)</th>\n",
" <th>sepal width (cm)</th>\n",
" <th>petal length (cm)</th>\n",
" <th>petal width (cm)</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>145</th>\n",
" <td>6.7</td>\n",
" <td>3.0</td>\n",
" <td>5.2</td>\n",
" <td>2.3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>146</th>\n",
" <td>6.3</td>\n",
" <td>2.5</td>\n",
" <td>5.0</td>\n",
" <td>1.9</td>\n",
" </tr>\n",
" <tr>\n",
" <th>147</th>\n",
" <td>6.5</td>\n",
" <td>3.0</td>\n",
" <td>5.2</td>\n",
" <td>2.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>148</th>\n",
" <td>6.2</td>\n",
" <td>3.4</td>\n",
" <td>5.4</td>\n",
" <td>2.3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>149</th>\n",
" <td>5.9</td>\n",
" <td>3.0</td>\n",
" <td>5.1</td>\n",
" <td>1.8</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sepal length (cm) sepal width (cm) petal length (cm) petal width (cm)\n",
"145 6.7 3.0 5.2 2.3\n",
"146 6.3 2.5 5.0 1.9\n",
"147 6.5 3.0 5.2 2.0\n",
"148 6.2 3.4 5.4 2.3\n",
"149 5.9 3.0 5.1 1.8"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris_df.tail()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "31kBWfyLgRr3"
},
"source": [
"Dalam praktiknya, sangat berguna untuk dapat dengan mudah memeriksa beberapa baris pertama atau beberapa baris terakhir dari sebuah `DataFrame`, terutama ketika Anda mencari nilai outlier dalam dataset yang terurut.\n",
"\n",
"Semua fungsi dan atribut yang ditunjukkan di atas dengan bantuan contoh kode membantu kita mendapatkan gambaran dan pemahaman tentang data.\n",
"\n",
"> **Kesimpulan:** Bahkan hanya dengan melihat metadata tentang informasi dalam sebuah DataFrame atau beberapa nilai pertama dan terakhir di dalamnya, Anda dapat langsung mendapatkan gambaran tentang ukuran, bentuk, dan isi data yang sedang Anda tangani.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TvurZyLSDxq_"
},
"source": [
"### Data yang Hilang\n",
"Mari kita bahas tentang data yang hilang. Data yang hilang terjadi ketika tidak ada nilai yang disimpan di beberapa kolom.\n",
"\n",
"Mari kita ambil contoh: misalnya seseorang sangat memperhatikan berat badannya dan tidak mengisi kolom berat badan dalam sebuah survei. Maka, nilai berat badan untuk orang tersebut akan hilang.\n",
"\n",
"Sebagian besar waktu, dalam dataset dunia nyata, nilai yang hilang sering terjadi.\n",
"\n",
"**Bagaimana Pandas Menangani Data yang Hilang**\n",
"\n",
"Pandas menangani nilai yang hilang dengan dua cara. Cara pertama yang sudah pernah Anda lihat di bagian sebelumnya adalah `NaN`, atau Not a Number. Ini sebenarnya adalah nilai khusus yang merupakan bagian dari spesifikasi floating-point IEEE dan hanya digunakan untuk menunjukkan nilai floating-point yang hilang.\n",
"\n",
"Untuk nilai yang hilang selain tipe float, pandas menggunakan objek Python `None`. Meskipun mungkin terlihat membingungkan bahwa Anda akan menemukan dua jenis nilai yang pada dasarnya menyampaikan hal yang sama, ada alasan programatik yang kuat untuk pilihan desain ini, dan dalam praktiknya, pendekatan ini memungkinkan pandas memberikan kompromi yang baik untuk sebagian besar kasus. Meskipun demikian, baik `None` maupun `NaN` memiliki batasan yang perlu Anda perhatikan terkait bagaimana mereka dapat digunakan.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lOHqUlZFgRr5"
},
"source": [
"### `None`: data hilang non-float\n",
"Karena `None` berasal dari Python, ia tidak dapat digunakan dalam array NumPy dan pandas yang bukan bertipe data `'object'`. Ingat, array NumPy (dan struktur data dalam pandas) hanya dapat berisi satu jenis data. Inilah yang memberikan kekuatan besar untuk pekerjaan data dan komputasi skala besar, tetapi juga membatasi fleksibilitasnya. Array semacam itu harus diubah ke \"penyebut umum terendah,\" yaitu tipe data yang dapat mencakup semua elemen dalam array. Ketika `None` ada dalam array, itu berarti Anda sedang bekerja dengan objek Python.\n",
"\n",
"Untuk melihat ini dalam praktik, perhatikan contoh array berikut (perhatikan `dtype` untuk array tersebut):\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "QIoNdY4ngRr7",
"outputId": "92779f18-62f4-4a03-eca2-e9a101604336",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"array([2, None, 6, 8], dtype=object)"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import numpy as np\n",
"\n",
"example1 = np.array([2, None, 6, 8])\n",
"example1"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pdlgPNbhgRr7"
},
"source": [
"Realitas tipe data yang di-upcast membawa dua efek samping bersamanya. Pertama, operasi akan dilakukan pada tingkat kode Python yang diinterpretasikan daripada kode NumPy yang dikompilasi. Pada dasarnya, ini berarti bahwa setiap operasi yang melibatkan `Series` atau `DataFrames` dengan `None` di dalamnya akan lebih lambat. Meskipun Anda mungkin tidak menyadari penurunan kinerja ini, untuk dataset yang besar, hal ini bisa menjadi masalah.\n",
"\n",
"Efek samping kedua berasal dari yang pertama. Karena `None` pada dasarnya membawa `Series` atau `DataFrame` kembali ke dunia Python biasa, menggunakan agregasi NumPy/pandas seperti `sum()` atau `min()` pada array yang mengandung nilai ``None`` umumnya akan menghasilkan error:\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 292
},
"id": "gWbx-KB9gRr8",
"outputId": "ecba710a-22ec-41d5-a39c-11f67e645b50",
"trusted": false
},
"outputs": [
{
"ename": "TypeError",
"evalue": "ignored",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-10-ce9901ad18bd>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mexample1\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m/usr/local/lib/python3.7/dist-packages/numpy/core/_methods.py\u001b[0m in \u001b[0;36m_sum\u001b[0;34m(a, axis, dtype, out, keepdims, initial, where)\u001b[0m\n\u001b[1;32m 45\u001b[0m def _sum(a, axis=None, dtype=None, out=None, keepdims=False,\n\u001b[1;32m 46\u001b[0m initial=_NoValue, where=True):\n\u001b[0;32m---> 47\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mumr_sum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkeepdims\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minitial\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwhere\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 48\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 49\u001b[0m def _prod(a, axis=None, dtype=None, out=None, keepdims=False,\n",
"\u001b[0;31mTypeError\u001b[0m: unsupported operand type(s) for +: 'int' and 'NoneType'"
]
}
],
"source": [
"example1.sum()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LcEwO8UogRr9"
},
"source": [
"**Inti utama**: Penjumlahan (dan operasi lainnya) antara bilangan bulat dan nilai `None` tidak terdefinisi, yang dapat membatasi apa yang dapat Anda lakukan dengan kumpulan data yang mengandungnya.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pWvVHvETgRr9"
},
"source": [
"### `NaN`: nilai float yang hilang\n",
"\n",
"Berbeda dengan `None`, NumPy (dan oleh karena itu pandas) mendukung `NaN` untuk operasi vektorisasi yang cepat dan ufuncs. Berita buruknya adalah bahwa setiap operasi aritmatika yang dilakukan pada `NaN` selalu menghasilkan `NaN`. Sebagai contoh:\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "rcFYfMG9gRr9",
"outputId": "699e81b7-5c11-4b46-df1d-06071768690f",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"nan"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.nan + 1"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "BW3zQD2-gRr-",
"outputId": "4525b6c4-495d-4f7b-a979-efce1dae9bd0",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"nan"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.nan * 0"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fU5IPRcCgRr-"
},
"source": [
"Berita baiknya: agregasi yang dijalankan pada array dengan `NaN` di dalamnya tidak menghasilkan error. Berita buruknya: hasilnya tidak selalu berguna:\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "LCInVgSSgRr_",
"outputId": "fa06495a-0930-4867-87c5-6023031ea8b5",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"(nan, nan, nan)"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example2 = np.array([2, np.nan, 6, 8]) \n",
"example2.sum(), example2.min(), example2.max()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nhlnNJT7gRr_"
},
"source": [
"### Latihan:\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": true,
"id": "yan3QRaOgRr_",
"trusted": false
},
"outputs": [],
"source": [
"# What happens if you add np.nan and None together?\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_iDvIRC8gRsA"
},
"source": [
"Ingat: `NaN` hanya untuk nilai floating-point yang hilang; tidak ada padanan `NaN` untuk bilangan bulat, string, atau Boolean.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kj6EKdsAgRsA"
},
"source": [
"### `NaN` dan `None`: nilai null dalam pandas\n",
"\n",
"Meskipun `NaN` dan `None` dapat berperilaku sedikit berbeda, pandas tetap dirancang untuk menangani keduanya secara bergantian. Untuk memahami maksudnya, perhatikan sebuah `Series` dari bilangan bulat:\n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Nji-KGdNgRsA",
"outputId": "36aa14d2-8efa-4bfd-c0ed-682991288822",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"0 1\n",
"1 2\n",
"2 3\n",
"dtype: int64"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"int_series = pd.Series([1, 2, 3], dtype=int)\n",
"int_series"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WklCzqb8gRsB"
},
"source": [
"### Latihan:\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": true,
"id": "Cy-gqX5-gRsB",
"trusted": false
},
"outputs": [],
"source": [
"# Now set an element of int_series equal to None.\n",
"# How does that element show up in the Series?\n",
"# What is the dtype of the Series?\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WjMQwltNgRsB"
},
"source": [
"Dalam proses menaikkan tipe data untuk memastikan keseragaman data dalam `Series` dan `DataFrame`, pandas dengan mudah mengganti nilai yang hilang antara `None` dan `NaN`. Karena fitur desain ini, akan membantu jika kita memandang `None` dan `NaN` sebagai dua jenis \"null\" yang berbeda dalam pandas. Bahkan, beberapa metode inti yang akan Anda gunakan untuk menangani nilai yang hilang dalam pandas mencerminkan ide ini dalam nama mereka:\n",
"\n",
"- `isnull()`: Menghasilkan masker Boolean yang menunjukkan nilai yang hilang\n",
"- `notnull()`: Kebalikan dari `isnull()`\n",
"- `dropna()`: Mengembalikan versi data yang telah difilter\n",
"- `fillna()`: Mengembalikan salinan data dengan nilai yang hilang diisi atau diimputasi\n",
"\n",
"Metode-metode ini sangat penting untuk dikuasai dan dipahami dengan baik, jadi mari kita bahas masing-masing secara mendalam.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Yh5ifd9FgRsB"
},
"source": [
"### Mendeteksi nilai null\n",
"\n",
"Setelah kita memahami pentingnya nilai yang hilang, kita perlu mendeteksinya dalam dataset kita sebelum mengatasinya. \n",
"Baik `isnull()` maupun `notnull()` adalah metode utama untuk mendeteksi data null. Keduanya mengembalikan masker Boolean pada data Anda.\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": true,
"id": "e-vFp5lvgRsC",
"trusted": false
},
"outputs": [],
"source": [
"example3 = pd.Series([0, np.nan, '', None])"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "1XdaJJ7PgRsC",
"outputId": "92fc363a-1874-471f-846d-f4f9ce1f51d0",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"0 False\n",
"1 True\n",
"2 False\n",
"3 True\n",
"dtype: bool"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example3.isnull()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PaSZ0SQygRsC"
},
"source": [
"Perhatikan dengan seksama hasilnya. Apakah ada yang mengejutkan Anda? Meskipun `0` adalah nol aritmatika, itu tetap merupakan bilangan bulat yang valid dan pandas memperlakukannya seperti itu. `''` sedikit lebih rumit. Meskipun kita menggunakannya di Bagian 1 untuk mewakili nilai string kosong, itu tetap merupakan objek string dan bukan representasi null sejauh yang pandas ketahui.\n",
"\n",
"Sekarang, mari kita balikkan ini dan gunakan metode-metode ini dengan cara yang lebih mirip dengan bagaimana Anda akan menggunakannya dalam praktik. Anda dapat menggunakan masker Boolean langsung sebagai indeks ``Series`` atau ``DataFrame``, yang bisa berguna saat mencoba bekerja dengan nilai yang hilang (atau ada) secara terpisah.\n",
"\n",
"Jika kita ingin jumlah total nilai yang hilang, kita cukup melakukan penjumlahan pada masker yang dihasilkan oleh metode `isnull()`.\n"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "JCcQVoPkHDUv",
"outputId": "001daa72-54f8-4bd5-842a-4df627a79d4d"
},
"outputs": [
{
"data": {
"text/plain": [
"2"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example3.isnull().sum()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PlBqEo3mgRsC"
},
"source": [
"### Latihan:\n"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"collapsed": true,
"id": "ggDVf5uygRsD",
"trusted": false
},
"outputs": [],
"source": [
"# Try running example3[example3.notnull()].\n",
"# Before you do so, what do you expect to see?\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "D_jWN7mHgRsD"
},
"source": [
"**Kesimpulan utama**: Baik metode `isnull()` maupun `notnull()` menghasilkan hasil yang serupa ketika Anda menggunakannya dalam DataFrame: mereka menunjukkan hasil dan indeks dari hasil tersebut, yang akan sangat membantu Anda saat Anda mengolah data Anda.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BvnoojWsgRr4"
},
"source": [
"### Menangani data yang hilang\n",
"\n",
"> **Tujuan pembelajaran:** Pada akhir bagian ini, Anda harus mengetahui cara dan kapan mengganti atau menghapus nilai null dari DataFrames.\n",
"\n",
"Model Machine Learning tidak dapat menangani data yang hilang secara langsung. Oleh karena itu, sebelum memberikan data ke model, kita perlu menangani nilai-nilai yang hilang ini.\n",
"\n",
"Cara menangani data yang hilang memiliki pertimbangan yang cukup rumit, dapat memengaruhi analisis akhir Anda dan hasil di dunia nyata.\n",
"\n",
"Ada dua cara utama untuk menangani data yang hilang:\n",
"\n",
"1. Menghapus baris yang mengandung nilai yang hilang\n",
"2. Mengganti nilai yang hilang dengan nilai lain\n",
"\n",
"Kita akan membahas kedua metode ini beserta kelebihan dan kekurangannya secara rinci.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3VaYC1TvgRsD"
},
"source": [
"### Menghapus nilai null\n",
"\n",
"Jumlah data yang kita berikan kepada model memiliki pengaruh langsung terhadap kinerjanya. Menghapus nilai null berarti kita mengurangi jumlah data, sehingga mengurangi ukuran dataset. Oleh karena itu, disarankan untuk menghapus baris dengan nilai null ketika dataset cukup besar.\n",
"\n",
"Contoh lainnya adalah ketika sebuah baris atau kolom memiliki banyak nilai yang hilang. Dalam kasus ini, baris atau kolom tersebut mungkin dihapus karena tidak akan memberikan banyak nilai pada analisis kita, mengingat sebagian besar datanya hilang.\n",
"\n",
"Selain mengidentifikasi nilai yang hilang, pandas menyediakan cara yang praktis untuk menghapus nilai null dari `Series` dan `DataFrame`. Untuk melihat cara kerjanya, mari kita kembali ke `example3`. Fungsi `DataFrame.dropna()` membantu dalam menghapus baris dengan nilai null.\n"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "7uIvS097gRsD",
"outputId": "c13fc117-4ca1-4145-a0aa-42ac89e6e218",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"0 0\n",
"2 \n",
"dtype: object"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example3 = example3.dropna()\n",
"example3"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hil2cr64gRsD"
},
"source": [
"Perhatikan bahwa ini seharusnya terlihat seperti output dari `example3[example3.notnull()]`. Perbedaannya di sini adalah, alih-alih hanya melakukan pengindeksan pada nilai yang dimask, `dropna` telah menghapus nilai-nilai yang hilang dari `Series` `example3`.\n",
"\n",
"Karena DataFrame memiliki dua dimensi, mereka memberikan lebih banyak opsi untuk menghapus data.\n"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "an-l74sPgRsE",
"outputId": "340876a0-63ad-40f6-bd54-6240cdae50ab",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>6.0</td>\n",
" <td>9</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 1.0 NaN 7\n",
"1 2.0 5.0 8\n",
"2 NaN 6.0 9"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4 = pd.DataFrame([[1, np.nan, 7], \n",
" [2, 5, 8], \n",
" [np.nan, 6, 9]])\n",
"example4"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "66wwdHZrgRsE"
},
"source": [
"(Apakah Anda menyadari bahwa pandas mengubah tipe dua kolom menjadi float untuk menyesuaikan dengan `NaN`?)\n",
"\n",
"Anda tidak dapat menghapus satu nilai dari sebuah `DataFrame`, jadi Anda harus menghapus seluruh baris atau kolom. Bergantung pada apa yang Anda lakukan, Anda mungkin ingin memilih salah satu dari keduanya, dan pandas memberikan opsi untuk keduanya. Karena dalam ilmu data, kolom umumnya mewakili variabel dan baris mewakili observasi, Anda lebih mungkin untuk menghapus baris data; pengaturan default untuk `dropna()` adalah menghapus semua baris yang mengandung nilai null:\n"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 80
},
"id": "jAVU24RXgRsE",
"outputId": "0b5e5aee-7187-4d3f-b583-a44136ae5f80",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"1 2.0 5.0 8"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4.dropna()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TrQRBuTDgRsE"
},
"source": [
"Jika perlu, Anda dapat menghapus nilai NA dari kolom. Gunakan `axis=1` untuk melakukannya:\n"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "GrBhxu9GgRsE",
"outputId": "ff4001f3-2e61-4509-d60e-0093d1068437",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>8</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>9</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 2\n",
"0 7\n",
"1 8\n",
"2 9"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4.dropna(axis='columns')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KWXiKTfMgRsF"
},
"source": [
"Perhatikan bahwa ini dapat menghapus banyak data yang mungkin ingin Anda simpan, terutama dalam dataset yang lebih kecil. Bagaimana jika Anda hanya ingin menghapus baris atau kolom yang mengandung beberapa atau bahkan semua nilai null? Anda dapat menentukan pengaturan tersebut dalam `dropna` dengan parameter `how` dan `thresh`.\n",
"\n",
"Secara default, `how='any'` (jika Anda ingin memeriksa sendiri atau melihat parameter lain yang dimiliki metode ini, jalankan `example4.dropna?` dalam sel kode). Sebagai alternatif, Anda dapat menentukan `how='all'` untuk hanya menghapus baris atau kolom yang mengandung semua nilai null. Mari kita perluas contoh `DataFrame` kita untuk melihat ini dalam aksi pada latihan berikutnya.\n"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "Bcf_JWTsgRsF",
"outputId": "72e0b1b8-52fa-4923-98ce-b6fbed6e44b1",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>7</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>6.0</td>\n",
" <td>9</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"0 1.0 NaN 7 NaN\n",
"1 2.0 5.0 8 NaN\n",
"2 NaN 6.0 9 NaN"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4[3] = np.nan\n",
"example4"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pNZer7q9JPNC"
},
"source": [
"> Poin-poin penting: \n",
"1. Menghapus nilai null adalah ide yang baik hanya jika dataset cukup besar. \n",
"2. Baris atau kolom penuh dapat dihapus jika sebagian besar datanya hilang. \n",
"3. Metode `DataFrame.dropna(axis=)` membantu dalam menghapus nilai null. Argumen `axis` menunjukkan apakah baris atau kolom yang akan dihapus. \n",
"4. Argumen `how` juga dapat digunakan. Secara default, nilainya diatur ke `any`. Jadi, hanya baris/kolom yang mengandung nilai null yang akan dihapus. Nilainya dapat diatur ke `all` untuk menentukan bahwa kita hanya akan menghapus baris/kolom di mana semua nilainya null. \n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oXXSfQFHgRsF"
},
"source": [
"### Latihan:\n"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"collapsed": true,
"id": "ExUwQRxpgRsF",
"trusted": false
},
"outputs": [],
"source": [
"# How might you go about dropping just column 3?\n",
"# Hint: remember that you will need to supply both the axis parameter and the how parameter.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "38kwAihWgRsG"
},
"source": [
"Parameter `thresh` memberikan kontrol yang lebih rinci: Anda menetapkan jumlah nilai *non-null* yang harus dimiliki oleh sebuah baris atau kolom agar tetap dipertahankan:\n"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 80
},
"id": "M9dCNMaagRsG",
"outputId": "8093713a-54d2-4e54-c73f-4eea315cb6f2",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"1 2.0 5.0 8 NaN"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4.dropna(axis='rows', thresh=3)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fmSFnzZegRsG"
},
"source": [
"Di sini, baris pertama dan terakhir telah dihapus, karena hanya berisi dua nilai non-null.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mCcxLGyUgRsG"
},
"source": [
"### Mengisi Nilai Null\n",
"\n",
"Kadang-kadang masuk akal untuk mengisi nilai yang hilang dengan nilai yang mungkin valid. Ada beberapa teknik untuk mengisi nilai null. Yang pertama adalah menggunakan Pengetahuan Domain (pengetahuan tentang subjek yang menjadi dasar dataset) untuk memperkirakan nilai yang hilang.\n",
"\n",
"Anda bisa menggunakan `isnull` untuk melakukan ini secara langsung, tetapi itu bisa melelahkan, terutama jika Anda memiliki banyak nilai yang harus diisi. Karena ini adalah tugas yang sangat umum dalam ilmu data, pandas menyediakan `fillna`, yang mengembalikan salinan dari `Series` atau `DataFrame` dengan nilai yang hilang diganti dengan nilai pilihan Anda. Mari kita buat contoh `Series` lain untuk melihat bagaimana ini bekerja dalam praktik.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CE8S7louLezV"
},
"source": [
"### Data Kategorikal (Non-numerik)\n",
"Pertama, mari kita bahas data non-numerik. Dalam dataset, kita memiliki kolom dengan data kategorikal. Contohnya: Jenis Kelamin, Benar atau Salah, dll.\n",
"\n",
"Dalam sebagian besar kasus ini, kita mengganti nilai yang hilang dengan `modus` dari kolom tersebut. Misalnya, kita memiliki 100 data, di mana 90 mengatakan Benar, 8 mengatakan Salah, dan 2 tidak mengisi. Maka, kita dapat mengisi 2 data yang kosong dengan Benar, dengan mempertimbangkan seluruh kolom.\n",
"\n",
"Sekali lagi, di sini kita dapat menggunakan pengetahuan domain. Mari kita lihat contoh pengisian dengan modus.\n"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "MY5faq4yLdpQ",
"outputId": "19ab472e-1eed-4de8-f8a7-db2a3af3cb1a"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" <td>None</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>5</td>\n",
" <td>6</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>7</td>\n",
" <td>8</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>9</td>\n",
" <td>10</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 1 2 True\n",
"1 3 4 None\n",
"2 5 6 False\n",
"3 7 8 True\n",
"4 9 10 True"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_mode = pd.DataFrame([[1,2,\"True\"],\n",
" [3,4,None],\n",
" [5,6,\"False\"],\n",
" [7,8,\"True\"],\n",
" [9,10,\"True\"]])\n",
"\n",
"fill_with_mode"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MLAoMQOfNPlA"
},
"source": [
"Sekarang, mari kita cari modus terlebih dahulu sebelum mengisi nilai `None` dengan modus.\n"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "WKy-9Y2tN5jv",
"outputId": "8da9fa16-e08c-447e-dea1-d4b1db2feebf"
},
"outputs": [
{
"data": {
"text/plain": [
"True 3\n",
"False 1\n",
"Name: 2, dtype: int64"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_mode[2].value_counts()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6iNz_zG_OKrx"
},
"source": [
"Jadi, kita akan mengganti None dengan True\n"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"id": "TxPKteRvNPOs"
},
"outputs": [],
"source": [
"fill_with_mode[2].fillna('True',inplace=True)"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "tvas7c9_OPWE",
"outputId": "ec3c8e44-d644-475e-9e22-c65101965850"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>5</td>\n",
" <td>6</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>7</td>\n",
" <td>8</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>9</td>\n",
" <td>10</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 1 2 True\n",
"1 3 4 True\n",
"2 5 6 False\n",
"3 7 8 True\n",
"4 9 10 True"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_mode"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SktitLxxOR16"
},
"source": [
"Seperti yang dapat kita lihat, nilai null telah digantikan. Tidak perlu dikatakan, kita bisa saja menulis apa pun sebagai pengganti atau `'True'` dan itu akan tersubstitusi.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "heYe1I0dOmQ_"
},
"source": [
"### Data Numerik\n",
"Sekarang, kita beralih ke data numerik. Di sini, ada dua cara umum untuk mengganti nilai yang hilang:\n",
"\n",
"1. Mengganti dengan Median dari baris\n",
"2. Mengganti dengan Mean dari baris\n",
"\n",
"Kita mengganti dengan Median jika data memiliki distribusi yang miring dengan outlier. Hal ini karena median lebih tahan terhadap outlier.\n",
"\n",
"Ketika data sudah dinormalisasi, kita bisa menggunakan mean, karena dalam kasus tersebut, mean dan median akan cukup dekat.\n",
"\n",
"Pertama, mari kita ambil sebuah kolom yang terdistribusi normal dan isi nilai yang hilang dengan mean dari kolom tersebut.\n"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "09HM_2feOj5Y",
"outputId": "7e309013-9acb-411c-9b06-4de795bbeeff"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>-2.0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>-1.0</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1.0</td>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2.0</td>\n",
" <td>8</td>\n",
" <td>9</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 -2.0 0 1\n",
"1 -1.0 2 3\n",
"2 NaN 4 5\n",
"3 1.0 6 7\n",
"4 2.0 8 9"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_mean = pd.DataFrame([[-2,0,1],\n",
" [-1,2,3],\n",
" [np.nan,4,5],\n",
" [1,6,7],\n",
" [2,8,9]])\n",
"\n",
"fill_with_mean"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ka7-wNfzSxbx"
},
"source": [
"Rata-rata dari kolom adalah\n"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "XYtYEf5BSxFL",
"outputId": "68a78d18-f0e5-4a9a-a959-2c3676a57c70"
},
"outputs": [
{
"data": {
"text/plain": [
"0.0"
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.mean(fill_with_mean[0])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oBSRGxKRS39K"
},
"source": [
"Mengisi dengan rata-rata\n"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "FzncQLmuS5jh",
"outputId": "00f74fff-01f4-4024-c261-796f50f01d2e"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>-2.0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>-1.0</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>0.0</td>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1.0</td>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2.0</td>\n",
" <td>8</td>\n",
" <td>9</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 -2.0 0 1\n",
"1 -1.0 2 3\n",
"2 0.0 4 5\n",
"3 1.0 6 7\n",
"4 2.0 8 9"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_mean[0].fillna(np.mean(fill_with_mean[0]),inplace=True)\n",
"fill_with_mean"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CwpVFCrPTC5z"
},
"source": [
"Seperti yang dapat kita lihat, nilai yang hilang telah digantikan dengan rata-ratanya.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jIvF13a1i00Z"
},
"source": [
"Sekarang mari kita coba dataframe lain, dan kali ini kita akan mengganti nilai None dengan median dari kolom tersebut.\n"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "DA59Bqo3jBYZ",
"outputId": "85dae6ec-7394-4c36-fda0-e04769ec4a32"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>-2</td>\n",
" <td>0.0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>-1</td>\n",
" <td>2.0</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>0</td>\n",
" <td>NaN</td>\n",
" <td>5</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1</td>\n",
" <td>6.0</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2</td>\n",
" <td>8.0</td>\n",
" <td>9</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 -2 0.0 1\n",
"1 -1 2.0 3\n",
"2 0 NaN 5\n",
"3 1 6.0 7\n",
"4 2 8.0 9"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_median = pd.DataFrame([[-2,0,1],\n",
" [-1,2,3],\n",
" [0,np.nan,5],\n",
" [1,6,7],\n",
" [2,8,9]])\n",
"\n",
"fill_with_median"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mM1GpXYmjHnc"
},
"source": [
"Median dari kolom kedua adalah\n"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "uiDy5v3xjHHX",
"outputId": "564b6b74-2004-4486-90d4-b39330a64b88"
},
"outputs": [
{
"data": {
"text/plain": [
"4.0"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_median[1].median()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "z9PLF75Jj_1s"
},
"source": [
"Mengisi dengan median\n"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "lFKbOxCMkBbg",
"outputId": "a8bd18fb-2765-47d4-e5fe-e965f57ed1f4"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>-2</td>\n",
" <td>0.0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>-1</td>\n",
" <td>2.0</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>0</td>\n",
" <td>4.0</td>\n",
" <td>5</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1</td>\n",
" <td>6.0</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2</td>\n",
" <td>8.0</td>\n",
" <td>9</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 -2 0.0 1\n",
"1 -1 2.0 3\n",
"2 0 4.0 5\n",
"3 1 6.0 7\n",
"4 2 8.0 9"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_median[1].fillna(fill_with_median[1].median(),inplace=True)\n",
"fill_with_median"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8JtQ53GSkKWC"
},
"source": [
"Seperti yang kita lihat, nilai NaN telah digantikan oleh median dari kolom.\n"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "0ybtWLDdgRsG",
"outputId": "b8c238ef-6024-4ee2-be2b-aa1f0fcac61d",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"a 1.0\n",
"b NaN\n",
"c 2.0\n",
"d NaN\n",
"e 3.0\n",
"dtype: float64"
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example5 = pd.Series([1, np.nan, 2, None, 3], index=list('abcde'))\n",
"example5"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yrsigxRggRsH"
},
"source": [
"Anda dapat mengisi semua entri null dengan satu nilai, seperti `0`:\n"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "KXMIPsQdgRsH",
"outputId": "aeedfa0a-a421-4c2f-cb0d-183ce8f0c91d",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"a 1.0\n",
"b 0.0\n",
"c 2.0\n",
"d 0.0\n",
"e 3.0\n",
"dtype: float64"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example5.fillna(0)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RRlI5f_hkfKe"
},
"source": [
"> Poin-poin penting:\n",
"1. Mengisi nilai yang hilang sebaiknya dilakukan ketika data yang tersedia sedikit atau ada strategi untuk mengisi data yang hilang.\n",
"2. Pengetahuan domain dapat digunakan untuk mengisi nilai yang hilang dengan cara memperkirakannya.\n",
"3. Untuk data kategorikal, biasanya nilai yang hilang diganti dengan modus dari kolom tersebut.\n",
"4. Untuk data numerik, nilai yang hilang biasanya diisi dengan rata-rata (untuk dataset yang sudah dinormalisasi) atau median dari kolom tersebut.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FI9MmqFJgRsH"
},
"source": [
"### Latihan:\n"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {
"collapsed": true,
"id": "af-ezpXdgRsH",
"trusted": false
},
"outputs": [],
"source": [
"# What happens if you try to fill null values with a string, like ''?\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kq3hw1kLgRsI"
},
"source": [
"Anda dapat **mengisi ke depan** nilai null, yaitu menggunakan nilai valid terakhir untuk mengisi null:\n"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "vO3BuNrggRsI",
"outputId": "e2bc591b-0b48-4e88-ee65-754f2737c196",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"a 1.0\n",
"b 1.0\n",
"c 2.0\n",
"d 2.0\n",
"e 3.0\n",
"dtype: float64"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example5.fillna(method='ffill')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nDXeYuHzgRsI"
},
"source": [
"Anda juga dapat **mengisi kembali** untuk menyebarkan nilai valid berikutnya ke belakang guna mengisi null:\n"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "4M5onHcEgRsI",
"outputId": "8f32b185-40dd-4a9f-bd85-54d6b6a414fe",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"a 1.0\n",
"b 2.0\n",
"c 2.0\n",
"d 3.0\n",
"e 3.0\n",
"dtype: float64"
]
},
"execution_count": 42,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example5.fillna(method='bfill')"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"id": "MbBzTom5gRsI"
},
"source": [
"Seperti yang mungkin Anda duga, ini bekerja sama dengan DataFrame, tetapi Anda juga dapat menentukan `axis` sepanjang mana untuk mengisi nilai null:\n"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "aRpIvo4ZgRsI",
"outputId": "905a980a-a808-4eca-d0ba-224bd7d85955",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>7</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>6.0</td>\n",
" <td>9</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"0 1.0 NaN 7 NaN\n",
"1 2.0 5.0 8 NaN\n",
"2 NaN 6.0 9 NaN"
]
},
"execution_count": 43,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "VM1qtACAgRsI",
"outputId": "71f2ad28-9b4e-4ff4-f5c3-e731eb489ade",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>7.0</td>\n",
" <td>7.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8.0</td>\n",
" <td>8.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>6.0</td>\n",
" <td>9.0</td>\n",
" <td>9.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"0 1.0 1.0 7.0 7.0\n",
"1 2.0 5.0 8.0 8.0\n",
"2 NaN 6.0 9.0 9.0"
]
},
"execution_count": 44,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4.fillna(method='ffill', axis=1)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZeMc-I1EgRsI"
},
"source": [
"Perhatikan bahwa ketika nilai sebelumnya tidak tersedia untuk pengisian ke depan, nilai null tetap ada.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eeAoOU0RgRsJ"
},
"source": [
"### Latihan:\n"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {
"collapsed": true,
"id": "e8S-CjW8gRsJ",
"trusted": false
},
"outputs": [],
"source": [
"# What output does example4.fillna(method='bfill', axis=1) produce?\n",
"# What about example4.fillna(method='ffill') or example4.fillna(method='bfill')?\n",
"# Can you think of a longer code snippet to write that can fill all of the null values in example4?\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YHgy0lIrgRsJ"
},
"source": [
"Anda dapat menjadi kreatif dalam menggunakan `fillna`. Sebagai contoh, mari kita lihat kembali `example4`, tetapi kali ini kita isi nilai yang hilang dengan rata-rata dari semua nilai dalam `DataFrame`:\n"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "OtYVErEygRsJ",
"outputId": "708b1e67-45ca-44bf-a5ee-8b2de09ece73",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>5.5</td>\n",
" <td>7</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>1.5</td>\n",
" <td>6.0</td>\n",
" <td>9</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"0 1.0 5.5 7 NaN\n",
"1 2.0 5.0 8 NaN\n",
"2 1.5 6.0 9 NaN"
]
},
"execution_count": 46,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4.fillna(example4.mean())"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zpMvCkLSgRsJ"
},
"source": [
"Perhatikan bahwa kolom 3 masih tidak memiliki nilai: arah default adalah mengisi nilai secara baris demi baris.\n",
"\n",
"> **Kesimpulan:** Ada berbagai cara untuk menangani nilai yang hilang dalam dataset Anda. Strategi spesifik yang Anda gunakan (menghapusnya, menggantinya, atau bahkan bagaimana Anda menggantinya) harus ditentukan oleh karakteristik data tersebut. Anda akan semakin memahami cara menangani nilai yang hilang seiring dengan semakin seringnya Anda bekerja dan berinteraksi dengan dataset.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bauDnESIl9FH"
},
"source": [
"### Mengkodekan Data Kategorikal\n",
"\n",
"Model pembelajaran mesin hanya dapat bekerja dengan angka dan segala bentuk data numerik. Model tidak akan bisa membedakan antara Ya dan Tidak, tetapi akan mampu membedakan antara 0 dan 1. Jadi, setelah mengisi nilai yang hilang, kita perlu mengkodekan data kategorikal ke dalam bentuk numerik agar model dapat memahaminya.\n",
"\n",
"Pengkodean dapat dilakukan dengan dua cara. Kita akan membahasnya selanjutnya.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uDq9SxB7mu5i"
},
"source": [
"**PENGKODEAN LABEL**\n",
"\n",
"Pengkodean label pada dasarnya adalah mengubah setiap kategori menjadi angka. Sebagai contoh, misalkan kita memiliki dataset penumpang pesawat dan terdapat kolom yang berisi kelas mereka di antara ['kelas bisnis', 'kelas ekonomi', 'kelas pertama']. Jika pengkodean label dilakukan pada data ini, maka akan diubah menjadi [0,1,2]. Mari kita lihat contohnya melalui kode. Karena kita akan mempelajari `scikit-learn` di notebook berikutnya, kita tidak akan menggunakannya di sini.\n"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 235
},
"id": "1vGz7uZyoWHL",
"outputId": "9e252855-d193-4103-a54d-028ea7787b34"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>ID</th>\n",
" <th>class</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>10</td>\n",
" <td>business class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>20</td>\n",
" <td>first class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>30</td>\n",
" <td>economy class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>40</td>\n",
" <td>economy class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>50</td>\n",
" <td>economy class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>60</td>\n",
" <td>business class</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" ID class\n",
"0 10 business class\n",
"1 20 first class\n",
"2 30 economy class\n",
"3 40 economy class\n",
"4 50 economy class\n",
"5 60 business class"
]
},
"execution_count": 47,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"label = pd.DataFrame([\n",
" [10,'business class'],\n",
" [20,'first class'],\n",
" [30, 'economy class'],\n",
" [40, 'economy class'],\n",
" [50, 'economy class'],\n",
" [60, 'business class']\n",
"],columns=['ID','class'])\n",
"label"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "IDHnkwTYov-h"
},
"source": [
"Untuk melakukan label encoding pada kolom pertama, kita harus terlebih dahulu mendeskripsikan pemetaan dari setiap kelas ke angka, sebelum menggantinya.\n"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 235
},
"id": "ZC5URJG3o1ES",
"outputId": "aab0f1e7-e0f3-4c14-8459-9f9168c85437"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>ID</th>\n",
" <th>class</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>10</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>20</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>30</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>40</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>50</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>60</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" ID class\n",
"0 10 0\n",
"1 20 2\n",
"2 30 1\n",
"3 40 1\n",
"4 50 1\n",
"5 60 0"
]
},
"execution_count": 48,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class_labels = {'business class':0,'economy class':1,'first class':2}\n",
"label['class'] = label['class'].replace(class_labels)\n",
"label"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ftnF-TyapOPt"
},
"source": [
"Seperti yang kita lihat, hasilnya sesuai dengan apa yang kita pikirkan akan terjadi. Jadi, kapan kita menggunakan label encoding? Label encoding digunakan dalam salah satu atau kedua kasus berikut:\n",
"1. Ketika jumlah kategori sangat banyak\n",
"2. Ketika kategori memiliki urutan tertentu.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eQPAPVwsqWT7"
},
"source": [
"**ONE HOT ENCODING**\n",
"\n",
"Jenis encoding lainnya adalah One Hot Encoding. Dalam jenis encoding ini, setiap kategori dari kolom akan ditambahkan sebagai kolom terpisah, dan setiap data akan mendapatkan nilai 0 atau 1 berdasarkan apakah data tersebut mengandung kategori tersebut. Jadi, jika ada n kategori yang berbeda, maka n kolom akan ditambahkan ke dataframe.\n",
"\n",
"Sebagai contoh, mari kita ambil contoh kelas pesawat yang sama. Kategorinya adalah: ['business class', 'economy class', 'first class']. Jadi, jika kita melakukan one hot encoding, tiga kolom berikut akan ditambahkan ke dataset: ['class_business class', 'class_economy class', 'class_first class'].\n"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 235
},
"id": "ZM0eVh0ArKUL",
"outputId": "83238a76-b3a5-418d-c0b6-605b02b6891b"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>ID</th>\n",
" <th>class</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>10</td>\n",
" <td>business class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>20</td>\n",
" <td>first class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>30</td>\n",
" <td>economy class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>40</td>\n",
" <td>economy class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>50</td>\n",
" <td>economy class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>60</td>\n",
" <td>business class</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" ID class\n",
"0 10 business class\n",
"1 20 first class\n",
"2 30 economy class\n",
"3 40 economy class\n",
"4 50 economy class\n",
"5 60 business class"
]
},
"execution_count": 49,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"one_hot = pd.DataFrame([\n",
" [10,'business class'],\n",
" [20,'first class'],\n",
" [30, 'economy class'],\n",
" [40, 'economy class'],\n",
" [50, 'economy class'],\n",
" [60, 'business class']\n",
"],columns=['ID','class'])\n",
"one_hot"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aVnZ7paDrWmb"
},
"source": [
"Mari kita lakukan one hot encoding pada kolom pertama\n"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {
"id": "RUPxf7egrYKr"
},
"outputs": [],
"source": [
"one_hot_data = pd.get_dummies(one_hot,columns=['class'])"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 235
},
"id": "TM37pHsFr4ge",
"outputId": "7be15f53-79b2-447a-979c-822658339a9e"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>ID</th>\n",
" <th>class_business class</th>\n",
" <th>class_economy class</th>\n",
" <th>class_first class</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>10</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>20</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>30</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>40</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>50</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>60</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" ID class_business class class_economy class class_first class\n",
"0 10 1 0 0\n",
"1 20 0 0 1\n",
"2 30 0 1 0\n",
"3 40 0 1 0\n",
"4 50 0 1 0\n",
"5 60 1 0 0"
]
},
"execution_count": 51,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"one_hot_data"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_zXRLOjXujdA"
},
"source": [
"Setiap kolom yang di-encode one hot berisi 0 atau 1, yang menunjukkan apakah kategori tersebut ada untuk titik data tersebut.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bDnC4NQOu0qr"
},
"source": [
"Kapan kita menggunakan one hot encoding? One hot encoding digunakan dalam salah satu atau kedua kasus berikut:\n",
"\n",
"1. Ketika jumlah kategori dan ukuran dataset lebih kecil.\n",
"2. Ketika kategori tidak mengikuti urutan tertentu.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XnUmci_4uvyu"
},
"source": [
"> Poin-Poin Penting:\n",
"1. Encoding dilakukan untuk mengubah data non-numerik menjadi data numerik.\n",
"2. Ada dua jenis encoding: Label encoding dan One Hot encoding, yang keduanya dapat dilakukan sesuai kebutuhan dataset.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "K8UXOJYRgRsJ"
},
"source": [
"## Menghapus Data Duplikat\n",
"\n",
"> **Tujuan pembelajaran:** Di akhir subbagian ini, Anda seharusnya merasa nyaman dalam mengidentifikasi dan menghapus nilai duplikat dari DataFrame.\n",
"\n",
"Selain data yang hilang, Anda sering kali akan menemukan data duplikat dalam dataset dunia nyata. Untungnya, pandas menyediakan cara yang mudah untuk mendeteksi dan menghapus entri duplikat.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qrEG-Wa0gRsJ"
},
"source": [
"### Mengidentifikasi duplikat: `duplicated`\n",
"\n",
"Anda dapat dengan mudah menemukan nilai duplikat menggunakan metode `duplicated` di pandas, yang mengembalikan masker Boolean yang menunjukkan apakah suatu entri dalam `DataFrame` adalah duplikat dari entri sebelumnya. Mari kita buat contoh `DataFrame` lain untuk melihat cara kerjanya.\n"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "ZLu6FEnZgRsJ",
"outputId": "376512d1-d842-4db1-aea3-71052aeeecaf",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>letters</th>\n",
" <th>numbers</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>B</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>A</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>B</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>B</td>\n",
" <td>3</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" letters numbers\n",
"0 A 1\n",
"1 B 2\n",
"2 A 1\n",
"3 B 3\n",
"4 B 3"
]
},
"execution_count": 52,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example6 = pd.DataFrame({'letters': ['A','B'] * 2 + ['B'],\n",
" 'numbers': [1, 2, 1, 3, 3]})\n",
"example6"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "cIduB5oBgRsK",
"outputId": "3da27b3d-4d69-4e1d-bb52-0af21bae87f2",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"0 False\n",
"1 False\n",
"2 True\n",
"3 False\n",
"4 True\n",
"dtype: bool"
]
},
"execution_count": 53,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example6.duplicated()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0eDRJD4SgRsK"
},
"source": [
"### Menghapus duplikat: `drop_duplicates`\n",
"`drop_duplicates` hanya mengembalikan salinan data di mana semua nilai `duplicated` adalah `False`:\n"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "w_YPpqIqgRsK",
"outputId": "ac66bd2f-8671-4744-87f5-8b8d96553dea",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>letters</th>\n",
" <th>numbers</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>B</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>B</td>\n",
" <td>3</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" letters numbers\n",
"0 A 1\n",
"1 B 2\n",
"3 B 3"
]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example6.drop_duplicates()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "69AqoCZAgRsK"
},
"source": [
"Baik `duplicated` maupun `drop_duplicates` secara default mempertimbangkan semua kolom tetapi Anda dapat menentukan agar mereka hanya memeriksa sebagian kolom dalam `DataFrame` Anda:\n"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 111
},
"id": "BILjDs67gRsK",
"outputId": "ef6dcc08-db8b-4352-c44e-5aa9e2bec0d3",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>letters</th>\n",
" <th>numbers</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>B</td>\n",
" <td>2</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" letters numbers\n",
"0 A 1\n",
"1 B 2"
]
},
"execution_count": 55,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example6.drop_duplicates(['letters'])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GvX4og1EgRsL"
},
"source": [
"> **Kesimpulan:** Menghapus data duplikat adalah bagian penting dari hampir setiap proyek ilmu data. Data duplikat dapat mengubah hasil analisis Anda dan memberikan hasil yang tidak akurat!\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pemeriksaan Kualitas Data di Dunia Nyata\n",
"\n",
"> **Tujuan pembelajaran:** Pada akhir bagian ini, Anda diharapkan mampu mendeteksi dan memperbaiki masalah kualitas data umum di dunia nyata, termasuk nilai kategori yang tidak konsisten, nilai numerik yang tidak wajar (outlier), dan entitas duplikat dengan variasi.\n",
"\n",
"Meskipun nilai yang hilang dan duplikat yang persis sama adalah masalah yang umum, dataset di dunia nyata sering kali mengandung masalah yang lebih halus:\n",
"\n",
"1. **Nilai kategori yang tidak konsisten**: Kategori yang sama ditulis dengan cara berbeda (misalnya, \"USA\", \"U.S.A\", \"United States\")\n",
"2. **Nilai numerik yang tidak wajar**: Outlier ekstrem yang menunjukkan kesalahan entri data (misalnya, usia = 999)\n",
"3. **Baris yang hampir duplikat**: Rekaman yang mewakili entitas yang sama dengan sedikit variasi\n",
"\n",
"Mari kita jelajahi teknik untuk mendeteksi dan menangani masalah-masalah ini.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Membuat Dataset \"Kotor\" Contoh\n",
"\n",
"Pertama, mari kita buat dataset contoh yang berisi jenis masalah yang sering kita temui dalam data dunia nyata:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"\n",
"# Create a sample dataset with quality issues\n",
"dirty_data = pd.DataFrame({\n",
" 'customer_id': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12],\n",
" 'name': ['John Smith', 'Jane Doe', 'John Smith', 'Bob Johnson', \n",
" 'Alice Williams', 'Charlie Brown', 'John Smith', 'Eva Martinez',\n",
" 'Bob Johnson', 'Diana Prince', 'Frank Castle', 'Alice Williams'],\n",
" 'age': [25, 32, 25, 45, 28, 199, 25, 31, 45, 27, -5, 28],\n",
" 'country': ['USA', 'UK', 'U.S.A', 'Canada', 'USA', 'United Kingdom',\n",
" 'United States', 'Mexico', 'canada', 'USA', 'UK', 'usa'],\n",
" 'purchase_amount': [100.50, 250.00, 105.00, 320.00, 180.00, 90.00,\n",
" 102.00, 275.00, 325.00, 195.00, 410.00, 185.00]\n",
"})\n",
"\n",
"print(\"Sample 'Dirty' Dataset:\")\n",
"print(dirty_data)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1. Mendeteksi Nilai Kategorikal yang Tidak Konsisten\n",
"\n",
"Perhatikan kolom `country` memiliki beberapa representasi untuk negara yang sama. Mari kita identifikasi ketidakkonsistenan ini:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Check unique values in the country column\n",
"print(\"Unique country values:\")\n",
"print(dirty_data['country'].unique())\n",
"print(f\"\\nTotal unique values: {dirty_data['country'].nunique()}\")\n",
"\n",
"# Count occurrences of each variation\n",
"print(\"\\nValue counts:\")\n",
"print(dirty_data['country'].value_counts())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Standarisasi Nilai Kategorikal\n",
"\n",
"Kita dapat membuat pemetaan untuk menstandarisasi nilai-nilai ini. Pendekatan sederhana adalah dengan mengubah ke huruf kecil dan membuat kamus pemetaan:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a standardization mapping\n",
"country_mapping = {\n",
" 'usa': 'USA',\n",
" 'u.s.a': 'USA',\n",
" 'united states': 'USA',\n",
" 'uk': 'UK',\n",
" 'united kingdom': 'UK',\n",
" 'canada': 'Canada',\n",
" 'mexico': 'Mexico'\n",
"}\n",
"\n",
"# Standardize the country column\n",
"dirty_data['country_clean'] = dirty_data['country'].str.lower().map(country_mapping)\n",
"\n",
"print(\"Before standardization:\")\n",
"print(dirty_data['country'].value_counts())\n",
"print(\"\\nAfter standardization:\")\n",
"print(dirty_data[['country_clean']].value_counts())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Alternatif: Menggunakan Pencocokan Fuzzy**\n",
"\n",
"Untuk kasus yang lebih kompleks, kita dapat menggunakan pencocokan string fuzzy dengan pustaka `rapidfuzz` untuk mendeteksi string yang mirip secara otomatis:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"try:\n",
" from rapidfuzz import process, fuzz\n",
"except ImportError:\n",
" print(\"rapidfuzz is not installed. Please install it with 'pip install rapidfuzz' to use fuzzy matching.\")\n",
" process = None\n",
" fuzz = None\n",
"\n",
"# Get unique countries\n",
"unique_countries = dirty_data['country'].unique()\n",
"\n",
"# For each country, find similar matches\n",
"if process is not None and fuzz is not None:\n",
" print(\"Finding similar country names (similarity > 70%):\")\n",
" for country in unique_countries:\n",
" matches = process.extract(country, unique_countries, scorer=fuzz.ratio, limit=3)\n",
" # Filter matches with similarity > 70 and not identical\n",
" similar = [m for m in matches if m[1] > 70 and m[0] != country]\n",
" if similar:\n",
" print(f\"\\n'{country}' is similar to:\")\n",
" for match, score, _ in similar:\n",
" print(f\" - '{match}' (similarity: {score}%)\")\n",
"else:\n",
" print(\"Skipping fuzzy matching because rapidfuzz is not available.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2. Mendeteksi Nilai Numerik yang Tidak Normal (Outlier)\n",
"\n",
"Melihat kolom `age`, terdapat beberapa nilai mencurigakan seperti 199 dan -5. Mari gunakan metode statistik untuk mendeteksi outlier ini.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Display basic statistics\n",
"print(\"Age column statistics:\")\n",
"print(dirty_data['age'].describe())\n",
"\n",
"# Identify impossible values using domain knowledge\n",
"print(\"\\nRows with impossible age values (< 0 or > 120):\")\n",
"impossible_ages = dirty_data[(dirty_data['age'] < 0) | (dirty_data['age'] > 120)]\n",
"print(impossible_ages[['customer_id', 'name', 'age']])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Menggunakan Metode IQR (Interquartile Range)\n",
"\n",
"Metode IQR adalah teknik statistik yang tangguh untuk mendeteksi pencilan yang kurang sensitif terhadap nilai ekstrem:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Calculate IQR for age (excluding impossible values)\n",
"valid_ages = dirty_data[(dirty_data['age'] >= 0) & (dirty_data['age'] <= 120)]['age']\n",
"\n",
"Q1 = valid_ages.quantile(0.25)\n",
"Q3 = valid_ages.quantile(0.75)\n",
"IQR = Q3 - Q1\n",
"\n",
"# Define outlier bounds\n",
"lower_bound = Q1 - 1.5 * IQR\n",
"upper_bound = Q3 + 1.5 * IQR\n",
"\n",
"print(f\"IQR-based outlier bounds for age: [{lower_bound:.2f}, {upper_bound:.2f}]\")\n",
"\n",
"# Identify outliers\n",
"age_outliers = dirty_data[(dirty_data['age'] < lower_bound) | (dirty_data['age'] > upper_bound)]\n",
"print(f\"\\nRows with age outliers:\")\n",
"print(age_outliers[['customer_id', 'name', 'age']])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Menggunakan Metode Z-Score\n",
"\n",
"Metode Z-score mengidentifikasi nilai pencilan berdasarkan deviasi standar dari rata-rata:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"try:\n",
" from scipy import stats\n",
"except ImportError:\n",
" print(\"scipy is required for Z-score calculation. Please install it with 'pip install scipy' and rerun this cell.\")\n",
"else:\n",
" # Calculate Z-scores for age, handling NaN values\n",
" age_nonan = dirty_data['age'].dropna()\n",
" zscores = np.abs(stats.zscore(age_nonan))\n",
" dirty_data['age_zscore'] = np.nan\n",
" dirty_data.loc[age_nonan.index, 'age_zscore'] = zscores\n",
"\n",
" # Typically, Z-score > 3 indicates an outlier\n",
" print(\"Rows with age Z-score > 3:\")\n",
" zscore_outliers = dirty_data[dirty_data['age_zscore'] > 3]\n",
" print(zscore_outliers[['customer_id', 'name', 'age', 'age_zscore']])\n",
"\n",
" # Clean up the temporary column\n",
" dirty_data = dirty_data.drop('age_zscore', axis=1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Menangani Outlier\n",
"\n",
"Setelah terdeteksi, outlier dapat ditangani dengan beberapa cara:\n",
"1. **Hapus**: Buang baris yang mengandung outlier (jika itu adalah kesalahan)\n",
"2. **Batasi**: Ganti dengan nilai batas\n",
"3. **Ganti dengan NaN**: Perlakukan sebagai data yang hilang dan gunakan teknik imputasi\n",
"4. **Pertahankan**: Jika itu adalah nilai ekstrem yang sah\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a cleaned version by replacing impossible ages with NaN\n",
"dirty_data['age_clean'] = dirty_data['age'].apply(\n",
" lambda x: np.nan if (x < 0 or x > 120) else x\n",
")\n",
"\n",
"print(\"Age column before and after cleaning:\")\n",
"print(dirty_data[['customer_id', 'name', 'age', 'age_clean']])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 3. Mendeteksi Baris yang Hampir Duplikat\n",
"\n",
"Perhatikan bahwa dataset kita memiliki beberapa entri untuk \"John Smith\" dengan nilai yang sedikit berbeda. Mari kita identifikasi kemungkinan duplikat berdasarkan kesamaan nama.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# First, let's look at exact name matches (ignoring extra whitespace)\n",
"dirty_data['name_normalized'] = dirty_data['name'].str.strip().str.lower()\n",
"\n",
"print(\"Checking for duplicate names:\")\n",
"duplicate_names = dirty_data[dirty_data.duplicated(['name_normalized'], keep=False)]\n",
"print(duplicate_names.sort_values('name_normalized')[['customer_id', 'name', 'age', 'country']])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Menemukan Duplikat yang Mirip dengan Pencocokan Fuzzy\n",
"\n",
"Untuk deteksi duplikat yang lebih canggih, kita dapat menggunakan pencocokan fuzzy untuk menemukan nama yang serupa:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"try:\n",
" from rapidfuzz import process, fuzz\n",
"\n",
" # Function to find potential duplicates\n",
" def find_near_duplicates(df, column, threshold=90):\n",
" \"\"\"\n",
" Find near-duplicate entries in a column using fuzzy matching.\n",
" \n",
" Parameters:\n",
" - df: DataFrame\n",
" - column: Column name to check for duplicates\n",
" - threshold: Similarity threshold (0-100)\n",
" \n",
" Returns: List of potential duplicate groups\n",
" \"\"\"\n",
" values = df[column].unique()\n",
" duplicate_groups = []\n",
" checked = set()\n",
" \n",
" for value in values:\n",
" if value in checked:\n",
" continue\n",
" \n",
" # Find similar values\n",
" matches = process.extract(value, values, scorer=fuzz.ratio, limit=len(values))\n",
" similar = [m[0] for m in matches if m[1] >= threshold]\n",
" \n",
" if len(similar) > 1:\n",
" duplicate_groups.append(similar)\n",
" checked.update(similar)\n",
" \n",
" return duplicate_groups\n",
"\n",
" # Find near-duplicate names\n",
" duplicate_groups = find_near_duplicates(dirty_data, 'name', threshold=90)\n",
"\n",
" print(\"Potential duplicate groups:\")\n",
" for i, group in enumerate(duplicate_groups, 1):\n",
" print(f\"\\nGroup {i}:\")\n",
" for name in group:\n",
" matching_rows = dirty_data[dirty_data['name'] == name]\n",
" print(f\" '{name}': {len(matching_rows)} occurrence(s)\")\n",
" for _, row in matching_rows.iterrows():\n",
" print(f\" - Customer {row['customer_id']}: age={row['age']}, country={row['country']}\")\n",
"except ImportError:\n",
" print(\"rapidfuzz is not installed. Skipping fuzzy matching for near-duplicates.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Menangani Duplikasi\n",
"\n",
"Setelah diidentifikasi, Anda perlu memutuskan bagaimana menangani duplikasi:\n",
"1. **Simpan kemunculan pertama**: Gunakan `drop_duplicates(keep='first')`\n",
"2. **Simpan kemunculan terakhir**: Gunakan `drop_duplicates(keep='last')`\n",
"3. **Gabungkan informasi**: Kombinasikan informasi dari baris yang duplikat\n",
"4. **Tinjauan manual**: Tandai untuk ditinjau oleh manusia\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Example: Remove duplicates based on normalized name, keeping first occurrence\n",
"cleaned_data = dirty_data.drop_duplicates(subset=['name_normalized'], keep='first')\n",
"\n",
"print(f\"Original dataset: {len(dirty_data)} rows\")\n",
"print(f\"After removing name duplicates: {len(cleaned_data)} rows\")\n",
"print(f\"Removed: {len(dirty_data) - len(cleaned_data)} duplicate rows\")\n",
"\n",
"print(\"\\nCleaned dataset:\")\n",
"print(cleaned_data[['customer_id', 'name', 'age', 'country_clean']])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Ringkasan: Pipeline Pembersihan Data Lengkap\n",
"\n",
"Mari kita gabungkan semuanya ke dalam pipeline pembersihan yang komprehensif:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def clean_dataset(df):\n",
" \"\"\"\n",
" Comprehensive data cleaning function.\n",
" \"\"\"\n",
" # Create a copy to avoid modifying the original\n",
" cleaned = df.copy()\n",
" \n",
" # 1. Standardize categorical values (country)\n",
" country_mapping = {\n",
" 'usa': 'USA', 'u.s.a': 'USA', 'united states': 'USA',\n",
" 'uk': 'UK', 'united kingdom': 'UK',\n",
" 'canada': 'Canada', 'mexico': 'Mexico'\n",
" }\n",
" cleaned['country'] = cleaned['country'].str.lower().map(country_mapping)\n",
" \n",
" # 2. Clean abnormal age values\n",
" cleaned['age'] = cleaned['age'].apply(\n",
" lambda x: np.nan if (x < 0 or x > 120) else x\n",
" )\n",
" \n",
" # 3. Remove near-duplicate names (normalize whitespace)\n",
" cleaned['name'] = cleaned['name'].str.strip()\n",
" cleaned = cleaned.drop_duplicates(subset=['name'], keep='first')\n",
" \n",
" return cleaned\n",
"\n",
"# Apply the cleaning pipeline\n",
"final_cleaned_data = clean_dataset(dirty_data)\n",
"\n",
"print(\"Before cleaning:\")\n",
"print(f\" Rows: {len(dirty_data)}\")\n",
"print(f\" Unique countries: {dirty_data['country'].nunique()}\")\n",
"print(f\" Invalid ages: {((dirty_data['age'] < 0) | (dirty_data['age'] > 120)).sum()}\")\n",
"\n",
"print(\"\\nAfter cleaning:\")\n",
"print(f\" Rows: {len(final_cleaned_data)}\")\n",
"print(f\" Unique countries: {final_cleaned_data['country'].nunique()}\")\n",
"print(f\" Invalid ages: {((final_cleaned_data['age'] < 0) | (final_cleaned_data['age'] > 120)).sum()}\")\n",
"\n",
"print(\"\\nCleaned dataset:\")\n",
"print(final_cleaned_data[['customer_id', 'name', 'age', 'country', 'purchase_amount']])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 🎯 Latihan Tantangan\n",
"\n",
"Sekarang giliran Anda! Di bawah ini terdapat baris data baru dengan beberapa masalah kualitas. Bisakah Anda:\n",
"\n",
"1. Mengidentifikasi semua masalah dalam baris ini\n",
"2. Menulis kode untuk membersihkan setiap masalah\n",
"3. Menambahkan baris yang telah dibersihkan ke dalam dataset\n",
"\n",
"Berikut adalah data yang bermasalah:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# New problematic row\n",
"new_row = pd.DataFrame({\n",
" 'customer_id': [13],\n",
" 'name': [' Diana Prince '], # Extra whitespace\n",
" 'age': [250], # Impossible age\n",
" 'country': ['U.S.A.'], # Inconsistent format\n",
" 'purchase_amount': [150.00]\n",
"})\n",
"\n",
"print(\"New row to clean:\")\n",
"print(new_row)\n",
"\n",
"# TODO: Your code here to clean this row\n",
"# Hints:\n",
"# 1. Strip whitespace from the name\n",
"# 2. Check if the name is a duplicate (Diana Prince already exists)\n",
"# 3. Handle the impossible age value\n",
"# 4. Standardize the country name\n",
"\n",
"# Example solution (uncomment and modify as needed):\n",
"# new_row_cleaned = new_row.copy()\n",
"# new_row_cleaned['name'] = new_row_cleaned['name'].str.strip()\n",
"# new_row_cleaned['age'] = np.nan # Invalid age\n",
"# new_row_cleaned['country'] = 'USA' # Standardized\n",
"# print(\"\\nCleaned row:\")\n",
"# print(new_row_cleaned)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Poin Penting\n",
"\n",
"1. **Kategori yang tidak konsisten** sering ditemukan dalam data dunia nyata. Selalu periksa nilai unik dan standarisasi menggunakan pemetaan atau pencocokan fuzzy.\n",
"\n",
"2. **Outlier** dapat secara signifikan memengaruhi analisis Anda. Gunakan pengetahuan domain yang dikombinasikan dengan metode statistik (IQR, Z-score) untuk mendeteksinya.\n",
"\n",
"3. **Hampir duplikat** lebih sulit dideteksi dibandingkan duplikat yang persis sama. Pertimbangkan menggunakan pencocokan fuzzy dan normalisasi data (huruf kecil, menghapus spasi) untuk mengidentifikasinya.\n",
"\n",
"4. **Pembersihan data bersifat iteratif**. Anda mungkin perlu menerapkan beberapa teknik dan meninjau hasilnya sebelum menyelesaikan dataset yang telah dibersihkan.\n",
"\n",
"5. **Dokumentasikan keputusan Anda**. Catat langkah-langkah pembersihan yang Anda terapkan dan alasannya, karena ini penting untuk reproduktibilitas dan transparansi.\n",
"\n",
"> **Praktik Terbaik:** Selalu simpan salinan data \"kotor\" asli Anda. Jangan pernah menimpa file data sumber Anda - buat versi yang telah dibersihkan dengan konvensi penamaan yang jelas seperti `data_cleaned.csv`.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n---\n\n**Penafian**: \nDokumen ini telah diterjemahkan menggunakan layanan penerjemahan AI [Co-op Translator](https://github.com/Azure/co-op-translator). Meskipun kami berusaha untuk memberikan hasil yang akurat, harap diketahui bahwa terjemahan otomatis mungkin mengandung kesalahan atau ketidakakuratan. Dokumen asli dalam bahasa aslinya harus dianggap sebagai sumber yang otoritatif. Untuk informasi yang bersifat kritis, disarankan menggunakan jasa penerjemahan manusia profesional. Kami tidak bertanggung jawab atas kesalahpahaman atau interpretasi yang keliru yang timbul dari penggunaan terjemahan ini.\n"
]
}
],
"metadata": {
"anaconda-cloud": {},
"colab": {
"name": "notebook.ipynb",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.4"
},
"coopTranslator": {
"original_hash": "6301339d1c9a301b00639c635dc9b731",
"translation_date": "2025-10-03T20:39:23+00:00",
"source_file": "2-Working-With-Data/08-data-preparation/notebook.ipynb",
"language_code": "id"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Reference in new issue View Git Blame Copy Permalink