msb-public
/
Data-Science-For-Beginners
mirror of https://github.com/microsoft/Data-Science-For-Beginners.git
1
0
Code Issues Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'main'
${ noResults }
Data-Science-For-Beginners/translations/id/2-Working-With-Data/08-data-preparation/notebook.ipynb

3713 lines
104 KiB
Raw Permalink Blame History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "rQ8UhzFpgRra"
},
"source": [
"# Persiapan Data\n",
"\n",
"[Sumber Notebook asli dari *Data Science: Introduction to Machine Learning for Data Science Python and Machine Learning Studio oleh Lee Stott*](https://github.com/leestott/intro-Datascience/blob/master/Course%20Materials/4-Cleaning_and_Manipulating-Reference.ipynb)\n",
"\n",
"## Mengeksplorasi informasi `DataFrame`\n",
"\n",
"> **Tujuan pembelajaran:** Di akhir subbagian ini, Anda diharapkan merasa nyaman dalam menemukan informasi umum tentang data yang disimpan dalam pandas DataFrames.\n",
"\n",
"Setelah Anda memuat data ke dalam pandas, kemungkinan besar data tersebut akan berada dalam bentuk `DataFrame`. Namun, jika data dalam `DataFrame` Anda memiliki 60.000 baris dan 400 kolom, bagaimana Anda mulai memahami apa yang sedang Anda kerjakan? Untungnya, pandas menyediakan beberapa alat yang praktis untuk dengan cepat melihat informasi keseluruhan tentang sebuah `DataFrame`, selain beberapa baris pertama dan terakhir.\n",
"\n",
"Untuk mengeksplorasi fungsi ini, kita akan mengimpor pustaka Python scikit-learn dan menggunakan dataset ikonik yang hampir semua data scientist pernah lihat ratusan kali: dataset *Iris* milik ahli biologi Inggris Ronald Fisher yang digunakan dalam makalahnya tahun 1936 \"The use of multiple measurements in taxonomic problems\":\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true,
"id": "hB1RofhdgRrp",
"trusted": false
},
"outputs": [],
"source": [
"import pandas as pd\n",
"from sklearn.datasets import load_iris\n",
"\n",
"iris = load_iris()\n",
"iris_df = pd.DataFrame(data=iris['data'], columns=iris['feature_names'])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AGA0A_Y8hMdz"
},
"source": [
"### `DataFrame.shape`\n",
"Kita telah memuat Dataset Iris ke dalam variabel `iris_df`. Sebelum mendalami data, akan sangat berguna untuk mengetahui jumlah data yang kita miliki dan ukuran keseluruhan dataset. Penting untuk melihat seberapa besar volume data yang sedang kita tangani.\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "LOe5jQohhulf",
"outputId": "fb0577ac-3b4a-4623-cb41-20e1b264b3e9"
},
"outputs": [
{
"data": {
"text/plain": [
"(150, 4)"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris_df.shape"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "smE7AGzOhxk2"
},
"source": [
"Jadi, kita sedang mengolah 150 baris dan 4 kolom data. Setiap baris mewakili satu titik data dan setiap kolom mewakili satu fitur yang terkait dengan kerangka data. Jadi, pada dasarnya ada 150 titik data yang masing-masing memiliki 4 fitur.\n",
"\n",
"`shape` di sini adalah atribut dari kerangka data dan bukan sebuah fungsi, itulah sebabnya tidak diakhiri dengan sepasang tanda kurung.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d3AZKs0PinGP"
},
"source": [
"### `DataFrame.columns`\n",
"Sekarang mari kita bahas 4 kolom data. Apa sebenarnya yang diwakili oleh masing-masing kolom tersebut? Atribut `columns` akan memberikan kita nama-nama kolom dalam dataframe.\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "YPGh_ziji-CY",
"outputId": "74e7a43a-77cc-4c80-da56-7f50767c37a0"
},
"outputs": [
{
"data": {
"text/plain": [
"Index(['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)',\n",
" 'petal width (cm)'],\n",
" dtype='object')"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris_df.columns"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TsobcU_VjCC_"
},
"source": [
"Seperti yang dapat kita lihat, ada empat(4) kolom. Atribut `columns` memberi tahu kita nama kolom dan pada dasarnya tidak ada yang lain. Atribut ini menjadi penting ketika kita ingin mengidentifikasi fitur yang terdapat dalam dataset.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2UTlvkjmgRrs"
},
"source": [
"### `DataFrame.info`\n",
"Jumlah data (diberikan oleh atribut `shape`) dan nama fitur atau kolom (diberikan oleh atribut `columns`) memberikan informasi tentang dataset. Sekarang, kita ingin menggali lebih dalam tentang dataset. Fungsi `DataFrame.info()` sangat berguna untuk ini.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "dHHRyG0_gRrt",
"outputId": "d8fb0c40-4f18-4e19-da48-c8db77d1d3a5",
"trusted": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 150 entries, 0 to 149\n",
"Data columns (total 4 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 sepal length (cm) 150 non-null float64\n",
" 1 sepal width (cm) 150 non-null float64\n",
" 2 petal length (cm) 150 non-null float64\n",
" 3 petal width (cm) 150 non-null float64\n",
"dtypes: float64(4)\n",
"memory usage: 4.8 KB\n"
]
}
],
"source": [
"iris_df.info()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1XgVMpvigRru"
},
"source": [
"Dari sini, kita dapat membuat beberapa pengamatan: \n",
"1. Tipe Data dari setiap kolom: Dalam dataset ini, semua data disimpan sebagai angka floating-point 64-bit. \n",
"2. Jumlah nilai Non-Null: Menangani nilai null adalah langkah penting dalam persiapan data. Hal ini akan ditangani nanti di notebook. \n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "IYlyxbpWFEF4"
},
"source": [
"### DataFrame.describe()\n",
"Misalkan kita memiliki banyak data numerik dalam dataset kita. Perhitungan statistik univariat seperti rata-rata, median, kuartil, dll. dapat dilakukan pada setiap kolom secara individual. Fungsi `DataFrame.describe()` memberikan kita ringkasan statistik dari kolom-kolom numerik dalam sebuah dataset.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 297
},
"id": "tWV-CMstFIRA",
"outputId": "4fc49941-bc13-4b0c-a412-cb39e7d3f289"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sepal length (cm)</th>\n",
" <th>sepal width (cm)</th>\n",
" <th>petal length (cm)</th>\n",
" <th>petal width (cm)</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>count</th>\n",
" <td>150.000000</td>\n",
" <td>150.000000</td>\n",
" <td>150.000000</td>\n",
" <td>150.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>mean</th>\n",
" <td>5.843333</td>\n",
" <td>3.057333</td>\n",
" <td>3.758000</td>\n",
" <td>1.199333</td>\n",
" </tr>\n",
" <tr>\n",
" <th>std</th>\n",
" <td>0.828066</td>\n",
" <td>0.435866</td>\n",
" <td>1.765298</td>\n",
" <td>0.762238</td>\n",
" </tr>\n",
" <tr>\n",
" <th>min</th>\n",
" <td>4.300000</td>\n",
" <td>2.000000</td>\n",
" <td>1.000000</td>\n",
" <td>0.100000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25%</th>\n",
" <td>5.100000</td>\n",
" <td>2.800000</td>\n",
" <td>1.600000</td>\n",
" <td>0.300000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>50%</th>\n",
" <td>5.800000</td>\n",
" <td>3.000000</td>\n",
" <td>4.350000</td>\n",
" <td>1.300000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>75%</th>\n",
" <td>6.400000</td>\n",
" <td>3.300000</td>\n",
" <td>5.100000</td>\n",
" <td>1.800000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>max</th>\n",
" <td>7.900000</td>\n",
" <td>4.400000</td>\n",
" <td>6.900000</td>\n",
" <td>2.500000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sepal length (cm) sepal width (cm) petal length (cm) petal width (cm)\n",
"count 150.000000 150.000000 150.000000 150.000000\n",
"mean 5.843333 3.057333 3.758000 1.199333\n",
"std 0.828066 0.435866 1.765298 0.762238\n",
"min 4.300000 2.000000 1.000000 0.100000\n",
"25% 5.100000 2.800000 1.600000 0.300000\n",
"50% 5.800000 3.000000 4.350000 1.300000\n",
"75% 6.400000 3.300000 5.100000 1.800000\n",
"max 7.900000 4.400000 6.900000 2.500000"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris_df.describe()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zjjtW5hPGMuM"
},
"source": [
"Output di atas menunjukkan jumlah total titik data, rata-rata, standar deviasi, nilai minimum, kuartil bawah (25%), median (50%), kuartil atas (75%), dan nilai maksimum dari setiap kolom.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-lviAu99gRrv"
},
"source": [
"### `DataFrame.head`\n",
"Dengan semua fungsi dan atribut di atas, kita telah mendapatkan gambaran umum tentang dataset. Kita tahu berapa banyak data yang ada, berapa banyak fitur yang ada, tipe data dari setiap fitur, dan jumlah nilai non-null untuk setiap fitur.\n",
"\n",
"Sekarang saatnya melihat data itu sendiri. Mari kita lihat seperti apa beberapa baris pertama (beberapa titik data pertama) dari `DataFrame` kita:\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "DZMJZh0OgRrw",
"outputId": "d9393ee5-c106-4797-f815-218f17160e00",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sepal length (cm)</th>\n",
" <th>sepal width (cm)</th>\n",
" <th>petal length (cm)</th>\n",
" <th>petal width (cm)</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>5.1</td>\n",
" <td>3.5</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>4.9</td>\n",
" <td>3.0</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>4.7</td>\n",
" <td>3.2</td>\n",
" <td>1.3</td>\n",
" <td>0.2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>4.6</td>\n",
" <td>3.1</td>\n",
" <td>1.5</td>\n",
" <td>0.2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>5.0</td>\n",
" <td>3.6</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sepal length (cm) sepal width (cm) petal length (cm) petal width (cm)\n",
"0 5.1 3.5 1.4 0.2\n",
"1 4.9 3.0 1.4 0.2\n",
"2 4.7 3.2 1.3 0.2\n",
"3 4.6 3.1 1.5 0.2\n",
"4 5.0 3.6 1.4 0.2"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris_df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EBHEimZuEFQK"
},
"source": [
"Sebagai output di sini, kita dapat melihat lima(5) entri dari dataset. Jika kita melihat indeks di sebelah kiri, kita menemukan bahwa ini adalah lima baris pertama.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oj7GkrTdgRry"
},
"source": [
"### Latihan:\n",
"\n",
"Dari contoh yang diberikan di atas, jelas bahwa secara default, `DataFrame.head` mengembalikan lima baris pertama dari sebuah `DataFrame`. Pada sel kode di bawah ini, bisakah Anda menemukan cara untuk menampilkan lebih dari lima baris?\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": true,
"id": "EKRmRFFegRrz",
"trusted": false
},
"outputs": [],
"source": [
"# Hint: Consult the documentation by using iris_df.head?"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BJ_cpZqNgRr1"
},
"source": [
"### `DataFrame.tail`\n",
"Cara lain untuk melihat data adalah dari bagian akhir (bukan dari awal). Kebalikan dari `DataFrame.head` adalah `DataFrame.tail`, yang mengembalikan lima baris terakhir dari sebuah `DataFrame`:\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 0
},
"id": "heanjfGWgRr2",
"outputId": "6ae09a21-fe09-4110-b0d7-1a1fbf34d7f3",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sepal length (cm)</th>\n",
" <th>sepal width (cm)</th>\n",
" <th>petal length (cm)</th>\n",
" <th>petal width (cm)</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>145</th>\n",
" <td>6.7</td>\n",
" <td>3.0</td>\n",
" <td>5.2</td>\n",
" <td>2.3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>146</th>\n",
" <td>6.3</td>\n",
" <td>2.5</td>\n",
" <td>5.0</td>\n",
" <td>1.9</td>\n",
" </tr>\n",
" <tr>\n",
" <th>147</th>\n",
" <td>6.5</td>\n",
" <td>3.0</td>\n",
" <td>5.2</td>\n",
" <td>2.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>148</th>\n",
" <td>6.2</td>\n",
" <td>3.4</td>\n",
" <td>5.4</td>\n",
" <td>2.3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>149</th>\n",
" <td>5.9</td>\n",
" <td>3.0</td>\n",
" <td>5.1</td>\n",
" <td>1.8</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sepal length (cm) sepal width (cm) petal length (cm) petal width (cm)\n",
"145 6.7 3.0 5.2 2.3\n",
"146 6.3 2.5 5.0 1.9\n",
"147 6.5 3.0 5.2 2.0\n",
"148 6.2 3.4 5.4 2.3\n",
"149 5.9 3.0 5.1 1.8"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris_df.tail()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "31kBWfyLgRr3"
},
"source": [
"Dalam praktiknya, sangat berguna untuk dapat dengan mudah memeriksa beberapa baris pertama atau beberapa baris terakhir dari sebuah `DataFrame`, terutama ketika Anda mencari nilai-nilai yang tidak biasa dalam dataset yang terurut.\n",
"\n",
"Semua fungsi dan atribut yang ditunjukkan di atas dengan bantuan contoh kode membantu kita mendapatkan gambaran dan pemahaman tentang data.\n",
"\n",
"> **Kesimpulan:** Bahkan hanya dengan melihat metadata tentang informasi dalam sebuah DataFrame atau beberapa nilai pertama dan terakhir di dalamnya, Anda dapat langsung mendapatkan gambaran tentang ukuran, bentuk, dan isi data yang sedang Anda hadapi.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TvurZyLSDxq_"
},
"source": [
"### Data yang Hilang\n",
"Mari kita bahas tentang data yang hilang. Data yang hilang terjadi ketika tidak ada nilai yang disimpan di beberapa kolom.\n",
"\n",
"Mari kita ambil contoh: misalnya seseorang sangat peduli dengan berat badannya dan tidak mengisi kolom berat badan dalam sebuah survei. Maka, nilai berat badan untuk orang tersebut akan hilang.\n",
"\n",
"Sebagian besar waktu, dalam dataset dunia nyata, nilai yang hilang sering terjadi.\n",
"\n",
"**Bagaimana Pandas Menangani Data yang Hilang**\n",
"\n",
"Pandas menangani nilai yang hilang dengan dua cara. Cara pertama sudah pernah Anda lihat di bagian sebelumnya: `NaN`, atau Not a Number. Ini sebenarnya adalah nilai khusus yang merupakan bagian dari spesifikasi floating-point IEEE dan hanya digunakan untuk menunjukkan nilai floating-point yang hilang.\n",
"\n",
"Untuk nilai yang hilang selain tipe float, pandas menggunakan objek Python `None`. Meskipun mungkin terlihat membingungkan bahwa Anda akan menemukan dua jenis nilai yang pada dasarnya mengatakan hal yang sama, ada alasan programatik yang kuat untuk pilihan desain ini dan, dalam praktiknya, pendekatan ini memungkinkan pandas memberikan kompromi yang baik untuk sebagian besar kasus. Meskipun demikian, baik `None` maupun `NaN` memiliki batasan yang perlu Anda perhatikan terkait bagaimana mereka dapat digunakan.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lOHqUlZFgRr5"
},
"source": [
"### `None`: data hilang non-float\n",
"Karena `None` berasal dari Python, nilai ini tidak dapat digunakan dalam array NumPy dan pandas yang bukan bertipe data `'object'`. Ingatlah, array NumPy (dan struktur data dalam pandas) hanya dapat berisi satu jenis data. Hal ini memberikan kekuatan besar untuk pekerjaan data dan komputasi berskala besar, tetapi juga membatasi fleksibilitasnya. Array semacam itu harus diubah ke \"denominator terendah,\" yaitu tipe data yang dapat mencakup semua elemen dalam array. Ketika `None` ada di dalam array, itu berarti Anda sedang bekerja dengan objek Python.\n",
"\n",
"Untuk melihat ini dalam praktik, perhatikan contoh array berikut (perhatikan `dtype` untuk array tersebut):\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "QIoNdY4ngRr7",
"outputId": "92779f18-62f4-4a03-eca2-e9a101604336",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"array([2, None, 6, 8], dtype=object)"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import numpy as np\n",
"\n",
"example1 = np.array([2, None, 6, 8])\n",
"example1"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pdlgPNbhgRr7"
},
"source": [
"Realitas dari tipe data upcast membawa dua efek samping bersamanya. Pertama, operasi akan dilakukan pada tingkat kode Python yang diinterpretasikan daripada kode NumPy yang dikompilasi. Pada dasarnya, ini berarti bahwa operasi apa pun yang melibatkan `Series` atau `DataFrame` dengan `None` di dalamnya akan lebih lambat. Meskipun Anda mungkin tidak menyadari penurunan kinerja ini, untuk dataset yang besar, hal ini bisa menjadi masalah.\n",
"\n",
"Efek samping kedua berasal dari yang pertama. Karena `None` pada dasarnya menarik `Series` atau `DataFrame` kembali ke dunia Python biasa, menggunakan agregasi NumPy/pandas seperti `sum()` atau `min()` pada array yang mengandung nilai ``None`` umumnya akan menghasilkan error:\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 292
},
"id": "gWbx-KB9gRr8",
"outputId": "ecba710a-22ec-41d5-a39c-11f67e645b50",
"trusted": false
},
"outputs": [
{
"ename": "TypeError",
"evalue": "ignored",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-10-ce9901ad18bd>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mexample1\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m/usr/local/lib/python3.7/dist-packages/numpy/core/_methods.py\u001b[0m in \u001b[0;36m_sum\u001b[0;34m(a, axis, dtype, out, keepdims, initial, where)\u001b[0m\n\u001b[1;32m 45\u001b[0m def _sum(a, axis=None, dtype=None, out=None, keepdims=False,\n\u001b[1;32m 46\u001b[0m initial=_NoValue, where=True):\n\u001b[0;32m---> 47\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mumr_sum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkeepdims\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minitial\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwhere\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 48\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 49\u001b[0m def _prod(a, axis=None, dtype=None, out=None, keepdims=False,\n",
"\u001b[0;31mTypeError\u001b[0m: unsupported operand type(s) for +: 'int' and 'NoneType'"
]
}
],
"source": [
"example1.sum()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LcEwO8UogRr9"
},
"source": [
"**Kesimpulan utama**: Penjumlahan (dan operasi lainnya) antara bilangan bulat dan nilai `None` tidak terdefinisi, yang dapat membatasi apa yang dapat Anda lakukan dengan kumpulan data yang mengandungnya.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pWvVHvETgRr9"
},
"source": [
"### `NaN`: nilai float yang hilang\n",
"\n",
"Berbeda dengan `None`, NumPy (dan oleh karena itu pandas) mendukung `NaN` untuk operasi vektorisasi dan ufuncs yang cepat. Berita buruknya adalah bahwa setiap operasi aritmatika yang dilakukan pada `NaN` selalu menghasilkan `NaN`. Sebagai contoh:\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "rcFYfMG9gRr9",
"outputId": "699e81b7-5c11-4b46-df1d-06071768690f",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"nan"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.nan + 1"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "BW3zQD2-gRr-",
"outputId": "4525b6c4-495d-4f7b-a979-efce1dae9bd0",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"nan"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.nan * 0"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fU5IPRcCgRr-"
},
"source": [
"Kabar baiknya: agregasi yang dijalankan pada array dengan `NaN` di dalamnya tidak menghasilkan error. Kabar buruknya: hasilnya tidak selalu berguna:\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "LCInVgSSgRr_",
"outputId": "fa06495a-0930-4867-87c5-6023031ea8b5",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"(nan, nan, nan)"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example2 = np.array([2, np.nan, 6, 8]) \n",
"example2.sum(), example2.min(), example2.max()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nhlnNJT7gRr_"
},
"source": [
"### Latihan:\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": true,
"id": "yan3QRaOgRr_",
"trusted": false
},
"outputs": [],
"source": [
"# What happens if you add np.nan and None together?\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_iDvIRC8gRsA"
},
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "kj6EKdsAgRsA"
},
"source": [
"### `NaN` dan `None`: nilai null dalam pandas\n",
"\n",
"Meskipun `NaN` dan `None` dapat berperilaku sedikit berbeda, pandas tetap dirancang untuk menangani keduanya secara bergantian. Untuk memahami maksudnya, perhatikan sebuah `Series` dari bilangan bulat:\n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Nji-KGdNgRsA",
"outputId": "36aa14d2-8efa-4bfd-c0ed-682991288822",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"0 1\n",
"1 2\n",
"2 3\n",
"dtype: int64"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"int_series = pd.Series([1, 2, 3], dtype=int)\n",
"int_series"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WklCzqb8gRsB"
},
"source": [
"### Latihan:\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": true,
"id": "Cy-gqX5-gRsB",
"trusted": false
},
"outputs": [],
"source": [
"# Now set an element of int_series equal to None.\n",
"# How does that element show up in the Series?\n",
"# What is the dtype of the Series?\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WjMQwltNgRsB"
},
"source": [
"Dalam proses menaikkan tipe data untuk memastikan keseragaman data dalam `Series` dan `DataFrame`, pandas dengan mudah akan mengganti nilai yang hilang antara `None` dan `NaN`. Karena fitur desain ini, akan membantu jika kita memandang `None` dan `NaN` sebagai dua jenis \"null\" yang berbeda dalam pandas. Memang, beberapa metode inti yang akan Anda gunakan untuk menangani nilai yang hilang di pandas mencerminkan gagasan ini dalam nama mereka:\n",
"\n",
"- `isnull()`: Menghasilkan masker Boolean yang menunjukkan nilai yang hilang\n",
"- `notnull()`: Kebalikan dari `isnull()`\n",
"- `dropna()`: Mengembalikan versi data yang telah difilter\n",
"- `fillna()`: Mengembalikan salinan data dengan nilai yang hilang diisi atau diimputasi\n",
"\n",
"Metode-metode ini penting untuk dikuasai dan dipahami dengan baik, jadi mari kita bahas masing-masing secara mendalam.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Yh5ifd9FgRsB"
},
"source": [
"### Mendeteksi nilai null\n",
"\n",
"Sekarang setelah kita memahami pentingnya nilai yang hilang, kita perlu mendeteksinya dalam dataset kita sebelum menanganinya. \n",
"Baik `isnull()` maupun `notnull()` adalah metode utama Anda untuk mendeteksi data null. Keduanya mengembalikan masker Boolean pada data Anda.\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": true,
"id": "e-vFp5lvgRsC",
"trusted": false
},
"outputs": [],
"source": [
"example3 = pd.Series([0, np.nan, '', None])"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "1XdaJJ7PgRsC",
"outputId": "92fc363a-1874-471f-846d-f4f9ce1f51d0",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"0 False\n",
"1 True\n",
"2 False\n",
"3 True\n",
"dtype: bool"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example3.isnull()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PaSZ0SQygRsC"
},
"source": [
"Perhatikan baik-baik hasilnya. Apakah ada yang mengejutkan Anda? Meskipun `0` adalah nilai nol dalam aritmatika, itu tetap merupakan bilangan bulat yang valid, dan pandas memperlakukannya seperti itu. `''` sedikit lebih halus. Meskipun kita menggunakannya di Bagian 1 untuk merepresentasikan nilai string kosong, itu tetap merupakan objek string dan bukan representasi null sejauh yang pandas ketahui.\n",
"\n",
"Sekarang, mari kita balikkan situasi ini dan gunakan metode-metode ini dengan cara yang lebih mirip dengan bagaimana Anda akan menggunakannya dalam praktik. Anda dapat menggunakan masker Boolean langsung sebagai indeks ``Series`` atau ``DataFrame``, yang bisa berguna saat mencoba bekerja dengan nilai yang hilang (atau ada) secara terisolasi.\n",
"\n",
"Jika kita ingin jumlah total nilai yang hilang, kita cukup melakukan penjumlahan pada masker yang dihasilkan oleh metode `isnull()`.\n"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "JCcQVoPkHDUv",
"outputId": "001daa72-54f8-4bd5-842a-4df627a79d4d"
},
"outputs": [
{
"data": {
"text/plain": [
"2"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example3.isnull().sum()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PlBqEo3mgRsC"
},
"source": [
"### Latihan:\n"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"collapsed": true,
"id": "ggDVf5uygRsD",
"trusted": false
},
"outputs": [],
"source": [
"# Try running example3[example3.notnull()].\n",
"# Before you do so, what do you expect to see?\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "D_jWN7mHgRsD"
},
"source": [
"**Kesimpulan utama**: Baik metode `isnull()` maupun `notnull()` menghasilkan hasil yang serupa ketika Anda menggunakannya dalam DataFrame: mereka menunjukkan hasil dan indeks dari hasil tersebut, yang akan sangat membantu Anda saat Anda mengolah data Anda.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BvnoojWsgRr4"
},
"source": [
"### Menangani Data yang Hilang\n",
"\n",
"> **Tujuan pembelajaran:** Di akhir subbagian ini, Anda seharusnya memahami bagaimana dan kapan mengganti atau menghapus nilai null dari DataFrame.\n",
"\n",
"Model Machine Learning tidak dapat menangani data yang hilang secara langsung. Oleh karena itu, sebelum data dimasukkan ke dalam model, kita perlu menangani nilai-nilai yang hilang ini.\n",
"\n",
"Cara menangani data yang hilang memiliki kompromi yang halus, dapat memengaruhi analisis akhir Anda dan hasil di dunia nyata.\n",
"\n",
"Ada dua cara utama untuk menangani data yang hilang:\n",
"\n",
"1. Menghapus baris yang mengandung nilai yang hilang\n",
"2. Mengganti nilai yang hilang dengan nilai lain\n",
"\n",
"Kita akan membahas kedua metode ini beserta kelebihan dan kekurangannya secara mendetail.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3VaYC1TvgRsD"
},
"source": [
"### Menghapus nilai null\n",
"\n",
"Jumlah data yang kita masukkan ke model memiliki pengaruh langsung terhadap kinerjanya. Menghapus nilai null berarti kita mengurangi jumlah data, sehingga mengurangi ukuran dataset. Oleh karena itu, disarankan untuk menghapus baris dengan nilai null ketika dataset cukup besar.\n",
"\n",
"Contoh lainnya adalah ketika sebuah baris atau kolom tertentu memiliki banyak nilai yang hilang. Dalam kasus ini, baris atau kolom tersebut mungkin dihapus karena tidak akan memberikan banyak nilai tambah pada analisis kita, mengingat sebagian besar datanya hilang.\n",
"\n",
"Selain mengidentifikasi nilai yang hilang, pandas menyediakan cara yang praktis untuk menghapus nilai null dari `Series` dan `DataFrame`. Untuk melihat ini dalam praktik, mari kita kembali ke `example3`. Fungsi `DataFrame.dropna()` membantu menghapus baris dengan nilai null.\n"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "7uIvS097gRsD",
"outputId": "c13fc117-4ca1-4145-a0aa-42ac89e6e218",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"0 0\n",
"2 \n",
"dtype: object"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example3 = example3.dropna()\n",
"example3"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hil2cr64gRsD"
},
"source": [
"Perhatikan bahwa ini seharusnya terlihat seperti output dari `example3[example3.notnull()]`. Perbedaannya di sini adalah, alih-alih hanya melakukan pengindeksan pada nilai yang dimask, `dropna` telah menghapus nilai yang hilang dari `Series` `example3`.\n",
"\n",
"Karena DataFrame memiliki dua dimensi, mereka memberikan lebih banyak opsi untuk menghapus data.\n"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "an-l74sPgRsE",
"outputId": "340876a0-63ad-40f6-bd54-6240cdae50ab",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>6.0</td>\n",
" <td>9</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 1.0 NaN 7\n",
"1 2.0 5.0 8\n",
"2 NaN 6.0 9"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4 = pd.DataFrame([[1, np.nan, 7], \n",
" [2, 5, 8], \n",
" [np.nan, 6, 9]])\n",
"example4"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "66wwdHZrgRsE"
},
"source": [
"(Apakah Anda memperhatikan bahwa pandas mengubah tipe dua kolom menjadi float untuk mengakomodasi `NaN`?)\n",
"\n",
"Anda tidak dapat menghapus satu nilai saja dari sebuah `DataFrame`, jadi Anda harus menghapus seluruh baris atau kolom. Tergantung pada apa yang Anda lakukan, Anda mungkin ingin memilih salah satu dari keduanya, dan karena itu pandas memberikan opsi untuk keduanya. Karena dalam ilmu data, kolom umumnya mewakili variabel dan baris mewakili observasi, Anda lebih mungkin untuk menghapus baris data; pengaturan default untuk `dropna()` adalah menghapus semua baris yang mengandung nilai null:\n"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 80
},
"id": "jAVU24RXgRsE",
"outputId": "0b5e5aee-7187-4d3f-b583-a44136ae5f80",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"1 2.0 5.0 8"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4.dropna()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TrQRBuTDgRsE"
},
"source": [
"Jika perlu, Anda dapat menghapus nilai NA dari kolom. Gunakan `axis=1` untuk melakukannya:\n"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "GrBhxu9GgRsE",
"outputId": "ff4001f3-2e61-4509-d60e-0093d1068437",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>8</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>9</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 2\n",
"0 7\n",
"1 8\n",
"2 9"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4.dropna(axis='columns')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KWXiKTfMgRsF"
},
"source": [
"Perhatikan bahwa ini dapat menghilangkan banyak data yang mungkin ingin Anda simpan, terutama dalam dataset yang lebih kecil. Bagaimana jika Anda hanya ingin menghapus baris atau kolom yang mengandung beberapa atau bahkan semua nilai null? Anda dapat menentukan pengaturan tersebut dalam `dropna` dengan parameter `how` dan `thresh`.\n",
"\n",
"Secara default, `how='any'` (jika Anda ingin memeriksa sendiri atau melihat parameter lain yang dimiliki metode ini, jalankan `example4.dropna?` dalam sel kode). Sebagai alternatif, Anda dapat menentukan `how='all'` untuk hanya menghapus baris atau kolom yang mengandung semua nilai null. Mari kita perluas contoh `DataFrame` kita untuk melihat ini dalam aksi pada latihan berikutnya.\n"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "Bcf_JWTsgRsF",
"outputId": "72e0b1b8-52fa-4923-98ce-b6fbed6e44b1",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>7</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>6.0</td>\n",
" <td>9</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"0 1.0 NaN 7 NaN\n",
"1 2.0 5.0 8 NaN\n",
"2 NaN 6.0 9 NaN"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4[3] = np.nan\n",
"example4"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pNZer7q9JPNC"
},
"source": [
"> Poin-poin penting: \n",
"1. Menghapus nilai null adalah ide yang baik hanya jika dataset cukup besar. \n",
"2. Baris atau kolom penuh dapat dihapus jika sebagian besar datanya hilang. \n",
"3. Metode `DataFrame.dropna(axis=)` membantu dalam menghapus nilai null. Argumen `axis` menunjukkan apakah baris atau kolom yang akan dihapus. \n",
"4. Argumen `how` juga dapat digunakan. Secara default, nilainya diatur ke `any`. Jadi, hanya baris/kolom yang mengandung nilai null yang akan dihapus. Nilai ini dapat diatur ke `all` untuk menentukan bahwa kita hanya akan menghapus baris/kolom di mana semua nilainya null. \n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oXXSfQFHgRsF"
},
"source": [
"### Latihan:\n"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"collapsed": true,
"id": "ExUwQRxpgRsF",
"trusted": false
},
"outputs": [],
"source": [
"# How might you go about dropping just column 3?\n",
"# Hint: remember that you will need to supply both the axis parameter and the how parameter.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "38kwAihWgRsG"
},
"source": [
"Parameter `thresh` memberi Anda kontrol yang lebih rinci: Anda menetapkan jumlah nilai *non-null* yang harus dimiliki oleh sebuah baris atau kolom agar tetap dipertahankan:\n"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 80
},
"id": "M9dCNMaagRsG",
"outputId": "8093713a-54d2-4e54-c73f-4eea315cb6f2",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"1 2.0 5.0 8 NaN"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4.dropna(axis='rows', thresh=3)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fmSFnzZegRsG"
},
"source": [
"Di sini, baris pertama dan terakhir telah dihapus, karena hanya berisi dua nilai non-null.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mCcxLGyUgRsG"
},
"source": [
"### Mengisi Nilai Null\n",
"\n",
"Terkadang masuk akal untuk mengisi nilai yang hilang dengan nilai yang mungkin valid. Ada beberapa teknik untuk mengisi nilai null. Yang pertama adalah menggunakan Pengetahuan Domain (pengetahuan tentang subjek yang menjadi dasar dataset) untuk memperkirakan nilai yang hilang.\n",
"\n",
"Anda bisa menggunakan `isnull` untuk melakukan ini secara langsung, tetapi itu bisa melelahkan, terutama jika Anda memiliki banyak nilai yang harus diisi. Karena ini adalah tugas yang sangat umum dalam data science, pandas menyediakan `fillna`, yang mengembalikan salinan `Series` atau `DataFrame` dengan nilai yang hilang diganti dengan nilai pilihan Anda. Mari kita buat contoh `Series` lain untuk melihat bagaimana ini bekerja dalam praktik.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CE8S7louLezV"
},
"source": [
"### Data Kategorikal (Non-numerik)\n",
"Pertama, mari kita bahas data non-numerik. Dalam dataset, kita memiliki kolom dengan data kategorikal. Contohnya: Jenis Kelamin, Benar atau Salah, dll.\n",
"\n",
"Dalam sebagian besar kasus ini, kita mengganti nilai yang hilang dengan `modus` dari kolom tersebut. Misalnya, kita memiliki 100 data, di mana 90 mengatakan Benar, 8 mengatakan Salah, dan 2 tidak diisi. Maka, kita dapat mengisi 2 data yang kosong dengan Benar, dengan mempertimbangkan keseluruhan kolom.\n",
"\n",
"Sekali lagi, di sini kita juga dapat menggunakan pengetahuan domain. Mari kita lihat contoh pengisian dengan modus.\n"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "MY5faq4yLdpQ",
"outputId": "19ab472e-1eed-4de8-f8a7-db2a3af3cb1a"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" <td>None</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>5</td>\n",
" <td>6</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>7</td>\n",
" <td>8</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>9</td>\n",
" <td>10</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 1 2 True\n",
"1 3 4 None\n",
"2 5 6 False\n",
"3 7 8 True\n",
"4 9 10 True"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_mode = pd.DataFrame([[1,2,\"True\"],\n",
" [3,4,None],\n",
" [5,6,\"False\"],\n",
" [7,8,\"True\"],\n",
" [9,10,\"True\"]])\n",
"\n",
"fill_with_mode"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MLAoMQOfNPlA"
},
"source": []
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "WKy-9Y2tN5jv",
"outputId": "8da9fa16-e08c-447e-dea1-d4b1db2feebf"
},
"outputs": [
{
"data": {
"text/plain": [
"True 3\n",
"False 1\n",
"Name: 2, dtype: int64"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_mode[2].value_counts()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6iNz_zG_OKrx"
},
"source": []
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"id": "TxPKteRvNPOs"
},
"outputs": [],
"source": [
"fill_with_mode[2].fillna('True',inplace=True)"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "tvas7c9_OPWE",
"outputId": "ec3c8e44-d644-475e-9e22-c65101965850"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>5</td>\n",
" <td>6</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>7</td>\n",
" <td>8</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>9</td>\n",
" <td>10</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 1 2 True\n",
"1 3 4 True\n",
"2 5 6 False\n",
"3 7 8 True\n",
"4 9 10 True"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_mode"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SktitLxxOR16"
},
"source": [
"Seperti yang kita lihat, nilai null telah digantikan. Tidak perlu dikatakan, kita bisa menulis apa saja sebagai pengganti `'True'` dan itu akan tersubstitusi.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "heYe1I0dOmQ_"
},
"source": [
"### Data Numerik\n",
"Sekarang, mari kita bahas tentang data numerik. Di sini, ada dua cara umum untuk menggantikan nilai yang hilang:\n",
"\n",
"1. Mengganti dengan Median dari baris\n",
"2. Mengganti dengan Mean dari baris\n",
"\n",
"Kita mengganti dengan Median jika datanya memiliki distribusi yang miring dengan outlier. Hal ini karena median lebih tahan terhadap outlier.\n",
"\n",
"Ketika data sudah dinormalisasi, kita dapat menggunakan mean, karena dalam kasus tersebut, mean dan median akan cukup dekat.\n",
"\n",
"Pertama, mari kita ambil sebuah kolom yang terdistribusi normal dan isi nilai yang hilang dengan mean dari kolom tersebut.\n"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "09HM_2feOj5Y",
"outputId": "7e309013-9acb-411c-9b06-4de795bbeeff"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>-2.0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>-1.0</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1.0</td>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2.0</td>\n",
" <td>8</td>\n",
" <td>9</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 -2.0 0 1\n",
"1 -1.0 2 3\n",
"2 NaN 4 5\n",
"3 1.0 6 7\n",
"4 2.0 8 9"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_mean = pd.DataFrame([[-2,0,1],\n",
" [-1,2,3],\n",
" [np.nan,4,5],\n",
" [1,6,7],\n",
" [2,8,9]])\n",
"\n",
"fill_with_mean"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ka7-wNfzSxbx"
},
"source": [
"Rata-rata dari kolom adalah\n"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "XYtYEf5BSxFL",
"outputId": "68a78d18-f0e5-4a9a-a959-2c3676a57c70"
},
"outputs": [
{
"data": {
"text/plain": [
"0.0"
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.mean(fill_with_mean[0])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oBSRGxKRS39K"
},
"source": []
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "FzncQLmuS5jh",
"outputId": "00f74fff-01f4-4024-c261-796f50f01d2e"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>-2.0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>-1.0</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>0.0</td>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1.0</td>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2.0</td>\n",
" <td>8</td>\n",
" <td>9</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 -2.0 0 1\n",
"1 -1.0 2 3\n",
"2 0.0 4 5\n",
"3 1.0 6 7\n",
"4 2.0 8 9"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_mean[0].fillna(np.mean(fill_with_mean[0]),inplace=True)\n",
"fill_with_mean"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CwpVFCrPTC5z"
},
"source": [
"Seperti yang dapat kita lihat, nilai yang hilang telah digantikan dengan rata-ratanya.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jIvF13a1i00Z"
},
"source": [
"Sekarang mari kita coba dataframe lain, dan kali ini kita akan mengganti nilai None dengan median dari kolom tersebut.\n"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "DA59Bqo3jBYZ",
"outputId": "85dae6ec-7394-4c36-fda0-e04769ec4a32"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>-2</td>\n",
" <td>0.0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>-1</td>\n",
" <td>2.0</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>0</td>\n",
" <td>NaN</td>\n",
" <td>5</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1</td>\n",
" <td>6.0</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2</td>\n",
" <td>8.0</td>\n",
" <td>9</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 -2 0.0 1\n",
"1 -1 2.0 3\n",
"2 0 NaN 5\n",
"3 1 6.0 7\n",
"4 2 8.0 9"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_median = pd.DataFrame([[-2,0,1],\n",
" [-1,2,3],\n",
" [0,np.nan,5],\n",
" [1,6,7],\n",
" [2,8,9]])\n",
"\n",
"fill_with_median"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mM1GpXYmjHnc"
},
"source": [
"Median dari kolom kedua adalah\n"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "uiDy5v3xjHHX",
"outputId": "564b6b74-2004-4486-90d4-b39330a64b88"
},
"outputs": [
{
"data": {
"text/plain": [
"4.0"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_median[1].median()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "z9PLF75Jj_1s"
},
"source": []
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "lFKbOxCMkBbg",
"outputId": "a8bd18fb-2765-47d4-e5fe-e965f57ed1f4"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>-2</td>\n",
" <td>0.0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>-1</td>\n",
" <td>2.0</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>0</td>\n",
" <td>4.0</td>\n",
" <td>5</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1</td>\n",
" <td>6.0</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2</td>\n",
" <td>8.0</td>\n",
" <td>9</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 -2 0.0 1\n",
"1 -1 2.0 3\n",
"2 0 4.0 5\n",
"3 1 6.0 7\n",
"4 2 8.0 9"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fill_with_median[1].fillna(fill_with_median[1].median(),inplace=True)\n",
"fill_with_median"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8JtQ53GSkKWC"
},
"source": [
"Seperti yang dapat kita lihat, nilai NaN telah digantikan oleh median dari kolom\n"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "0ybtWLDdgRsG",
"outputId": "b8c238ef-6024-4ee2-be2b-aa1f0fcac61d",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"a 1.0\n",
"b NaN\n",
"c 2.0\n",
"d NaN\n",
"e 3.0\n",
"dtype: float64"
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example5 = pd.Series([1, np.nan, 2, None, 3], index=list('abcde'))\n",
"example5"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yrsigxRggRsH"
},
"source": [
"Anda dapat mengisi semua entri null dengan satu nilai, seperti `0`:\n"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "KXMIPsQdgRsH",
"outputId": "aeedfa0a-a421-4c2f-cb0d-183ce8f0c91d",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"a 1.0\n",
"b 0.0\n",
"c 2.0\n",
"d 0.0\n",
"e 3.0\n",
"dtype: float64"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example5.fillna(0)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RRlI5f_hkfKe"
},
"source": [
"> Poin-poin penting:\n",
"1. Mengisi nilai yang hilang sebaiknya dilakukan ketika data yang tersedia sedikit atau ada strategi untuk mengisi data yang hilang.\n",
"2. Pengetahuan domain dapat digunakan untuk mengisi nilai yang hilang dengan cara memperkirakannya.\n",
"3. Untuk data kategorikal, biasanya nilai yang hilang digantikan dengan modus dari kolom tersebut.\n",
"4. Untuk data numerik, nilai yang hilang biasanya diisi dengan rata-rata (untuk dataset yang sudah dinormalisasi) atau median dari kolom tersebut.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FI9MmqFJgRsH"
},
"source": [
"### Latihan:\n"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {
"collapsed": true,
"id": "af-ezpXdgRsH",
"trusted": false
},
"outputs": [],
"source": [
"# What happens if you try to fill null values with a string, like ''?\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kq3hw1kLgRsI"
},
"source": [
"Anda dapat **forward-fill** nilai null, yaitu menggunakan nilai valid terakhir untuk mengisi null:\n"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "vO3BuNrggRsI",
"outputId": "e2bc591b-0b48-4e88-ee65-754f2737c196",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"a 1.0\n",
"b 1.0\n",
"c 2.0\n",
"d 2.0\n",
"e 3.0\n",
"dtype: float64"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example5.fillna(method='ffill')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nDXeYuHzgRsI"
},
"source": [
"Anda juga dapat **mengisi kembali** untuk menyebarkan nilai valid berikutnya ke belakang guna mengisi null:\n"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "4M5onHcEgRsI",
"outputId": "8f32b185-40dd-4a9f-bd85-54d6b6a414fe",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"a 1.0\n",
"b 2.0\n",
"c 2.0\n",
"d 3.0\n",
"e 3.0\n",
"dtype: float64"
]
},
"execution_count": 42,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example5.fillna(method='bfill')"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"id": "MbBzTom5gRsI"
},
"source": [
"Seperti yang mungkin Anda tebak, ini bekerja dengan cara yang sama pada DataFrame, tetapi Anda juga dapat menentukan `axis` untuk mengisi nilai null:\n"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "aRpIvo4ZgRsI",
"outputId": "905a980a-a808-4eca-d0ba-224bd7d85955",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>7</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>6.0</td>\n",
" <td>9</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"0 1.0 NaN 7 NaN\n",
"1 2.0 5.0 8 NaN\n",
"2 NaN 6.0 9 NaN"
]
},
"execution_count": 43,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "VM1qtACAgRsI",
"outputId": "71f2ad28-9b4e-4ff4-f5c3-e731eb489ade",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>7.0</td>\n",
" <td>7.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8.0</td>\n",
" <td>8.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>6.0</td>\n",
" <td>9.0</td>\n",
" <td>9.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"0 1.0 1.0 7.0 7.0\n",
"1 2.0 5.0 8.0 8.0\n",
"2 NaN 6.0 9.0 9.0"
]
},
"execution_count": 44,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4.fillna(method='ffill', axis=1)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZeMc-I1EgRsI"
},
"source": [
"Perhatikan bahwa ketika nilai sebelumnya tidak tersedia untuk pengisian ke depan, nilai null tetap ada.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eeAoOU0RgRsJ"
},
"source": [
"### Latihan:\n"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {
"collapsed": true,
"id": "e8S-CjW8gRsJ",
"trusted": false
},
"outputs": [],
"source": [
"# What output does example4.fillna(method='bfill', axis=1) produce?\n",
"# What about example4.fillna(method='ffill') or example4.fillna(method='bfill')?\n",
"# Can you think of a longer code snippet to write that can fill all of the null values in example4?\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YHgy0lIrgRsJ"
},
"source": [
"Anda dapat menjadi kreatif dalam menggunakan `fillna`. Sebagai contoh, mari kita lihat kembali `example4`, tetapi kali ini mari kita isi nilai yang hilang dengan rata-rata dari semua nilai dalam `DataFrame`:\n"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "OtYVErEygRsJ",
"outputId": "708b1e67-45ca-44bf-a5ee-8b2de09ece73",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.0</td>\n",
" <td>5.5</td>\n",
" <td>7</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>8</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>1.5</td>\n",
" <td>6.0</td>\n",
" <td>9</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3\n",
"0 1.0 5.5 7 NaN\n",
"1 2.0 5.0 8 NaN\n",
"2 1.5 6.0 9 NaN"
]
},
"execution_count": 46,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example4.fillna(example4.mean())"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zpMvCkLSgRsJ"
},
"source": [
"Perhatikan bahwa kolom 3 masih tidak memiliki nilai: arah defaultnya adalah mengisi nilai secara baris demi baris.\n",
"\n",
"> **Intisari:** Ada berbagai cara untuk menangani nilai yang hilang dalam dataset Anda. Strategi spesifik yang Anda gunakan (menghapusnya, menggantinya, atau bahkan bagaimana Anda menggantinya) harus ditentukan oleh karakteristik data tersebut. Anda akan mengembangkan pemahaman yang lebih baik tentang cara menangani nilai yang hilang seiring dengan semakin seringnya Anda bekerja dan berinteraksi dengan dataset.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bauDnESIl9FH"
},
"source": [
"### Mengkodekan Data Kategorikal\n",
"\n",
"Model pembelajaran mesin hanya dapat bekerja dengan angka dan segala bentuk data numerik. Model tidak akan bisa membedakan antara Ya dan Tidak, tetapi akan mampu membedakan antara 0 dan 1. Jadi, setelah mengisi nilai yang hilang, kita perlu mengkodekan data kategorikal ke dalam bentuk numerik agar model dapat memahaminya.\n",
"\n",
"Pengkodean dapat dilakukan dengan dua cara. Kita akan membahasnya selanjutnya.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uDq9SxB7mu5i"
},
"source": [
"**PENGODEAN LABEL** \n",
"\n",
"Pengodean label pada dasarnya adalah mengonversi setiap kategori menjadi sebuah angka. Sebagai contoh, misalkan kita memiliki dataset penumpang pesawat dan terdapat sebuah kolom yang berisi kelas mereka di antara ['business class', 'economy class', 'first class']. Jika pengodean label dilakukan pada kolom ini, maka akan diubah menjadi [0,1,2]. Mari kita lihat contohnya melalui kode. Karena kita akan mempelajari `scikit-learn` di notebook berikutnya, kita tidak akan menggunakannya di sini.\n"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 235
},
"id": "1vGz7uZyoWHL",
"outputId": "9e252855-d193-4103-a54d-028ea7787b34"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>ID</th>\n",
" <th>class</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>10</td>\n",
" <td>business class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>20</td>\n",
" <td>first class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>30</td>\n",
" <td>economy class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>40</td>\n",
" <td>economy class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>50</td>\n",
" <td>economy class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>60</td>\n",
" <td>business class</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" ID class\n",
"0 10 business class\n",
"1 20 first class\n",
"2 30 economy class\n",
"3 40 economy class\n",
"4 50 economy class\n",
"5 60 business class"
]
},
"execution_count": 47,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"label = pd.DataFrame([\n",
" [10,'business class'],\n",
" [20,'first class'],\n",
" [30, 'economy class'],\n",
" [40, 'economy class'],\n",
" [50, 'economy class'],\n",
" [60, 'business class']\n",
"],columns=['ID','class'])\n",
"label"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "IDHnkwTYov-h"
},
"source": [
"Untuk melakukan label encoding pada kolom pertama, kita harus terlebih dahulu mendeskripsikan pemetaan dari setiap kelas ke sebuah angka, sebelum mengganti\n"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 235
},
"id": "ZC5URJG3o1ES",
"outputId": "aab0f1e7-e0f3-4c14-8459-9f9168c85437"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>ID</th>\n",
" <th>class</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>10</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>20</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>30</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>40</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>50</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>60</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" ID class\n",
"0 10 0\n",
"1 20 2\n",
"2 30 1\n",
"3 40 1\n",
"4 50 1\n",
"5 60 0"
]
},
"execution_count": 48,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class_labels = {'business class':0,'economy class':1,'first class':2}\n",
"label['class'] = label['class'].replace(class_labels)\n",
"label"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ftnF-TyapOPt"
},
"source": [
"Seperti yang kita lihat, hasilnya sesuai dengan apa yang kita perkirakan. Jadi, kapan kita menggunakan label encoding? Label encoding digunakan dalam salah satu atau kedua kasus berikut:\n",
"1. Ketika jumlah kategori sangat banyak\n",
"2. Ketika kategori memiliki urutan tertentu.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eQPAPVwsqWT7"
},
"source": [
"**ONE HOT ENCODING**\n",
"\n",
"Jenis encoding lainnya adalah One Hot Encoding. Dalam jenis encoding ini, setiap kategori dari kolom akan ditambahkan sebagai kolom terpisah, dan setiap data akan mendapatkan nilai 0 atau 1 berdasarkan apakah data tersebut mengandung kategori tersebut. Jadi, jika ada n kategori yang berbeda, maka n kolom akan ditambahkan ke dataframe.\n",
"\n",
"Sebagai contoh, mari kita ambil contoh kelas pesawat yang sama. Kategorinya adalah: ['business class', 'economy class', 'first class']. Jadi, jika kita melakukan one hot encoding, tiga kolom berikut akan ditambahkan ke dataset: ['class_business class', 'class_economy class', 'class_first class'].\n"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 235
},
"id": "ZM0eVh0ArKUL",
"outputId": "83238a76-b3a5-418d-c0b6-605b02b6891b"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>ID</th>\n",
" <th>class</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>10</td>\n",
" <td>business class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>20</td>\n",
" <td>first class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>30</td>\n",
" <td>economy class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>40</td>\n",
" <td>economy class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>50</td>\n",
" <td>economy class</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>60</td>\n",
" <td>business class</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" ID class\n",
"0 10 business class\n",
"1 20 first class\n",
"2 30 economy class\n",
"3 40 economy class\n",
"4 50 economy class\n",
"5 60 business class"
]
},
"execution_count": 49,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"one_hot = pd.DataFrame([\n",
" [10,'business class'],\n",
" [20,'first class'],\n",
" [30, 'economy class'],\n",
" [40, 'economy class'],\n",
" [50, 'economy class'],\n",
" [60, 'business class']\n",
"],columns=['ID','class'])\n",
"one_hot"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aVnZ7paDrWmb"
},
"source": [
"Mari kita lakukan one hot encoding pada kolom pertama\n"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {
"id": "RUPxf7egrYKr"
},
"outputs": [],
"source": [
"one_hot_data = pd.get_dummies(one_hot,columns=['class'])"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 235
},
"id": "TM37pHsFr4ge",
"outputId": "7be15f53-79b2-447a-979c-822658339a9e"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>ID</th>\n",
" <th>class_business class</th>\n",
" <th>class_economy class</th>\n",
" <th>class_first class</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>10</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>20</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>30</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>40</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>50</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>60</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" ID class_business class class_economy class class_first class\n",
"0 10 1 0 0\n",
"1 20 0 0 1\n",
"2 30 0 1 0\n",
"3 40 0 1 0\n",
"4 50 0 1 0\n",
"5 60 1 0 0"
]
},
"execution_count": 51,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"one_hot_data"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_zXRLOjXujdA"
},
"source": [
"Setiap kolom hot encoded berisi 0 atau 1, yang menunjukkan apakah kategori tersebut ada untuk titik data tersebut.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bDnC4NQOu0qr"
},
"source": [
"Kapan kita menggunakan one hot encoding? One hot encoding digunakan dalam salah satu atau kedua kasus berikut:\n",
"\n",
"1. Ketika jumlah kategori dan ukuran dataset lebih kecil.\n",
"2. Ketika kategori tidak memiliki urutan tertentu.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XnUmci_4uvyu"
},
"source": [
"> Poin Penting:\n",
"1. Encoding dilakukan untuk mengubah data non-numerik menjadi data numerik.\n",
"2. Ada dua jenis encoding: Label encoding dan One Hot encoding, yang keduanya dapat dilakukan berdasarkan kebutuhan dataset.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "K8UXOJYRgRsJ"
},
"source": [
"## Menghapus data duplikat\n",
"\n",
"> **Tujuan pembelajaran:** Pada akhir subbagian ini, Anda diharapkan dapat dengan mudah mengidentifikasi dan menghapus nilai duplikat dari DataFrames.\n",
"\n",
"Selain data yang hilang, Anda sering kali akan menemukan data duplikat dalam dataset dunia nyata. Untungnya, pandas menyediakan cara yang mudah untuk mendeteksi dan menghapus entri duplikat.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qrEG-Wa0gRsJ"
},
"source": [
"### Mengidentifikasi duplikat: `duplicated`\n",
"\n",
"Anda dapat dengan mudah menemukan nilai duplikat menggunakan metode `duplicated` di pandas, yang mengembalikan masker Boolean yang menunjukkan apakah sebuah entri dalam `DataFrame` adalah duplikat dari entri sebelumnya. Mari kita buat contoh `DataFrame` lain untuk melihat cara kerjanya.\n"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"id": "ZLu6FEnZgRsJ",
"outputId": "376512d1-d842-4db1-aea3-71052aeeecaf",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>letters</th>\n",
" <th>numbers</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>B</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>A</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>B</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>B</td>\n",
" <td>3</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" letters numbers\n",
"0 A 1\n",
"1 B 2\n",
"2 A 1\n",
"3 B 3\n",
"4 B 3"
]
},
"execution_count": 52,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example6 = pd.DataFrame({'letters': ['A','B'] * 2 + ['B'],\n",
" 'numbers': [1, 2, 1, 3, 3]})\n",
"example6"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "cIduB5oBgRsK",
"outputId": "3da27b3d-4d69-4e1d-bb52-0af21bae87f2",
"trusted": false
},
"outputs": [
{
"data": {
"text/plain": [
"0 False\n",
"1 False\n",
"2 True\n",
"3 False\n",
"4 True\n",
"dtype: bool"
]
},
"execution_count": 53,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example6.duplicated()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0eDRJD4SgRsK"
},
"source": [
"### Menghapus duplikat: `drop_duplicates`\n",
"`drop_duplicates` hanya mengembalikan salinan data di mana semua nilai `duplicated` adalah `False`:\n"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 142
},
"id": "w_YPpqIqgRsK",
"outputId": "ac66bd2f-8671-4744-87f5-8b8d96553dea",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>letters</th>\n",
" <th>numbers</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>B</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>B</td>\n",
" <td>3</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" letters numbers\n",
"0 A 1\n",
"1 B 2\n",
"3 B 3"
]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example6.drop_duplicates()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "69AqoCZAgRsK"
},
"source": [
"Baik `duplicated` maupun `drop_duplicates` secara default mempertimbangkan semua kolom tetapi Anda dapat menentukan agar mereka hanya memeriksa sebagian kolom dalam `DataFrame` Anda:\n"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 111
},
"id": "BILjDs67gRsK",
"outputId": "ef6dcc08-db8b-4352-c44e-5aa9e2bec0d3",
"trusted": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>letters</th>\n",
" <th>numbers</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>B</td>\n",
" <td>2</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" letters numbers\n",
"0 A 1\n",
"1 B 2"
]
},
"execution_count": 55,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example6.drop_duplicates(['letters'])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GvX4og1EgRsL"
},
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n---\n\n**Penafian**: \nDokumen ini telah diterjemahkan menggunakan layanan penerjemahan AI [Co-op Translator](https://github.com/Azure/co-op-translator). Meskipun kami berusaha untuk memberikan hasil yang akurat, harap diingat bahwa terjemahan otomatis mungkin mengandung kesalahan atau ketidakakuratan. Dokumen asli dalam bahasa aslinya harus dianggap sebagai sumber yang otoritatif. Untuk informasi yang bersifat kritis, disarankan menggunakan jasa penerjemahan profesional oleh manusia. Kami tidak bertanggung jawab atas kesalahpahaman atau penafsiran yang keliru yang timbul dari penggunaan terjemahan ini.\n"
]
}
],
"metadata": {
"anaconda-cloud": {},
"colab": {
"name": "notebook.ipynb",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.4"
},
"coopTranslator": {
"original_hash": "8533b3a2230311943339963fc7f04c21",
"translation_date": "2025-09-01T22:00:06+00:00",
"source_file": "2-Working-With-Data/08-data-preparation/notebook.ipynb",
"language_code": "id"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Reference in new issue View Git Blame Copy Permalink