"Let us dive into missing data. Missing data occurs, when no value is sotred in some of the columns. \n",
"\n",
"Let us take an example: say someone is concious about his/her weight and doesn't fill the weight field in a survey. Then, the weight value for that certain person will be missing. \n",
"\n",
"Most of the time, in real world datasets, missing values occur.\n",
"\n",
"> **Learning goal:** By the end of this subsection, you should know how to replace or remove null values from DataFrames.\n",
"**How Pandas Handles missing data**\n",
"\n",
"Most of the time the datasets you want to use (of have to use) have missing values in them. How missing data is handled carries with it subtle tradeoffs that can affect your final analysis and real-world outcomes.\n",
"\n",
"Pandas handles missing values in two ways. The first you've seen before in previous sections: `NaN`, or Not a Number. This is a actually a special value that is part of the IEEE floating-point specification and it is only used to indicate missing floating-point values.\n",
"\u001b[0;32m<ipython-input-10-ce9901ad18bd>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mexample1\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"Look closely at the output. Does any of it surprise you? While `0` is an arithmetic null, it's nevertheless a perfectly good integer and pandas treats it as such. `''` is a little more subtle. While we used it in Section 1 to represent an empty string value, it is nevertheless a string object and not a representation of null as far as pandas is concerned.\n",
"\n",
"Now, let's turn this around and use these methods in a manner more like you will use them in practice. You can use Boolean masks directly as a ``Series`` or ``DataFrame`` index, which can be useful when trying to work with isolated missing (or present) values."
"Now, let's turn this around and use these methods in a manner more like you will use them in practice. You can use Boolean masks directly as a ``Series`` or ``DataFrame`` index, which can be useful when trying to work with isolated missing (or present) values.\n",
"\n",
"If we want the total number of missing values, we can just do a sum over the mask produced by the `isnull()` method."
"# Before you do so, what do you expect to see?\n"
],
"execution_count": null,
"execution_count": 20,
"outputs": []
},
{
@ -919,7 +1068,31 @@
"id": "D_jWN7mHgRsD"
},
"source": [
"**Key takeaway**: Both the `isnull()` and `notnull()` methods produce similar results when you use them in `DataFrame`s: they show the results and the index of those results, which will help you enormously as you wrestle with your data."
"**Key takeaway**: Both the `isnull()` and `notnull()` methods produce similar results when you use them in DataFrames: they show the results and the index of those results, which will help you enormously as you wrestle with your data."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BvnoojWsgRr4"
},
"source": [
"### Dealing with missing data\n",
"\n",
"> **Learning goal:** By the end of this subsection, you should know how and when to replace or remove null values from DataFrames.\n",
"\n",
"Machine Learning models can't deal with missing data themselves. So, before passing the data into the model, we need to deal with these missing values.\n",
"\n",
"How missing data is handled carries with it subtle tradeoffs, can affect your final analysis and real-world outcomes.\n",
"\n",
"There are primarily two ways of dealing with missing data:\n",
"\n",
"\n",
"1. Drop the row containing the missing value\n",
"2. Replace the missing value with some other value\n",
"\n",
"We will discuss both these methods and their pros and cons in details.\n",
Nirmalya Misra
3 years ago