" <figcaption>Infographic by Dasani Madipalli</figcaption>\n",
"\n",
"\n",
"<!--{width=\"600\"}-->"
"Alternately, the script below checks whether you have the packages required to complete this module and installs them for you in case they are missing."
],
"metadata": {
"id": "KPmut75XkmXY"
}
"Alternately, the script below checks whether you have the packages required to complete this module and installs them for you in case they are missing.\n"
"For our purposes, we will express this as a binary: 'Orange' or 'Not Orange'. There is also a 'striped' category in our dataset but there are few instances of it, so we will not use it. It disappears once we remove null values from the dataset, anyway.\n",
"For our purposes, we will express this as a binary: 'White' or 'Not White'. There is also a 'striped' category in our dataset but there are few instances of it, so we will not use it. It disappears once we remove null values from the dataset, anyway.\n",
"\n",
"> 🎃 Fun fact, we sometimes call white pumpkins 'ghost' pumpkins. They aren't very easy to carve, so they aren't as popular as the orange ones but they are cool looking!\n",
"> 🎃 Fun fact, we sometimes call white pumpkins 'ghost' pumpkins. They aren't very easy to carve, so they aren't as popular as the orange ones but they are cool looking! So we could also reformulate our question as: 'Ghost' or 'Not Ghost'. 👻\n",
"\n",
"## **About logistic regression**\n",
"\n",
@ -115,45 +74,7 @@
"\n",
"Logistic regression does not offer the same features as linear regression. The former offers a prediction about a `binary category` (\"orange or not orange\") whereas the latter is capable of predicting `continual values`, for example given the origin of a pumpkin and the time of harvest, *how much its price will rise*.\n",
" <figcaption>Infographic by Dasani Madipalli</figcaption>\n",
"\n",
"<!--{width=\"600\"}-->"
],
"metadata": {
"id": "ws-hP_SXk2O6"
}
},
{
"cell_type": "markdown",
"source": [
"#### **Other classifications**\n",
"\n",
"There are other types of logistic regression, including multinomial and ordinal:\n",
"\n",
"- **Multinomial**, which involves having more than one category - \"Orange, White, and Striped\".\n",
"\n",
"- **Ordinal**, which involves ordered categories, useful if we wanted to order our outcomes logically, like our pumpkins that are ordered by a finite number of sizes (mini,sm,med,lg,xl,xxl).\n",
" <figcaption>Infographic by Dasani Madipalli</figcaption>\n",
"\n",
"<!--{width=\"600\"}-->"
],
"metadata": {
"id": "LkLN-ZgDlBEc"
}
},
{
"cell_type": "markdown",
"source": [
"**It's still linear**\n",
"\n",
"Even though this type of Regression is all about 'category predictions', it still works best when there is a clear linear relationship between the dependent variable (color) and the other independent variables (the rest of the dataset, like city name and size). It's good to get an idea of whether there is any linearity dividing these variables or not.\n",
"{width=\"600\"}\n",
"\n",
"#### **Variables DO NOT have to correlate**\n",
"\n",
@ -163,26 +84,18 @@
"\n",
"Logistic regression will give more accurate results if you use more data; our small dataset is not optimal for this task, so keep that in mind.\n",
"\n",
"✅ Think about the types of data that would lend themselves well to logistic regression\n"
],
"metadata": {
"id": "D8_JoVZtlHUt"
}
},
{
"cell_type": "markdown",
"source": [
"✅ Think about the types of data that would lend themselves well to logistic regression\n",
"\n",
"## 1. Tidy the data\n",
"\n",
"Now, the fun begins! Let's start by importing the data, cleaning the data a bit, dropping rows containing missing values and selecting only some of the columns:"
],
"metadata": {
"id": "LPj8Ib1AlIua"
}
"Now, the fun begins! Let's start by importing the data, cleaning the data a bit, dropping rows containing missing values and selecting only some of the columns:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Load the core tidyverse packages\n",
"library(tidyverse)\n",
@ -203,59 +116,49 @@
"# View the first few rows\n",
"pumpkins_select %>% \n",
" slice_head(n = 5)\n"
],
"outputs": [],
"metadata": {
"id": "Q8oKJ8PAlLM0"
}
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Sometimes, we may want some little more information on our data. We can have a look at the `data`, `its structure` and the `data type` of its features by using the [*glimpse()*](https://pillar.r-lib.org/reference/glimpse.html) function as below:"
],
"metadata": {
"id": "tKY5eN8alPNn"
}
"Sometimes, we may want some little more information on our data. We can have a look at the `data`, `its structure` and the `data type` of its features by using the [*glimpse()*](https://pillar.r-lib.org/reference/glimpse.html) function as below:\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pumpkins_select %>% \n",
" glimpse()"
],
"outputs": [],
"metadata": {
"id": "wDpatL1WlShu"
}
" glimpse()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Wow! Seems that all our columns are all of type *character*, further alluding that they are all categorical.\n",
"\n",
"Let's confirm that we will actually be doing a binary classification problem:"
],
"metadata": {
"id": "QbdC2b0JlU2G"
}
"Let's confirm that we will actually be doing a binary classification problem:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Subset distinct observations in outcome column\n",
"pumpkins_select %>% \n",
" distinct(color)"
],
"outputs": [],
"metadata": {
"id": "Gys-Q18rlZpE"
}
" distinct(color)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"🥳🥳 That went down well!\n",
"\n",
@ -263,98 +166,112 @@
"\n",
"The goal of data exploration is to try to understand the `relationships` between its attributes; in particular, any apparent correlation between the *features* and the *label* your model will try to predict. One way of doing this is by using data visualization.\n",
"\n",
"Given our the data types of our columns, we can `encode` them and be on our way to making some visualizations. This simply involves `translating` a column with `categorical values` for example our columns of type *char*, into one or more `numeric columns` that take the place of the original. - Something we did in our [last lesson](https://github.com/microsoft/ML-For-Beginners/blob/main/2-Regression/3-Linear/solution/lesson_3-R.ipynb).\n",
"Given our the data types of our columns, we can `encode` them and be on our way to making some visualizations. This simply involves `translating` a column with `categorical values` for example our columns of type *char*, into one or more `numeric columns` that take the place of the original. - Something we did in our [last lesson](https://github.com/microsoft/ML-For-Beginners/blob/main/2-Regression/3-Linear/solution/lesson_3.html).\n",
"\n",
"For feature encoding there are two main types of encoders:\n",
"\n",
"1. Ordinal encoder: it suits well for ordinal variables, which are categorical variables where their data follows a logical ordering, like the `item_size` column in our dataset. It creates a mapping such that each category is represented by a number, which is the order of the category in the column.\n",
"\n",
"2. Categorical encoder: it suits well for nominal variables, which are categorical variables where their data does not follow a logical ordering, like all the features different from `item_size` in our dataset. It is a one-hot encoding, which means that each category is represented by a binary column: the encoded variable is equal to 1 if the pumpkin belongs to that Variety and 0 otherwise.\n",
"\n",
"Tidymodels provides yet another neat package: [recipes](https://recipes.tidymodels.org/)- a package for preprocessing data. We'll define a `recipe` that specifies that all predictor columns should be encoded into a set of integers , `prep` it to estimates the required quantities and statistics needed by any operations and finally `bake` to apply the computations to new data.\n",
"\n",
"> Normally, recipes is usually used as a preprocessor for modelling where it defines what steps should be applied to a data set in order to get it ready for modelling. In that case it is **highly recommend** that you use a `workflow()` instead of manually estimating a recipe using prep and bake. We'll see all this in just a moment.\n",
">\n",
"> However for now, we are using recipes + prep + bake to specify what steps should be applied to a data set in order to get it ready for data analysis and then extract the preprocessed data with the steps applied."
],
"metadata": {
"id": "kn_20wSPldVH"
}
"> However for now, we are using recipes + prep + bake to specify what steps should be applied to a data set in order to get it ready for data analysis and then extract the preprocessed data with the steps applied.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Preprocess and extract data to allow some data analysis\n",
"baked_pumpkins <- recipe(color ~ ., data = pumpkins_select) %>% \n",
"# Display the first few rows of preprocessed data\n",
"baked_pumpkins %>% \n",
" slice_head(n = 5)"
],
"outputs": [],
"metadata": {
"id": "syaCgFQ_lijg"
}
" slice_head(n = 5)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's compare the feature distributions for each label value using box plots. We'll begin by formatting the data to a *long* format to make it somewhat easier to make multiple `facets`."
],
"metadata": {
"id": "RlkOZ_C5lldq"
}
"Now, let's make a categorical plot showing the distribution of the predictors with respect to the outcome color!\n",
"# Specify colors for each value of the hue variable\n",
"palette <- c(ORANGE = \"orange\", WHITE = \"wheat\")\n",
"\n",
"# Print out restructured data\n",
"baked_pumpkins_long %>% \n",
" slice_head(n = 10)\n"
],
"outputs": [],
"metadata": {
"id": "putq8DagltUQ"
}
"# Create the bar plot\n",
"ggplot(pumpkins_select, aes(y = variety, fill = color)) +\n",
" geom_bar(position = \"dodge\") +\n",
" scale_fill_manual(values = palette) +\n",
" labs(y = \"Variety\", fill = \"Color\") +\n",
" theme_minimal()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, let's make some boxplots showing the distribution of the predictors with respect to the outcome color."
],
"metadata": {
"id": "-RHm-12zlt-B"
}
"Amazing🤩! For some of the features, there's a noticeable difference in the distribution for each color label. For instance, it seems the white pumpkins can be found in smaller packages and in some particular varieties of pumpkins. The *item_size* category also seems to make a difference in the color distribution. These features may help predict the color of a pumpkin.\n",
"\n",
"### **Analysing relationships between features and label**\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"theme_set(theme_light())\n",
"#Make a box plot for each predictor feature\n",
"baked_pumpkins_long %>% \n",
" mutate(color = factor(color)) %>% \n",
" ggplot(mapping = aes(x = color, y = values, fill = features)) +\n",
"Amazing🤩! For some of the features, there's a noticeable difference in the distribution for each color label. For instance, it seems the white pumpkins can be found in smaller packages and in some particular varieties of pumpkins. The *item_size* category also seems to make a difference in the color distribution. These features may help predict the color of a pumpkin.\n",
"Let's now focus on a specific relationship: Item Size and Color!\n",
"\n",
"#### **Use a swarm plot**\n",
"\n",
@ -362,15 +279,14 @@
"\n",
"Try a `swarm plot` to show the distribution of color with respect to the item_size.\n",
"\n",
"We'll use the [ggbeeswarm package](https://github.com/eclarke/ggbeeswarm) which provides methods to create beeswarm-style plots using ggplot2. Beeswarm plots are a way of plotting points that would ordinarily overlap so that they fall next to each other instead."
],
"metadata": {
"id": "2LSj6_LCl68V"
}
"We'll use the [ggbeeswarm package](https://github.com/eclarke/ggbeeswarm) which provides methods to create beeswarm-style plots using ggplot2. Beeswarm plots are a way of plotting points that would ordinarily overlap so that they fall next to each other instead.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create beeswarm plots of color and item_size\n",
"baked_pumpkins %>% \n",
@ -378,76 +294,27 @@
" ggplot(mapping = aes(x = color, y = item_size, color = color)) +\n",
" geom_quasirandom() +\n",
" scale_color_brewer(palette = \"Dark2\", direction = -1) +\n",
" theme(legend.position = \"none\")"
],
"outputs": [],
"metadata": {
"id": "hGKeRgUemMTb"
}
},
{
"cell_type": "markdown",
"source": [
"#### **Violin plot**\n",
"\n",
"A 'violin' type plot is useful as you can easily visualize the way that data in the two categories is distributed. [`Violin plots`](https://en.wikipedia.org/wiki/Violin_plot) are similar to box plots, except that they also show the probability density of the data at different values. Violin plots don't work so well with smaller datasets as the distribution is displayed more 'smoothly'."
],
"metadata": {
"id": "_9wdZJH5mOvN"
}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"# Create a violin plot of color and item_size\n",
"baked_pumpkins %>%\n",
" mutate(color = factor(color)) %>% \n",
" ggplot(mapping = aes(x = color, y = item_size, fill = color)) +\n",
" scale_fill_brewer(palette = \"Dark2\", direction = -1) +\n",
" theme(legend.position = \"none\")"
],
"outputs": [],
"metadata": {
"id": "LFFFymujmTAZ"
}
" theme(legend.position = \"none\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we have an idea of the relationship between the binary categories of color and the larger group of sizes, let's explore logistic regression to determine a given pumpkin's likely color.\n",
" <figcaption>Infographic by Dasani Madipalli</figcaption>\n",
"\n",
"> **🧮 Show Me The Math**\n",
">\n",
"> Remember how `linear regression` often used `ordinary least squares` to arrive at a value? `Logistic regression` relies on the concept of 'maximum likelihood' using [`sigmoid functions`](https://wikipedia.org/wiki/Sigmoid_function). A Sigmoid Function on a plot looks like an `S shape`. It takes a value and maps it to somewhere between 0 and 1. Its curve is also called a 'logistic curve'. Its formula looks like this:\n",
">\n",
"> \n",
"<p >\n",
" <img src=\"../../images/sigmoid.png\">\n",
"\n",
"\n",
"> where the sigmoid's midpoint finds itself at x's 0 point, L is the curve's maximum value, and k is the curve's steepness. If the outcome of the function is more than 0.5, the label in question will be given the class 1 of the binary choice. If not, it will be classified as 0.\n",
"## 3. Build your model\n",
"\n",
"Let's begin by splitting the data into `training` and `test` sets. The training set is used to train a classifier so that it finds a statistical relationship between the features and the label value.\n",
"\n",
"It is best practice to hold out some of your data for **testing** in order to get a better estimate of how your models will perform on new data by comparing the predicted labels with the already known labels in the test set. [rsample](https://rsample.tidymodels.org/), a package in Tidymodels, provides infrastructure for efficient data splitting and resampling:"
],
"metadata": {
"id": "RA_bnMS9mVo8"
}
"It is best practice to hold out some of your data for **testing** in order to get a better estimate of how your models will perform on new data by comparing the predicted labels with the already known labels in the test set. [rsample](https://rsample.tidymodels.org/), a package in Tidymodels, provides infrastructure for efficient data splitting and resampling:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Split data into 80% for training and 20% for testing\n",
"set.seed(2056)\n",
@ -460,59 +327,52 @@
"\n",
"# Print out the first 5 rows of the training set\n",
"pumpkins_train %>% \n",
" slice_head(n = 5)"
],
"outputs": [],
"metadata": {
"id": "PQdpEYYPmdGW"
}
" slice_head(n = 5)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"🙌 We are now ready to train a model by fitting the training features to the training label (color).\n",
"\n",
"We'll begin by creating a recipe that specifies the preprocessing steps that should be carried out on our data to get it ready for modelling i.e: encoding categorical variables into a set of integers.\n",
"We'll begin by creating a recipe that specifies the preprocessing steps that should be carried out on our data to get it ready for modelling i.e: encoding categorical variables into a set of integers. Just like `baked_pumpkins`, we create a `pumpkins_recipe` but do not `prep` and `bake` since it would be bundled into a workflow, which you will see in just a few steps from now. \n",
"\n",
"There are quite a number of ways to specify a logistic regression model in Tidymodels. See `?logistic_reg()` For now, we'll specify a logistic regression model via the default `stats::glm()` engine."
],
"metadata": {
"id": "MX9LipSimhn0"
}
"There are quite a number of ways to specify a logistic regression model in Tidymodels. See `?logistic_reg()` For now, we'll specify a logistic regression model via the default `stats::glm()` engine.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a recipe that specifies preprocessing steps for modelling\n",
"pumpkins_recipe <- recipe(color ~ ., data = pumpkins_train) %>% \n",
"Now that we have a recipe and a model specification, we need to find a way of bundling them together into an object that will first preprocess the data (prep+bake behind the scenes), fit the model on the preprocessed data and also allow for potential post-processing activities.\n",
"\n",
"In Tidymodels, this convenient object is called a [`workflow`](https://workflows.tidymodels.org/) and conveniently holds your modeling components."
],
"metadata": {
"id": "G599GKhXmqWf"
}
"In Tidymodels, this convenient object is called a [`workflow`](https://workflows.tidymodels.org/) and conveniently holds your modeling components.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Bundle modelling components in a workflow\n",
"log_reg_wf <- workflow() %>% \n",
@ -521,51 +381,44 @@
"\n",
"# Print out the workflow\n",
"log_reg_wf\n"
],
"outputs": [],
"metadata": {
"id": "cRoU0tpbmu1T"
}
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After a workflow has been *specified*, a model can be `trained` using the [`fit()`](https://tidymodels.github.io/parsnip/reference/fit.html) function. The workflow will estimate a recipe and preprocess the data before training, so we won't have to manually do that using prep and bake."
],
"metadata": {
"id": "JnRXKmREnEpd"
}
"After a workflow has been *specified*, a model can be `trained` using the [`fit()`](https://tidymodels.github.io/parsnip/reference/fit.html) function. The workflow will estimate a recipe and preprocess the data before training, so we won't have to manually do that using prep and bake.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Train the model\n",
"wf_fit <- log_reg_wf %>% \n",
" fit(data = pumpkins_train)\n",
"\n",
"# Print the trained workflow\n",
"wf_fit"
],
"outputs": [],
"metadata": {
"id": "ehFwfkjWnNCb"
}
"wf_fit\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The model print out shows the coefficients learned during training.\n",
"\n",
"Now we've trained the model using the training data, we can make predictions on the test data using [parsnip::predict()](https://parsnip.tidymodels.org/reference/predict.model_fit.html). Let's start by using the model to predict labels for our test set and the probabilities for each label. When the probability is more than 0.5, the predict class is `ORANGE` else `WHITE`."
],
"metadata": {
"id": "w01dGNZjnOJQ"
}
"Now we've trained the model using the training data, we can make predictions on the test data using [parsnip::predict()](https://parsnip.tidymodels.org/reference/predict.model_fit.html). Let's start by using the model to predict labels for our test set and the probabilities for each label. When the probability is more than 0.5, the predict class is `WHITE` else `ORANGE`.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Make predictions for color and corresponding probabilities\n",
"Very nice! This provides some more insights into how logistic regression works.\n",
"\n",
@ -592,55 +442,43 @@
"\n",
"One performance metric associated with classification problems is the [`confusion matrix`](https://wikipedia.org/wiki/Confusion_matrix). A confusion matrix describes how well a classification model performs. A confusion matrix tabulates how many examples in each class were correctly classified by a model. In our case, it will show you how many orange pumpkins were classified as orange and how many white pumpkins were classified as white; the confusion matrix also shows you how many were classified into the **wrong** categories.\n",
"\n",
"The [**`conf_mat()`**](https://tidymodels.github.io/yardstick/reference/conf_mat.html) function from yardstick calculates this cross-tabulation of observed and predicted classes."
],
"metadata": {
"id": "N3J-yW0wngKo"
}
"The [**`conf_mat()`**](https://tidymodels.github.io/yardstick/reference/conf_mat.html) function from yardstick calculates this cross-tabulation of observed and predicted classes.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Confusion matrix for prediction results\n",
"conf_mat(data = results, truth = color, estimate = .pred_class)"
],
"outputs": [],
"metadata": {
"id": "0RD77Dq1nl2j"
}
"conf_mat(data = results, truth = color, estimate = .pred_class)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's interpret the confusion matrix. Our model is asked to classify pumpkins between two binary categories, category `orange` and category `not-orange`\n",
"\n",
"- If your model predicts a pumpkin as orange and it belongs to category 'orange' in reality we call it a `true positive`, shown by the top left number.\n",
"Let's interpret the confusion matrix. Our model is asked to classify pumpkins between two binary categories, category `white` and category `not-white`\n",
"\n",
"- If your model predicts a pumpkin as not orange and it belongs to category 'orange' in reality we call it a `false negative`, shown by the bottom left number.\n",
"- If your model predicts a pumpkin as white and it belongs to category 'white' in reality we call it a `true positive`, shown by the top left number.\n",
"\n",
"- If your model predicts a pumpkin as orange and it belongs to category 'not-orange' in reality we call it a `false positive`, shown by the top right number.\n",
"- If your model predicts a pumpkin as not white and it belongs to category 'white' in reality we call it a `false negative`, shown by the bottom left number.\n",
"\n",
"- If your model predicts a pumpkin as not orange and it belongs to category 'not-orange' in reality we call it a `true negative`, shown by the bottom right number.\n",
"- If your model predicts a pumpkin as white and it belongs to category 'not-white' in reality we call it a `false positive`, shown by the top right number.\n",
"\n",
"- If your model predicts a pumpkin as not white and it belongs to category 'not-white' in reality we call it a `true negative`, shown by the bottom right number.\n",
"\n",
"| **Truth** |\n",
"| Truth |\n",
"|:-----:|\n",
"\n",
"\n",
"| | | |\n",
"|---------------|--------|-------|\n",
"| **Predicted** | ORANGE | WHITE |\n",
"| ORANGE | TP | FP |\n",
"| WHITE | FN | TN |"
],
"metadata": {
"id": "H61sFwdOnoiO"
}
},
{
"cell_type": "markdown",
"source": [
"| **Predicted** | WHITE | ORANGE |\n",
"| WHITE | TP | FP |\n",
"| ORANGE | FN | TN |\n",
"\n",
"As you might have guessed it's preferable to have a larger number of true positives and true negatives and a lower number of false positives and false negatives, which implies that the model performs better.\n",
"\n",
"The confusion matrix is helpful since it gives rise to other metrics that can help us better evaluate the performance of a classification model. Let's go through some of them:\n",
@ -655,80 +493,68 @@
"\n",
"🎓 F Measure: A weighted average of the precision and recall, with best being 1 and worst being 0.\n",
"\n",
"Let's calculate these metrics!"
],
"metadata": {
"id": "Yc6QUie2oQUr"
}
"Let's calculate these metrics!\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Combine metric functions and calculate them all at once\n",
"eval_metrics(data = results, truth = color, estimate = .pred_class)"
],
"outputs": [],
"metadata": {
"id": "p6rXx_T3oVxX"
}
"eval_metrics(data = results, truth = color, estimate = .pred_class)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### **Visualize the ROC curve of this model**\n",
"\n",
"For a start, this is not a bad model; its precision, recall, F measure and accuracy are in the 80% range so ideally you could use it to predict the color of a pumpkin given a set of variables. It also seems that our model was not really able to identify the white pumpkins 🧐. Could you guess why? One reason could be because of the high prevalence of ORANGE pumpkins in our training set making our model more inclined to predict the majority class.\n",
"For a start, this is not a bad model; its precision, recall, F measure and accuracy are in the 90% range so ideally you could use it to predict the color of a pumpkin given a set of variables. It also seems that our model was not really able to identify the white pumpkins 🧐. Could you guess why? One reason could be because of the high prevalence of ORANGE pumpkins in our training set making our model more inclined to predict the majority class.\n",
"\n",
"Let's do one more visualization to see the so-called [`ROC score`](https://en.wikipedia.org/wiki/Receiver_operating_characteristic):"
],
"metadata": {
"id": "JcenzZo1oaKR"
}
"Let's do one more visualization to see the so-called [`ROC score`](https://en.wikipedia.org/wiki/Receiver_operating_characteristic):\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Make a roc_curve\n",
"results %>% \n",
" roc_curve(color, .pred_ORANGE) %>% \n",
" autoplot()"
],
"outputs": [],
"metadata": {
"id": "BcmkHHHwogRB"
}
" autoplot()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"ROC curves are often used to get a view of the output of a classifier in terms of its true vs. false positives. ROC curves typically feature `True Positive Rate`/Sensitivity on the Y axis, and `False Positive Rate`/1-Specificity on the X axis. Thus, the steepness of the curve and the space between the midpoint line and the curve matter: you want a curve that quickly heads up and over the line. In our case, there are false positives to start with, and then the line heads up and over properly.\n",
"\n",
"Finally, let's use `yardstick::roc_auc()` to calculate the actual Area Under the Curve. One way of interpreting AUC is as the probability that the model ranks a random positive example more highly than a random negative example."
],
"metadata": {
"id": "P_an3vc1oqjI"
}
"Finally, let's use `yardstick::roc_auc()` to calculate the actual Area Under the Curve. One way of interpreting AUC is as the probability that the model ranks a random positive example more highly than a random negative example.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Calculate area under curve\n",
"results %>% \n",
" roc_auc(color, .pred_ORANGE)"
],
"outputs": [],
"metadata": {
"id": "SZyy5BT8ovew"
}
" roc_auc(color, .pred_ORANGE)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The result is around `0.67053`. Given that the AUC ranges from 0 to 1, you want a big score, since a model that is 100% correct in its predictions will have an AUC of 1; in this case, the model is *pretty good*.\n",
"The result is around `0.947`. Given that the AUC ranges from 0 to 1, you want a big score, since a model that is 100% correct in its predictions will have an AUC of 1; in this case, the model is *pretty good*.\n",
"\n",
"In future lessons on classifications, you will learn how to improve your model's scores (such as dealing with imbalanced data in this case).\n",
@ -72,21 +72,6 @@ Logistic regression does not offer the same features as linear regression. The f
{width="600"}
#### **Other classifications**
There are other types of logistic regression, including multinomial and ordinal:
- **Multinomial**, which involves having more than one category - "Orange, White, and Striped".
- **Ordinal**, which involves ordered categories, useful if we wanted to order our outcomes logically, like our pumpkins that are ordered by a finite number of sizes (mini,sm,med,lg,xl,xxl).
\
**It's still linear**
Even though this type of Regression is all about 'category predictions', it still works best when there is a clear linear relationship between the dependent variable (color) and the other independent variables (the rest of the dataset, like city name and size). It's good to get an idea of whether there is any linearity dividing these variables or not.
#### **Variables DO NOT have to correlate**
Remember how linear regression worked better with more correlated variables? Logistic regression is the opposite - the variables don't have to align. That works for this data which has somewhat weak correlations.
@ -179,36 +164,52 @@ baked_pumpkins %>%
slice_head(n = 5)
```
Now let's compare the feature distributions for each label value using box plots. We'll begin by formatting the data to a *long* format to make it somewhat easier to make multiple `facets`.
Now, let's make a categorical plot showing the distribution of the predictors with respect to the outcome color!
# Specify colors for each value of the hue variable
palette <- c(ORANGE = "orange", WHITE = "wheat")
# Create the bar plot
ggplot(pumpkins_select, aes(y = variety, fill = color)) +
geom_bar(position = "dodge") +
scale_fill_manual(values = palette) +
labs(y = "Variety", fill = "Color") +
theme_minimal()
```
# Print out restructured data
baked_pumpkins_long %>%
slice_head(n = 10)
Amazing🤩! For some of the features, there's a noticeable difference in the distribution for each color label. For instance, it seems the white pumpkins can be found in smaller packages and in some particular varieties of pumpkins. The *item_size* category also seems to make a difference in the color distribution. These features may help predict the color of a pumpkin.
```
### **Analysing relationships between features and label**
```{r}
Now, let's make some boxplots showing the distribution of the predictors with respect to the outcome color!
# Define the color palette
palette <- c(ORANGE = "orange", WHITE = "wheat")
```{r boxplots}
theme_set(theme_light())
#Make a box plot for each predictor feature
baked_pumpkins_long %>%
mutate(color = factor(color)) %>%
ggplot(mapping = aes(x = color, y = values, fill = features)) +
# We need the encoded Item Size column to use it as the x-axis values in the plot
Amazing🤩! For some of the features, there's a noticeable difference in the distribution for each color label. For instance, it seems the white pumpkins can be found in smaller packages and in some particular varieties of pumpkins. The *item_size* category also seems to make a difference in the color distribution. These features may help predict the color of a pumpkin.
Let's now focus on a specific relationship: Item Size and Color!
#### **Use a swarm plot**
@ -227,19 +228,10 @@ baked_pumpkins %>%
scale_color_brewer(palette = "Dark2", direction = -1) +
theme(legend.position = "none")
```
Now that we have an idea of the relationship between the binary categories of color and the larger group of sizes, let's explore logistic regression to determine a given pumpkin's likely color.
## 3. Build your model
> **🧮 Show Me The Math**
>
> Remember how `linear regression` often used `ordinary least squares` to arrive at a value? `Logistic regression` relies on the concept of 'maximum likelihood' using [`sigmoid functions`](https://wikipedia.org/wiki/Sigmoid_function). A Sigmoid Function on a plot looks like an `S shape`. It takes a value and maps it to somewhere between 0 and 1. Its curve is also called a 'logistic curve'. Its formula looks like this:
>
> 
>
> where the sigmoid's midpoint finds itself at x's 0 point, L is the curve's maximum value, and k is the curve's steepness. If the outcome of the function is more than 0.5, the label in question will be given the class 1 of the binary choice. If not, it will be classified as 0.
Let's begin by splitting the data into `training` and `test` sets. The training set is used to train a classifier so that it finds a statistical relationship between the features and the label value.
It is best practice to hold out some of your data for **testing** in order to get a better estimate of how your models will perform on new data by comparing the predicted labels with the already known labels in the test set. [rsample](https://rsample.tidymodels.org/), a package in Tidymodels, provides infrastructure for efficient data splitting and resampling:
🙌 We are now ready to train a model by fitting the training features to the training label (color).
@ -294,11 +284,11 @@ log_reg_wf <- workflow() %>%
# Print out the workflow
log_reg_wf
```
After a workflow has been *specified*, a model can be `trained` using the [`fit()`](https://tidymodels.github.io/parsnip/reference/fit.html) function. The workflow will estimate a recipe and preprocess the data before training, so we won't have to manually do that using prep and bake.
```{r train}
# Train the model
wf_fit <- log_reg_wf %>%
@ -307,6 +297,7 @@ wf_fit <- log_reg_wf %>%
# Print the trained workflow
wf_fit
```
The model print out shows the coefficients learned during training.
@ -338,8 +329,6 @@ The [**`conf_mat()`**](https://tidymodels.github.io/yardstick/reference/conf_mat
```{r conf_mat}
# Confusion matrix for prediction results
conf_mat(data = results, truth = color, estimate = .pred_class)
```
Let's interpret the confusion matrix. Our model is asked to classify pumpkins between two binary categories, category `white` and category `not-white`
@ -418,5 +407,3 @@ But for now, congratulations 🎉🎉🎉! You've completed these regression les
You R awesome!
Vidushi Gupta
1 year ago
committed by
GitHub