You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
676 lines
35 KiB
676 lines
35 KiB
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Build a logistic regression model - Lesson 4\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"#### **[Pre-lecture quiz](https://gray-sand-07a10f403.1.azurestaticapps.net/quiz/15/)**\n",
|
|
"\n",
|
|
"#### Introduction\n",
|
|
"\n",
|
|
"In this final lesson on Regression, one of the basic *classic* ML techniques, we will take a look at Logistic Regression. You would use this technique to discover patterns to predict binary categories. Is this candy chocolate or not? Is this disease contagious or not? Will this customer choose this product or not?\n",
|
|
"\n",
|
|
"In this lesson, you will learn:\n",
|
|
"\n",
|
|
"- Techniques for logistic regression\n",
|
|
"\n",
|
|
"✅ Deepen your understanding of working with this type of regression in this [Learn module](https://learn.microsoft.com/training/modules/introduction-classification-models/?WT.mc_id=academic-77952-leestott)\n",
|
|
"\n",
|
|
"## Prerequisite\n",
|
|
"\n",
|
|
"Having worked with the pumpkin data, we are now familiar enough with it to realize that there's one binary category that we can work with: `Color`.\n",
|
|
"\n",
|
|
"Let's build a logistic regression model to predict that, given some variables, *what color a given pumpkin is likely to be* (orange 🎃 or white 👻).\n",
|
|
"\n",
|
|
"> Why are we talking about binary classification in a lesson grouping about regression? Only for linguistic convenience, as logistic regression is [really a classification method](https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression), albeit a linear-based one. Learn about other ways to classify data in the next lesson group.\n",
|
|
"\n",
|
|
"For this lesson, we'll require the following packages:\n",
|
|
"\n",
|
|
"- `tidyverse`: The [tidyverse](https://www.tidyverse.org/) is a [collection of R packages](https://www.tidyverse.org/packages) designed to makes data science faster, easier and more fun!\n",
|
|
"\n",
|
|
"- `tidymodels`: The [tidymodels](https://www.tidymodels.org/) framework is a [collection of packages](https://www.tidymodels.org/packages/) for modeling and machine learning.\n",
|
|
"\n",
|
|
"- `janitor`: The [janitor package](https://github.com/sfirke/janitor) provides simple little tools for examining and cleaning dirty data.\n",
|
|
"\n",
|
|
"- `ggbeeswarm`: The [ggbeeswarm package](https://github.com/eclarke/ggbeeswarm) provides methods to create beeswarm-style plots using ggplot2.\n",
|
|
"\n",
|
|
"You can have them installed as:\n",
|
|
"\n",
|
|
"`install.packages(c(\"tidyverse\", \"tidymodels\", \"janitor\", \"ggbeeswarm\"))`\n",
|
|
"\n",
|
|
"Alternately, the script below checks whether you have the packages required to complete this module and installs them for you in case they are missing.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "r"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"suppressWarnings(if (!require(\"pacman\"))install.packages(\"pacman\"))\n",
|
|
"\n",
|
|
"pacman::p_load(tidyverse, tidymodels, janitor, ggbeeswarm)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## **Define the question**\n",
|
|
"\n",
|
|
"For our purposes, we will express this as a binary: 'White' or 'Not White'. There is also a 'striped' category in our dataset but there are few instances of it, so we will not use it. It disappears once we remove null values from the dataset, anyway.\n",
|
|
"\n",
|
|
"> 🎃 Fun fact, we sometimes call white pumpkins 'ghost' pumpkins. They aren't very easy to carve, so they aren't as popular as the orange ones but they are cool looking! So we could also reformulate our question as: 'Ghost' or 'Not Ghost'. 👻\n",
|
|
"\n",
|
|
"## **About logistic regression**\n",
|
|
"\n",
|
|
"Logistic regression differs from linear regression, which you learned about previously, in a few important ways.\n",
|
|
"\n",
|
|
"#### **Binary classification**\n",
|
|
"\n",
|
|
"Logistic regression does not offer the same features as linear regression. The former offers a prediction about a `binary category` (\"orange or not orange\") whereas the latter is capable of predicting `continual values`, for example given the origin of a pumpkin and the time of harvest, *how much its price will rise*.\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"### Other classifications\n",
|
|
"\n",
|
|
"There are other types of logistic regression, including multinomial and ordinal:\n",
|
|
"\n",
|
|
"- **Multinomial**, which involves having more than one category - \"Orange, White, and Striped\".\n",
|
|
"\n",
|
|
"- **Ordinal**, which involves ordered categories, useful if we wanted to order our outcomes logically, like our pumpkins that are ordered by a finite number of sizes (mini,sm,med,lg,xl,xxl).\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"#### **Variables DO NOT have to correlate**\n",
|
|
"\n",
|
|
"Remember how linear regression worked better with more correlated variables? Logistic regression is the opposite - the variables don't have to align. That works for this data which has somewhat weak correlations.\n",
|
|
"\n",
|
|
"#### **You need a lot of clean data**\n",
|
|
"\n",
|
|
"Logistic regression will give more accurate results if you use more data; our small dataset is not optimal for this task, so keep that in mind.\n",
|
|
"\n",
|
|
"✅ Think about the types of data that would lend themselves well to logistic regression\n",
|
|
"\n",
|
|
"## Exercise - tidy the data\n",
|
|
"\n",
|
|
"First, clean the data a bit, dropping null values and selecting only some of the columns:\n",
|
|
"\n",
|
|
"1. Add the following code:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "r"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Load the core tidyverse packages\n",
|
|
"library(tidyverse)\n",
|
|
"\n",
|
|
"# Import the data and clean column names\n",
|
|
"pumpkins <- read_csv(file = \"https://raw.githubusercontent.com/microsoft/ML-For-Beginners/main/2-Regression/data/US-pumpkins.csv\") %>% \n",
|
|
" clean_names()\n",
|
|
"\n",
|
|
"# Select desired columns\n",
|
|
"pumpkins_select <- pumpkins %>% \n",
|
|
" select(c(city_name, package, variety, origin, item_size, color)) \n",
|
|
"\n",
|
|
"# Drop rows containing missing values and encode color as factor (category)\n",
|
|
"pumpkins_select <- pumpkins_select %>% \n",
|
|
" drop_na() %>% \n",
|
|
" mutate(color = factor(color))\n",
|
|
"\n",
|
|
"# View the first few rows\n",
|
|
"pumpkins_select %>% \n",
|
|
" slice_head(n = 5)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"You can always take a peek at your new dataframe, by using the [*glimpse()*](https://pillar.r-lib.org/reference/glimpse.html) function as below:\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "r"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"pumpkins_select %>% \n",
|
|
" glimpse()\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Let's confirm that we will actually be doing a binary classification problem:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "r"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Subset distinct observations in outcome column\n",
|
|
"pumpkins_select %>% \n",
|
|
" distinct(color)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Visualization - categorical plot\n",
|
|
"By now you have loaded up the pumpkin data once again and cleaned it so as to preserve a dataset containing a few variables, including Color. Let's visualize the dataframe in the notebook using ggplot library.\n",
|
|
"\n",
|
|
"The ggplot library offers some neat ways to visualize your data. For example, you can compare distributions of the data for each Variety and Color in a categorical plot.\n",
|
|
"\n",
|
|
"1. Create such a plot by using the geombar function, using our pumpkin data, and specifying a color mapping for each pumpkin category (orange or white):"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "python"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Specify colors for each value of the hue variable\n",
|
|
"palette <- c(ORANGE = \"orange\", WHITE = \"wheat\")\n",
|
|
"\n",
|
|
"# Create the bar plot\n",
|
|
"ggplot(pumpkins_select, aes(y = variety, fill = color)) +\n",
|
|
" geom_bar(position = \"dodge\") +\n",
|
|
" scale_fill_manual(values = palette) +\n",
|
|
" labs(y = \"Variety\", fill = \"Color\") +\n",
|
|
" theme_minimal()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"By observing the data, you can see how the Color data relates to Variety.\n",
|
|
"\n",
|
|
"✅ Given this categorical plot, what are some interesting explorations you can envision?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Data pre-processing: feature encoding\n",
|
|
"\n",
|
|
"Our pumpkins dataset contains string values for all its columns. Working with categorical data is intuitive for humans but not for machines. Machine learning algorithms work well with numbers. That's why encoding is a very important step in the data pre-processing phase, since it enables us to turn categorical data into numerical data, without losing any information. Good encoding leads to building a good model.\n",
|
|
"\n",
|
|
"For feature encoding there are two main types of encoders:\n",
|
|
"\n",
|
|
"1. Ordinal encoder: it suits well for ordinal variables, which are categorical variables where their data follows a logical ordering, like the `item_size` column in our dataset. It creates a mapping such that each category is represented by a number, which is the order of the category in the column.\n",
|
|
"\n",
|
|
"2. Categorical encoder: it suits well for nominal variables, which are categorical variables where their data does not follow a logical ordering, like all the features different from `item_size` in our dataset. It is a one-hot encoding, which means that each category is represented by a binary column: the encoded variable is equal to 1 if the pumpkin belongs to that Variety and 0 otherwise.\n",
|
|
"\n",
|
|
"Tidymodels provides yet another neat package: [recipes](https://recipes.tidymodels.org/)- a package for preprocessing data. We'll define a `recipe` that specifies that all predictor columns should be encoded into a set of integers , `prep` it to estimates the required quantities and statistics needed by any operations and finally `bake` to apply the computations to new data.\n",
|
|
"\n",
|
|
"> Normally, recipes is usually used as a preprocessor for modelling where it defines what steps should be applied to a data set in order to get it ready for modelling. In that case it is **highly recommend** that you use a `workflow()` instead of manually estimating a recipe using prep and bake. We'll see all this in just a moment.\n",
|
|
">\n",
|
|
"> However for now, we are using recipes + prep + bake to specify what steps should be applied to a data set in order to get it ready for data analysis and then extract the preprocessed data with the steps applied.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "r"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Preprocess and extract data to allow some data analysis\n",
|
|
"baked_pumpkins <- recipe(color ~ ., data = pumpkins_select) %>%\n",
|
|
" # Define ordering for item_size column\n",
|
|
" step_mutate(item_size = ordered(item_size, levels = c('sml', 'med', 'med-lge', 'lge', 'xlge', 'jbo', 'exjbo'))) %>%\n",
|
|
" # Convert factors to numbers using the order defined above (Ordinal encoding)\n",
|
|
" step_integer(item_size, zero_based = F) %>%\n",
|
|
" # Encode all other predictors using one hot encoding\n",
|
|
" step_dummy(all_nominal(), -all_outcomes(), one_hot = TRUE) %>%\n",
|
|
" prep(data = pumpkin_select) %>%\n",
|
|
" bake(new_data = NULL)\n",
|
|
"\n",
|
|
"# Display the first few rows of preprocessed data\n",
|
|
"baked_pumpkins %>% \n",
|
|
" slice_head(n = 5)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"✅ What are the advantages of using an ordinal encoder for the Item Size column?\n",
|
|
"\n",
|
|
"### Analyse relationships between variables\n",
|
|
"\n",
|
|
"Now that we have pre-processed our data, we can analyse the relationships between the features and the label to grasp an idea of how well the model will be able to predict the label given the features. The best way to perform this kind of analysis is plotting the data. \n",
|
|
"We'll be using again the ggplot geom_boxplot_ function, to visualize the relationships between Item Size, Variety and Color in a categorical plot. To better plot the data we'll be using the encoded Item Size column and the unencoded Variety column.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "r"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Define the color palette\n",
|
|
"palette <- c(ORANGE = \"orange\", WHITE = \"wheat\")\n",
|
|
"\n",
|
|
"# We need the encoded Item Size column to use it as the x-axis values in the plot\n",
|
|
"pumpkins_select_plot<-pumpkins_select\n",
|
|
"pumpkins_select_plot$item_size <- baked_pumpkins$item_size\n",
|
|
"\n",
|
|
"# Create the grouped box plot\n",
|
|
"ggplot(pumpkins_select_plot, aes(x = `item_size`, y = color, fill = color)) +\n",
|
|
" geom_boxplot() +\n",
|
|
" facet_grid(variety ~ ., scales = \"free_x\") +\n",
|
|
" scale_fill_manual(values = palette) +\n",
|
|
" labs(x = \"Item Size\", y = \"\") +\n",
|
|
" theme_minimal() +\n",
|
|
" theme(strip.text = element_text(size = 12)) +\n",
|
|
" theme(axis.text.x = element_text(size = 10)) +\n",
|
|
" theme(axis.title.x = element_text(size = 12)) +\n",
|
|
" theme(axis.title.y = element_blank()) +\n",
|
|
" theme(legend.position = \"bottom\") +\n",
|
|
" guides(fill = guide_legend(title = \"Color\")) +\n",
|
|
" theme(panel.spacing = unit(0.5, \"lines\"))+\n",
|
|
" theme(strip.text.y = element_text(size = 4, hjust = 0)) \n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Use a swarm plot\n",
|
|
"\n",
|
|
"Since Color is a binary category (White or Not), it needs 'a [specialized approach](https://github.com/rstudio/cheatsheets/blob/main/data-visualization.pdf) to visualization'.\n",
|
|
"\n",
|
|
"Try a `swarm plot` to show the distribution of color with respect to the item_size.\n",
|
|
"\n",
|
|
"We'll use the [ggbeeswarm package](https://github.com/eclarke/ggbeeswarm) which provides methods to create beeswarm-style plots using ggplot2. Beeswarm plots are a way of plotting points that would ordinarily overlap so that they fall next to each other instead.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "r"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Create beeswarm plots of color and item_size\n",
|
|
"baked_pumpkins %>% \n",
|
|
" mutate(color = factor(color)) %>% \n",
|
|
" ggplot(mapping = aes(x = color, y = item_size, color = color)) +\n",
|
|
" geom_quasirandom() +\n",
|
|
" scale_color_brewer(palette = \"Dark2\", direction = -1) +\n",
|
|
" theme(legend.position = \"none\")\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Now that we have an idea of the relationship between the binary categories of color and the larger group of sizes, let's explore logistic regression to determine a given pumpkin's likely color.\n",
|
|
"\n",
|
|
"## Build your model\n",
|
|
"\n",
|
|
"Select the variables you want to use in your classification model and split the data into training and test sets. [rsample](https://rsample.tidymodels.org/), a package in Tidymodels, provides infrastructure for efficient data splitting and resampling:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "r"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Split data into 80% for training and 20% for testing\n",
|
|
"set.seed(2056)\n",
|
|
"pumpkins_split <- pumpkins_select %>% \n",
|
|
" initial_split(prop = 0.8)\n",
|
|
"\n",
|
|
"# Extract the data in each split\n",
|
|
"pumpkins_train <- training(pumpkins_split)\n",
|
|
"pumpkins_test <- testing(pumpkins_split)\n",
|
|
"\n",
|
|
"# Print out the first 5 rows of the training set\n",
|
|
"pumpkins_train %>% \n",
|
|
" slice_head(n = 5)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"🙌 We are now ready to train a model by fitting the training features to the training label (color).\n",
|
|
"\n",
|
|
"We'll begin by creating a recipe that specifies the preprocessing steps that should be carried out on our data to get it ready for modelling i.e: encoding categorical variables into a set of integers. Just like `baked_pumpkins`, we create a `pumpkins_recipe` but do not `prep` and `bake` since it would be bundled into a workflow, which you will see in just a few steps from now. \n",
|
|
"\n",
|
|
"There are quite a number of ways to specify a logistic regression model in Tidymodels. See `?logistic_reg()` For now, we'll specify a logistic regression model via the default `stats::glm()` engine.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "r"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Create a recipe that specifies preprocessing steps for modelling\n",
|
|
"pumpkins_recipe <- recipe(color ~ ., data = pumpkins_train) %>% \n",
|
|
" step_mutate(item_size = ordered(item_size, levels = c('sml', 'med', 'med-lge', 'lge', 'xlge', 'jbo', 'exjbo'))) %>%\n",
|
|
" step_integer(item_size, zero_based = F) %>% \n",
|
|
" step_dummy(all_nominal(), -all_outcomes(), one_hot = TRUE)\n",
|
|
"\n",
|
|
"# Create a logistic model specification\n",
|
|
"log_reg <- logistic_reg() %>% \n",
|
|
" set_engine(\"glm\") %>% \n",
|
|
" set_mode(\"classification\")\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Now that we have a recipe and a model specification, we need to find a way of bundling them together into an object that will first preprocess the data (prep+bake behind the scenes), fit the model on the preprocessed data and also allow for potential post-processing activities.\n",
|
|
"\n",
|
|
"In Tidymodels, this convenient object is called a [`workflow`](https://workflows.tidymodels.org/) and conveniently holds your modeling components.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "r"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Bundle modelling components in a workflow\n",
|
|
"log_reg_wf <- workflow() %>% \n",
|
|
" add_recipe(pumpkins_recipe) %>% \n",
|
|
" add_model(log_reg)\n",
|
|
"\n",
|
|
"# Print out the workflow\n",
|
|
"log_reg_wf\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"After a workflow has been *specified*, a model can be `trained` using the [`fit()`](https://tidymodels.github.io/parsnip/reference/fit.html) function. The workflow will estimate a recipe and preprocess the data before training, so we won't have to manually do that using prep and bake.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "r"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Train the model\n",
|
|
"wf_fit <- log_reg_wf %>% \n",
|
|
" fit(data = pumpkins_train)\n",
|
|
"\n",
|
|
"# Print the trained workflow\n",
|
|
"wf_fit\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"The model print out shows the coefficients learned during training.\n",
|
|
"\n",
|
|
"Now we've trained the model using the training data, we can make predictions on the test data using [parsnip::predict()](https://parsnip.tidymodels.org/reference/predict.model_fit.html). Let's start by using the model to predict labels for our test set and the probabilities for each label. When the probability is more than 0.5, the predict class is `WHITE` else `ORANGE`.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "r"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Make predictions for color and corresponding probabilities\n",
|
|
"results <- pumpkins_test %>% select(color) %>% \n",
|
|
" bind_cols(wf_fit %>% \n",
|
|
" predict(new_data = pumpkins_test)) %>%\n",
|
|
" bind_cols(wf_fit %>%\n",
|
|
" predict(new_data = pumpkins_test, type = \"prob\"))\n",
|
|
"\n",
|
|
"# Compare predictions\n",
|
|
"results %>% \n",
|
|
" slice_head(n = 10)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Very nice! This provides some more insights into how logistic regression works.\n",
|
|
"\n",
|
|
"### Better comprehension via a confusion matrix\n",
|
|
"\n",
|
|
"Comparing each prediction with its corresponding \"ground truth\" actual value isn't a very efficient way to determine how well the model is predicting. Fortunately, Tidymodels has a few more tricks up its sleeve: [`yardstick`](https://yardstick.tidymodels.org/) - a package used to measure the effectiveness of models using performance metrics.\n",
|
|
"\n",
|
|
"One performance metric associated with classification problems is the [`confusion matrix`](https://wikipedia.org/wiki/Confusion_matrix). A confusion matrix describes how well a classification model performs. A confusion matrix tabulates how many examples in each class were correctly classified by a model. In our case, it will show you how many orange pumpkins were classified as orange and how many white pumpkins were classified as white; the confusion matrix also shows you how many were classified into the **wrong** categories.\n",
|
|
"\n",
|
|
"The [**`conf_mat()`**](https://tidymodels.github.io/yardstick/reference/conf_mat.html) function from yardstick calculates this cross-tabulation of observed and predicted classes.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "r"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Confusion matrix for prediction results\n",
|
|
"conf_mat(data = results, truth = color, estimate = .pred_class)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Let's interpret the confusion matrix. Our model is asked to classify pumpkins between two binary categories, category `white` and category `not-white`\n",
|
|
"\n",
|
|
"- If your model predicts a pumpkin as white and it belongs to category 'white' in reality we call it a `true positive`, shown by the top left number.\n",
|
|
"\n",
|
|
"- If your model predicts a pumpkin as not white and it belongs to category 'white' in reality we call it a `false negative`, shown by the bottom left number.\n",
|
|
"\n",
|
|
"- If your model predicts a pumpkin as white and it belongs to category 'not-white' in reality we call it a `false positive`, shown by the top right number.\n",
|
|
"\n",
|
|
"- If your model predicts a pumpkin as not white and it belongs to category 'not-white' in reality we call it a `true negative`, shown by the bottom right number.\n",
|
|
"\n",
|
|
"| Truth |\n",
|
|
"|:-----:|\n",
|
|
"\n",
|
|
"\n",
|
|
"| | | |\n",
|
|
"|---------------|--------|-------|\n",
|
|
"| **Predicted** | WHITE | ORANGE |\n",
|
|
"| WHITE | TP | FP |\n",
|
|
"| ORANGE | FN | TN |\n",
|
|
"\n",
|
|
"As you might have guessed it's preferable to have a larger number of true positives and true negatives and a lower number of false positives and false negatives, which implies that the model performs better.\n",
|
|
"\n",
|
|
"The confusion matrix is helpful since it gives rise to other metrics that can help us better evaluate the performance of a classification model. Let's go through some of them:\n",
|
|
"\n",
|
|
"🎓 Precision: `TP/(TP + FP)` defined as the proportion of predicted positives that are actually positive. Also called [positive predictive value](https://en.wikipedia.org/wiki/Positive_predictive_value \"Positive predictive value\")\n",
|
|
"\n",
|
|
"🎓 Recall: `TP/(TP + FN)` defined as the proportion of positive results out of the number of samples which were actually positive. Also known as `sensitivity`.\n",
|
|
"\n",
|
|
"🎓 Specificity: `TN/(TN + FP)` defined as the proportion of negative results out of the number of samples which were actually negative.\n",
|
|
"\n",
|
|
"🎓 Accuracy: `TP + TN/(TP + TN + FP + FN)` The percentage of labels predicted accurately for a sample.\n",
|
|
"\n",
|
|
"🎓 F Measure: A weighted average of the precision and recall, with best being 1 and worst being 0.\n",
|
|
"\n",
|
|
"Let's calculate these metrics!\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "r"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Combine metric functions and calculate them all at once\n",
|
|
"eval_metrics <- metric_set(ppv, recall, spec, f_meas, accuracy)\n",
|
|
"eval_metrics(data = results, truth = color, estimate = .pred_class)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Visualize the ROC curve of this model\n",
|
|
"\n",
|
|
"Let's do one more visualization to see the so-called [`ROC curve`](https://en.wikipedia.org/wiki/Receiver_operating_characteristic):\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "r"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Make a roc_curve\n",
|
|
"results %>% \n",
|
|
" roc_curve(color, .pred_ORANGE) %>% \n",
|
|
" autoplot()\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"ROC curves are often used to get a view of the output of a classifier in terms of its true vs. false positives. ROC curves typically feature `True Positive Rate`/Sensitivity on the Y axis, and `False Positive Rate`/1-Specificity on the X axis. Thus, the steepness of the curve and the space between the midpoint line and the curve matter: you want a curve that quickly heads up and over the line. In our case, there are false positives to start with, and then the line heads up and over properly.\n",
|
|
"\n",
|
|
"Finally, let's use `yardstick::roc_auc()` to calculate the actual Area Under the Curve. One way of interpreting AUC is as the probability that the model ranks a random positive example more highly than a random negative example.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "r"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Calculate area under curve\n",
|
|
"results %>% \n",
|
|
" roc_auc(color, .pred_ORANGE)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"The result is around `0.975`. Given that the AUC ranges from 0 to 1, you want a big score, since a model that is 100% correct in its predictions will have an AUC of 1; in this case, the model is *pretty good*.\n",
|
|
"\n",
|
|
"In future lessons on classifications, you will learn how to improve your model's scores (such as dealing with imbalanced data in this case).\n",
|
|
"\n",
|
|
"## 🚀Challenge\n",
|
|
"\n",
|
|
"There's a lot more to unpack regarding logistic regression! But the best way to learn is to experiment. Find a dataset that lends itself to this type of analysis and build a model with it. What do you learn? tip: try [Kaggle](https://www.kaggle.com/search?q=logistic+regression+datasets) for interesting datasets.\n",
|
|
"\n",
|
|
"## Review & Self Study\n",
|
|
"\n",
|
|
"Read the first few pages of [this paper from Stanford](https://web.stanford.edu/~jurafsky/slp3/5.pdf) on some practical uses for logistic regression. Think about tasks that are better suited for one or the other type of regression tasks that we have studied up to this point. What would work best?\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"anaconda-cloud": "",
|
|
"kernelspec": {
|
|
"display_name": "R",
|
|
"langauge": "R",
|
|
"name": "ir"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": "r",
|
|
"file_extension": ".r",
|
|
"mimetype": "text/x-r-source",
|
|
"name": "R",
|
|
"pygments_lexer": "r",
|
|
"version": "3.4.1"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 1
|
|
}
|