{ "nbformat": 4, "nbformat_minor": 2, "metadata": { "colab": { "name": "lesson_2-R.ipynb", "provenance": [], "collapsed_sections": [], "toc_visible": true }, "kernelspec": { "name": "ir", "display_name": "R" }, "language_info": { "name": "R" }, "coopTranslator": { "original_hash": "f3c335f9940cfd76528b3ef918b9b342", "translation_date": "2025-09-06T15:32:51+00:00", "source_file": "2-Regression/2-Data/solution/R/lesson_2-R.ipynb", "language_code": "en" } }, "cells": [ { "cell_type": "markdown", "source": [ "# Build a regression model: prepare and visualize data\n", "\n", "## **Linear Regression for Pumpkins - Lesson 2**\n", "#### Introduction\n", "\n", "Now that you have the tools needed to start building machine learning models using Tidymodels and the Tidyverse, you're ready to begin asking questions about your data. When working with data and applying ML solutions, it's crucial to know how to ask the right questions to fully unlock the potential of your dataset.\n", "\n", "In this lesson, you will learn:\n", "\n", "- How to prepare your data for building models.\n", "\n", "- How to use `ggplot2` for visualizing data.\n", "\n", "The type of question you want answered will determine which ML algorithms you use. Additionally, the quality of the answer you receive will largely depend on the characteristics of your data.\n", "\n", "Let's explore this through a practical exercise.\n", "\n", "

\n", " \n", "

Artwork by @allison_horst
\n", "\n", "\n", "\n" ], "metadata": { "id": "Pg5aexcOPqAZ" } }, { "cell_type": "markdown", "source": [ "## 1. Importing pumpkins data and loading the Tidyverse\n", "\n", "We'll need the following packages to work through this lesson:\n", "\n", "- `tidyverse`: The [tidyverse](https://www.tidyverse.org/) is a [set of R packages](https://www.tidyverse.org/packages) created to make data science quicker, simpler, and more enjoyable!\n", "\n", "You can install them using:\n", "\n", "`install.packages(c(\"tidyverse\"))`\n", "\n", "The script below verifies if you already have the necessary packages for this module and installs any missing ones for you.\n" ], "metadata": { "id": "dc5WhyVdXAjR" } }, { "cell_type": "code", "execution_count": null, "source": [ "suppressWarnings(if(!require(\"pacman\")) install.packages(\"pacman\"))\n", "pacman::p_load(tidyverse)" ], "outputs": [], "metadata": { "id": "GqPYUZgfXOBt" } }, { "cell_type": "markdown", "source": [ "Now, let's fire up some packages and load the [data](https://github.com/microsoft/ML-For-Beginners/blob/main/2-Regression/data/US-pumpkins.csv) provided for this lesson!\n" ], "metadata": { "id": "kvjDTPDSXRr2" } }, { "cell_type": "code", "execution_count": null, "source": [ "# Load the core Tidyverse packages\n", "library(tidyverse)\n", "\n", "# Import the pumpkins data\n", "pumpkins <- read_csv(file = \"https://raw.githubusercontent.com/microsoft/ML-For-Beginners/main/2-Regression/data/US-pumpkins.csv\")\n", "\n", "\n", "# Get a glimpse and dimensions of the data\n", "glimpse(pumpkins)\n", "\n", "\n", "# Print the first 50 rows of the data set\n", "pumpkins %>% \n", " slice_head(n =50)" ], "outputs": [], "metadata": { "id": "VMri-t2zXqgD" } }, { "cell_type": "markdown", "source": [ "A quick `glimpse()` immediately reveals that there are missing values and a mix of strings (`chr`) and numeric data (`dbl`). The `Date` column is stored as a character type, and there's an unusual column called `Package` where the data includes a mix of `sacks`, `bins`, and other values. In short, the dataset is a bit messy 😤.\n", "\n", "It's actually quite rare to receive a dataset that's perfectly ready to use for building a machine learning model right away. But don't worry—this lesson will teach you how to clean and prepare a raw dataset using standard R libraries 🧑‍🔧. You'll also learn different techniques to visualize the data. 📈📊\n", "
\n", "\n", "> Quick reminder: The pipe operator (`%>%`) allows you to perform operations in a logical sequence by passing an object forward into a function or expression. You can think of the pipe operator as saying \"and then\" in your code.\n" ], "metadata": { "id": "REWcIv9yX29v" } }, { "cell_type": "markdown", "source": [ "## 2. Check for missing data\n", "\n", "One of the most common challenges data scientists face is handling incomplete or missing data. R uses a special sentinel value, `NA` (Not Available), to represent missing or unknown values.\n", "\n", "How can we determine if the data frame contains missing values?\n", "
\n", "- A simple approach is to use the base R function `anyNA`, which returns the logical values `TRUE` or `FALSE`.\n" ], "metadata": { "id": "Zxfb3AM5YbUe" } }, { "cell_type": "code", "execution_count": null, "source": [ "pumpkins %>% \n", " anyNA()" ], "outputs": [], "metadata": { "id": "G--DQutAYltj" } }, { "cell_type": "markdown", "source": [ "Great, there seems to be some missing data! That's a good place to start.\n", "\n", "- Another approach would be to use the function `is.na()` which identifies the missing elements in each column with a logical `TRUE`.\n" ], "metadata": { "id": "mU-7-SB6YokF" } }, { "cell_type": "code", "execution_count": null, "source": [ "pumpkins %>% \n", " is.na() %>% \n", " head(n = 7)" ], "outputs": [], "metadata": { "id": "W-DxDOR4YxSW" } }, { "cell_type": "markdown", "source": [ "Okay, got the job done, but with a large data frame like this, reviewing all the rows and columns individually would be inefficient and practically impossible 😴.\n", "\n", "- A more practical approach would be to calculate the total number of missing values for each column:\n" ], "metadata": { "id": "xUWxipKYY0o7" } }, { "cell_type": "code", "execution_count": null, "source": [ "pumpkins %>% \n", " is.na() %>% \n", " colSums()" ], "outputs": [], "metadata": { "id": "ZRBWV6P9ZArL" } }, { "cell_type": "markdown", "source": [ "Much better! There is missing data, but maybe it won't matter for the task at hand. Let's see what further analysis brings forth.\n", "\n", "> In addition to its impressive collection of packages and functions, R also offers excellent documentation. For example, you can use `help(colSums)` or `?colSums` to learn more about the function.\n" ], "metadata": { "id": "9gv-crB6ZD1Y" } }, { "cell_type": "markdown", "source": [ "## 3. Dplyr: A Grammar of Data Manipulation\n", "\n", "

\n", " \n", "

Illustration by @allison_horst
\n", "\n", "\n", "\n" ], "metadata": { "id": "o4jLY5-VZO2C" } }, { "cell_type": "markdown", "source": [ "[`dplyr`](https://dplyr.tidyverse.org/), a package in the Tidyverse, is a framework for data manipulation that offers a consistent set of functions to address the most common challenges in handling data. In this section, we will dive into some of the key functions provided by dplyr!\n" ], "metadata": { "id": "i5o33MQBZWWw" } }, { "cell_type": "markdown", "source": [ "#### dplyr::select()\n", "\n", "`select()` is a function in the `dplyr` package that allows you to choose which columns to keep or remove.\n", "\n", "To simplify working with your data frame, you can use `select()` to drop multiple columns and retain only the ones you need.\n", "\n", "For example, in this exercise, our analysis will focus on the columns `Package`, `Low Price`, `High Price`, and `Date`. Let's select these columns.\n" ], "metadata": { "id": "x3VGMAGBZiUr" } }, { "cell_type": "code", "execution_count": null, "source": [ "# Select desired columns\n", "pumpkins <- pumpkins %>% \n", " select(Package, `Low Price`, `High Price`, Date)\n", "\n", "\n", "# Print data set\n", "pumpkins %>% \n", " slice_head(n = 5)" ], "outputs": [], "metadata": { "id": "F_FgxQnVZnM0" } }, { "cell_type": "markdown", "source": [ "#### dplyr::mutate()\n", "\n", "`mutate()` is a function in the `dplyr` package that allows you to create or modify columns while keeping the existing ones intact.\n", "\n", "The general structure of `mutate` is:\n", "\n", "`data %>% mutate(new_column_name = what_it_contains)`\n", "\n", "Let's explore `mutate` using the `Date` column by performing the following operations:\n", "\n", "1. Convert the dates (currently stored as character type) into a month format (these are US dates, so the format is `MM/DD/YYYY`).\n", "\n", "2. Extract the month from the dates into a new column.\n", "\n", "In R, the [lubridate](https://lubridate.tidyverse.org/) package simplifies working with Date-time data. So, we'll use `dplyr::mutate()`, `lubridate::mdy()`, and `lubridate::month()` to accomplish the above tasks. We can drop the `Date` column since it won't be needed for further operations.\n" ], "metadata": { "id": "2KKo0Ed9Z1VB" } }, { "cell_type": "code", "execution_count": null, "source": [ "# Load lubridate\n", "library(lubridate)\n", "\n", "pumpkins <- pumpkins %>% \n", " # Convert the Date column to a date object\n", " mutate(Date = mdy(Date)) %>% \n", " # Extract month from Date\n", " mutate(Month = month(Date)) %>% \n", " # Drop Date column\n", " select(-Date)\n", "\n", "# View the first few rows\n", "pumpkins %>% \n", " slice_head(n = 7)" ], "outputs": [], "metadata": { "id": "5joszIVSZ6xe" } }, { "cell_type": "markdown", "source": [ "Woohoo! 🤩\n", "\n", "Next, let's create a new column `Price`, which represents the average price of a pumpkin. Now, calculate the average of the `Low Price` and `High Price` columns to fill the new Price column.\n" ], "metadata": { "id": "nIgLjNMCZ-6Y" } }, { "cell_type": "code", "execution_count": null, "source": [ "# Create a new column Price\n", "pumpkins <- pumpkins %>% \n", " mutate(Price = (`Low Price` + `High Price`)/2)\n", "\n", "# View the first few rows of the data\n", "pumpkins %>% \n", " slice_head(n = 5)" ], "outputs": [], "metadata": { "id": "Zo0BsqqtaJw2" } }, { "cell_type": "markdown", "source": [ "Yeees!💪\n", "\n", "\"But wait!\", you'll say after quickly browsing through the entire dataset with `View(pumpkins)`, \"Something seems off here!\"🤔\n", "\n", "If you examine the `Package` column, you'll notice that pumpkins are sold in various formats. Some are sold in `1 1/9 bushel` units, others in `1/2 bushel` units, some by the pumpkin, some by the pound, and some in large boxes of different sizes.\n", "\n", "Let's check this:\n" ], "metadata": { "id": "p77WZr-9aQAR" } }, { "cell_type": "code", "execution_count": null, "source": [ "# Verify the distinct observations in Package column\n", "pumpkins %>% \n", " distinct(Package)" ], "outputs": [], "metadata": { "id": "XISGfh0IaUy6" } }, { "cell_type": "markdown", "source": [ "Amazing!👏\n", "\n", "Pumpkins appear to be quite difficult to weigh consistently, so let's narrow them down by selecting only pumpkins that contain the word *bushel* in the `Package` column and store this in a new data frame `new_pumpkins`.\n" ], "metadata": { "id": "7sMjiVujaZxY" } }, { "cell_type": "markdown", "source": [ "#### dplyr::filter() and stringr::str_detect()\n", "\n", "[`dplyr::filter()`](https://dplyr.tidyverse.org/reference/filter.html): creates a subset of the data that includes only the **rows** meeting your specified conditions. In this case, it filters for pumpkins with the string *bushel* in the `Package` column.\n", "\n", "[stringr::str_detect()](https://stringr.tidyverse.org/reference/str_detect.html): checks whether a specific pattern exists in a string.\n", "\n", "The [`stringr`](https://github.com/tidyverse/stringr) package offers straightforward functions for common string manipulation tasks.\n" ], "metadata": { "id": "L8Qfcs92ageF" } }, { "cell_type": "code", "execution_count": null, "source": [ "# Retain only pumpkins with \"bushel\"\n", "new_pumpkins <- pumpkins %>% \n", " filter(str_detect(Package, \"bushel\"))\n", "\n", "# Get the dimensions of the new data\n", "dim(new_pumpkins)\n", "\n", "# View a few rows of the new data\n", "new_pumpkins %>% \n", " slice_head(n = 5)" ], "outputs": [], "metadata": { "id": "hy_SGYREampd" } }, { "cell_type": "markdown", "source": [ "You can see that we have approximately 415 rows of data focused on pumpkins by the bushel. 🤩\n" ], "metadata": { "id": "VrDwF031avlR" } }, { "cell_type": "markdown", "source": [ "#### dplyr::case_when()\n", "\n", "**But wait! There's one more thing to do**\n", "\n", "Did you notice that the bushel amount changes for each row? You need to adjust the pricing so that it reflects the cost per bushel, rather than per 1 1/9 or 1/2 bushel. Time to do some calculations to standardize it.\n", "\n", "We'll use the function [`case_when()`](https://dplyr.tidyverse.org/reference/case_when.html) to *modify* the Price column based on certain conditions. `case_when` lets you apply multiple `if_else()` statements in a vectorized way.\n" ], "metadata": { "id": "mLpw2jH4a0tx" } }, { "cell_type": "code", "execution_count": null, "source": [ "# Convert the price if the Package contains fractional bushel values\n", "new_pumpkins <- new_pumpkins %>% \n", " mutate(Price = case_when(\n", " str_detect(Package, \"1 1/9\") ~ Price/(1 + 1/9),\n", " str_detect(Package, \"1/2\") ~ Price/(1/2),\n", " TRUE ~ Price))\n", "\n", "# View the first few rows of the data\n", "new_pumpkins %>% \n", " slice_head(n = 30)" ], "outputs": [], "metadata": { "id": "P68kLVQmbM6I" } }, { "cell_type": "markdown", "source": [ "Now, we can analyze the pricing per unit based on their bushel measurement. All this study of bushels of pumpkins, however, highlights how crucial it is to truly understand your data!\n", "\n", "> ✅ According to [The Spruce Eats](https://www.thespruceeats.com/how-much-is-a-bushel-1389308), the weight of a bushel depends on the type of produce, as it's a measurement of volume. \"For instance, a bushel of tomatoes is supposed to weigh 56 pounds... Leaves and greens take up more space with less weight, so a bushel of spinach is only 20 pounds.\" It's all pretty complex! Instead of converting bushels to pounds, let's stick to pricing by the bushel. This whole exploration of pumpkin bushels, however, underscores how essential it is to understand the nature of your data!\n", ">\n", "> ✅ Did you notice that pumpkins sold by the half-bushel are quite expensive? Can you figure out why? Hint: smaller pumpkins tend to be much pricier than larger ones, likely because there are far more of them per bushel, given the extra space taken up by one large hollow pie pumpkin.\n" ], "metadata": { "id": "pS2GNPagbSdb" } }, { "cell_type": "markdown", "source": [ "Now finally, just for fun 💁‍♀️, let's move the Month column to the first position, i.e., before the Package column.\n", "\n", "You can use `dplyr::relocate()` to adjust column positions.\n" ], "metadata": { "id": "qql1SowfbdnP" } }, { "cell_type": "code", "execution_count": null, "source": [ "# Create a new data frame new_pumpkins\n", "new_pumpkins <- new_pumpkins %>% \n", " relocate(Month, .before = Package)\n", "\n", "new_pumpkins %>% \n", " slice_head(n = 7)" ], "outputs": [], "metadata": { "id": "JJ1x6kw8bixF" } }, { "cell_type": "markdown", "source": [ "Good job!👌 You now have a clean, organized dataset ready to build your new regression model!\n" ], "metadata": { "id": "y8TJ0Za_bn5Y" } }, { "cell_type": "markdown", "source": [ "## 4. Data visualization with ggplot2\n", "\n", "

\n", " \n", "

Infographic by Dasani Madipalli
\n", "\n", "\n", "\n", "\n", "There’s a *wise* saying that goes like this:\n", "\n", "> \"The simple graph has brought more information to the data analyst's mind than any other device.\" --- John Tukey\n", "\n", "A key part of a data scientist's job is to showcase the quality and characteristics of the data they are working with. To achieve this, they often create engaging visualizations—such as plots, graphs, and charts—that highlight various aspects of the data. These visualizations help uncover relationships and gaps that might otherwise remain hidden.\n", "\n", "Visualizations can also guide the selection of the most suitable machine learning technique for the data. For instance, a scatterplot that appears to follow a linear pattern suggests that the data might be well-suited for a linear regression model.\n", "\n", "R provides several systems for creating graphs, but [`ggplot2`](https://ggplot2.tidyverse.org/index.html) stands out as one of the most elegant and versatile options. `ggplot2` enables you to build graphs by **combining independent components**.\n", "\n", "Let’s begin with a simple scatterplot for the Price and Month columns.\n", "\n", "In this case, we’ll start with [`ggplot()`](https://ggplot2.tidyverse.org/reference/ggplot.html), provide a dataset and aesthetic mapping (using [`aes()`](https://ggplot2.tidyverse.org/reference/aes.html)), and then add layers (like [`geom_point()`](https://ggplot2.tidyverse.org/reference/geom_point.html)) to create scatterplots.\n" ], "metadata": { "id": "mYSH6-EtbvNa" } }, { "cell_type": "code", "execution_count": null, "source": [ "# Set a theme for the plots\n", "theme_set(theme_light())\n", "\n", "# Create a scatter plot\n", "p <- ggplot(data = new_pumpkins, aes(x = Price, y = Month))\n", "p + geom_point()" ], "outputs": [], "metadata": { "id": "g2YjnGeOcLo4" } }, { "cell_type": "markdown", "source": [ "Is this a useful plot 🤷? Does anything about it surprise you?\n", "\n", "It's not particularly useful since all it does is show your data as a scatter of points for a specific month.\n", "
\n" ], "metadata": { "id": "Ml7SDCLQcPvE" } }, { "cell_type": "markdown", "source": [ "### **How do we make it useful?**\n", "\n", "To display meaningful data in charts, you often need to organize the data in some way. For example, in our case, calculating the average price of pumpkins for each month would reveal more insights into the patterns within our data. This brings us to another quick look at **dplyr**:\n", "\n", "#### `dplyr::group_by() %>% summarize()`\n", "\n", "Grouped aggregation in R can be easily performed using\n", "\n", "`dplyr::group_by() %>% summarize()`\n", "\n", "- `dplyr::group_by()` shifts the focus of analysis from the entire dataset to specific groups, such as by month.\n", "\n", "- `dplyr::summarize()` generates a new data frame with one column for each grouping variable and one column for each summary statistic you specify.\n", "\n", "For instance, we can use `dplyr::group_by() %>% summarize()` to group the pumpkins based on the **Month** column and then calculate the **average price** for each month.\n" ], "metadata": { "id": "jMakvJZIcVkh" } }, { "cell_type": "code", "execution_count": null, "source": [ "# Find the average price of pumpkins per month\r\n", "new_pumpkins %>%\r\n", " group_by(Month) %>% \r\n", " summarise(mean_price = mean(Price))" ], "outputs": [], "metadata": { "id": "6kVSUa2Bcilf" } }, { "cell_type": "markdown", "source": [ "Succinct!✨\n", "\n", "Categorical features like months are best visualized with a bar plot 📊. The layers used for creating bar charts are `geom_bar()` and `geom_col()`. Check `?geom_bar` for more details.\n", "\n", "Let’s create one!\n" ], "metadata": { "id": "Kds48GUBcj3W" } }, { "cell_type": "code", "execution_count": null, "source": [ "# Find the average price of pumpkins per month then plot a bar chart\r\n", "new_pumpkins %>%\r\n", " group_by(Month) %>% \r\n", " summarise(mean_price = mean(Price)) %>% \r\n", " ggplot(aes(x = Month, y = mean_price)) +\r\n", " geom_col(fill = \"midnightblue\", alpha = 0.7) +\r\n", " ylab(\"Pumpkin Price\")" ], "outputs": [], "metadata": { "id": "VNbU1S3BcrxO" } }, { "cell_type": "markdown", "source": [ "🤩🤩 This is a much more useful data visualization! It appears to show that pumpkin prices peak in September and October. Does that align with your expectations? Why or why not?\n", "\n", "Well done on completing the second lesson 👏! You prepared your data for building a model and discovered additional insights through visualizations!\n" ], "metadata": { "id": "zDm0VOzzcuzR" } }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n---\n\n**Disclaimer**: \nThis document has been translated using the AI translation service [Co-op Translator](https://github.com/Azure/co-op-translator). While we aim for accuracy, please note that automated translations may include errors or inaccuracies. The original document in its native language should be regarded as the authoritative source. For critical information, professional human translation is advised. We are not responsible for any misunderstandings or misinterpretations resulting from the use of this translation.\n" ] } ] }