16 KiB

Raw Permalink Blame History

Build a regression model using Scikit-learn: prepare and visualize data

Pre-lecture quiz

Dis lesson dey available for R!

Introduction

Now wey you don get all di tools wey you need to start to dey build machine learning model with Scikit-learn, you don ready to start dey ask questions about your data. As you dey work with data and apply ML solutions, e important well well to sabi how to ask correct question to fit unlock all di powers wey dey inside your dataset.

For dis lesson, you go learn:

How to prepare your data for model-building.
How to take use Matplotlib for data visualization.
How to take use Seaborn for more expressive data visualization.

How to ask the correct question about your data

The question wey you wan find answer go determine which kind ML algorithms you go use. And how correct di answer go be depend wella on di nature of your data.

Look di data wey dem provide for dis lesson. You fit open dis .csv file for VS Code. If you look am quickly, e show say e get blanks and mix of strings and numeric data. E still get one kain strange column wey dem call 'Package' wey di data na mix of 'sacks', 'bins' and other values. Di data, truth be told, na small gbege.

🎥 Click di picture wey dey above for short video wey dey show how to prepare di data for dis lesson.

For fact, e no too common to get dataset wey ready complete to use build ML model straight from di beginning. For dis lesson, you go learn how to prepare raw dataset with standard Python libraries. You go learn plenty ways to visualize di data too.

Case study: 'the pumpkin market'

Inside dis folder, you go find .csv file for di root data folder called US-pumpkins.csv wey get 1757 lines of data about pumpkin market, sorted by city groupings. Dis na raw data wey dem comot from di Specialty Crops Terminal Markets Standard Reports wey United States Department of Agriculture dey distribute.

Preparing data

Dis data dey public domain. You fit download am as many small small files, per city, for USDA website. To avoid too many small files, we don join all di city data into one spreadsheet, so we don already prepare di data small. Next, make we look di data well well.

Di pumpkin data - early conclusions

Wetin you notice for dis data? You see say e get mix of strings, numbers, blanks and strange values wey you go need make sense.

Which kind question you fit ask this data with Regression technique? Wetin about "Predict di price of pumpkin wey dem dey sell for given month". If you look di data again, e get some changes wey you go do make data fit the structure wey this work need.

Exercise - analyze di pumpkin data

Make we use Pandas, (di name mean Python Data Analysis) wey na tool wey make am easy to shape data, to analyze and prepare dis pumpkin data.

First, check for missing dates

You go first need do step to check if any date dey miss:

Change di dates to month format (dis na US dates, so format na MM/DD/YYYY).
Cut di month come put for new column.

Open notebook.ipynb file for Visual Studio Code and import di spreadsheet into new Pandas dataframe.

Use di head() function to see di first five rows.
```
import pandas as pd
pumpkins = pd.read_csv('../data/US-pumpkins.csv')
pumpkins.head()
```
✅ Wetin function you go use to see last five rows?
Check if any data dey miss for current dataframe:
```
pumpkins.isnull().sum()
```
Some data dey miss, but e fit no matter for dis work.
To make your dataframe easy to work with, select only di columns wey you need, using loc function wey go pick some rows (first parameter) and columns (second parameter) from di original dataframe. : for dis case mean "all rows".
```
columns_to_select = ['Package', 'Low Price', 'High Price', 'Date']
pumpkins = pumpkins.loc[:, columns_to_select]
```

Second, find average price of pumpkin

Think how to find average price of pumpkin for one month. Which columns you go choose for dis work? Hint: you go need 3 columns.

Solution: take average of Low Price and High Price columns to make new Price column, and change Date column to show only month. Luckily, from di check above, no date or price data dey missing.

To calculate average, add dis code:

price = (pumpkins['Low Price'] + pumpkins['High Price']) / 2

month = pd.DatetimeIndex(pumpkins['Date']).month

✅ You fit print any data you wan check with print(month).

Now, copy your converted data enter new Pandas dataframe:

new_pumpkins = pd.DataFrame({'Month': month, 'Package': pumpkins['Package'], 'Low Price': pumpkins['Low Price'],'High Price': pumpkins['High Price'], 'Price': price})

If you print your dataframe, e go show you clean, tidy dataset wey you fit use build your new regression model.

But wait! Something dey strange here

If you check Package column, pumpkins dey sell for many different ways. Some dey sell for '1 1/9 bushel' measurements, some for '1/2 bushel', some per pumpkin, some per pound, and some dey large boxes wey get different widths.

E be like say e hard well well to weigh pumpkins the same way always

When you dey check original data, e get one thing interest - anything wey get Unit of Sale wey equal 'EACH' or 'PER BIN' still get Package type wey be per inch, per bin or 'each'. E be like say e dey really hard to weigh pumpkins the same consistent way, so make we use filter take only pumpkins wey get 'bushel' string for their Package column.

Add one filter for top of di file, under di initial .csv import:
```
pumpkins = pumpkins[pumpkins['Package'].str.contains('bushel', case=True, regex=True)]
```
If you print di data now, you fit see say you dey only get di 415 or something rows of data wey get pumpkins by di bushel.

But wait! E still get one more thing to do

You notice say di bushel amount dey change for each row? You need to normalize di pricing so dat e go show di price per bushel, so make you do some math to standardize am.

Add these lines after di block wey dey create di new_pumpkins dataframe:

new_pumpkins.loc[new_pumpkins['Package'].str.contains('1 1/9'), 'Price'] = price/(1 + 1/9)

new_pumpkins.loc[new_pumpkins['Package'].str.contains('1/2'), 'Price'] = price/(1/2)

✅ According to The Spruce Eats, weight of bushel dey depend on di kind produce, as na volume measurement e be. "A bushel of tomatoes, for example, suppose weigh 56 pounds... Leaves and greens dey take more space but get less weight, so bushel of spinach na only 20 pounds." E dey quite complicated! Make we no try convert bushel to pound, but instead price am by di bushel. All dis study of bushels of pumpkins show well well how e important to sabi di nature of your data!

Now, you fit analyze di pricing per unit based on their bushel measurement. If you print di data one more time, you go fit see how e standarize.

✅ You notice say pumpkins wey dem sell by di half-bushel dey very expensive? You fit figure why? Hint: small pumpkins dey cost pass big ones, probably because plenty small pumpkins dey one bushel, while one big hollow pie pumpkin dey waste space.

Visualization Strategies

Part of di work wey data scientist dey do na to show di quality and nature of di data wey dem dey work with. To do dis, dem dey often create correct visualizations, like plots, graphs, and charts, wey go show different sides of di data. With dis kind taf taf, dem fit visually show relationships and gaps wey hard to see normally.

🎥 Click di picture above for short video wey dey show how to visualize di data for dis lesson.

Visualizations fit also help decide which machine learning method go best for di data. For example, if scatterplot dey follow one line, e show say di data good for linear regression.

One data visualization library wey dey work well for Jupyter notebooks na Matplotlib (wey you also see for di last lesson).

Get more experience with data visualization inside these tutorials.

Exercise - experiment with Matplotlib

Try create some basic plots to show di new dataframe wey you just create. Wetin basic line plot fit show?

Import Matplotlib for top of file under Pandas import:
```
import matplotlib.pyplot as plt
```
Run di complete notebook again to refresh.
For bottom of di notebook, add one cell to plot di data as box:
```
price = new_pumpkins.Price
month = new_pumpkins.Month
plt.scatter(price, month)
plt.show()
```
Na useful plot dis? E surprise you for any way?

E no too useful as e only show your data spread as points for different months.

Make am useful

To make charts show useful data, you usually need to group di data somehow. Make we try create one plot wey di y axis dey show di months and di data dey show how data dey distribute.

Add one cell to create grouped bar chart:
```
new_pumpkins.groupby(['Month'])['Price'].mean().plot(kind='bar')
plt.ylabel("Pumpkin Price")
```
Dis one na more useful data visualization! E seem to show say highest price for pumpkin dey happen for September and October. E match your expectation? Why or why not?

Exercise - experiment with Seaborn

Matplotlib strong well well, but e fit need plenty code to make one fine fine chart. Seaborn na library wey build on top of Matplotlib designed for statistical data visualization. E dey work directly with Pandas dataframes, get nice default styles, and let you create correct plots with less code. Because Seaborn dey return Matplotlib objects, you fit still use everything wey you sabi about Matplotlib to fine-tune di result.

If you never get Seaborn installed, install am with pip install seaborn.

Import Seaborn for top of di notebook, under di other imports. Dem normally dey import am as sns:
```
import seaborn as sns
```

Scatter plots to show relationships

One big part of exploring data before you build model na to look for relationships between variables. Scatter plot na one of di best tools for dis: if points dey follow one line, di two variables fit dey correlated, which be good sign say linear regression model fit work.

Recreate di price-to-month scatter plot from before, dis time use Seaborn's relplot() (relational plot), wey dey work directly with your dataframe columns:
```
sns.relplot(x="Price", y="Month", data=new_pumpkins)
```
Notice how you pass di column names and di dataframe, and Seaborn go handle di axis labels for you.
You fit change am to line plot by passing kind="line". Seaborn even go draw one shaded band wey dey show di confidence interval around di line:
```
sns.relplot(x="Price", y="Month", kind="line", data=new_pumpkins)
```
Dis data too noisy small, so line plot no be di clearest choice here — but e show how e easy you fit change chart types inside Seaborn.

Bar charts to show distributions

Before, you bin group di data by hand to create bar chart wit Matplotlib. Seaborn's catplot() (categorical plot) fit do di grouping and aggregation for you. By default kind="bar" dey show di mean of each category plus one black line wey dey show di confidence interval.

Make bar chart of average price per month:
```
sns.catplot(x="Month", y="Price", data=new_pumpkins, kind="bar")
```
Dis dey confirm wetin you see wit Matplotlib — prices dey peak around September and October — but Seaborn also dey show how much di price dey change inside each month.

Heatmaps to show correlations

Scatter plots dey compare two variables at once. When you get plenty numeric columns, one heatmap go let you see how strong di relationship between every pair of columns be at the same time. Dis na common way to find which features get strong correlation before you choose wetin to put inside model (and di same kain chart dey used later to show confusion matrices for classification).

Make correlation matrix wit Pandas, then draw am wit Seaborn's heatmap(). Di annot=True option dey print di correlation values for each cell:
```
correlations = new_pumpkins[['Month', 'Low Price', 'High Price', 'Price']].corr()
sns.heatmap(correlations, annot=True, cmap="coolwarm")
```
Values wey close to 1 (or -1) mean say di columns get strong linear correlation. See as Low Price and High Price almost get perfect correlation. Month, however, only show small linear correlation with price — even though di bar chart above show one clear seasonal peak for September and October. Dat important lesson be say: di correlation coefficient only measure straight-line relationship, so e fit miss seasonal or other non-linear patterns. ✅ Why e useful to look both heatmap and charts like di bar chart before you decide which columns to use?

Matplotlib or Seaborn?

Both libraries worth to sabi:

Matplotlib dey give you fine control over every part of chart and na di foundation nearly every other Python plotting library build on top.
Seaborn dey provide higher-level functions and better defaults for statistical charts, e work direct wit dataframes, and e quick pass sometimes for exploratory data analysis.

Normal workflow na to use Seaborn first to explore your data quick, then fall back to Matplotlib if you need customize details.

🚀Challenge

Explore di different types of visualization wey Matplotlib and Seaborn dey offer. Which ones best for regression problems?

Post-lecture quiz

Review & Self Study

Look di many ways wey you fit visualize data. Make list of the libraries wey dey available and note which one best for particular task, like 2D visualizations vs 3D visualizations. Wetin you discover?

Assignment

Exploring visualization

Disclaimer: Dis document don translate wit AI translation service Co-op Translator. Even tho we dey try make am correct, abeg make you know say automated translation fit get errors or mistakes. Di original document for dia own language na im be di correct source. For important info, make person wey sabi human translation do am. We no go responsible for any misunderstanding or wrong understanding wey fit happen because of dis translation.

16 KiB Raw Permalink Blame History