{ "cells": [ { "cell_type": "markdown", "source": [ "## **Nigerian Music wey dem scrape from Spotify - one analysis**\n", "\n", "Clustering na one kain [Unsupervised Learning](https://wikipedia.org/wiki/Unsupervised_learning) wey dey assume say dataset no get label or say e input no dey match with any predefined output. E dey use different algorithms to arrange data wey no get label and group dem based on patterns wey e see for di data.\n", "\n", "[**Pre-lecture quiz**](https://gray-sand-07a10f403.1.azurestaticapps.net/quiz/27/)\n", "\n", "### **Introduction**\n", "\n", "[Clustering](https://link.springer.com/referenceworkentry/10.1007%2F978-0-387-30164-8_124) dey very useful for data exploration. Make we see whether e fit help us discover trends and patterns for how Nigerian people dey enjoy music.\n", "\n", "> ✅ Take small time think about wetin clustering fit do. For real life, clustering dey happen anytime you get pile of clothes wey you wan sort out for your family members 🧦👕👖🩲. For data science, clustering dey happen when you dey try analyze wetin person like or to find di characteristics of any dataset wey no get label. Clustering dey help make sense of confusion, like how you go arrange sock drawer.\n", "\n", "For work matter, clustering fit help for things like market segmentation, to know di age group wey dey buy certain items. Another example na anomaly detection, like to catch fraud for dataset wey get credit card transactions. Or you fit use clustering to find tumors for medical scans.\n", "\n", "✅ Think small about how you don see clustering 'for di wild', maybe for banking, e-commerce, or business.\n", "\n", "> 🎓 E funny say cluster analysis start for Anthropology and Psychology for di 1930s. You fit imagine how dem take use am?\n", "\n", "Another way you fit use am na to group search results - like shopping links, images, or reviews. Clustering dey useful if you get big dataset wey you wan reduce and analyze well well, so di technique fit help you sabi di data before you build other models.\n", "\n", "✅ Once you don arrange your data inside clusters, you go give am cluster Id. Dis technique fit help keep di dataset private; you fit dey refer to di data point by di cluster Id instead of di original data wey fit expose di person. You fit think of other reasons why you go use cluster Id instead of di original data?\n", "\n", "### How to start with clustering\n", "\n", "> 🎓 How we dey create clusters dey depend on how we dey gather di data points into groups. Make we break down di vocabulary:\n", ">\n", "> 🎓 ['Transductive' vs. 'inductive'](https://wikipedia.org/wiki/Transduction_(machine_learning))\n", ">\n", "> Transductive inference dey come from training cases wey dem observe wey dey match specific test cases. Inductive inference dey come from training cases wey dey form general rules wey dem go later apply to test cases.\n", ">\n", "> Example: Imagine say you get dataset wey no complete label. Some things na 'records', some na 'cds', and some no get label. Your work na to give label to di ones wey no get. If you use inductive approach, you go train model wey dey look for 'records' and 'cds', then apply di labels to di data wey no get label. Dis approach go struggle to classify things wey be 'cassettes'. But transductive approach go handle di unknown data better because e dey group similar items together before e give di group label. For dis case, clusters fit be 'round musical things' and 'square musical things'.\n", ">\n", "> 🎓 ['Non-flat' vs. 'flat' geometry](https://datascience.stackexchange.com/questions/52260/terminology-flat-geometry-in-the-context-of-clustering)\n", ">\n", "> Dis one dey come from mathematics, non-flat vs. flat geometry dey measure di distance between points either 'flat' ([Euclidean](https://wikipedia.org/wiki/Euclidean_geometry)) or 'non-flat' (non-Euclidean) way.\n", ">\n", "> 'Flat' for dis context na Euclidean geometry (di one wey dem dey teach as 'plane' geometry), and non-flat na non-Euclidean geometry. Wetin geometry get to do with machine learning? Well, since di two na mathematics, dem need common way to measure distance between points for clusters, and e fit be 'flat' or 'non-flat' way, depending on di data. [Euclidean distances](https://wikipedia.org/wiki/Euclidean_distance) dey measure di length of line between two points. [Non-Euclidean distances](https://wikipedia.org/wiki/Non-Euclidean_geometry) dey measure distance along curve. If your data no dey for plane when you visualize am, you go need special algorithm to handle am.\n", "\n", "

\n", " \n", "

Infographic by Dasani Madipalli
\n", "\n", "\n", "\n", "> 🎓 ['Distances'](https://web.stanford.edu/class/cs345a/slides/12-clustering.pdf)\n", ">\n", "> Clusters dey defined by di distance matrix, like di distance between points. Dis distance fit dey measured in different ways. Euclidean clusters dey defined by di average of di point values, and dem get 'centroid' or center point. Distance dey measured by di distance to di centroid. Non-Euclidean distances dey refer to 'clustroids', di point wey dey closest to other points. Clustroids fit dey defined in different ways.\n", ">\n", "> 🎓 ['Constrained'](https://wikipedia.org/wiki/Constrained_clustering)\n", ">\n", "> [Constrained Clustering](https://web.cs.ucdavis.edu/~davidson/Publications/ICDMTutorial.pdf) dey add 'semi-supervised' learning to dis unsupervised method. Di relationships between points dey flagged as 'cannot link' or 'must-link' so some rules go dey forced on di dataset.\n", ">\n", "> Example: If algorithm dey free to work on batch of data wey no get label or wey get small label, di clusters wey e go produce fit no make sense. For di example above, di clusters fit group 'round music things', 'square music things', 'triangular things', and 'cookies'. But if you give am some rules like (\"di item must be plastic\", \"di item must fit produce music\"), e go help di algorithm make better choices.\n", ">\n", "> 🎓 'Density'\n", ">\n", "> Data wey dey 'noisy' na di one wey dem dey call 'dense'. Di distance between points for each cluster fit dey more or less dense, or 'crowded', so dis data go need di correct clustering method. [Dis article](https://www.kdnuggets.com/2020/02/understanding-density-based-clustering.html) dey show di difference between using K-Means clustering and HDBSCAN algorithms to analyze noisy dataset wey get uneven cluster density.\n", "\n", "Learn more about clustering techniques for dis [Learn module](https://docs.microsoft.com/learn/modules/train-evaluate-cluster-models?WT.mc_id=academic-77952-leestott)\n", "\n", "### **Clustering algorithms**\n", "\n", "Plenty clustering algorithms dey, more than 100, and di one wey you go use dey depend on di data wey you get. Make we talk about di main ones:\n", "\n", "- **Hierarchical clustering**. If object dey classified by how e near another object, instead of di one wey far, clusters go form based on di distance between di members. Hierarchical clustering dey combine two clusters repeatedly.\n", "\n", "

\n", " \n", "

Infographic by Dasani Madipalli
\n", "\n", "\n", "\n", "- **Centroid clustering**. Dis popular algorithm dey require make you choose 'k', di number of clusters wey you wan form, then di algorithm go find di center point of di cluster and gather data around di point. [K-means clustering](https://wikipedia.org/wiki/K-means_clustering) na popular version of centroid clustering wey dey divide dataset into K groups wey you don define. Di center dey determined by di nearest mean, na why dem call am so. Di squared distance from di cluster dey minimized.\n", "\n", "

\n", " \n", "

Infographic by Dasani Madipalli
\n", "\n", "\n", "\n", "- **Distribution-based clustering**. Dis one dey use statistical modeling, e dey focus on di probability say data point belong to cluster, then e go assign am. Gaussian mixture methods dey belong to dis type.\n", "\n", "- **Density-based clustering**. Data points dey assigned to clusters based on di density, or how dem dey group around each other. Data points wey dey far from di group na outliers or noise. DBSCAN, Mean-shift, and OPTICS dey belong to dis type of clustering.\n", "\n", "- **Grid-based clustering**. For datasets wey get plenty dimensions, dem go create grid and divide di data among di grid cells, so di clusters go form.\n", "\n", "Di best way to learn about clustering na to try am yourself, so na wetin you go do for dis exercise.\n", "\n", "We go need some packages to complete dis module. You fit install dem like dis: `install.packages(c('tidyverse', 'tidymodels', 'DataExplorer', 'summarytools', 'plotly', 'paletteer', 'corrplot', 'patchwork'))`\n", "\n", "Or you fit use di script below to check whether you get di packages wey you need for dis module, and e go install di ones wey dey miss.\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "suppressWarnings(if(!require(\"pacman\")) install.packages(\"pacman\"))\r\n", "\r\n", "pacman::p_load('tidyverse', 'tidymodels', 'DataExplorer', 'summarytools', 'plotly', 'paletteer', 'corrplot', 'patchwork')\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "## Exercise - cluster your data\n", "\n", "Clustering na one technique wey dey work well if we fit see wetin dey happen well well, so make we start by showing our music data. Dis exercise go help us decide which method of clustering go work pass for dis kind data.\n", "\n", "Make we start sharp sharp by bringing in the data.\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# Load the core tidyverse and make it available in your current R session\r\n", "library(tidyverse)\r\n", "\r\n", "# Import the data into a tibble\r\n", "df <- read_csv(file = \"https://raw.githubusercontent.com/microsoft/ML-For-Beginners/main/5-Clustering/data/nigerian-songs.csv\")\r\n", "\r\n", "# View the first 5 rows of the data set\r\n", "df %>% \r\n", " slice_head(n = 5)\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "Sometimes, we fit wan know small more tins about our data. We fit check di `data` and `di structure` by usin di [*glimpse()*](https://pillar.r-lib.org/reference/glimpse.html) function:\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# Glimpse into the data set\r\n", "df %>% \r\n", " glimpse()\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "Good job!💪\n", "\n", "We fit see say `glimpse()` go show you di total number of rows (observations) and columns (variables), den, di first few entries of each variable for one row after di variable name. Plus, di *data type* of di variable go dey show immediately after each variable name inside `< >`.\n", "\n", "`DataExplorer::introduce()` fit summarize dis information well:\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# Describe basic information for our data\r\n", "df %>% \r\n", " introduce()\r\n", "\r\n", "# A visual display of the same\r\n", "df %>% \r\n", " plot_intro()\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "Nice one! We don sabi say our data no get any missing values.\n", "\n", "As we dey here, we fit check common central tendency statistics (like [mean](https://en.wikipedia.org/wiki/Arithmetic_mean) and [median](https://en.wikipedia.org/wiki/Median)) plus measures of dispersion (like [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation)) using `summarytools::descr()`\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# Describe common statistics\r\n", "df %>% \r\n", " descr(stats = \"common\")\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "Make we check the general values wey dey inside di data. Make you sabi say popularity fit be `0`, wey mean say na songs wey no get ranking. We go remove dem soon.\n", "\n", "> 🤔 If we dey use clustering, wey be unsupervised method wey no need labeled data, why we dey show dis data with labels? For di data exploration phase, e dey useful, but e no dey necessary for di clustering algorithms to work.\n", "\n", "### 1. Check di popular genres\n", "\n", "Make we go ahead find di most popular genres 🎶 by counting how many times dem show.\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# Popular genres\r\n", "top_genres <- df %>% \r\n", " count(artist_top_genre, sort = TRUE) %>% \r\n", "# Encode to categorical and reorder the according to count\r\n", " mutate(artist_top_genre = factor(artist_top_genre) %>% fct_inorder())\r\n", "\r\n", "# Print the top genres\r\n", "top_genres\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "Dat one go well! Dem dey talk say one picture fit talk pass thousand rows for one data frame (even though nobody dey really talk am like dat 😅). But you sabi wetin I mean, abi?\n", "\n", "One way wey you fit take show categorical data (like character or factor variables) na to use barplots. Make we do barplot for di top 10 genres:\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# Change the default gray theme\r\n", "theme_set(theme_light())\r\n", "\r\n", "# Visualize popular genres\r\n", "top_genres %>%\r\n", " slice(1:10) %>% \r\n", " ggplot(mapping = aes(x = artist_top_genre, y = n,\r\n", " fill = artist_top_genre)) +\r\n", " geom_col(alpha = 0.8) +\r\n", " paletteer::scale_fill_paletteer_d(\"rcartocolor::Vivid\") +\r\n", " ggtitle(\"Top genres\") +\r\n", " theme(plot.title = element_text(hjust = 0.5),\r\n", " # Rotates the X markers (so we can read them)\r\n", " axis.text.x = element_text(angle = 90))\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "Now e don dey easy to sabi say we get `missing` genres 🧐!\n", "\n", "> Beta visualisation go show you things wey you no expect, or e go make you ask new questions about the data - Hadley Wickham and Garrett Grolemund, [R For Data Science](https://r4ds.had.co.nz/introduction.html)\n", "\n", "Make you sabi say, when dem talk say the top genre na `Missing`, e mean say Spotify no classify am, so make we comot am.\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# Visualize popular genres\r\n", "top_genres %>%\r\n", " filter(artist_top_genre != \"Missing\") %>% \r\n", " slice(1:10) %>% \r\n", " ggplot(mapping = aes(x = artist_top_genre, y = n,\r\n", " fill = artist_top_genre)) +\r\n", " geom_col(alpha = 0.8) +\r\n", " paletteer::scale_fill_paletteer_d(\"rcartocolor::Vivid\") +\r\n", " ggtitle(\"Top genres\") +\r\n", " theme(plot.title = element_text(hjust = 0.5),\r\n", " # Rotates the X markers (so we can read them)\r\n", " axis.text.x = element_text(angle = 90))\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "From di small data wey we don check, we see say di top three genres dey control dis dataset. Make we focus on `afro dancehall`, `afropop`, and `nigerian pop`, plus make we filter di dataset make e commot anything wey get 0 popularity value (dis one mean say dem no classify am wit popularity for di dataset and we fit see am as noise for wetin we wan do):\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "nigerian_songs <- df %>% \r\n", " # Concentrate on top 3 genres\r\n", " filter(artist_top_genre %in% c(\"afro dancehall\", \"afropop\",\"nigerian pop\")) %>% \r\n", " # Remove unclassified observations\r\n", " filter(popularity != 0)\r\n", "\r\n", "\r\n", "\r\n", "# Visualize popular genres\r\n", "nigerian_songs %>%\r\n", " count(artist_top_genre) %>%\r\n", " ggplot(mapping = aes(x = artist_top_genre, y = n,\r\n", " fill = artist_top_genre)) +\r\n", " geom_col(alpha = 0.8) +\r\n", " paletteer::scale_fill_paletteer_d(\"ggsci::category10_d3\") +\r\n", " ggtitle(\"Top genres\") +\r\n", " theme(plot.title = element_text(hjust = 0.5))\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "Make we see if any clear linear relationship dey among di numerical variables wey dey our data set. Dis relationship na wetin di [correlation statistic](https://en.wikipedia.org/wiki/Correlation) dey measure mathematically.\n", "\n", "Di correlation statistic na number wey dey between -1 and 1, and e dey show how strong di relationship be. If di value dey above 0, e mean say di correlation na *positive* (when one variable high, di other one go high too). But if di value dey below 0, e mean say di correlation na *negative* (when one variable high, di other one go low).\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# Narrow down to numeric variables and fid correlation\r\n", "corr_mat <- nigerian_songs %>% \r\n", " select(where(is.numeric)) %>% \r\n", " cor()\r\n", "\r\n", "# Visualize correlation matrix\r\n", "corrplot(corr_mat, order = 'AOE', col = c('white', 'black'), bg = 'gold2') \r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "Di data no dey strongly connect, except for `energy` and `loudness`, wey make sense, because loud music dey usually get plenty energy. `Popularity` get connection wit `release date`, wey also make sense, as new songs dey likely dey more popular. Length and energy sef dey show say dem get connection.\n", "\n", "E go dey interesting to see wetin clustering algorithm fit do wit dis data!\n", "\n", "> 🎓 Make sure say you sabi say correlation no mean say e dey cause am! We get proof say dem dey connect but we no get proof say one dey cause di oda. One [funny website](https://tylervigen.com/spurious-correlations) get some pictures wey dey show dis point well.\n", "\n", "### 2. Check how di data dey spread\n", "\n", "Make we ask some more deep questions. Di genres dem dey really different for how people dey see dia danceability, based on dia popularity? Make we look di way di data for our top three genres dey spread for popularity and danceability along one x and y axis using [density plots](https://www.khanacademy.org/math/ap-statistics/density-curves-normal-distribution-ap/density-curves/v/density-curves).\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# Perform 2D kernel density estimation\r\n", "density_estimate_2d <- nigerian_songs %>% \r\n", " ggplot(mapping = aes(x = popularity, y = danceability, color = artist_top_genre)) +\r\n", " geom_density_2d(bins = 5, size = 1) +\r\n", " paletteer::scale_color_paletteer_d(\"RSkittleBrewer::wildberry\") +\r\n", " xlim(-20, 80) +\r\n", " ylim(0, 1.2)\r\n", "\r\n", "# Density plot based on the popularity\r\n", "density_estimate_pop <- nigerian_songs %>% \r\n", " ggplot(mapping = aes(x = popularity, fill = artist_top_genre, color = artist_top_genre)) +\r\n", " geom_density(size = 1, alpha = 0.5) +\r\n", " paletteer::scale_fill_paletteer_d(\"RSkittleBrewer::wildberry\") +\r\n", " paletteer::scale_color_paletteer_d(\"RSkittleBrewer::wildberry\") +\r\n", " theme(legend.position = \"none\")\r\n", "\r\n", "# Density plot based on the danceability\r\n", "density_estimate_dance <- nigerian_songs %>% \r\n", " ggplot(mapping = aes(x = danceability, fill = artist_top_genre, color = artist_top_genre)) +\r\n", " geom_density(size = 1, alpha = 0.5) +\r\n", " paletteer::scale_fill_paletteer_d(\"RSkittleBrewer::wildberry\") +\r\n", " paletteer::scale_color_paletteer_d(\"RSkittleBrewer::wildberry\")\r\n", "\r\n", "\r\n", "# Patch everything together\r\n", "library(patchwork)\r\n", "density_estimate_2d / (density_estimate_pop + density_estimate_dance)\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "We dey see say concentric circles dey align, no matter di genre. E fit be say Nigerian people taste dey meet for one level of danceability for dis genre?\n", "\n", "Generally, di three genres dey align for how dem popular and how dem fit dance to. To find clusters for dis kind data wey no too align go hard small. Make we see if scatter plot go fit help us.\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# A scatter plot of popularity and danceability\r\n", "scatter_plot <- nigerian_songs %>% \r\n", " ggplot(mapping = aes(x = popularity, y = danceability, color = artist_top_genre, shape = artist_top_genre)) +\r\n", " geom_point(size = 2, alpha = 0.8) +\r\n", " paletteer::scale_color_paletteer_d(\"futurevisions::mars\")\r\n", "\r\n", "# Add a touch of interactivity\r\n", "ggplotly(scatter_plot)\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "Scatterplot wey dey use di same axes dey show similar pattern of convergence.\n", "\n", "For general, for clustering, you fit use scatterplots take show di clusters of data, so to sabi dis kain visualization well well go help you. For di next lesson, we go carry dis filtered data use k-means clustering take find groups for di data wey dey overlap in interesting ways.\n", "\n", "## **🚀 Challenge**\n", "\n", "To prepare for di next lesson, try make chart about di different clustering algorithms wey you fit discover and use for production environment. Wetin be di kain problems wey clustering dey try solve?\n", "\n", "## [**Post-lecture quiz**](https://gray-sand-07a10f403.1.azurestaticapps.net/quiz/28/)\n", "\n", "## **Review & Self Study**\n", "\n", "Before you go apply clustering algorithms, as we don learn, e good make you understand how your dataset be. You fit read more about dis topic [here](https://www.kdnuggets.com/2019/10/right-clustering-algorithm.html)\n", "\n", "Make you sabi clustering techniques well well:\n", "\n", "- [Train and Evaluate Clustering Models using Tidymodels and friends](https://rpubs.com/eR_ic/clustering)\n", "\n", "- Bradley Boehmke & Brandon Greenwell, [*Hands-On Machine Learning with R*](https://bradleyboehmke.github.io/HOML/)*.*\n", "\n", "## **Assignment**\n", "\n", "[Research other visualizations for clustering](https://github.com/microsoft/ML-For-Beginners/blob/main/5-Clustering/1-Visualize/assignment.md)\n", "\n", "## THANK YOU TO:\n", "\n", "[Jen Looper](https://www.twitter.com/jenlooper) wey create di original Python version of dis module ♥️\n", "\n", "[`Dasani Madipalli`](https://twitter.com/dasani_decoded) wey create di amazing illustrations wey make machine learning concepts easy to understand.\n", "\n", "Happy Learning,\n", "\n", "[Eric](https://twitter.com/ericntay), Gold Microsoft Learn Student Ambassador.\n" ], "metadata": {} }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n\n\n**Disclaimer**: \nDis dokyument don use AI transle-shon service [Co-op Translator](https://github.com/Azure/co-op-translator) do di transle-shon. Even as we dey try make am accurate, abeg make you sabi say transle-shon wey machine do fit get mistake or no dey correct well. Di original dokyument for im native language na di one wey you go take as di correct source. For important mata, e good make you use professional human transle-shon. We no go fit take blame for any misunderstanding or wrong interpretation wey fit happen because you use dis transle-shon.\n\n" ] } ], "metadata": { "anaconda-cloud": "", "kernelspec": { "display_name": "R", "language": "R", "name": "ir" }, "language_info": { "codemirror_mode": "r", "file_extension": ".r", "mimetype": "text/x-r-source", "name": "R", "pygments_lexer": "r", "version": "3.4.1" }, "coopTranslator": { "original_hash": "99c36449cad3708a435f6798cfa39972", "translation_date": "2025-11-18T19:29:32+00:00", "source_file": "5-Clustering/1-Visualize/solution/R/lesson_14-R.ipynb", "language_code": "pcm" } }, "nbformat": 4, "nbformat_minor": 1 }