\n", " \n", " 圖表由 Dasani Madipalli 製作\n", "\n", "> 🎓 ['Distances'](https://web.stanford.edu/class/cs345a/slides/12-clustering.pdf)\n", ">\n", "> 聚類由其距離矩陣定義，例如點之間的距離。這些距離可以通過幾種方式測量。歐幾里得聚類由點值的平均值定義，並包含一個“質心”或中心點。因此，距離是通過到該質心的距離來測量的。非歐幾里得距離則指“聚心”，即最接近其他點的點。聚心可以通過多種方式定義。\n", ">\n", "> 🎓 ['Constrained'](https://wikipedia.org/wiki/Constrained_clustering)\n", ">\n", "> [約束聚類](https://web.cs.ucdavis.edu/~davidson/Publications/ICDMTutorial.pdf) 在這種無監督方法中引入了“半監督”學習。點之間的關係被標記為“不能鏈接”或“必須鏈接”，因此對數據集施加了一些規則。\n", ">\n", "> 舉例：如果一個算法在一批未標記或部分標記的數據上自由運行，它生成的聚類可能質量較差。在上述例子中，聚類可能會將“圓形音樂物品”、“方形音樂物品”、“三角形物品”和“餅乾”分組。如果給定一些約束或規則（例如“物品必須是塑料製成的”、“物品需要能夠產生音樂”），這可以幫助“約束”算法做出更好的選擇。\n", ">\n", "> 🎓 'Density'\n", ">\n", "> 被認為是“密集”的數據是“噪聲”數據。每個聚類中點之間的距離可能在檢查時顯示為更密集或更稀疏，因此需要使用適當的聚類方法來分析這些數據。[這篇文章](https://www.kdnuggets.com/2020/02/understanding-density-based-clustering.html) 展示了使用 K-Means 聚類與 HDBSCAN 算法探索具有不均勻聚類密度的噪聲數據集的區別。\n", "\n", "在這個 [學習模組](https://docs.microsoft.com/learn/modules/train-evaluate-cluster-models?WT.mc_id=academic-77952-leestott) 中深入了解聚類技術。\n", "\n", "### **聚類算法**\n", "\n", "有超過 100 種聚類算法，其使用取決於手頭數據的性質。我們來討論一些主要的算法：\n", "\n", "- **層次聚類**。如果一個對象是通過其與附近對象的接近程度而分類的，而不是與更遠的對象分類，則聚類是基於其成員與其他對象的距離形成的。層次聚類的特點是反覆合併兩個聚類。\n", "\n", "
\n", " \n", " 圖表由 Dasani Madipalli 製作\n", "\n", "- **質心聚類**。這種流行的算法需要選擇“k”，即要形成的聚類數量，然後算法確定聚類的中心點並圍繞該點收集數據。[K-means 聚類](https://wikipedia.org/wiki/K-means_clustering) 是質心聚類的一種流行版本，它將數據集分為預定義的 K 組。中心由最近的平均值確定，因此得名。從聚類的平方距離被最小化。\n", "\n", "
\n", " \n", " 圖表由 Dasani Madipalli 製作\n", "\n", "- **基於分佈的聚類**。基於統計建模，分佈式聚類集中於確定數據點屬於某個聚類的概率，並據此分配。高斯混合方法屬於這種類型。\n", "\n", "- **基於密度的聚類**。數據點根據其密度或圍繞彼此的分組分配到聚類中。遠離群體的數據點被認為是異常值或噪聲。DBSCAN、Mean-shift 和 OPTICS 屬於這種類型的聚類。\n", "\n", "- **基於網格的聚類**。對於多維數據集，創建一個網格，並將數據劃分到網格的單元中，從而創建聚類。\n", "\n", "學習聚類的最佳方式是親自嘗試，所以這就是你在這個練習中要做的。\n", "\n", "我們需要一些套件來完成這個模組。你可以通過以下方式安裝它們：`install.packages(c('tidyverse', 'tidymodels', 'DataExplorer', 'summarytools', 'plotly', 'paletteer', 'corrplot', 'patchwork'))`\n", "\n", "或者，下面的腳本會檢查你是否擁有完成此模組所需的套件，並在缺少某些套件時為你安裝它們。\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "suppressWarnings(if(!require(\"pacman\")) install.packages(\"pacman\"))\r\n", "\r\n", "pacman::p_load('tidyverse', 'tidymodels', 'DataExplorer', 'summarytools', 'plotly', 'paletteer', 'corrplot', 'patchwork')\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "## 練習 - 將你的數據進行分群\n", "\n", "分群作為一種技術，透過適當的視覺化可以大大提升效果，因此讓我們從視覺化音樂數據開始。本次練習將幫助我們決定哪種分群方法最適合這些數據的特性。\n", "\n", "讓我們立即開始，先匯入數據。\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# Load the core tidyverse and make it available in your current R session\r\n", "library(tidyverse)\r\n", "\r\n", "# Import the data into a tibble\r\n", "df <- read_csv(file = \"https://raw.githubusercontent.com/microsoft/ML-For-Beginners/main/5-Clustering/data/nigerian-songs.csv\")\r\n", "\r\n", "# View the first 5 rows of the data set\r\n", "df %>% \r\n", " slice_head(n = 5)\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "有時候，我們可能希望獲得更多關於資料的資訊。我們可以使用 [*glimpse()*](https://pillar.r-lib.org/reference/glimpse.html) 函數來查看 `資料` 和 `其結構`：\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# Glimpse into the data set\r\n", "df %>% \r\n", " glimpse()\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "做得好！💪\n", "\n", "我們可以看到 `glimpse()` 會提供總行數（觀測值）和列數（變數），接著在變數名稱後面顯示每個變數的前幾個條目。此外，變數的*資料型態*會直接顯示在每個變數名稱後的 `< >` 中。\n", "\n", "`DataExplorer::introduce()` 可以將這些資訊整齊地摘要如下：\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# Describe basic information for our data\r\n", "df %>% \r\n", " introduce()\r\n", "\r\n", "# A visual display of the same\r\n", "df %>% \r\n", " plot_intro()\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "太棒了！我們剛剛了解到我們的數據沒有缺失值。\n", "\n", "既然如此，我們可以探索一些常見的集中趨勢統計（例如[平均值](https://en.wikipedia.org/wiki/Arithmetic_mean)和[中位數](https://en.wikipedia.org/wiki/Median)）以及分散程度的測量（例如[標準差](https://en.wikipedia.org/wiki/Standard_deviation)），使用`summarytools::descr()`。\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# Describe common statistics\r\n", "df %>% \r\n", " descr(stats = \"common\")\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "讓我們來看看資料的一般值。請注意，流行度可以為 `0`，這表示歌曲沒有排名。我們稍後會移除這些。\n", "\n", "> 🤔 如果我們正在使用分群分析，一種不需要標籤資料的非監督式方法，為什麼我們要展示這些帶有標籤的資料呢？在資料探索階段，這些標籤很有用，但它們並不是分群演算法運作所必需的。\n", "\n", "### 1. 探索流行的音樂類型\n", "\n", "讓我們繼續找出最受歡迎的音樂類型 🎶，方法是統計它出現的次數。\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# Popular genres\r\n", "top_genres <- df %>% \r\n", " count(artist_top_genre, sort = TRUE) %>% \r\n", "# Encode to categorical and reorder the according to count\r\n", " mutate(artist_top_genre = factor(artist_top_genre) %>% fct_inorder())\r\n", "\r\n", "# Print the top genres\r\n", "top_genres\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "那很順利！人們常說一張圖片勝過千行數據框（其實沒人真的這麼說 😅）。但你明白我的意思，對吧？\n", "\n", "視覺化分類數據（字符或因子變數）的一種方法是使用長條圖。讓我們來製作一個前10大類型的長條圖：\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# Change the default gray theme\r\n", "theme_set(theme_light())\r\n", "\r\n", "# Visualize popular genres\r\n", "top_genres %>%\r\n", " slice(1:10) %>% \r\n", " ggplot(mapping = aes(x = artist_top_genre, y = n,\r\n", " fill = artist_top_genre)) +\r\n", " geom_col(alpha = 0.8) +\r\n", " paletteer::scale_fill_paletteer_d(\"rcartocolor::Vivid\") +\r\n", " ggtitle(\"Top genres\") +\r\n", " theme(plot.title = element_text(hjust = 0.5),\r\n", " # Rotates the X markers (so we can read them)\r\n", " axis.text.x = element_text(angle = 90))\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "現在更容易辨識出我們有「缺失」的音樂類型 🧐！\n", "\n", "> 一個好的視覺化能讓你看到意想不到的事物，或者對數據提出新的問題 —— Hadley Wickham 和 Garrett Grolemund，《R For Data Science》([R For Data Science](https://r4ds.had.co.nz/introduction.html))\n", "\n", "注意，當主要音樂類型被描述為「缺失」時，這表示 Spotify 沒有對其進行分類，所以讓我們把它移除吧。\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# Visualize popular genres\r\n", "top_genres %>%\r\n", " filter(artist_top_genre != \"Missing\") %>% \r\n", " slice(1:10) %>% \r\n", " ggplot(mapping = aes(x = artist_top_genre, y = n,\r\n", " fill = artist_top_genre)) +\r\n", " geom_col(alpha = 0.8) +\r\n", " paletteer::scale_fill_paletteer_d(\"rcartocolor::Vivid\") +\r\n", " ggtitle(\"Top genres\") +\r\n", " theme(plot.title = element_text(hjust = 0.5),\r\n", " # Rotates the X markers (so we can read them)\r\n", " axis.text.x = element_text(angle = 90))\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "從簡單的數據探索中，我們了解到前三大音樂類型在這個數據集中占據主導地位。我們來專注於 `afro dancehall`、`afropop` 和 `nigerian pop`，並進一步篩選數據集，移除任何受歡迎度值為 0 的項目（這意味著它在數據集中未被歸類為受歡迎，對於我們的目的來說可以視為噪音）：\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "nigerian_songs <- df %>% \r\n", " # Concentrate on top 3 genres\r\n", " filter(artist_top_genre %in% c(\"afro dancehall\", \"afropop\",\"nigerian pop\")) %>% \r\n", " # Remove unclassified observations\r\n", " filter(popularity != 0)\r\n", "\r\n", "\r\n", "\r\n", "# Visualize popular genres\r\n", "nigerian_songs %>%\r\n", " count(artist_top_genre) %>%\r\n", " ggplot(mapping = aes(x = artist_top_genre, y = n,\r\n", " fill = artist_top_genre)) +\r\n", " geom_col(alpha = 0.8) +\r\n", " paletteer::scale_fill_paletteer_d(\"ggsci::category10_d3\") +\r\n", " ggtitle(\"Top genres\") +\r\n", " theme(plot.title = element_text(hjust = 0.5))\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "讓我們來看看數據集中數值變數之間是否存在明顯的線性關係。這種關係可以通過[相關係數](https://en.wikipedia.org/wiki/Correlation)來進行數學量化。\n", "\n", "相關係數是一個介於 -1 和 1 之間的值，用來表示關係的強度。大於 0 的值表示*正相關*（一個變數的高值通常伴隨著另一個變數的高值），而小於 0 的值表示*負相關*（一個變數的高值通常伴隨著另一個變數的低值）。\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# Narrow down to numeric variables and fid correlation\r\n", "corr_mat <- nigerian_songs %>% \r\n", " select(where(is.numeric)) %>% \r\n", " cor()\r\n", "\r\n", "# Visualize correlation matrix\r\n", "corrplot(corr_mat, order = 'AOE', col = c('white', 'black'), bg = 'gold2') \r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "數據之間的相關性並不強，除了 `energy` 和 `loudness` 之間的關係，這是可以理解的，因為嘈雜的音樂通常更有活力。`Popularity` 與 `release date` 之間也有一定的關聯，這也合理，因為較新的歌曲可能更受歡迎。`Length` 和 `energy` 似乎也有一定的相關性。\n", "\n", "這些數據交給聚類算法處理可能會很有趣！\n", "\n", "> 🎓 請注意，相關性並不意味著因果關係！我們有相關性的證據，但沒有因果關係的證據。一個[有趣的網站](https://tylervigen.com/spurious-correlations)提供了一些視覺化內容來強調這一點。\n", "\n", "### 2. 探索數據分佈\n", "\n", "讓我們提出一些更微妙的問題。基於流行度，音樂類型在舞蹈性上的感知是否有顯著差異？讓我們使用[密度圖](https://www.khanacademy.org/math/ap-statistics/density-curves-normal-distribution-ap/density-curves/v/density-curves)來檢視我們前三大音樂類型在流行度和舞蹈性上的數據分佈，沿著給定的 x 和 y 軸進行分析。\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# Perform 2D kernel density estimation\r\n", "density_estimate_2d <- nigerian_songs %>% \r\n", " ggplot(mapping = aes(x = popularity, y = danceability, color = artist_top_genre)) +\r\n", " geom_density_2d(bins = 5, size = 1) +\r\n", " paletteer::scale_color_paletteer_d(\"RSkittleBrewer::wildberry\") +\r\n", " xlim(-20, 80) +\r\n", " ylim(0, 1.2)\r\n", "\r\n", "# Density plot based on the popularity\r\n", "density_estimate_pop <- nigerian_songs %>% \r\n", " ggplot(mapping = aes(x = popularity, fill = artist_top_genre, color = artist_top_genre)) +\r\n", " geom_density(size = 1, alpha = 0.5) +\r\n", " paletteer::scale_fill_paletteer_d(\"RSkittleBrewer::wildberry\") +\r\n", " paletteer::scale_color_paletteer_d(\"RSkittleBrewer::wildberry\") +\r\n", " theme(legend.position = \"none\")\r\n", "\r\n", "# Density plot based on the danceability\r\n", "density_estimate_dance <- nigerian_songs %>% \r\n", " ggplot(mapping = aes(x = danceability, fill = artist_top_genre, color = artist_top_genre)) +\r\n", " geom_density(size = 1, alpha = 0.5) +\r\n", " paletteer::scale_fill_paletteer_d(\"RSkittleBrewer::wildberry\") +\r\n", " paletteer::scale_color_paletteer_d(\"RSkittleBrewer::wildberry\")\r\n", "\r\n", "\r\n", "# Patch everything together\r\n", "library(patchwork)\r\n", "density_estimate_2d / (density_estimate_pop + density_estimate_dance)\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "我們可以看到，無論是什麼類型，都有同心圓排列的現象。是否可能是尼日利亞的品味在這個類型的某個舞蹈性水平上趨於一致？\n", "\n", "整體而言，這三個類型在受歡迎程度和舞蹈性方面是相符的。在這些鬆散排列的數據中確定聚類將是一個挑戰。我們來看看散佈圖是否能提供支持。\n" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "# A scatter plot of popularity and danceability\r\n", "scatter_plot <- nigerian_songs %>% \r\n", " ggplot(mapping = aes(x = popularity, y = danceability, color = artist_top_genre, shape = artist_top_genre)) +\r\n", " geom_point(size = 2, alpha = 0.8) +\r\n", " paletteer::scale_color_paletteer_d(\"futurevisions::mars\")\r\n", "\r\n", "# Add a touch of interactivity\r\n", "ggplotly(scatter_plot)\r\n" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "散佈圖使用相同的坐標軸顯示了類似的收斂模式。\n", "\n", "一般來說，對於聚類分析，可以使用散佈圖來展示數據的群集，因此掌握這種可視化方法非常有用。在下一課中，我們將使用這些篩選後的數據，並利用 k-means 聚類來發現數據中有趣的重疊群組。\n", "\n", "## **🚀 挑戰**\n", "\n", "為了準備下一課，請製作一個關於各種聚類算法的圖表，這些算法可能會在生產環境中被發現和使用。聚類試圖解決哪些類型的問題？\n", "\n", "## [**課後測驗**](https://gray-sand-07a10f403.1.azurestaticapps.net/quiz/28/)\n", "\n", "## **回顧與自學**\n", "\n", "在應用聚類算法之前，正如我們所學的，了解數據集的特性是個好主意。可以在[這裡](https://www.kdnuggets.com/2019/10/right-clustering-algorithm.html)閱讀更多相關內容。\n", "\n", "加深對聚類技術的理解：\n", "\n", "- [使用 Tidymodels 和相關工具訓練並評估聚類模型](https://rpubs.com/eR_ic/clustering)\n", "\n", "- Bradley Boehmke & Brandon Greenwell, [*Hands-On Machine Learning with R*](https://bradleyboehmke.github.io/HOML/)*.*\n", "\n", "## **作業**\n", "\n", "[研究其他聚類的可視化方法](https://github.com/microsoft/ML-For-Beginners/blob/main/5-Clustering/1-Visualize/assignment.md)\n", "\n", "## 特別感謝：\n", "\n", "[Jen Looper](https://www.twitter.com/jenlooper) 創建了此模組的原始 Python 版本 ♥️\n", "\n", "[`Dasani Madipalli`](https://twitter.com/dasani_decoded) 創作了令人驚嘆的插圖，使機器學習概念更易於理解。\n", "\n", "祝學習愉快，\n", "\n", "[Eric](https://twitter.com/ericntay)，Gold Microsoft Learn 學生大使。\n" ], "metadata": {} }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n---\n\n**免責聲明**： \n本文件已使用 AI 翻譯服務 [Co-op Translator](https://github.com/Azure/co-op-translator) 進行翻譯。儘管我們致力於提供準確的翻譯，請注意自動翻譯可能包含錯誤或不準確之處。原始文件的母語版本應被視為權威來源。對於關鍵資訊，建議使用專業人工翻譯。我們對因使用此翻譯而引起的任何誤解或錯誤解釋不承擔責任。\n" ] } ], "metadata": { "anaconda-cloud": "", "kernelspec": { "display_name": "R", "language": "R", "name": "ir" }, "language_info": { "codemirror_mode": "r", "file_extension": ".r", "mimetype": "text/x-r-source", "name": "R", "pygments_lexer": "r", "version": "3.4.1" }, "coopTranslator": { "original_hash": "99c36449cad3708a435f6798cfa39972", "translation_date": "2025-09-03T20:09:49+00:00", "source_file": "5-Clustering/1-Visualize/solution/R/lesson_14-R.ipynb", "language_code": "tw" } }, "nbformat": 4, "nbformat_minor": 1 }
![](\"../../images/flat-nonflat.png\"\n",)
\n",
" \n",
" ![](\"../../images/hierarchical.png\"\n",)
\n",
" ![](\"../../images/centroid.png\"\n",)
\n",
"