msb-public-java
/
AiLearning-Theory-Applying
mirror of https://github.com/ben1234560/AiLearning-Theory-Applying
2
6
Code Issues Projects Releases Wiki Activity

Add. Outlier Removal / Smooth

Browse Source
master
benjas 4 years ago
parent c3b408ee69
commit 008e2d5689
2 changed files with 60 additions and 38 deletions
Show Stats Download Patch File Download Diff File

49
竞赛优胜技巧/.ipynb_checkpoints/Feature Engineering Techniques-checkpoint.ipynb
Unescape View File

@ -243,7 +243,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "0ef70ec2", "id": "f72a6efa",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## 分类特征\n", "## 分类特征\n",
@ -253,7 +253,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 17, "execution_count": 17,
"id": "38d54603", "id": "224ba994",
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [
{ {
@ -280,7 +280,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "7cca3ea7", "id": "ba369f7a",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Splitting\n", "## Splitting\n",
@ -291,7 +291,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "735d477e", "id": "6512d8e2",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## 组合/转化/交互\n", "## 组合/转化/交互\n",
@ -301,7 +301,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "f79c747e", "id": "7e1bbabe",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -310,7 +310,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "12240f96", "id": "f195a2c7",
"metadata": {}, "metadata": {},
"source": [ "source": [
"这有助于LGBM将card1和card2一起去与目标关联，并不会在树节点分裂他们。\n", "这有助于LGBM将card1和card2一起去与目标关联，并不会在树节点分裂他们。\n",
@ -321,7 +321,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "baed14ad", "id": "8f2bea13",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -330,7 +330,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "56db7969", "id": "a8ab1bcb",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## 频率编码\n", "## 频率编码\n",
@ -340,7 +340,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 19, "execution_count": 19,
"id": "30ff5b0b", "id": "87cca857",
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [
{ {
@ -420,7 +420,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "13d14185", "id": "7986b33c",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## 聚合/组统计\n", "## 聚合/组统计\n",
@ -432,7 +432,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 20, "execution_count": 20,
"id": "373291cf", "id": "e8f7106e",
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [
{ {
@ -518,7 +518,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "e638bb48", "id": "9da30631",
"metadata": {}, "metadata": {},
"source": [ "source": [
"此处的功能向每一行添加color_counts该行color组的平均值。因此，LGBM 现在可以判断color_counts对它们的color组是否为极少数的部分。" "此处的功能向每一行添加color_counts该行color组的平均值。因此，LGBM 现在可以判断color_counts对它们的color组是否为极少数的部分。"
@ -526,7 +526,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "90d66547", "id": "790eb030",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## 标准化\n", "## 标准化\n",
@ -536,7 +536,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 22, "execution_count": 22,
"id": "b60781c7", "id": "14474d08",
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [
{ {
@ -611,7 +611,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "5e7c1b67", "id": "cdfb237a",
"metadata": {}, "metadata": {},
"source": [ "source": [
"或者你可以针对一列标准化另一列。例如，如果你创建一个组统计数据（如上所述）来指示D3每周的平均值。然后你可以通过" "或者你可以针对一列标准化另一列。例如，如果你创建一个组统计数据（如上所述）来指示D3每周的平均值。然后你可以通过"
@ -620,7 +620,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "19520699", "id": "d426927e",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -629,7 +629,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "28892593", "id": "78fc4991",
"metadata": {}, "metadata": {},
"source": [ "source": [
"D3_remove_time随着时间的推移，新变量不再增加，因为我们已经针对时间的影响对其进行了标准化。" "D3_remove_time随着时间的推移，新变量不再增加，因为我们已经针对时间的影响对其进行了标准化。"
@ -637,11 +637,22 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "ecf0eb32", "id": "da699903",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## 离群值去除/平滑" "## 离群值去除/平滑\n",
"通常，你希望从数据中删除异常，因为它们会混淆你的模型。然而，在风控等比赛中，我们想要发现异常，所以要谨慎使用平滑技术。\n",
"\n",
"这些方法背后的想法是确定和删除不常见的值。例如，通过使用变量的频率编码，你可以删除所有出现小于 0.1% 的值，方法是将它们替换为 -9999 之类的新值（请注意，您应该使用与 NAN 使用的值不同的值）。"
] ]
},
{
"cell_type": "code",
"execution_count": null,
"id": "19087e59",
"metadata": {},
"outputs": [],
"source": []
} }
], ],
"metadata": { "metadata": {

49
竞赛优胜技巧/Feature Engineering Techniques.ipynb
Unescape View File

@ -243,7 +243,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "0ef70ec2", "id": "f72a6efa",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## 分类特征\n", "## 分类特征\n",
@ -253,7 +253,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 17, "execution_count": 17,
"id": "38d54603", "id": "224ba994",
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [
{ {
@ -280,7 +280,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "7cca3ea7", "id": "ba369f7a",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Splitting\n", "## Splitting\n",
@ -291,7 +291,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "735d477e", "id": "6512d8e2",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## 组合/转化/交互\n", "## 组合/转化/交互\n",
@ -301,7 +301,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "f79c747e", "id": "7e1bbabe",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -310,7 +310,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "12240f96", "id": "f195a2c7",
"metadata": {}, "metadata": {},
"source": [ "source": [
"这有助于LGBM将card1和card2一起去与目标关联，并不会在树节点分裂他们。\n", "这有助于LGBM将card1和card2一起去与目标关联，并不会在树节点分裂他们。\n",
@ -321,7 +321,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "baed14ad", "id": "8f2bea13",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -330,7 +330,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "56db7969", "id": "a8ab1bcb",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## 频率编码\n", "## 频率编码\n",
@ -340,7 +340,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 19, "execution_count": 19,
"id": "30ff5b0b", "id": "87cca857",
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [
{ {
@ -420,7 +420,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "13d14185", "id": "7986b33c",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## 聚合/组统计\n", "## 聚合/组统计\n",
@ -432,7 +432,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 20, "execution_count": 20,
"id": "373291cf", "id": "e8f7106e",
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [
{ {
@ -518,7 +518,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "e638bb48", "id": "9da30631",
"metadata": {}, "metadata": {},
"source": [ "source": [
"此处的功能向每一行添加color_counts该行color组的平均值。因此，LGBM 现在可以判断color_counts对它们的color组是否为极少数的部分。" "此处的功能向每一行添加color_counts该行color组的平均值。因此，LGBM 现在可以判断color_counts对它们的color组是否为极少数的部分。"
@ -526,7 +526,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "90d66547", "id": "790eb030",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## 标准化\n", "## 标准化\n",
@ -536,7 +536,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 22, "execution_count": 22,
"id": "b60781c7", "id": "14474d08",
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [
{ {
@ -611,7 +611,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "5e7c1b67", "id": "cdfb237a",
"metadata": {}, "metadata": {},
"source": [ "source": [
"或者你可以针对一列标准化另一列。例如，如果你创建一个组统计数据（如上所述）来指示D3每周的平均值。然后你可以通过" "或者你可以针对一列标准化另一列。例如，如果你创建一个组统计数据（如上所述）来指示D3每周的平均值。然后你可以通过"
@ -620,7 +620,7 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "19520699", "id": "d426927e",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
@ -629,7 +629,7 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "28892593", "id": "78fc4991",
"metadata": {}, "metadata": {},
"source": [ "source": [
"D3_remove_time随着时间的推移，新变量不再增加，因为我们已经针对时间的影响对其进行了标准化。" "D3_remove_time随着时间的推移，新变量不再增加，因为我们已经针对时间的影响对其进行了标准化。"
@ -637,11 +637,22 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "ecf0eb32", "id": "da699903",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## 离群值去除/平滑" "## 离群值去除/平滑\n",
"通常，你希望从数据中删除异常，因为它们会混淆你的模型。然而，在风控等比赛中，我们想要发现异常，所以要谨慎使用平滑技术。\n",
"\n",
"这些方法背后的想法是确定和删除不常见的值。例如，通过使用变量的频率编码，你可以删除所有出现小于 0.1% 的值，方法是将它们替换为 -9999 之类的新值（请注意，您应该使用与 NAN 使用的值不同的值）。"
] ]
},
{
"cell_type": "code",
"execution_count": null,
"id": "19087e59",
"metadata": {},
"outputs": [],
"source": []
} }
], ],
"metadata": { "metadata": {

Write Preview
Loading…
Cancel
Save