【算法+机器学习】关于算法、机器学习的一些读书笔记，思维导图.zip

# outlier detection异常点识别方法 ## 1. isolation forest 孤立森林 ### 1.1 测试样本示例 文件 [test.pkl](https://github.com/cjn-chen/machine_learn_reading_notes/blob/master/outliner_detection/test.pkl) | index | x | y | | ----- | -------- | -------- | | 0 | -2.24055 | -2.21173 | | 1 | -1.66227 | -1.79528 | | 2 | -1.65948 | -1.58545 | | 3 | -1.65629 | -1.59716 | | 4 | -1.64114 | -1.82453 | ### 1.2 孤立森林 demo **孤立森林原理** 通过对特征进行随机划分，建立随机森林，将经过较少次数进行划分就可以划分出来的点认为时异常点。 ```python # 参考https://blog.csdn.net/ye1215172385/article/details/79762317 # 官方例子https://scikit-learn.org/stable/auto_examples/ensemble/plot_isolation_forest.html#sphx-glr-auto-examples-ensemble-plot-isolation-forest-py import numpy as np import matplotlib.pyplot as plt from sklearn.ensemble import IsolationForest rng = np.random.RandomState(42) # 构造训练样本 n_samples = 200 #样本总数 outliers_fraction = 0.25 #异常样本比例 n_inliers = int((1. - outliers_fraction) * n_samples) n_outliers = int(outliers_fraction * n_samples) X = 0.3 * rng.randn(n_inliers // 2, 2) X_train = np.r_[X + 2, X - 2] #正常样本 X_train = np.r_[X_train, np.random.uniform(low=-6, high=6, size=(n_outliers, 2))] #正常样本加上异常样本 # 构造模型并拟合 clf = IsolationForest(max_samples=n_samples, random_state=rng, contamination=outliers_fraction) clf.fit(X_train) # 计算得分并设置阈值 scores_pred = clf.decision_function(X_train) threshold = np.percentile(scores_pred, 100 * outliers_fraction) #根据训练样本中异常样本比例，得到阈值，用于绘图 # plot the line, the samples, and the nearest vectors to the plane xx, yy = np.meshgrid(np.linspace(-7, 7, 50), np.linspace(-7, 7, 50)) Z = clf.decision_function(np.c_[xx.ravel(), yy.ravel()]) Z = Z.reshape(xx.shape) plt.title("IsolationForest") # plt.contourf(xx, yy, Z, cmap=plt.cm.Blues_r) plt.contourf(xx, yy, Z, levels=np.linspace(Z.min(), threshold, 7), cmap=plt.cm.Blues_r) #绘制异常点区域，值从最小的到阈值的那部分 a = plt.contour(xx, yy, Z, levels=[threshold], linewidths=2, colors='red') #绘制异常点区域和正常点区域的边界 plt.contourf(xx, yy, Z, levels=[threshold, Z.max()], colors='palevioletred') #绘制正常点区域，值从阈值到最大的那部分 b = plt.scatter(X_train[:-n_outliers, 0], X_train[:-n_outliers, 1], c='white', s=20, edgecolor='k') c = plt.scatter(X_train[-n_outliers:, 0], X_train[-n_outliers:, 1], c='black', s=20, edgecolor='k') plt.axis('tight') plt.xlim((-7, 7)) plt.ylim((-7, 7)) plt.legend([a.collections[0], b, c], ['learned decision function', 'true inliers', 'true outliers'], loc="upper left") plt.show() ```  ### 1.3 自己修改的，X_train能够改成自己需要的数据 此处没有进行标准化，可以先进行标准化再在标准化的基础上去除异常点， from sklearn.preprocessing import StandardScaler ```python import numpy as np import matplotlib.pyplot as plt from sklearn.ensemble import IsolationForest from scipy import stats rng = np.random.RandomState(42) X_train = X_train_demo.values outliers_fraction = 0.1 n_samples = 500 # 构造模型并拟合 clf = IsolationForest(max_samples=n_samples, random_state=rng, contamination=outliers_fraction) clf.fit(X_train) # 计算得分并设置阈值 scores_pred = clf.decision_function(X_train) threshold = stats.scoreatpercentile(scores_pred, 100 * outliers_fraction) #根据训练样本中异常样本比例，得到阈值，用于绘图 # plot the line, the samples, and the nearest vectors to the plane range_max_min0 = (X_train[:,0].max()-X_train[:,0].min())*0.2 range_max_min1 = (X_train[:,1].max()-X_train[:,1].min())*0.2 xx, yy = np.meshgrid(np.linspace(X_train[:,0].min()-range_max_min0, X_train[:,0].max()+range_max_min0, 500), np.linspace(X_train[:,1].min()-range_max_min1, X_train[:,1].max()+range_max_min1, 500)) Z = clf.decision_function(np.c_[xx.ravel(), yy.ravel()]) Z = Z.reshape(xx.shape) plt.title("IsolationForest") # plt.contourf(xx, yy, Z, cmap=plt.cm.Blues_r) plt.contourf(xx, yy, Z, levels=np.linspace(Z.min(), threshold, 7), cmap=plt.cm.Blues_r) #绘制异常点区域，值从最小的到阈值的那部分 a = plt.contour(xx, yy, Z, levels=[threshold], linewidths=2, colors='red') #绘制异常点区域和正常点区域的边界 plt.contourf(xx, yy, Z, levels=[threshold, Z.max()], colors='palevioletred') #绘制正常点区域，值从阈值到最大的那部分 is_in = clf.predict(X_train)>0 b = plt.scatter(X_train[is_in, 0], X_train[is_in, 1], c='white', s=20, edgecolor='k') c = plt.scatter(X_train[~is_in, 0], X_train[~is_in, 1], c='black', s=20, edgecolor='k') plt.axis('tight') plt.xlim((X_train[:,0].min()-range_max_min0, X_train[:,0].max()+range_max_min0,)) plt.ylim((X_train[:,1].min()-range_max_min1, X_train[:,1].max()+range_max_min1,)) plt.legend([a.collections[0], b, c], ['learned decision function', 'inliers', 'outliers'], loc="upper left") plt.show() ```  ### 1.4 核心代码 #### 1.4.1 示例样本 ```python import numpy as np # 构造训练样本 n_samples = 200 #样本总数 outliers_fraction = 0.25 #异常样本比例 n_inliers = int((1. - outliers_fraction) * n_samples) n_outliers = int(outliers_fraction * n_samples) X = 0.3 * rng.randn(n_inliers // 2, 2) X_train = np.r_[X + 2, X - 2] #正常样本 X_train = np.r_[X_train, np.random.uniform(low=-6, high=6, size=(n_outliers, 2))] #正常样本加上异常样本 ``` #### 1.4.2 核心代码实现 **clf = IsolationForest(max_samples=0.8, contamination=0.25)** ```python from sklearn.ensemble import IsolationForest # fit the model # max_samples 构造一棵树使用的样本数，输入大于1的整数则使用该数字作为构造的最大样本数目， # 如果数字属于(0,1]则使用该比例的数字作为构造iforest # outliers_fraction 多少比例的样本可以作为异常值 clf = IsolationForest(max_samples=0.8, contamination=0.25) clf.fit(X_train) # y_pred_train = clf.predict(X_train) scores_pred = clf.decision_function(X_train) threshold = np.percentile(scores_pred, 100 * outliers_fraction) #根据训练样本中异常样本比例，得到阈值，用于绘图 ## 以下两种方法的筛选结果，完全相同 X_train_predict1 = X_train[clf.predict(X_train)==1] X_train_predict2 = X_train[scores_pred>=threshold,:] ``` ```python # 其中，1的表示非异常点，-1的表示为异常点 clf.predict(X_train) array([ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, -1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]) ``` ## 2. DBSCAN **DBSCAN(Density-Based Spatial Clustering of Applications with Noise) 原理** 以每个点为中心，设定邻域及邻域内需要有多少个点，如果样本点大于指定要求，则认为该点与邻域内的点属于同一类，如果小于指定值，若该