Machine Learning 014- Building nonlinear classification Models using SVM

(Python libraries and versions used in this article: Python 3.5, Numpy 1.14, Scikit-learn 0.19, matplotlib 2.2)

Support Vector Machine (SVM) is a common discriminant method. Its basic model is to find the optimal separation hyperplane in the feature space to maximize the positive and negative sample interval in the data set. SVM is a supervised learning algorithm used to solve binary classification problems, which can solve linear problems or nonlinear problems by introducing kernel functions.

This project aims to use SVM to construct a nonlinear classification model to discriminate different categories of data sets.

1. Prepare data sets

Start by loading and looking at some of the features of the dataset. The following code loads the dataset and views its basic information

# Prepare the data set
data_path='E:\PyProjects\DataSet\FireAI/data_multivar_2_class.txt'
df=pd.read_csv(data_path,header=None)
# print(df.head()) # no problem
print(df.info()) Check the data to make sure there are no errors
dataset_X,dataset_y=df.iloc[:,:- 1],df.iloc[:,- 1]
# print(dataset_X.head())
# print(dataset_X.info())
# print(dataset_y.head()
dataset_X=dataset_X.values
dataset_y=dataset_y.values
Copy the code

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — – — – a — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

<class ‘pandas.core.frame.DataFrame’> RangeIndex: 300 entries, 0 to 299 Data columns (total 3 columns): 0 300 non-null float64 1 300 non-null float64 2 300 non-null int64 dtypes: float64(2), int64(1) memory usage: 7.1 KB None

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — – — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — –

As you can see from the results of the dx.info () function above, this data set has two characteristic attributes (0,1 column, continuous float), one tag (2 columns, discrete int, only two categories), and no missing values in each column. Then take a look at the distribution of data points in this dataset, as shown in the figure below:

# Data set visualization
def visual_2D_dataset(dataset_X,dataset_y):
    Display the two-dimensional dataset dataset_X and its corresponding category DATASet_y in a scatter graph.
    assert dataset_X.shape[1] = =2.'only support dataset with 2 features'
    plt.figure()
    classes=list(set(dataset_y)) 
    markers=['. '.', '.'o'.'v'.A '^'.'<'.'>'.'1'.'2'.'3'.'4'.'8'
             ,'s'.'p'.The '*'.'h'.'H'.'+'.'x'.'D'.'d'.'|']
    colors=['b'.'c'.'g'.'k'.'m'.'w'.'r'.'y']
    for class_id in classes:
        one_class=np.array([feature for (feature,label) in 
                   zip(dataset_X,dataset_y) if label==class_id])
        plt.scatter(one_class[:,0],one_class[:,1],marker=np.random.choice(markers,1) [0],
                    c=np.random.choice(colors,1) [0],label='class_'+str(class_id))
    plt.legend()
    
visual_2D_dataset(dataset_X,dataset_y)
Copy the code

Many of my previous articles have talked about data set processing, splitting, preparation, etc. This data set is relatively simple, so I will briefly describe it.

2. Build linear classifier with SVM

You are right, I just want to use SVM to build a linear classifier to discriminate this data set. Of course, even entry-level machine learning battering lions can see that this data set is a linearly indivisible type that needs to be solved with nonlinear classifiers. So, here, I’m using a linear classifier to fit in and see what the “bad” results are.

# As can be seen from the distribution of the data set, this data set cannot be separated by straight lines
# In order to verify our judgment, SVM is used to construct a linear classifier to classify them

Divide the entire data set into train set and test set
from sklearn.model_selection import train_test_split
train_X, test_X, train_y, test_y=train_test_split(
    dataset_X,dataset_y,test_size=0.25,random_state=42)

# print(train_X.shape) # (225, 2)
# print(train_y.shape) # (225,)
# print(test_X.shape) # (75, 2)

Initialize a SVM object with a linear kernel function.
from sklearn.svm import SVC
classifier=SVC(kernel='linear') # Build linear classifiers
classifier.fit(train_X,train_y)
Copy the code

SVC(C=1.0, cache_size=200, class_weight=None, coEF0 =0.0, decision_function_shape=’ovr’, degree=3, gamma=’auto’, Kernel =’linear’, max_iter=-1, probability=False, random_state=None, shrinking=True, TOl =0.001, verbose=False)

Then check the performance of the trained SVM linear classifier on the training set and test set. The performance report on the training set is as follows:

# Model performance report on training set:
from sklearn.metrics import classification_report
plot_classifier(classifier,train_X,train_y)  # Classification effect of classifier on training set
target_names = ['Class-0'.'Class-1']
y_pred=classifier.predict(train_X)
print(classification_report(train_y, y_pred, target_names=target_names))
Copy the code

precision recall f1-score support

Class-0 0.60 0.96 0.74 114 Class-1 0.89 0.35 0.50 111

Avg/Total 0.74 0.66 0.62 225

It is clear from the classification renderings and performance reports that this model is very bad, as bad as grandma’s house… So, not to mention, performance on the test set is, of course, a poor word…

# Classification effect of classifier on test set
plot_classifier(classifier,test_X,test_y)  

target_names = ['Class-0'.'Class-1']
y_pred=classifier.predict(test_X)
print(classification_report(test_y, y_pred, target_names=target_names))
Copy the code

precision recall f1-score support

Class-1 1.00 0.31 0.47 39

Avg/Total 0.79 0.64 0.59 75

3. Use SVM to construct nonlinear classifier

It is clear that linear classifiers cannot solve this problem, so let’s use nonlinear classifiers instead.

The use of SVM to construct nonlinear classifier mainly considers the use of different kernel functions, here refers to two kernel functions: polynomial kernel function and radial basis function.

# As you can see from the performance report above, the classification is not very good
# Therefore, we use SVM to establish nonlinear classifier and see its classification effect
# Using SVM to build nonlinear classifier mainly uses different kernel functions
# 1: Use polynomial kernel functions:
classifier_poly=SVC(kernel='poly',degree=3) Cubic polynomial equation
classifier_poly.fit(train_X,train_y)

# Performance on training set is:
plot_classifier(classifier_poly,train_X,train_y)  

target_names = ['Class-0'.'Class-1']
y_pred=classifier_poly.predict(train_X)
print(classification_report(train_y, y_pred, target_names=target_names))
Copy the code

precision recall f1-score support

Class-0 0.92 0.85 0.89 114 Class-1 0.86 0.93 0.89 111

Avg/Total 0.89 0.89 0.89 225

# The second kind: using radial basis function to build nonlinear classifier
classifier_rbf=SVC(kernel='rbf') 
classifier_rbf.fit(train_X,train_y)

# Performance on training set is:
plot_classifier(classifier_rbf,train_X,train_y)  

target_names = ['Class-0'.'Class-1']
y_pred=classifier_rbf.predict(train_X)
print(classification_report(train_y, y_pred, target_names=target_names))
Copy the code

precision recall f1-score support

Class-0 0.96 0.96 0.96 114 Class-1 0.96 0.95 0.96 111

Avg/Total 0.96 0.96 0.96 225

# # # # # # # # # # # # # # # # # # # # # # # # small * * * * * * * * * * and # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

1. It is easy to construct nonlinear classifier with SVM, as long as different kernel functions are used.

2. For this data set, the classification effect has been greatly improved after the use of nonlinear classifier, which can be seen from the performance report. Moreover, it is obvious that the classification effect of two nonlinear kernel functions, radial basis function RBF, is better than that of polynomial kernel function.

3. The model may be able to further optimize some hyperparameters to get better classification effect.

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

Note: This part of the code has been uploaded to (my Github), welcome to download.

References:

1, Classic Examples of Python machine learning, by Prateek Joshi, translated by Tao Junjie and Chen Xiaoli

Machine Learning 014- Building nonlinear classification Models using SVM