Reviewing the model scores

When the training of the machine learning models is done, you can review the data. See if the score can be improved by changing the configuration and training another version.

Analyzing the model

On the model view, we can see that the XGBoost Classification algorithm has a . This means that is the top-performing model based on the F1 score.

Do the following:

1. In the top right of the table, click . Here we can see all the available metrics for our problem. Select any metrics you want to show in the table.

2. Select XGBoost Classification, and then click Show training data metrics. You can now see the metrics from the cross-validation training (marked with a "t") and compare them to the holdout metrics.

   

3. Sort the models by performance by clicking the F1 column header. We can see that Lasso Regression, Logistic Regression, and Elastic Net Regression don't give as high scores as the other three algorithms. We might choose to exclude the low-performing algorithms or focus only on the best one—XGBoost Classification—to get faster results in the next iteration of the training.

4. Scroll down below the metrics table to see visualizations for the selected model.

   

5. Look at the Permutation importance chart. The bottom five features—DeviceType, CurrentPeriodUsage, ServiceTickets, StartMonth, and CustomerTenure—provide little or no influence on our model. They are of no value for this use case and can be seen as statistical noise. However, given the identification of data leakage in the step above, it is best to Let's remove those features to see if this improves the model scores.

   

Refining the model

Now that you have seen the results of the first run of the experiment, you can adjust which features and algorithms to include in the next iteration of the training.

Do the following:

1. With the XGBoost Classification model selected, click Configure v2 in the bottom right to open the Experiment configuration pane.

2. Under Features, clear the checkboxes for DeviceType, CurrentPeriodUsage, ServiceTickets, StartMonth, and CustomerTenure.

3. Under Algorithms, clear the checkboxes for Lasso Regression, Logistic Regression, and Elastic Net Regression.

It's time to see what effect your changes have on the model scores. The next step is to train another version of your experiment and compare the results with the first version.