Scoring regression models
Regression models predict outcomes as a number, indicating the model’s best estimate of the target variable. Several metrics are generated for you to evaluate regressions models.
During the training of a regression experiment, the following charts are auto-generated to provide quick analysis of the generated models:
-
Permutation importance: A chart in which features are displayed in order from highest influence (biggest impact on model performance) to lowest influence (smallest impact on model performance). For more information, see Permutation importance.
-
SHAP importance: A chart representing how much each feature influences the predicted outcome. For more information, see SHAP importance in experiment training.
R2
R squared (R2) is a unitless measure of correlation of the features to the target. It can be expressed as: what percent closer to perfect are the model’s predictions compared to using the average value of the target?
R2 ranges from negative infinity to 1. The closer to 1, the more the variance in the target value can be explained by the feature variables. In other words, the more likely it is that you have feature important variables that would lead to accurate predictions.
The charts show an example of height plotted against age. Height is more closely correlated to age in the first chart, where R2 is 0.97, than in the second chart, where R2 is 0.56.
Two charts plotting height versus age with different R2 values
Note that a lower R2 value doesn't necessarily mean that it is a bad model. How you interpret R2 depends on the use case and the data. When considering regression model scoring, it's important to remember that a linear relationship isn't always critical. If linear regression is producing poor results while other algorithms have better performance, it might simply mean that your data can't be modeled well enough by a linear relationship.
RMSE
Root mean squared error (RMSE) can be interpreted as the average +/- difference expected between a predicted value and the actual value. It is the standard deviation of residuals (the difference between the observed value and the predicted value for a feature). RMSE is measured in the same unit as the target value.
As an example, say that our target is to predict contract value and we get RMSE = 1250. This means that, on average, the predicted value differs +/- $1,250 from the actual value.
MSE
Mean squared error (MSE) can be interpreted as the squared +/- difference between the predicted value and the actual value that we would expect to see on average. It is measured in the same unit as the target value squared.
In the example with contract value prediction, an MSE value of 1562500 would mean that model is off by +/- 1,562,500 $2. Note that the unit is dollars squared.
MAE
The mean absolute error (MAE) is the average of all absolute prediction errors, where the prediction error is the difference between the actual and the predicted value. Using the absolute value of prediction errors prevents +/- errors from canceling each other out. MAE is measured in the same unit as the target value.