Intelligent model optimization

Intelligent model optimization provides automatic refinement of the models you train in an experiment. With intelligent model optimization, features that can negatively affect model performance are automatically excluded from model training. With a well-prepared training dataset that includes all relevant features, you can expect intelligent model optimization to train ready-to-deploy models within a single version.

What is intelligent model optimization?

Intelligent model optimization automates many aspects of the model refinement process. Intelligent model optimization identifies and removes features that decrease a model's predictive potential.

Using intelligent model optimization

Intelligent model optimization is turned on by default in new ML experiments. You can turn it on or off for each version of the experiment that you run.

After you run an experiment version with intelligent optimization turned on, the results of the optimization can be viewed in the Model training summary. This summary is shown in the Models tab under Model insights. Hover your cursor over underlined terms to view a tooltip with a detailed description.

The Model training summary is different for each model trained in an experiment version.

During intelligent model optimization, a feature might be dropped for any of the following reasons:

• Target leakage: The feature is suspected of being affected by target leakage. Features affected target leakage include information about the target column that you are trying to predict. For example, the feature is derived directly from the target, or includes information that would not be known at time of prediction. Features causing target leakage can give you a false sense of assurance about model performance. In real-world predictions, they cause the model to perform very poorly.

• Low permutation importance: The feature does not have much, if any, influence on the model predictions. Removing these features improves model performance by reducing statistical noise.

• Highly correlated: The feature is highly correlated with one or more other features in the experiment. Features that are too highly correlated are not suitable for use in training models.

In the Data tab within the experiment, you can view insights about dropped features for each model. The Insights also refer to features that were dropped outside of the intelligent model optimization process. For more information about each insight, see Interpreting dataset insights.

How intelligent model optimization works

With intelligent model optimization:

• More models are trained than with manual optimization. Feature selection is handled at the model level. This means that unlike manual optimization, each model in a version can have different feature selection.
• For quality assurance, a baseline model – a model trained on the entire feature set you configured for the version – is still trained. This helps to check whether the intelligent optimization is, in fact, improving model scores.
• For larger training datasets, models are trained on a variety of sampling ratios. This helps to speed up the training process. For more information, see Sampling of training data.

Sampling of training data

When you are training models with a large amount of data, AutoML uses sampling to train models on a variety of subsets (sampling ratios) of the original dataset. Sampling is used to speed up the training process. At the start of the training, models are trained on a small sampling ratio. As training continues, models are gradually trained on larger portions of the data. Eventually, models are trained on the entire dataset (a sampling ratio of 100%).

When you start analyzing your models, models trained with less than 100% of the training dataset are hidden from some views. However, you can select to show these models in your results if you want to analyze them.

Turning off intelligent optimization

With intelligent optimization turned off, you are optimizing the training manually. Manual optimization can be helpful if you need more control over the training process. In particular, you might want to run a version with intelligent model optimization, then turn the setting off if you need to make a small set of manual adjustments.

Do the following:

1. In an experiment, click View configuration.

   The experiment configuration panel opens.

2. If you have already run at least one version of the experiment, click New version.

3. In the panel, expand Model optimization.

4. Switch from Intelligent to Manual.

Considerations

When working with intelligent model optimization, consider the following:

• Using intelligent model optimization does not guarantee that your training will produce high-quality models. The dataset preparation and experiment configuration stages are also essential to producing reliable models. If you do not have a well-prepared dataset, or if your configuration is missing key features, your models are not guaranteed to perform well in production use cases. For more information about these stages, see:

  • Getting your dataset ready for training

  • Configuring experiments

• When intelligent model optimization is turned on for a version, each model from this version will have a separate set of included features. On the other hand, all models from a version trained with manual optimization will have the same set of included features.

• Intelligent model optimization only uses the features and algorithms you've included in the configuration for the version.

Hyperparameter optimization

Hyperparameter optimization is not available when intelligent model optimization is turned on. To activate hyperparameter optimization, you need to set the model optimization to Manual.

For more information, see Hyperparameter optimization.

Example

For an example demonstrating the benefits of intelligent model optimization, see Example – Training models with automated machine learning.