Azure Automated ML(interface) choosing primary metrics to handle imbalanced data

Question

question

JiinJeong-9636 asked Jun 29, '20 ThaboMofokeng-8091 commented Apr 30, '21

Azure Automated ML(interface) choosing primary metrics to handle imbalanced data

I figured out that there are some primary metrics I can choose when I run an automated ML experiment. Yet the number of primary metrics is fewer than the run metrics in the result page. I want to deal with imbalanced data(10:1 or 20:1) and

looked up the links below:
https://docs.microsoft.com/en-us/azure/machine-learning/concept-manage-ml-pitfalls#identify-models-with-imbalanced-data
and
https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-auto-train

It seems F1 score is recommended to evaluate each model with imbalanced data.

Here are my questions:

Is there any way to set F1 score or multiple measures as a primary metric?
If there is no such way, should I do it manually?
Of all the given primary metrics, which primary metric is the most appropriate(to build a Classification model with imbalanced data)?

Thanks.

azure-machine-learning

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

ramr-msft · Jun 30, 2020 at 08:03 AM

@JiinJeong-9636 Here is the useful link that discussed about the automl settings(primary metric, number of cross-validation folds etc.).If you select another model from a sweep, choosing based on a different metric, that model is likely reasonable, though less optimized than it could be. Many metric are well correlated so choosing one tends to co-optimize the other (but to a lesser extent); some other metric pairs can be at odds. For multiple measures as a primary metric forwarded to the product team to check.

0 ·

Answer 1 · 2020-06-30T09:20:12.347Z

ramr-msft answered Jun 30, '20 ThaboMofokeng-8091 commented Apr 30, '21

For imbalanced data, it is preferred to choose AUC Weighted. Also user should then choose a metric that is appropriate to work well for imbalance. E.g. F1, micro averaged AUC, balanced accuracy for model evaluation. For primary metric (metric used for model optimization) the user should preferably choose AUC Weighted instead of accuracy.
Currently from the ml.azure.com the following metrics are supported. To add F1 score metric forwarded to product team to check on this.
https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-auto-train#primary-metric

screenshot-162.png (19.5 KiB)

· 3

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

ramr-msft · Jul 01, 2020 at 08:19 AM

@JiinJeong-9636 We have plan to support optimizing on metrics like F1 in near future.You can try AUC_weighted or norm_macro_recall depending on the use case, but for simplicity AUC_weighted should be good.

1 ·

JiinJeong-9636 ramr-msft · Jul 03, 2020 at 05:18 AM

Thank you.

2 ·

ThaboMofokeng-8091 JiinJeong-9636 · Apr 30, 2021 at 04:15 PM

Hi all,

I agree. I am presently working on a data set that is split 60/40 between the two binary outcomes 0 and 1. I find that optimizing for AUC returns better performance results compared to other metrics.

0 ·

question