question

SadeghHosseinpoor-9324 avatar image
0 Votes"
SadeghHosseinpoor-9324 asked ·

TensorFlow 2.2.0 update for the tensorflow estimator for Azure ML, or disable horovod?

I would like to use the tensorflow hub to retrain existing models, however tensorflow supports the hub library only on their 2.2 version. And The Estimator azure presents supports tf 2.0.

When I list tensorflow 2.2 as a required dependency as a pip package, during docker image creation the system fails - it seems like horovod is responsible, - that it cannot find the correct libraries.


Is this possible to be fixed? as in either an Estimator with tf 2.2 support, or an esitmator without the horovod - as I do not need a distributed system for my solution.

azure-machine-learning
· 3
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I would assume creating my own image from a tf 2.2 base with cuda support and then add/update conda/pip/azureml to azure ml requirements, should do it. However I have not found any documentation on custom-docker-images being able to use the available gpu on the cluster (even though I only need 1 gpu). I am fairly new to containerization - pointers would be much appreciated!

0 Votes 0 ·
romungi-MSFT avatar image romungi-MSFT SadeghHosseinpoor-9324 ·

@SadeghHosseinpoor-9324 The steps to create a custom docker image for the model is available here.

The images for tensorrt and cuda are these from the registry.

 mcr.microsoft.com/azureml/onnxruntime:latest-cuda
 mcr.microsoft.com/azureml/onnxruntime:latest-tensorrt

The repo with the custom docker images is here.



1 Vote 1 ·

When creating a custom docker image, how do I address using available gpu? when using the tensorflow estimator I am able to raise a flag (see below), but do I need to worry about this when creating my own docker image?

"If no custom_docker_image was specified, Azure ML decides between CPU base images and GPU base images based on the use_gpu flag."


Any comments on the initial question? support for tf 2.2? or an estimator without horovod enabled?

0 Votes 0 ·

1 Answer

SadeghHosseinpoor-9324 avatar image
0 Votes"
SadeghHosseinpoor-9324 answered ·

Following the pointers from @romungi-MSFT, defining estimator with gpubase image; "mcr.microsoft.com/azureml/base-gpu:openmpi3.1.2-cuda10.1-cudnn7-ubuntu18.04" solves the problem, and Tensorflow 2.2 can be included. Tensorflow uses GPU by default when available.


 estimator = Estimator(source_directory=experiment_folder,
                       compute_target=compute_target,
                       script_params=script_params,
                       entry_script='rps_efn_b0.py',
                       node_count=1,        
                       conda_packages=['ipykernel'],
                       pip_packages = ['azureml-sdk',
                                       'pyarrow',
                                       'pyspark',
                                       'azureml-mlflow',
                                       'joblib',
                                       'matplotlib',
                                       'Pillow',
                                       'tensorflow==2.2',
                                       'tensorflow-datasets',
                                       'tensorflow-hub',
                                       'azureml-defaults',
                                       'azureml-dataprep[fuse,pandas]'],
                       custom_docker_image='mcr.microsoft.com/azureml/base-gpu:openmpi3.1.2-cuda10.1-cudnn7-ubuntu18.04')

·
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.