question

AlexanderPakakis avatar image
0 Votes"
AlexanderPakakis asked AlexanderPakakis commented

Run experiment fails when using a pre-build Docker image as environment

My co-workers are using pre-build docker images for our developing environment in Azure Machine Learning Service.
In a separate script, they have registered these environments with the command myenv.register(workspace=ws). In another script, I should use their environment for testing our model.

In order to get one of their environments, I use the command registered_env = Environment.get(ws, 'the-specific-environment-name')

Unfortunately, this does not work when I use registered_env for the experiment. I get the error "Authentication failed for container registry name_of_their_container_registry.azurecr.io". The experiment run works perfectly when I copy their environment definition code into my script instead of using the command registered_env = Environment.get(ws, 'the-specific-environment-name').

However, I cannot copy everytime their environment definition code into my script.
How can I get the environment into my script which has been defined in another script?

This StackOverFlow post is quite related to my problem:
https://stackoverflow.com/questions/71131403/registering-and-getting-an-environment-in-azure-machine-learning-studio-that-der



To illustrate what my problem is, here are some code samples.

This code sample is working:

 registry = ContainerRegistry()
 registry.address = <DockerRegistryAddress>
 registry.username = <UserName>
 registry.password = <Password>
 exemplarily_env_docker_image = Environment.from_docker_image('exemplarily-env_Docker-image-AzureRegistry', <DockerImageAddress>, container_registry=registry, conda_specification=None, pip_requirements=None)
    
 exemplarily_env_docker_image.python.user_managed_dependencies = True
    
 # Registering and getting of an environment that derives from a Docker Image is not working because the credentials are not saved
 exemplarily_env_docker_image.register(workspace=ws)

 model = Model(ws, 'exemplarily_model')
    
 inference_config = InferenceConfig(environment=exemplarily_env_docker_image, 
                                    source_directory='./source_dir', 
                                    entry_script='./score.py') 


 deployment_config = LocalWebservice.deploy_configuration(port=6789)
    
 service = Model.deploy(
     ws,
     "myservice",
     [model],
     inference_config,
     deployment_config,
     overwrite=True,
 )
    
 service.wait_for_deployment(show_output=True)
 print(service.get_logs())


Now, I do a small change and the code sample is not working anymore:

 registry = ContainerRegistry()
 registry.address = <DockerRegistryAddress>
 registry.username = <UserName>
 registry.password = <Password>
 exemplarily_env_docker_image = Environment.from_docker_image('exemplarily-env_Docker-image-AzureRegistry', <DockerImageAddress>, container_registry=registry, conda_specification=None, pip_requirements=None)
    
 exemplarily_env_docker_image.python.user_managed_dependencies = True

 # Registering and getting of an environment that derives from a Docker Image is not working because the credentials are not saved
 exemplarily_env_docker_image.register(workspace=ws)

 model = Model(ws, 'exemplarily_model')
    
 reg_env = Environment.get(ws, "exemplarily-env_Docker-image-AzureRegistry")
 inference_config = InferenceConfig(environment=reg_env, 
                                    source_directory='./source_dir', 
                                    entry_script='./score.py') 
    
 deployment_config = LocalWebservice.deploy_configuration(port=6789)
    
 service = Model.deploy(
     ws,
     "myservice",
     [model],
     inference_config,
     deployment_config,
     overwrite=True,
 )
    
 service.wait_for_deployment(show_output=True)
 print(service.get_logs())


What is working:

 registry = ContainerRegistry()
 registry.address = <DockerRegistryAddress>
 registry.username = <UserName>
 registry.password = <Password>
 exemplarily_env_docker_image = Environment.from_docker_image('exemplarily-env_Docker-image-AzureRegistry', <DockerImageAddress>, container_registry=registry, conda_specification=None, pip_requirements=None)
    
 exemplarily_env_docker_image.python.user_managed_dependencies = True

 # Registering and getting of an environment that derives from a Docker Image is not working because the credentials are not saved
 exemplarily_env_docker_image.save_to_directory(path="./env", overwrite=True)

 model = Model(ws, 'exemplarily_model')
    
 reg_env = Environment.load_from_directory(path="./env")
 inference_config = InferenceConfig(environment=reg_env, 
                                    source_directory='./source_dir', 
                                    entry_script='./score.py') 
    
 deployment_config = LocalWebservice.deploy_configuration(port=6789)
    
 service = Model.deploy(
     ws,
     "myservice",
     [model],
     inference_config,
     deployment_config,
     overwrite=True,
 )
    
 service.wait_for_deployment(show_output=True)
 print(service.get_logs())


Why is the middle code sample not working? Is this a bug?

azure-machine-learning
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@AlexanderPakakis-0994 Thanks for the question. We have forwarded to the product team to check on this.

0 Votes 0 ·

Thank you, Sir!

0 Votes 0 ·

1 Answer

ramr-msft avatar image
0 Votes"
ramr-msft answered AlexanderPakakis commented

@AlexanderPakakis-0994 Thanks, register() returns a registered instance of environment. If environment was previously registered the corresponding version will be returned, otherwise AzureML service will create a new version and return back to the client. Now, when you do Environment.get() with no version specified, it returns you latest registered, that might be different that you would expect playing a lot with your environment. So, can you please just see repr of the environment returned by register() and by get(). or simply check env.register(ws).version and Environment.get(ws, name).version. I believe that will shed the light on the mystery. Please do not rely on the latest version, but specify the exact version or label, latter is more flexible and preferred.

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@ramr-msft: I have added some code samples to illustrate my problem.
Why is the middle code sample not working? Is this a bug?

0 Votes 0 ·

@ramr-msft any updates?

0 Votes 0 ·