python dependencies in batch api

cory simpson 1 Reputation point
2020-05-18T06:57:21.007+00:00

Hello,

I am trying to run a python script that has some package dependencies beyond that of the standard libraries. Specifically, I need pandas and numpy. Im in a windows environment. Ive been hunting for hours and cant quite find it. Here is my pool creation for now, and it doesnt work. Am I just not going about it the right way? Do I need make an "application package" or something? That seems overkill if Im able to call the command line directly. 

   task_commands = ["curl -fSsL https://bootstrap.pypa.io/get-pip.py | python", "pip install numpy", "pip install 
        pandas", "python myScript.py 1 3"]

    pool = batchmodels.PoolAddParameter(
        id=pool_id,
        virtual_machine_configuration=batchmodels.VirtualMachineConfiguration(
            image_reference=image_ref_to_use,
            node_agent_sku_id=sku_to_use),
        vm_size=vm_size,
        target_dedicated_nodes=vm_count,
        start_task=batchmodels.StartTask(
            command_line=common.helpers.wrap_commands_in_shell(windows,task_commands),
            resource_files=[batchmodels.ResourceFile(
                            file_path=task_name,
                            http_url=sas_url)]))
Azure Virtual Machines
Azure Virtual Machines
An Azure service that is used to provision Windows and Linux virtual machines.
7,125 questions
Azure Batch
Azure Batch
An Azure service that provides cloud-scale job scheduling and compute management.
301 questions

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Xing Wu 6 Reputation points Microsoft Employee
    2020-05-29T06:48:28.903+00:00

    Hi,

    I assume you try to run this start task command in standard windows image. However, the standard windows doesn't have python pre-installed. You have to install the python first. Also "curl" is not a standard windows command.

    Thanks

    1 person found this answer helpful.
    0 comments
  2. IrinaSchweisser 1 Reputation point
    2021-09-02T11:34:55.627+00:00

    Hi, I just changed the code a bit - it should work now 

      task_commands = ["curl -fSsL https://bootstrap.pypa.io/get-pip.py | python", "pip install numpy", "pip install 
         pandas", "python myScript.py 1 3"]
     pool = batchmodels.PoolAddParameter(
         id=pool_id,
         virtual_machine_configuration=batchmodels.VirtualMachineConfiguration(
             image_reference=image_ref_to_use,
             node_agent_sku_id=sku_to_use),
         vm_size=vm_size,
         target_dedicated_nodes=vm_count,
         start_task=batchmodels.StartTask(
             command_line=common.helpers.wrap_commands_in_shell(windows,task_commands),
             resource_files=[batchmodels.ResourceFile(
                             file_path=task_name,
                             http_url=sas_url)]))

    Plus os path expanduser method

    0 comments