question

LuukvanVliet-7731 avatar image
0 Votes"
LuukvanVliet-7731 asked Alden-5604 answered

Error reaching blob container from azure runbooks python 3

Hello,

I need to pull a json file from azure blob container using python 3 in azure runbooks.

  1. First I added all python packages to the automation account.

  2. I ran the following script:

from azure.storage.blob import BlobServiceClient
import json
import pandas as pd
from pandas import json_normalize
from datetime import timedelta, datetime
from sqlalchemy import create_engine
import urllib

BLOB_CONNECTION_STRING = ""

def fetch_json_from_blob(container, blob, decoder = 'utf-8-sig'):
    """ Fetches the json from blob container and returns a dictionary """
    blob_service_client = BlobServiceClient.from_connection_string(BLOB_CONNECTION_STRING)
    container_client = blob_service_client.get_container_client(container)
    json_blob = container_client.get_blob_client(blob).download_blob().readall()
    return json.loads(json_blob.decode(decoder))

def convert_timestamp(df):
    df_rows = []
    for col in df.columns:
        series = df[col]
        split = series.str.split('#', expand = True)
        split[2] = split[1].apply(lambda x: str(timedelta(milliseconds = int(x))) if x is not None else '')
        df[col] = split[0] + ' ' +  split[2]
        df_rows.append(df[col])
    return pd.concat(df_rows, axis=1)

def main():
   df_rows = []
   dict_from_blob = fetch_json_from_blob('system', 'timeclockings/2021-01-05.json') 

   for row in dict_from_blob.get('dayDetails'):
       df = json_normalize({key: row[key] for key in row.keys() & {'employee', 'clockings'}},record_path = 'clockings', meta='employee')
       df_rows.append(df)

   print(pd.concat(df_rows))


  1. I get the following error. Traceback (most recent call last): File "C:\Temp\ichlqq25.mz4\59fb6e25-60a1-46b1-b3ec-9976a2d6d2ac", line 1, in <module> from azure.storage.blob import BlobServiceClient File "C:\WPy64-3800\python-3.8.0.amd64\lib\site-packages\azure\storage\blob\__init__.py", line 10, in <module> from ._blob_client import BlobClient File "C:\WPy64-3800\python-3.8.0.amd64\lib\site-packages\azure\storage\blob\_blob_client.py", line 23, in <module> from ._shared import encode_base64ModuleNotFoundError: No module named 'azure.storage.blob._shared' However I downloaded all packages and dependencies and added them to the python packages in automation account. Please advise.





azure-automation
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@LuukvanVliet-7731 Thanks for your comment. Can you please share sample json file so that I can test it and advise you accordingly?

0 Votes 0 ·

@SwathiDhanwada-MSFT, before I proceed I was told today that Azure runbooks are not designed to perform data cleaning jobs but instead I should use azure batch to execute a python script. Am I right or would you advise the opposite?

0 Votes 0 ·

@LuukvanVliet-7731 Yes you are right in a way. For more information with regards to it, kindly refer this document where its mentioned that azure automation is used to automate, configure and update your resources.


0 Votes 0 ·

1 Answer

Alden-5604 avatar image
0 Votes"
Alden-5604 answered

I got a similar error

"""

 Traceback (most recent call last): File "C:\Temp\censored", line 2, in <module> from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient, __version__ File "C:\WPy64-3800\python-3.8.0.amd64\lib\site-packages\azure\storage\blob\__init__.py", line 10, in <module> from ._blob_client import BlobClient File "C:\WPy64-3800\python-3.8.0.amd64\lib\site-packages\azure\storage\blob\_blob_client.py", line 20, in <module> from azure.core.tracing.decorator import distributed_traceModuleNotFoundError: No module named 'azure.core'

"""

TLDR, Basically, azure runbook can't find azure.core module


Go to Azure Dashboard -> Automation Accounts -> Under Shared Resources -> Python Packages -> Click on Python 3 Packages -> Click on + sign to add package


Add azure_core-1.12.0-py2.py3-none-any.whl

File can be downloaded from official source, https://pypi.org/project/azure-core/


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.