question

Swapnilkumar-8722 avatar image
0 Votes"
Swapnilkumar-8722 asked ·

Event Based trigger on Synapse is not working when the blob is created in the container through some external source.

Hello guys!
I have been trying to create an Event based Trigger and have attached it to a pipeline. Now, what is happening is when we upload a file of the format of the blob path specified through the Synapse UI, the trigger works fine.
Here's the UI I am talking about from where when we upload, the upload goes fine
66887-trigger1.png


But, when the same file is created through a script run on some other environment, the file gets created the same way, but the trigger doesn't work and the pipeline execution doesn't start.

Here's the definition of the trigger:
66913-trigger.png



Please guide me on the same.

Thanks
Swapnil kumar


azure-synapse-analytics
trigger1.png (11.2 KiB)
trigger.png (35.2 KiB)
trigger.png (23.1 KiB)
· 14
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @Swapnilkumar-8722 ,

Thanks for bringing this to our attention. My understanding is that when you upload a file to the event trigger blob location manually, then the event trigger fires as expected and runs your pipeline. But when file is created by a script run in the event trigger blob location, the event trigger didn't fire. Please correct me if I misunderstood your requirement.
If this is the case, then it clears out that the issue is not with Synapse event trigger but the way the files are being written by the script.

Could you please share few additional details about the script run to better understand the functionality of that? What is the script language? If any version, what is the specific version of script language used? Is it regular Azure Blob Storage or ADLS Gen2? If ADLS Gen2, then the event is triggered when clients use the CreateFile and FlushWithClose operations that are available in the Azure Data Lake Storage Gen2 REST API.



0 Votes 0 ·

Continuation to above comment:

If it a regular blob storage then the event is triggered when clients use the PutBlob, PutBlockList, or CopyBlob operations that are available in the Blob REST API.

67110-image.png

For ADLS Gen 2:

67147-image.png

To explore more about Event grid event schema, please refer to this doc: Event grid event schema

Thank you

0 Votes 0 ·
image.png (53.3 KiB)
image.png (35.5 KiB)

Hi @Swapnilkumar-8722 ,

Following up to see if the above information was helpful? If you need further assistance, please do share few additional details as called out in my previous comments.

Thank you

0 Votes 0 ·

Hi @KranthiPakala-MSFT
Thanks for the prompt response. the account is ADLS Gen 2.
We upload the file in a way like this:
def upload_file_to_directory():
try:

         file_system_client = service_client.get_file_system_client(file_system="my-file-system")
    
         directory_client = file_system_client.get_directory_client("my-directory")
            
         file_client = directory_client.create_file("uploaded-file.txt")
         local_file = open("C:\\file-to-upload.txt",'r')
    
         file_contents = local_file.read()
    
         file_client.append_data(data=file_contents, offset=0, length=len(file_contents))
    
         file_client.flush_data(len(file_contents))
    
     except Exception as e:
       print(e)

I tried looking online for FlushWithClose method so i could incorporate it in my code but I couldn't find much on it. it'll be great if you could guide me on the same.

0 Votes 0 ·

Hi @Swapnilkumar-8722 ,

Thanks for your response and additional details. I will look into it and will get back to you soon.

Thank you for your patience.

0 Votes 0 ·

Sure, @KranthiPakala-MSFT . Thank you so much for treating this with priority!

0 Votes 0 ·
Show more comments

0 Answers