Event Based trigger on Synapse is not working when the blob is created in the container through some external source.

Swapnil kumar 26

Hello guys!
I have been trying to create an Event based Trigger and have attached it to a pipeline. Now, what is happening is when we upload a file of the format of the blob path specified through the Synapse UI, the trigger works fine.
Here's the UI I am talking about from where when we upload, the upload goes fine

But, when the same file is created through a script run on some other environment, the file gets created the same way, but the trigger doesn't work and the pipeline execution doesn't start.

Here's the definition of the trigger:

Please guide me on the same.

Thanks
Swapnil kumar

KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-02-12T00:01:53.507+00:00

Hi @Swapnil kumar ,

Thanks for bringing this to our attention. My understanding is that when you upload a file to the event trigger blob location manually, then the event trigger fires as expected and runs your pipeline. But when file is created by a script run in the event trigger blob location, the event trigger didn't fire. Please correct me if I misunderstood your requirement.
If this is the case, then it clears out that the issue is not with Synapse event trigger but the way the files are being written by the script.

Could you please share few additional details about the script run to better understand the functionality of that? What is the script language? If any version, what is the specific version of script language used? Is it regular Azure Blob Storage or ADLS Gen2? If ADLS Gen2, then the event is triggered when clients use the CreateFile and FlushWithClose operations that are available in the Azure Data Lake Storage Gen2 REST API.
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-02-12T00:10:46.32+00:00

Continuation to above comment:

If it a regular blob storage then the event is triggered when clients use the PutBlob, PutBlockList, or CopyBlob operations that are available in the Blob REST API.

For ADLS Gen 2:

To explore more about Event grid event schema, please refer to this doc: Event grid event schema

Thank you
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-02-12T20:13:31.593+00:00

Hi @Swapnil kumar ,

Following up to see if the above information was helpful? If you need further assistance, please do share few additional details as called out in my previous comments.

Thank you

Swapnil kumar 26

Hi @KranthiPakala-MSFT
Thanks for the prompt response. the account is ADLS Gen 2.
We upload the file in a way like this:
def upload_file_to_directory():
try:

        file_system_client = service_client.get_file_system_client(file_system="my-file-system")  
  
        directory_client = file_system_client.get_directory_client("my-directory")  
          
        file_client = directory_client.create_file("uploaded-file.txt")  
        local_file = open("C:\\file-to-upload.txt",'r')  
  
        file_contents = local_file.read()  
  
        file_client.append_data(data=file_contents, offset=0, length=len(file_contents))  
  
        file_client.flush_data(len(file_contents))  
  
    except Exception as e:  
      print(e)

I tried looking online for FlushWithClose method so i could incorporate it in my code but I couldn't find much on it. it'll be great if you could guide me on the same.

KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-02-16T23:29:04.813+00:00

Hi @Swapnil kumar ,

Thanks for your response and additional details. I will look into it and will get back to you soon.

Thank you for your patience.
Swapnil kumar 26 Reputation points

2021-02-17T06:13:51.917+00:00

Sure, @KranthiPakala-MSFT . Thank you so much for treating this with priority!
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-02-19T23:32:49.093+00:00

HI @Swapnil kumar ,

Sorry for the delay. I tried with the sample you have provided, and the event trigger is being fired once the file is generated. Could you please double check your script with the below from this doc:

Also please try adding file_client.close() at the end and see if that helps.

I tried the steps mentioned in that doc to generate a file in ADLS and associated a event trigger to get fired once the file is created and it worked as expected. You might be doing something different than that described in the doc. Could you please verify.

Thank you
Swapnil kumar 26 Reputation points

2021-02-22T13:33:40.03+00:00

Hey @KranthiPakala-MSFT !
I tried the steps mentioned in the doc and the problem seems to persist. What we are doing is we are creating a dataframe, then with that dataframe as data we are creating a csv file according to the above mentioned. I also added the file_client.close() method but it wasn't too helpful.
For your reference, I am attaching the code we have used. the trigger has the ignore empty blobs condition set to true
Code:
if isinstance(data, pd.DataFrame):
data = data.to_csv(index=False)
file_name = f"{file}.csv"
else:
data = data.str()
file_name = f"{file}.json"
self.directory_client = self.filesystem_client.get_directory_client(folder)
file_client = self.directory_client.create_file(file_name)
file_client.append_data(data, offset=0, length=len(data))
file_client.flush_data(len(data))
file_client.close()
Swapnil kumar 26 Reputation points

2021-02-22T13:36:14.143+00:00

Hey @KranthiPakala-MSFT !
I tried the steps mentioned in the doc and the problem seems to persist. What we are doing is we are creating a dataframe, then with that dataframe as data we are creating a csv file according to the above mentioned. I also added the file_client.close() method but it wasn't too helpful.
For your reference, I am attaching the code we have used. the trigger has the ignore empty blobs condition set to true
Code:
if isinstance(data, pd.DataFrame):
data = data.to_csv(index=False)
file_name = f"{file}.csv"
else:
data = data.str()
file_name = f"{file}.json"
self.directory_client = self.filesystem_client.get_directory_client(folder)
file_client = self.directory_client.create_file(file_name)
file_client.append_data(data, offset=0, length=len(data))
file_client.flush_data(len(data))
file_client.close()
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-02-25T06:39:50.233+00:00

Hi @Swapnil kumar ,

Sorry for the delay in response. This might need deeper analysis. For deeper investigation and immediate assistance, If you have a support plan you may file a support ticket, else could you please send an email to AzCommunity[at]Microsoft[dot]com with the below details, so that we can work closely on this matter.

Email subject: <Attn - Kranthi : Microsoft Q&A Thread title>
Thread URL: <Microsoft Q&A Thread>
Subscription ID: <your subscription id>

Looking forward to your reply.

Thanks
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-02-26T22:37:30.073+00:00

Hi @Swapnil kumar ,

Following up to see if you have got a chance to see my previous comment. If so could you please share the details requested.

Thank you
Swapnil kumar 26 Reputation points

2021-02-27T18:52:02.007+00:00

Hey @KranthiPakala-MSFT . I have sent out an email with the information requested
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-03-02T00:18:25.83+00:00

Thanks for sharing the details @Swapnil kumar . I have shared the details on how to create a support ticket for further investigation. Please do follow the steps and share the SR# with me for tracking it internally.

Thank you
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-03-03T06:29:06.273+00:00

Hi @Swapnil kumar

Following up to see if you have got a chance to file a support ticket for this issue, if so could you please share it with us for internal tracking.

Thank you