Collect synapse pipeline, trigger, run status into Azure log analytics and ingest data into Azure database.

Question

question

NC-1234 asked Apr 18, '22 PRADEEPCHEEKATLA-MSFT commented May 10, '22

Collect synapse pipeline, trigger, run status into Azure log analytics and ingest data into Azure database.

Hi,
I am trying to make a monitoring system that collect information from Azure synapse into log analytics and then try to ingest data into Azure database.
I created a log analytics workspace and turn on diagnostic in azure synapse and connect to log analytics workspace but can not find and example on how to run the query
to get the job information pipeline info in log analytics and ingest info to database.

How can I do that? Do you have any example? And how can I save this information into azure sql

azure-synapse-analytics

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PRADEEPCHEEKATLA-MSFT · Apr 26 at 06:07 AM

Hello @NC-1234,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.

Please don't forget to click on or upvote button whenever the information provided helps you.

0 ·

Answer 1 · 2022-04-19T10:51:37.81Z

PRADEEPCHEEKATLA-MSFT answered Apr 19, '22 PRADEEPCHEEKATLA-MSFT commented May 10, '22

Hello @NC-1234,

Thanks for the question and using MS Q&A platform.

Azure Log Analytics Workspace:

Once the Diagnostics settings configured on the Synapse workspace, you can query directly on the job information as shown below:

Note: In Log Analytics workspace, you can query the job information pipeline info and export to local file and then you can bulk insert into the Azure SQL Database.

For more details, refer to Use Azure Monitor with your Azure Synapse Analytics workspace and

Azure Data Explorer:

For seamless integration, you can create a Azure Data Explorer.

The Azure Data Explorer supports cross-service queries between Azure Data Explorer, Application Insights (AI), and Log Analytics (LA). You can query your Log Analytics or Application Insights workspace using Azure Data Explorer query tools and in a cross-service query. The article shows you how to create a cross-service query and add the Log Analytics or Application Insights workspace to the Azure Data Explorer Web UI.

Once the Log Analytics workspace connected to the Azure Data Explorer, you can directly export data to SQL.

Export data to SQL allows you to run a query and have its results sent to a table in a SQL database, such as a SQL database hosted by the Azure SQL Database service.

For more details, refer to Query data in Azure Monitor using Azure Data Explorer and Export data to SQL.

Hope this will help. Please let us know if any further queries.

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

synapse-la-ws.gif (1.7 MiB)

image.png (62.8 KiB)

· 9

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

NC-1234 · Apr 19 at 07:30 PM

Thank you for the answer. I added a diagnostic setting in Synapse workspace with this set up but some how I got just only two tables under LogManagemnt. I got only Usage and SynapseGatewayApiRequests

Anything else that I should do in order to activate IntergrationPipelineRuns, IntergrationTriggerRuns and IntergrationActivityRun.

0 ·

image.png (9.9 KiB)

PRADEEPCHEEKATLA-MSFT NC-1234 · Apr 20 at 03:34 AM

Hello @NC-1234,

After Diagnostics settings configured on the Synapse workspace, did you ran any pipelines?

Note: Make sure you have pipeline runs in the Monitoring tab in Azure Synapse Studio, so that the same IntergrationPipelineRun will be shown in Log Analytics workspace.

0 ·

image.png (74.3 KiB)

NC-1234 · Apr 20 at 06:37 PM

I ran a pipeline after diagnostics settings configured but there is no IntergrationPipelineRuns, in log analytics.

I choose azure synapse workspace in select scope

0 ·

pipeline.png (130.7 KiB)

log-analytics.png (160.2 KiB)

PRADEEPCHEEKATLA-MSFT NC-1234 · Apr 21 at 04:26 AM

Hello @NC-1234,

Could you please checkout` LogManagement` section under Tables section as shown below:

Hope this will help. Please let us know if any further queries.

0 ·

image.png (139.8 KiB)

NC-1234 · Apr 21 at 10:15 AM

Her is my LogManagement as you can see I got only two tables and not include Integration tables.

0 ·

log.png (84.5 KiB)

PRADEEPCHEEKATLA-MSFT NC-1234 · Apr 25 at 05:23 AM

Hello @NC-1234,

Could you please re-configure the diagnostics settings and check if you are able to see the IntergrationPipelineRuns in log analytics?

0 ·

NC-1234 · May 04 at 08:01 PM

Hi @PRADEEPCHEEKATLA-MSFT,
I did re-configure the diagnostics settings as you can see in the pictures above but I still can not see Synapse tables in logManagement.
Anything else that I should do in order to solve this problem?

0 ·

PRADEEPCHEEKATLA-MSFT NC-1234 · May 05 at 06:35 AM

Hello @NC-1234,

I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation.

0 ·

PRADEEPCHEEKATLA-MSFT NC-1234 · May 10 at 06:03 AM

Hello @NC-1234,

Did you get a chance to open a support ticket on the above issue?

0 ·

question