question

NC-1234 avatar image
0 Votes"
NC-1234 asked PRADEEPCHEEKATLA-MSFT commented

Collect synapse pipeline, trigger, run status into Azure log analytics and ingest data into Azure database.

Hi,
I am trying to make a monitoring system that collect information from Azure synapse into log analytics and then try to ingest data into Azure database.
I created a log analytics workspace and turn on diagnostic in azure synapse and connect to log analytics workspace but can not find and example on how to run the query
to get the job information pipeline info in log analytics and ingest info to database.

How can I do that? Do you have any example? And how can I save this information into azure sql

azure-synapse-analytics
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @NC-1234,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.

0 Votes 0 ·

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
1 Vote"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @NC-1234,

Thanks for the question and using MS Q&A platform.

Azure Log Analytics Workspace:

Once the Diagnostics settings configured on the Synapse workspace, you can query directly on the job information as shown below:

Note: In Log Analytics workspace, you can query the job information pipeline info and export to local file and then you can bulk insert into the Azure SQL Database.

194292-synapse-la-ws.gif

For more details, refer to Use Azure Monitor with your Azure Synapse Analytics workspace and

Azure Data Explorer:

For seamless integration, you can create a Azure Data Explorer.

The Azure Data Explorer supports cross-service queries between Azure Data Explorer, Application Insights (AI), and Log Analytics (LA). You can query your Log Analytics or Application Insights workspace using Azure Data Explorer query tools and in a cross-service query. The article shows you how to create a cross-service query and add the Log Analytics or Application Insights workspace to the Azure Data Explorer Web UI.

194273-image.png

Once the Log Analytics workspace connected to the Azure Data Explorer, you can directly export data to SQL.

Export data to SQL allows you to run a query and have its results sent to a table in a SQL database, such as a SQL database hosted by the Azure SQL Database service.

For more details, refer to Query data in Azure Monitor using Azure Data Explorer and Export data to SQL.

Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators


synapse-la-ws.gif (1.7 MiB)
image.png (62.8 KiB)
· 9
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you for the answer. I added a diagnostic setting in Synapse workspace with this set up but some how I got just only two tables under LogManagemnt. I got only Usage and SynapseGatewayApiRequests


194287-image.png




Anything else that I should do in order to activate IntergrationPipelineRuns, IntergrationTriggerRuns and IntergrationActivityRun.

0 Votes 0 ·
image.png (9.9 KiB)

Hello @NC-1234,

After Diagnostics settings configured on the Synapse workspace, did you ran any pipelines?

Note: Make sure you have pipeline runs in the Monitoring tab in Azure Synapse Studio, so that the same IntergrationPipelineRun will be shown in Log Analytics workspace.

194484-image.png


0 Votes 0 ·
image.png (74.3 KiB)

I ran a pipeline after diagnostics settings configured but there is no IntergrationPipelineRuns, in log analytics.
194796-pipeline.png

I choose azure synapse workspace in select scope

194871-log-analytics.png


0 Votes 0 ·
pipeline.png (130.7 KiB)
log-analytics.png (160.2 KiB)

Hello @NC-1234,

Could you please checkout` LogManagement` section under Tables section as shown below:

194982-image.png

Hope this will help. Please let us know if any further queries.

0 Votes 0 ·
image.png (139.8 KiB)

Her is my LogManagement as you can see I got only two tables and not include Integration tables.
195124-log.png


0 Votes 0 ·
log.png (84.5 KiB)

Hello @NC-1234,

Could you please re-configure the diagnostics settings and check if you are able to see the IntergrationPipelineRuns in log analytics?

0 Votes 0 ·

Hi @PRADEEPCHEEKATLA-MSFT,
I did re-configure the diagnostics settings as you can see in the pictures above but I still can not see Synapse tables in logManagement.
Anything else that I should do in order to solve this problem?

0 Votes 0 ·

Hello @NC-1234,

I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation.

0 Votes 0 ·

Hello @NC-1234,

Did you get a chance to open a support ticket on the above issue?

0 Votes 0 ·