Copy Activity ADF pipeline data not showing up

Rishabh Srivastava 60 Microsoft Employee

Hi Team I created pipeline to copy activity from one table to other table in same log analytics workspace. I am using source Linked service Rest connector using managed Identity to get data from exceptions table then I am using Destination linked service rest connector to copy data to exceptions aggregated table.

When I click on preview data its showing up

User's image

when I trigger pipeline its getting succeeded

User's image

However when I go to log analytics workspace to run query on destination table data is not showing up. Can anyone please help?

Harishga 4,160 Reputation points Microsoft Vendor

2024-04-30T05:43:05.11+00:00
@Rishabh Srivastava
Welcome to Microsoft Q&A platform and thanks for posting your question here.

When you click on preview data, it shows up because the data is being fetched from the source table and displayed in the preview window. This states that the source linked service is correctly configured and able to fetch data from the source table.

When you trigger the pipeline, it gets succeeded because the pipeline is able to copy the data from the source table to the destination table without any errors. This states that the pipeline is correctly configured and able to copy data from the source table to the destination table.

when you go to the Log Analytics workspace to run a query on the destination table, the data is not showing up. This could be due to several reasons, such as incorrect table name, incorrect mapping of columns, incorrect data type, missing permissions, incorrect query syntax, or incorrect workspace.

To resolve the issue of data not showing up in the destination table in Log Analytics workspace, you can follow these steps:

First, check that the table name in the destination linked service matches the table name in the Log Analytics workspace. If the table name is incorrect, the data won't be copied to the correct table.

Next, ensure that the columns in the source table are correctly mapped to the columns in the destination table. If the mapping is incorrect, the data may not be copied to the correct columns. It's also important to verify that the data type of the columns in the source table matches the data type of the columns in the destination table. If the data type is incorrect, the data may not be copied to the destination table.

Another thing to check is that the managed identity used in the source and destination linked services has the necessary permissions to read from the source table and write to the destination table. If the managed identity does not have the necessary permissions, the data will not be copied to the destination table.

It's also possible that the data may take some time to be copied to the destination table. In this case, you can wait for some time and try querying the destination table again.

If the query syntax is incorrect, the data will not be displayed in the query results. Finally, ensure that you are running the query on the correct Log Analytics workspace where the destination table is located. If you are running the query on the wrong workspace, the data will not be displayed in the query results.

If none of these solutions work, you can try enabling diagnostic logging for the pipeline and the destination table. This will help you identify any issues with the pipeline or the table.

I hope this information helps you. Let me know if you have any further questions or concerns.
Harishga 4,160 Reputation points Microsoft Vendor

2024-05-01T05:56:20.8633333+00:00

Hi @Rishabh Srivastava
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Rishabh Srivastava 60 Reputation points Microsoft Employee

2024-05-01T21:24:30.64+00:00

Hi Harishga

SO, I can see data after I added transformKQL to my DCR
but its only showing the column which I added to transformKQL
but not all columns are being populated. Is there any policy that's preventing it?
Harishga 4,160 Reputation points Microsoft Vendor

2024-05-02T11:02:42.31+00:00
Hi @Rishabh Srivastava

It's possible that there might be an issue with the transformation query itself. To troubleshoot this issue, you can try the following steps:

Check the transformation query in your DCR to ensure that it's correctly transforming the data.

Check if there are any errors or warnings in the Azure portal or in the logs that might indicate an issue with the data ingestion process.

Verify that the columns in the source table are correctly mapped to the columns in the destination table. If the mapping is incorrect, the data may not be copied to the correct columns. It's also important to verify that the data type of the columns in the source table matches the data type of the columns in the destination table. If the data type is incorrect, the data may not be copied to the destination table.

Reference:
https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-transformations#combination-of-azure-and-custom-tables
I hope this information helps you. Let me know if you have any further questions or concerns.

Share via

Copy Activity ADF pipeline data not showing up