question

mao-5640 avatar image
0 Votes"
mao-5640 asked MartinJaffer-MSFT commented

Azure Data Factory Pipeline trigger not showing same values in UI as it does via CLI and CLI commands not updating some triggers

Hi,

I have 2 data factories with several pipelines and scheduled triggers to call these. One of the data factories runs in live mode whilst the other is it GIT mode and tied to a repo/branch.

Recently I tried to update the schedule the triggers run on via Azure CLI with the following commands to stop / update / start each of the triggers:

 az datafactory trigger stop --factory-name <factory name> --resource-group <rg-name> --name <trigger name>
    
 az datafactory trigger stop --factory-name <factory name> --resource-group <rg-name> --name <trigger name> --set properties.recurrence.schedule.hours=[9]
    
 az datafactory trigger stop --factory-name <factory name> --resource-group <rg-name> --name <trigger name> --set properties.recurrence.schedule.minutes=[3]
    
 az datafactory trigger stop --factory-name <factory name> --resource-group <rg-name> --name <trigger name>

The responses all seem to be correct with --debug enabled on the above commands, including the updated values for the trigger schedules. This command also shows the triggers with updated values:

 az datafactory trigger show --factory-name <factory name> --resource-group <rg-name> --name <trigger name>

I have found that although most work some of the triggers do not run to the expected updated schedule and none of the triggers in the Azure portal Datafactory UI show the updated values.

Can someone please explain where I'm going wrong here or what the correct process is for working with trigger schedules in CLI and web UI?

Is the Data factory tied to GIT maybe showing the GIT schedule values in the UI but accepting changes via CLI in the background? The live mode is also behaving the same so I'm not sure whats going on.

Thanks

azure-data-factory
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@mao-5640 if my response solved your issue, please mark as accepted answer. If not, please let me know how I may better assist.

0 Votes 0 ·

1 Answer

MartinJaffer-MSFT avatar image
1 Vote"
MartinJaffer-MSFT answered MartinJaffer-MSFT commented

Hello @mao-5640 and welcome to Microsoft Q&A. Please allow me to explain some things.

As you have noticed, there are two modes, GIT and Factory mode.

GIT mode reflects what is held in the GIT repo, it loads all objects using GIT as the source of truth. This is only available when you have a GIT /ADO integration.

Factory mode reflects what triggers and pipelines are running off of. This is updated on Publishing. The is the source of truth for anything running (except debug/sandbox).

These two modes are separate. Only when the collaboration branch of GIT is published, is the Factory mode updated to match the GIT mode.

With these two modes and their relation explained, I now move on to how the CLI , Powershell, SDK, or API relate.

The CLI, Powershell, SDKs and REST API, all directly work on the Factory mode. They have no way to interact with the GIT repo, and so have no affect on GIT mode. The only exception, is altering or removing the repo integration settings. Still, this doesn't change the repo, it only changes which repo will be used by the UI.

Only the UI can work with GIT mode. Only the UI can push changes to the repo. Only the UI can read from the repo.

This means when you use the CLI, you are making a partial publish, and ignoring the GIT repo. This causes the two to fall out of sync and will likely cause problems later.

This also means, the CLI is useful if for some reason you need to stop a trigger, but are not in a position to do a full publish. I advise against using the CLI to author while a GIT/ADO integration is present.

Does this explain what you are seeing?

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you, I really appreciate the detailed explanation. It almost explains the my situation with the exception of two things:

I have something like 8 triggers in the GIT mode data factory, I understand the UI is out of sync now because I used the CLI but I did experience that the CLI commands working as expected on all but one trigger. Could be something difficult to diagnose related to that particular triggers configuration but it is almost identical to the others.

In the factory mode data factory I noticed that 3 triggers showed as stopped in the UI but they had been running fine for several weeks. I have inherited this project rather than it being something I created but I wonder could this be explained by this data factory once being in GIT mode and losing sync because of CLI use and then being switched to factory mode?

Thanks again for the help

0 Votes 0 ·

@mao-5640 Whatever you see in the factory mode is the current state of affairs.

Changes can be saved to GIT mode without affecting the factory mode. This is the difference between "Save" and "Publish". Save only writes to GIT, whereas Publish also pushes to factory mode.

So, while the CLI use could be responsible for causing factory mode to not match GIT mode, it could also be caused by updates to GIT mode not published to factory mode. Or some combination thereof.

If you are interested in finding the cause, be aware that Factory mode does not store or make available a history of changes. That is the role which GIT mode is to fill. There may be some logs elsewhere, but getting things running again is probably a better use of time.

0 Votes 0 ·