Azure Data Factory won't publish cloned pipeline

JAL 571 Reputation points
2021-10-05T11:21:21.593+00:00

I've spent a whole day on this. I'm a newbie to ADF.

My first pipeline published just fine, but I cloned it and enhanced the cloned version. This clone won't publish.

Since then, I've tried cloning multiple times, without any changes/enhancements. NONE of the clones will publish.

I've tried renaming, resaving, making small changes, resaving again, removing the git-repo, switching to a new collaboration branch, merging the old branch into the new one and vice versa, switching out publish branches, playing with the Import Repository options - I literally don't know what else to try. I always get the message "Published successfully" but I've manually inspected the repo files in AzureDevOps and the clones simply do not exist on the published branch.

  • BTW I'm using AKV for all secrets (not sure if that's relevant).

This is frustrating. Is there a better way to do cloning? (Please don't ask me to remove/recreate my entire repo since I lack permission to do so).

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,525 questions
Accepted answer
  1. JAL 571 Reputation points
    2021-10-07T14:27:36.077+00:00

    Ok I'm back in sync now, I think. I downloaded the files from the collaboration branch into a backup folder on my machine. Then I used a powershell script from Stack Overflow to delete all my objects from ADF. I also disconnected the repo from ADF. I also deleted the collaboration branch and the publish branch from the git repo. Then I recreated the git-config in ADF, using the New option in the wizard to create a new collaboration branch. I downloaded this empty collaboration branch, and moved my backup files into it, and pushed it to the git repo. Now the objects are displayed again in ADF, and the publishing seems to be working even for cloned pipelines.

    Hoping my eyes are not deceiving me, I think I'm back on track.

4 additional answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. MartinJaffer-MSFT 26,021 Reputation points
    2021-10-06T23:36:29.467+00:00

    @JAL I see you are having a frustrating time. I suspect there is some confusion over what publish does. Perhaps some clarification would help.

    "Publish" pushes assets from the collaboration branch of Git/DevOps Mode to the Live Factory Mode. The process of publishing also creates a new artifact in the form of ARM templates in the adf_publish branch.

    "Publish" does not write things to the other branches of your repo. "Save" is used to write to the selected repo branch, whatever is currently loaded in your browser.

    The adf_publish branch is not used for development. It only stores the post-publish artefacts. This branch will always be disabled. There is no reason to write to this branch.

    "Trigger Now" starts a pipeline run using the "Live Factory Mode". It does not use the repo definitions. To start a run using whatever is on your screen at the time, "Debug" can be used. "Debug" runs are considered different from "Trigger Now" or scheduled runs. "Debug" is best used for development, without having to commit changes.

    The collaboration branch (name is chosen when repo integration was made) is effectively the main or master branch. All other branches are effectively feature branches. Only the collaboration branch can publish. Your feature branch must be merged / pulled into the collaboration branch for your changes to show up after publishing.

    It is possible to do development directly on the collaboration branch. This is suitable only when there is a single developer doing 1 change at a time. In all other cases, creating and then merging feature branches is the way to go.

    2 people found this answer helpful.
  2. JAL 571 Reputation points
    2021-10-05T14:55:17.193+00:00

    I had a hopeful idea but maybe I didn't execute it properly. Since ADF only publishes the pending/recent changes, I removed all the files in the branch, and then re-added them to put them back in pending/changes. Then I re-published.

    But I can't tell if it worked because now the publish branch is disabled/greyed in the ADF dropdown, I can't select it. I tried this with 2 different publish-branches, and they both are disabled. Anyone know how to enable them?

    0 comments
  3. JAL 571 Reputation points
    2021-10-06T12:05:46.787+00:00

    Well it's gone from bad to worse. I started all over and rebuilt the pipeline from scratch, one icon at a time, on a new/empty branch, and it still says "Publish succeeded" but:
    (1) No files in the adf_publish branch
    (2) In ADF the adf_publish branch is disabled/unselectable
    (3) I can't run the pipeline even manually, not even from the collaboration branch using Trigger Now (tells me I need to do this on the publish branch).

    0 comments
  4. JAL 571 Reputation points
    2021-10-07T04:22:46.587+00:00

    @MartinJaffer-MSFT

    Thanks for the help. However, while I'm indeed a bit lost here, I don't think my understanding is quite as skewed as you seem to think. Let's take a look at what you said:

    The process of publishing also creates a new artifact in the form of ARM templates in the adf_publish branch.

    Yes, I was already aware of that. As I tried to convey:
    (1) The first time I created a pipeline (built from the collab branch), it produced the Arm Templates in my publish branch. And I was able to run Trigger Now.
    (2) Then I cloned the pipeline while positioned in the collaboration branch. This clone seemed to publish as I recall (i.e. it seemed to add more files in the publish branch) but, in any case, I couldn't see the clone(s) in ADF when I selected the publish branch.
    (3) Then I started over from scratch and re-built the pipeline (from the collaboration branch once again). This time it got even worse, as I explained above. (A) The publish branch got NO files, meaning NO Arm templates. (B) In ADF, that branch became greyed/unselectable.

    "Publish" does not write things to the other branches of your repo.

    Yes, I was already aware of that.

    "Save" is used to write to the selected repo branch, whatever is currently loaded in your browser.

    Yes, I was already aware of that.

    The adf_publish branch is not used for development. It only stores the post-publish artefacts.

    Yes, I was already aware of that. It stores the Arm Templates.

    This branch will always be disabled. There is no reason to write to this branch.

    I wasn't trying to write to it. I thought I was allowed to select it and then do a Trigger Now. I thought I already did so once before, but maybe my memory is serving me wrong (as it often does).

    "Trigger Now" starts a pipeline run using the "Live Factory Mode". It does not use the repo definitions.

    Ok, but pretty much the same thing, right? When I open the ADF window, where does it get the pipeline from, if not the repo definitions?

    The collaboration branch (name is chosen when repo integration was made) is effectively the main or master branch. All other branches are effectively feature branches. Only the collaboration branch can publish. Your feature branch must be merged / pulled into the collaboration branch for your changes to show up after publishing.

    Yes, I was already aware of that.

    It is possible to do development directly on the collaboration branch. This is suitable only when there is a single developer doing 1 change at a time. In all other cases, creating and then merging feature branches is the way to go.

    Yes, I was already aware of that.

    To summarize, Publish doesn't seem to be working correctly for me.

    0 comments