question

MinhNGUYEN-2713 avatar image
0 Votes"
MinhNGUYEN-2713 asked MinhNGUYEN-2713 commented

Synapse Spark pools get omitted in Synapse Workspace template

Hi,

I'm using Azure DevOps to deploy Synapse Workspace in a release pipeline using 'Synapse workspace deployment' plugin. There's a Spark pool in my source Workspace but the target Workspace deployment fails half way through, when deploying a Spark job definition, with error: "The document creation or update failed because of invalid reference \'sparkpool1\'. Earlier in the logs, it deployed Dedicated SQL pool and SQL scripts successfully.

So I looked at the TemplateForWorkspace.json on the workspace_publish branch on GIT repo. There's no declaration of SparkPool1 as a resource. It is only referred once when it comes to the Spark job definition as "referenceName" in "properties".

It seems like an issue when generating the template when publishing the source Workspace.

Could you please advise on how I can solve this?

azure-synapse-analytics
· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @MinhNGUYEN-2713 and welcome to Microsoft Q&A. Thank you for bringing this to our attention.

If I understand correctly, the deployment first creates the spark pool successfully, but then later fails when creating the spark job definitions.

What name is the spark pool created with? Sounds like it does not match the one in the spark job definition.

I will reach out internally for more information on the issue.

0 Votes 0 ·

Hi @MartinJaffer-MSFT,

I've got 1 dedicated SQL pool, and 1 Spark pool named 'sparkpool1'. The dedicated SQL Pool was created successfully. It can deploy the SQL pool and those associated SQL scripts. Once it got to the Spark job, it failed with the mentioned error about the Spark pool named 'sparkpool1'.

The ARM template that was generated in the 'workspace_publish' branch, does declare the SQL pool, but not the Spark pool. So I guess that for some reason it ignored the Spark pool earlier in the script. Then when it came to deploy the Spark job, it doesn't know what 'sparkpool1' is.

0 Votes 0 ·

Hi @MartinJaffer-MSFT,

Have you got any updates for me on this issue?

0 Votes 0 ·

Hi @MartinJaffer-MSFT , @PRADEEPCHEEKATLA-MSFT , @KranthiPakala-MSFT and Microsoft supports,

Have you got any updates for me on this issue?
Just to briefly reiterate, it seems that after studio publishing, the TemplateForWorkspace.json does not include the Spark Pools created in Synapse Workspace.
If it is a bug then it might be critical. If not, could you please let me know what I'm missing here?

0 Votes 0 ·

Thank you for the reminder. I am following up hard now to get a resolution.

0 Votes 0 ·

1 Answer

MartinJaffer-MSFT avatar image
1 Vote"
MartinJaffer-MSFT answered MinhNGUYEN-2713 commented

@MinhNGUYEN-2713 I heard back.

The workspace deployment task in the devops is only for the artifacts deployment(non-arm call). For pools, the workspace resources(arm call), you need to have them ready before the artifacts deployment.
Users can custom the parameters in the parameter file, and parameterize the pools’ properties.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @MartinJaffer-MSFT ,

Thanks for your answer. So according to your explanation, the pools (ARM calls) need to be ready before artifact deployment
However I notice that the dedicated (provisioned) SQL pools (also ARM call if I'm not mistaken) get deployed successfully in the task. This is inconsistent compare to the Spark pools. Why is that?

0 Votes 0 ·