Need help on setting up Env variable in Azure Synapse Analytics Spark cluster

Question

question

ShikhilKumarGuptaMSFT-2699 asked Jan 4, '22 PRADEEPCHEEKATLA-MSFT commented May 9, '22

Need help on setting up Env variable in Azure Synapse Analytics Spark cluster

Hi Team, I have dev, ppe and prod Azure Synapse analytics workspace. I will be deploying my jar for some computation. Now I need to insert the ENV variable for ppe, prod and dev enviorment. So that my jar running in spark cluster can refer env variable value and based on it, it can load dev, ppe or prod config. But I am unable to find out how to pass custom env variable for Synapse workspace.

Can you guys help me to know that?

azure-synapse-analytics

· 3

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

ShikhilKumarGuptaMSFT-2699 · Jan 04 at 01:38 PM

@PRADEEPCHEEKATLA-MSFT Could you please help me here?

0 ·

PRADEEPCHEEKATLA-MSFT ShikhilKumarGuptaMSFT-2699 · Jan 05 at 03:59 AM

Hello @ShikhilKumarGuptaMSFT-2699,

Thanks for the question and using MS Q&A platform.

We are reaching out to the internal team to confirm if the custom environment variables supported in Spark pool. I will be update you once I hear back from the team.

0 ·

PRADEEPCHEEKATLA-MSFT · Jan 12 at 07:40 AM

Hello @ShikhilKumarGuptaMSFT-2699,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.

Please don't forget to click on or upvote button whenever the information provided helps you.

0 ·

Answer 1 · 2022-01-05T12:05:01.29Z

PRADEEPCHEEKATLA-MSFT answered Jan 5, '22 PRADEEPCHEEKATLA-MSFT commented May 9, '22

Hello @ShikhilKumarGuptaMSFT-2699,

Thanks for the question and using the MS Q&A platform.

You can read custom config values in a spark notebook as shown below.

Step1: Add your key-value pairs inside the config.txt file:

 spark.executorEnv.environmentName ppe

Step2: You can upload a config file to your spark pool.

Step3: Which you are then able to access inside your notebooks:

 envName: str = spark.sparkContext.environment.get('environmentName', 'get')

Hope this will help. Please let us know if any further queries.

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

image.png (32.5 KiB)

· 3

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PRADEEPCHEEKATLA-MSFT · Jan 07 at 09:40 AM

Hello @ShikhilKumarGuptaMSFT-2699,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

0 ·

Rupeshgupta-1646 PRADEEPCHEEKATLA-MSFT · May 09 at 08:10 AM

@PRADEEPCHEEKATLA-MSFT Is it possible to share the content of config.txt ? I would like to see how to define it inside the file.

0 ·

PRADEEPCHEEKATLA-MSFT Rupeshgupta-1646 · May 09 at 08:57 AM

Hello @Rupeshgupta-1646,

It's simple key-value pairs inside the config.txt file as shown below:

0 ·

image.png (5.1 KiB)

question