How can I use MSSparkUtils in Python from PyCharm?

Question

question

VictorSeifert-1141 asked May 10, '22 PRADEEPCHEEKATLA-MSFT commented May 11, '22

How can I use MSSparkUtils in Python from PyCharm?

I have a script that works in Synapse by utilizing the mssparkutils to retrieve credentials via a LinkedService:

storage_token_source = mssparkutils.credentials.getConnectionStringOrCreds(linked_service_name_source)

However, if I add this code to my local repository, PyCharm throws an Error as mssparkutils is unknown - I couldn't find the package in PyPi, how can I add it to the repository as an import/dependency?

I package my code later on and deploy it as a whl file to the spark pools.

Best regards,
Victor

azure-synapse-analytics

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 1 · 2022-05-10T13:14:04.393Z

PRADEEPCHEEKATLA-MSFT answered May 10, '22 PRADEEPCHEEKATLA-MSFT commented May 11, '22

Hello @VictorSeifert-1141,

Thanks for the question and using MS Q&A platform.

You can download the mssparkutils package from below MVN repo and import the jar into your local project. With that, you should be able to compile the project successfully: https://mvnrepository.com/artifact/com.microsoft.azure.synapse/synapseutils

Hope this will help. Please let us know if any further queries.

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

· 2

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

VictorSeifert-1141 · May 11 at 07:52 AM

Correct me if I'm wrong, but isn't the .jar mainly for Java/Scala projects? I don't know of any way to import a .jar file in Python?

Of course I can add the jar to my spark config, but that won't solve the issues I have with my code-editor + tests not recognizing mssparkutils as a valid python reference (to the package) and throwing errors.

I solved it currently that my custom class expects two a parameters:

param a: flag whether the code is executed locally or in Synapse. (so any use of mssparkutils in local-execution-mode is omitted)
param b: mssparkutils as a reference to the package (which gets stored in a variable)

Once I execute it in Synapse, I pass the reference to the mssparkutils package to my custom package's class via the parameter which enables me to use it like a variable in the code.

This feels very ugly, but I don't see any other way not to have errors all over my code. If there is a better solution, please add to the answer, but the existing answer above is not actually a solution.

0 ·

PRADEEPCHEEKATLA-MSFT VictorSeifert-1141 · May 11 at 08:11 AM

Hello @VictorSeifert-1141,

The answer was provided by the internal product team.

We are reaching out to the internal team to get the more details on the better solution. I will be update you once I hear back from the team.

0 ·

question