question

JoeKastner-2406 avatar image
JoeKastner-2406 asked ·

Synapse Analytics - Unable to start cluster when I upload requirements.txt file - cluster fails.

Hi, I'm trying to start a cluster with some additional packages (tried output of pip freeze > requirements.txt and also added all of the packages that come with the anaconda distribution that is installed on standard cluster in synapse analytics). When I start the cluster it fails, with the following message: "[plugins.neusynapse.jksparkcluster.17] Attempt=[1]/[3]Cluster was in terminal state=[Cancelled] before it reached 'Ready' state. Cluster job has WorkspaceName=[neusynapse], SpecName=[jksparkcluster], and JobId=[1b4a0282-742d-4235-b379-603d9b77e60c]."

When I remove the requirements.txt file, the cluster starts up no problem.

Any idea on what can be going on here?

8721-requirements.txt


azure-synapse-analytics
requirements.txt (5.4 KiB)
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

euangMS avatar image
euangMS answered ·

Joe,
There is a 20 min timeout currently on the install of packages this is a pretty long list and some of these packages take a long time to install, can you shrink the list down to the min list and perhaps use the versions we ship in the image for some of them just to debug?

-Euan

5 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi Euan, I only added four packages to what's included in the anaconda distribution (which to my knowledge is included in every cluster?). I also tried a requirements file with only those four packages and it failed as well. To your point, lastly I tried with just the base packages included in Anaconda and that failed as well. Any other thoughts?

0 Votes 0 · ·
euangMS avatar image euangMS JoeKastner-2406 ·

Can you provide the 4 package version of the requirements.txt?

Also if you can send me information about the workspace name and spark pool name to euang at microsoft dot com we can take a look at the telem and debug logs

0 Votes 0 · ·

Sure, please see attached file. I'll run this again now so you can use the latest logs.

Workspace Name: neusynapse
Spark Pool: synapseSpark

Please let me know if you need any additional info.

Thanks!

0 Votes 0 · ·

Well that's interesting, I ran it again with that requirements file and the cluster started up. Hmm.. I'm not exactly sure what happened but if you see anything in the logs that would be very helpful.

0 Votes 0 · ·
Show more comments