Synapse Analytics - Unable to start cluster when I upload requirements.txt file - cluster fails.

Joe Kastner 21 Reputation points
2020-05-26T02:09:14.497+00:00

Hi, I'm trying to start a cluster with some additional packages (tried output of pip freeze > requirements.txt and also added all of the packages that come with the anaconda distribution that is installed on standard cluster in synapse analytics). When I start the cluster it fails, with the following message: "[plugins.neusynapse.jksparkcluster.17] Attempt=1/[3]Cluster was in terminal state=[Cancelled] before it reached 'Ready' state. Cluster job has WorkspaceName=[neusynapse], SpecName=[jksparkcluster], and JobId=[1b4a0282-742d-4235-b379-603d9b77e60c]."

When I remove the requirements.txt file, the cluster starts up no problem.

Any idea on what can be going on here?

8721-requirements.txt

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,364 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Euan Garden 136 Reputation points
    2020-05-26T02:59:06.73+00:00

    Joe,
    There is a 20 min timeout currently on the install of packages this is a pretty long list and some of these packages take a long time to install, can you shrink the list down to the min list and perhaps use the versions we ship in the image for some of them just to debug?

    -Euan

    1 person found this answer helpful.