question

rawwar avatar image
1 Vote"
rawwar asked PRADEEPCHEEKATLA-MSFT commented

Will existing cluster with databricks runtime stop working because of out of support

Hi,
I am currently using an old data bricks cluster that uses a 6.4 ML version. Will it ever stop working? I understand that its out of support. But, if I never delete this cluster, can I keep using the same cluster? Will i ever be forced to upgrade my cluster runtime version?

azure-databricks
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PRADEEPCHEEKATLA-MSFT avatar image
1 Vote"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @rawwar,

Will it ever stop working?

You can continue to run until you delete the cluster manually.

Note: You can manually terminate a cluster or configure the cluster to automatically terminate after a specified period of inactivity. Azure Databricks records information whenever a cluster is terminated. When the number of terminated clusters exceeds 150, the oldest clusters are deleted.

Unless a cluster is pinned, 30 days after the cluster is terminated, it is automatically and permanently deleted.

For more details, refer Azure Databricks - Terminate a cluster.

Support for Databricks Runtime 6.4 ended on April 1, 2021 due to the end of support for its underlying Ubuntu operating system. Databricks recommends upgrading to Databricks Runtime 7.x or 8.x to get the benefits of Apache Spark 3.x and the many new features and improvements built into these newer runtimes.

For migration information, see Databricks Runtime 7.x migration guide.

You have the option to migrate to Databricks Runtime 6.4 Extended Support, which is identical to Databricks Runtime 6.4 apart from the operating system version, JDK, and a small number of R and Python libraries. See Databricks Runtime 6.4 Extended Support.

For more details, refer Databricks Runtime 6.4 for Machine Learning (Unsupported).

Hope this helps. Do let us know if you any further queries.


Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @PRADEEPCHEEKATLA-MSFT , The reason I asked is that we have old notebooks integrated with ADF pipelines and we don't intend to update these notebooks. ADF Pipelines run every day and they start and terminate this specific cluster every day on UAT and prod environments. But, On dev, we might not use it for a while. Thanks for the info regarding "Pinning" a cluster. I just did that. So, this ensures that the cluster is never deleted even if we don't start it for a while(probably another 6-8 months)

Also, can you please clarify what you meant by "When the number of terminated clusters exceeds 150, the oldest clusters are deleted.". I only have two clusters in my workspace and i just pinned both of them. So, i should not be facing any problem here right?

0 Votes 0 ·

Hello @rawwar,

Just to clarify, the cluster will be removed, if it meets any of the below condition:

  • When the number of terminated clusters exceeds 150, the oldest clusters are deleted.

  • More than 30 days after the cluster is terminated

You will not face any problem, since you already pinned both of your clusters.

If a cluster is pinned, it will be there forever until you manually delete it.

Hope this helps.

0 Votes 0 ·
BeliveScala-1539 avatar image
1 Vote"
BeliveScala-1539 answered BeliveScala-1539 published

According to this Stack overflow post- https://stackoverflow.com/questions/65122072/azure-databricks-this-clusters-runtime-version-is-out-of-support... you can just leave the cluster and it will work just fine. But, i'll wait for someone from Microsoft to confirm this

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.