question

SashankGuduru-0588 avatar image
0 Votes"
SashankGuduru-0588 asked KranthiPakala-MSFT commented

Azure Data Factory - Spark cluster needs to be for ever - TTL to be for ever

In ADF, is there a way that we can have the spark cluster for data flow not to go down at all (can we set TTL to forever). So that we can use that one cluster for multiple dataflow even if the next dataflow happens after 4 hours. We see the option for TTL only up to 4 hours, is there a way to set it more than that.

Any thoughts?

azure-data-factory
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HI @SashankGuduru-0588,

Just checking in to see if the below information was helpful. If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Thank you

0 Votes 0 ·

HI @SashankGuduru-0588,

We still have not heard back from you. Just wanted to check if the below suggestion was helpful? If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

0 Votes 0 ·
MarkKromer-2402 avatar image
0 Votes"
MarkKromer-2402 answered

TTL defines how much idle time you would like the Azure IR to allow for your cluster. If you want the cluster to be available to execute jobs immediately all day. then set the TTL to the lowest amount of time between your scheduled jobs. For example: if you execute data flows 1x/hour, and those jobs take 10 mins, you'll have 50 mins of idle time. If you don't want to wait for another cluster cold startup, set the Azure IR TTL to 1 hour.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered KranthiPakala-MSFT edited

HI @SashankGuduru-0588,

Welcome to Microsoft Q&A forum and thanks for your query.

Unfortunately 4 hours is the max TTL value you can set currently in Azure IR.


But if you have any specific feedback regarding any improvement, please provide your feedback in ADF user voice forum: https://feedback.azure.com/forums/270578-data-factory
This forum is actively monitored and reviewed by Azure Engineering teams. Also please do share the feedback link here once it is posted so that others with similar idea can up-vote and/or comment on your suggestion which would help to increase the priority of feature request.

Please note: Billing for data flows is based upon the type of compute you select and the number of cores selected per hour. If you set a TTL, then the minimum billing time will be that amount of time. Otherwise, the time billed will be based on the execution time of your data flows and the time of your debug sessions. Note that debug sessions will incur a minimum of 60 minutes of billing time unless you switch off the debug session manually. For further details, please click here for the pricing page.

One important point to remember here is if we set the Cluster idle time (i.e., TTL) to 4 hrs then you will be charged for 4 hrs. So we need to think about 4hrs billing vs 5min cluster start up time.

Incase if you have jobs executing within short period of time then you can utilize the TTL feature efficiently with minimum value set and that will make the cluster available all day.

Hope this helps.




Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.