question

ShwetaChandramouli-8836 avatar image
0 Votes"
ShwetaChandramouli-8836 asked PRADEEPCHEEKATLA-MSFT commented

Setting StatusFolder for HDInsight Spark job triggered through Data Factory

Currently our Spark job runs result in a number of folders with random guid names being created in the root directory of the container we use as our HDInsight cluster storage. This seems to be the folder in the context of which the job runs, it has a copy of the script being used. Is there a way to specify a folder within which these guid folders for job state can be created?

azure-data-factoryazure-hdinsightdotnet-ml-big-data
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @ShwetaChandramouli-8836,

Thanks for the ask and also for using the Microsoft Q&A forum.

We are reaching out to the internal team to get more help on this, I will update you once we hear back from them.

0 Votes 0 ·

Hello @ShwetaChandramouli-8836,

We are still waiting for an update from internal team. I will get back to you once we hear back from them.

Stay tuned!

0 Votes 0 ·

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @ShwetaChandramouli-8836,

Unfortunately, you cannot specific a Status Folder for HDInsight Spark Job trigged through Data Factory.

Reason: Azure Data Factory creates [auto-generated-GUID] once HDInsight Spark job completes.

I would suggest you to provide feedback on the same:

https://feedback.azure.com/forums/270578-data-factory

All of the feedback you share in these forums will be monitored and reviewed by the Microsoft engineering teams responsible for building Azure.

Hope this helps. Do let us know if you any further queries.


Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @ShwetaChandramouli-8836,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

0 Votes 0 ·

Hello @ShwetaChandramouli-8836,
Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.
Take care & stay safe!

0 Votes 0 ·