question

RyanAbbey-0701 avatar image
1 Vote"
RyanAbbey-0701 asked RyanAbbey-0701 commented

Call notebook from Spark job

Something I didn't think would be particularly complex but no details of how to do such in the MS help files apache-spark-job-definitions



I've created a Spark notebook that I want to call from a Spark job definition, how to do such? Or am I overthinking and should follow exactly how it's done for DataBricks?

azure-synapse-analytics
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered RyanAbbey-0701 commented

Hello @RyanAbbey-0701 ,
Thanks for the ask and using the Microsoft Q&A platform .
You can do it from Synapse Studio . the below screen shot should help .

121845-image.png

Please do let me know how it goes .
Thanks
Himanshu
Please do consider clicking on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members








image.png (168.4 KiB)
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Yes, it's known that I can run a notebook from a pipeline but I want to be able to run a notebook from a Spark job

0 Votes 0 ·

Thanks @RyanAbbey-0701 for clarifying and I don't think at this you can run a notebook from the spark job . But I am just curiuos as to what is the miss if I trigger a notebook from a pipeline then if I do the same from Spark job . I am sure I am missing out something here .

Yes in Adb we can just browse to the notebook while creating the spark job ,
Thanks
Himanshu

0 Votes 0 ·

I'd say what's the point of a spark job if it can't kick off a spark notebook... having to create functionality outside of Synapse, upload it and repeat until we get settings right is pretty ludicrous... all the more so because we can't seem to pass parameters from Logic Apps to it from a REST API call...

0 Votes 0 ·