question

CatherineMeng-5865 avatar image
0 Votes"
CatherineMeng-5865 asked PRADEEPCHEEKATLA-MSFT edited

Magic configuration can't work in spark session

85352-image.png





I set the configuration in my notebook as the attached screenshot.
the numExecutors=8, but it still use 2 executors when running the query in spark pool

azure-synapse-analytics
image.png (89.3 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
1 Vote"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT edited

Hello @CatherineMeng-5865,

As per the above screenshot, you have configured the custom configs in Synapse Workspace spark pool at 19:25 and you have ran the query before the custom configs in Synapse Workspace spark pool at 19:11 and that's the reason you are seeing number of executors as 2.

85669-image.png

I would request to run the query after the custom configs in Synapse Workspace spark pool, you will see the exact number of executors as per the configuration.

You can use this command to check number of executors configured: spark.conf.get("spark.executor.instances")

85658-image.png

Hope this helps. Do let us know if you any further queries.


Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.



image.png (139.7 KiB)
image.png (56.3 KiB)
image.png (59.2 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.