
I set the configuration in my notebook as the attached screenshot.
the numExecutors=8, but it still use 2 executors when running the query in spark pool

I set the configuration in my notebook as the attached screenshot.
the numExecutors=8, but it still use 2 executors when running the query in spark pool
Hello @CatherineMeng-5865,
As per the above screenshot, you have configured the custom configs in Synapse Workspace spark pool at 19:25 and you have ran the query before the custom configs in Synapse Workspace spark pool at 19:11 and that's the reason you are seeing number of executors as 2.

I would request to run the query after the custom configs in Synapse Workspace spark pool, you will see the exact number of executors as per the configuration.
You can use this command to check number of executors configured: spark.conf.get("spark.executor.instances")

Hope this helps. Do let us know if you any further queries.
Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.
5 people are following this question.