Share via

Proper Cores/Executors Configuration in HD-Insight

Joaquin Chemile 41 Reputation points
2020-08-04T15:19:31.557+00:00

Proper Cores/Executors Configuration in HD-Insight

15574-1.jpg

And for this cluster i've this configuration
15582-config.jpg

Which is the best way to make a proper configuration in order to run efficiently a job in Spark. Is it Ok this configuration? Thanks!

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
199 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. PRADEEPCHEEKATLA-MSFT 78,986 Reputation points Microsoft Employee
    2020-08-05T07:51:21.923+00:00

    Hello @Joaquin Chemile ,

    Welcome to Microsoft Q&A platform.

    Depending on your Spark workload, you may determine that a non-default Spark configuration provides more optimized Spark job executions. Do benchmark testing with sample workloads to validate any non-default cluster configurations. Some of the common parameters that you may consider adjusting are:

    Here are some common parameters you can adjust:

    15698-image.png

    This article discusses how to optimize the configuration of your Apache Spark cluster for best performance on Azure HDInsight.

    For more information on using Ambari to configure executors, see Apache Spark settings - Spark executors.

    Hope this helps. Do let us know if you any further queries.

    ----------------------------------------------------------------------------------------

    Do click on "Accept Answer" and Upvote on the post that helps you, this can be beneficial to other community members.