Share via


SparkSessionExtensions.GetAssemblyInfo(SparkSession, Int32) Method

Definition

Get the Microsoft.Spark.Utils.AssemblyInfoProvider.AssemblyInfo for the "Microsoft.Spark" assembly running on the Spark Driver and make a "best effort" attempt in determining the Microsoft.Spark.Utils.AssemblyInfoProvider.AssemblyInfo of "Microsoft.Spark.Worker" assembly on the Spark Executors.

There is no guarantee that a Spark Executor will be run on all the nodes in a cluster. To increase the likelyhood, the spark conf spark.executor.instances and the numPartitions settings should be adjusted to a reasonable number relative to the number of nodes in the Spark cluster.

public static Microsoft.Spark.Sql.DataFrame GetAssemblyInfo (this Microsoft.Spark.Sql.SparkSession session, int numPartitions = 10);
static member GetAssemblyInfo : Microsoft.Spark.Sql.SparkSession * int -> Microsoft.Spark.Sql.DataFrame
<Extension()>
Public Function GetAssemblyInfo (session As SparkSession, Optional numPartitions As Integer = 10) As DataFrame

Parameters

numPartitions
Int32

Number of partitions

Returns

A DataFrame containing the Microsoft.Spark.Utils.AssemblyInfoProvider.AssemblyInfo

Applies to