SparkSessionExtensions.GetAssemblyInfo(SparkSession, Int32) Method

  • Reference

Definition

Namespace:
Microsoft.Spark.Experimental.Sql
Assembly:
Microsoft.Spark.dll
Package:
Microsoft.Spark v1.0.0

Get the Microsoft.Spark.Utils.AssemblyInfoProvider.AssemblyInfo for the "Microsoft.Spark" assembly running on the Spark Driver and make a "best effort" attempt in determining the Microsoft.Spark.Utils.AssemblyInfoProvider.AssemblyInfo of "Microsoft.Spark.Worker" assembly on the Spark Executors.

There is no guarantee that a Spark Executor will be run on all the nodes in a cluster. To increase the likelyhood, the spark conf spark.executor.instances and the numPartitions settings should be adjusted to a reasonable number relative to the number of nodes in the Spark cluster.

public static Microsoft.Spark.Sql.DataFrame GetAssemblyInfo (this Microsoft.Spark.Sql.SparkSession session, int numPartitions = 10);
static member GetAssemblyInfo : Microsoft.Spark.Sql.SparkSession * int -> Microsoft.Spark.Sql.DataFrame
<Extension()>
Public Function GetAssemblyInfo (session As SparkSession, Optional numPartitions As Integer = 10) As DataFrame

Parameters

session
SparkSession

The SparkSession

numPartitions
Int32

Number of partitions

Returns

DataFrame

A DataFrame containing the Microsoft.Spark.Utils.AssemblyInfoProvider.AssemblyInfo

Applies to