Upravit

rx_get_pyspark_connection

  • Článek

Usage

revoscalepy.rx_get_pyspark_connection(compute_context: revoscalepy.computecontext.RxSpark.RxSpark)

Description

Gets a PySpark connection from the current Spark compute context.

Arguments

compute_context

Compute context get created by rx_spark_connect.

Returns

Object of python.context.SparkContext.

Example

from revoscalepy import rx_spark_connect, rx_get_pyspark_connection
from pyspark.sql import SparkSession
cc = rx_spark_connect(interop = "pyspark")
sc = rx_get_pyspark_connection(cc)
spark = SparkSession(sc)
df = spark.createDataFrame([('Monday',3012),('Friday',1354),('Sunday',5452)], ["DayOfWeek", "Sale"])