Use external packages with Jupyter notebooks in Apache Spark clusters on HDInsight

Learn how to configure a Jupyter notebook in Apache Spark cluster on HDInsight to use external, community-contributed maven packages that are not included out-of-the-box in the cluster.

You can search the Maven repository for the complete list of packages that are available. You can also get a list of available packages from other sources. For example, a complete list of community-contributed packages is available at Spark Packages.

In this article, you will learn how to use the spark-csv package with the Jupyter notebook.

Prerequisites

You must have the following:

Use external packages with Jupyter notebooks

  1. From the Azure Portal, from the startboard, click the tile for your Spark cluster (if you pinned it to the startboard). You can also navigate to your cluster under Browse All > HDInsight Clusters.
  2. From the Spark cluster blade, click Quick Links, and then from the Cluster Dashboard blade, click Jupyter Notebook. If prompted, enter the admin credentials for the cluster.

    Note

    You may also reach the Jupyter Notebook for your cluster by opening the following URL in your browser. Replace CLUSTERNAME with the name of your cluster:

    https://CLUSTERNAME.azurehdinsight.net/jupyter

  3. Create a new notebook. Click New, and then click Spark.

    Create a new Jupyter notebook

  4. A new notebook is created and opened with the name Untitled.pynb. Click the notebook name at the top, and enter a friendly name.

    Provide a name for the notebook

  5. You will use the %%configure magic to configure the notebook to use an external package. In notebooks that use external packages, make sure you call the %%configure magic in the first code cell. This ensures that the kernel is configured to use the package before the session starts.

    Important

    If you forget to configure the kernel in the first cell, you can use the %%configure with the -f parameter, but that will restart the session and all progress will be lost.

    HDInsight version Command
    For HDInsight 3.3 and HDInsight 3.4 %%configure
    { "packages":["com.databricks:spark-csv_2.10:1.4.0"] }
    For HDInsight 3.5 and HDInsight 3.6 %%configure
    { "conf": {"spark.jars.packages": "com.databricks:spark-csv_2.10:1.4.0" }}
  6. The snippet above expects the maven coordinates for the external package in Maven Central Repository. In this snippet, com.databricks:spark-csv_2.10:1.4.0 is the maven coordinate for spark-csv package. Here's how you construct the coordinates for a package.

    a. Locate the package in the Maven Repository. For this tutorial, we use spark-csv.

    b. From the repository, gather the values for GroupId, ArtifactId, and Version. Make sure that the values you gather match your cluster. In this case, we are using a Scala 2.10 and Spark 1.4.0 package, but you may need to select different versions for the appropriate Scala or Spark version in your cluster. You can find out the Scala version on your cluster by running scala.util.Properties.versionString on the Spark Jupyter kernel or on Spark submit. You can find out the Spark version on your cluster by running sc.version on Jupyter notebooks.

    Use external packages with Jupyter notebook

    c. Concatenate the three values, separated by a colon (:).

     com.databricks:spark-csv_2.10:1.4.0
    
  7. Run the code cell with the %%configure magic. This will configure the underlying Livy session to use the package you provided. In the subsequent cells in the notebook, you can now use the package, as shown below.

     val df = sqlContext.read.format("com.databricks.spark.csv").
     option("header", "true").
     option("inferSchema", "true").
     load("wasb:///HdiSamples/HdiSamples/SensorSampleData/hvac/HVAC.csv")
    

    For HDInsight 3.6, you should use the following snippet.

     val df = spark.read.format("com.databricks.spark.csv").
     option("header", "true").
     option("inferSchema", "true").
     load("wasb:///HdiSamples/HdiSamples/SensorSampleData/hvac/HVAC.csv")
    
  8. You can then run the snippets, like shown below, to view the data from the dataframe you created in the previous step.

     df.show()
    
     df.select("Time").count()
    

See also

Scenarios

Create and run applications

Tools and extensions

Manage resources