How to install jars /libraries related to spark-redis in azure databricks cluster?

Question

question

Shivasai asked Mar 10, '22 PRADEEPCHEEKATLA-MSFT commented Mar 17, '22

How to install jars /libraries related to spark-redis in azure databricks cluster?

I am trying to connect to Azure cache for redis using python from azure databricks .

I have installed this package com.redislabs:spark-redis:2.3.0 from maven package in databricks. I have created a spark session with below code

SparkSession\
.builder\
.appName("myApp")\
.config("spark.redis.host", "my host")\
.config("spark.redis.port", "6379")\
.config("spark.redis.auth", "passwd")\
.getOrCreate()

But when I ran df.write.format("org.apache.spark.sql.redis").option("table", "people").option("key.column", "name").save()

I am getting below error.

Py4JJavaError: An error occurred while calling o390.save.
: java.lang.ClassNotFoundException:
Failed to find data source: org.apache.spark.sql.redis. Please find packages at
http://spark.apache.org/third-party-projects.html

Could you please let me know the detailed steps to install all necessary libraries/jars to access redis in databricks.

I have seen below code in spark-redis-python.md but I don't know how to run it in databricks.

$ ./bin/pyspark --jars <path-to>/spark-redis-<version>-jar-with-dependencies.jar

And also please let me know what is the latest spark-redis version.

azure-databricks azure-cache-redis

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PRADEEPCHEEKATLA-MSFT · Mar 16 at 10:27 AM

Hello @Shivasai,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.

Please don't forget to click on or upvote button whenever the information provided helps you.

0 ·

Answer 1 · 2022-03-10T11:21:30.653Z

PRADEEPCHEEKATLA-MSFT answered Mar 10, '22 PRADEEPCHEEKATLA-MSFT commented Mar 17, '22

Hello @Shivasai ,

Thanks for the question and using MS Q&A platform.

Redis has a Spark Package that you can download and attach to your cluster

The following notebook shows how to use Redis with Apache Spark in Azure Databricks.

For more details, refer to Azure Databricks - Redis.

Hope this will help. Please let us know if any further queries.

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

· 5

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Shivasai · Mar 10 at 02:59 PM

Hi @PRADEEPCHEEKATLA-MSFT ,

I would be using python/ pyspark to write my dataframe to redis. The instructions you provided are related to scala. Could you please provide instruction on how to install jars files as mentioned in the question.

0 ·

PRADEEPCHEEKATLA-MSFT Shivasai · Mar 11 at 03:59 AM

Hello @Shivasai,

Unfortunately, I cannot find the equivalent notebooks in Python. You can modify the scala code into python and run the code.

Hope this will help. Please let us know if any further queries.

0 ·

Shivasai PRADEEPCHEEKATLA-MSFT · Mar 11 at 02:35 PM

Hi Pradeep,

I cannot use scala spark-redis libraries for python code. like I mentioned in my question I have already installed com.redislabs:spark-redis:2.3.0 on my cluster and I am getting the error specified in question while writing a pyspark dataframe to redis.

And also please provide instruction on how to install the necessary jars

0 ·

Show more comments

question