Using SynapseML v0.9.5 in Spark pool

Question

We aim to use the geocoding features of SynapseML library v0.9.5 over Synapse Spark pools. However, this library needs Spark 3.2+ while Spark pools just support Spark 3.1 & 2.7. So we can just install SynapseML v0.9.4 but it does not have the geospatial features and we face an error when using "from synapse.ml.geospatial import *". So, does it mean that for now we can't do geocoding over Spark pools?

Answer 1

Hi @Ali Davoudian ,

Thank you for posting query in Microsoft Q&A Platform.

As I understand query here, your goal is to use geocoding features of SynapeML library v0.9.5 over Synapse Spark pools. Please correct me if my understanding is wrong.

Below version is the latest available version for SynapseML on Spark3.1 pool. Internal team will be updating our official doc as well soon. Please use this configuration magic instead to make this work.

%%configure -f  
{  
  "name": "synapseml",  
  "conf": {  
      "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:0.9.5-13-d1b51517-SNAPSHOT",  
      "spark.jars.repositories": https://mmlspark.azureedge.net/maven,  
      "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12",  
      "spark.yarn.user.classpath.first": "true"  
  }  
}  

Hope this helps. Please let us know how it goes.

----------------

Please consider hitting Accept Answer button. Accepted answers helps community as well.