Unable to run Sql queries in Azure Synapse. Error: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException

Question

Hi,
I have deployed a Synapse workspace through devops and running some sql queries inside it. I can run simple SQL queries but can not run anything related to Delta Table. Commands like -

1. SHOW TABLES
2. %%sql
   CREATE DATABASE AdventureWorksLT2019
3. DROP TABLE IF EXISTS table_name

all fail with 'Error: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException'. I originally want to save a delta table to my ADLS but my saveAsTable command fails. Running new_target_df.write.format("delta") \ .mode('append').option("overwriteSchema", "true") \ .option("path", delta_table_path) \ .partitionBy('subscriptionId','year','month','day') \ .saveAsTable(delta_table_name) # External table
It gives -

AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
Traceback (most recent call last):

  File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 1158, in saveAsTable
    self._jwrite.saveAsTable(name)

  File "/home/trusted-service-user/cluster-env/env/lib/python3.8/site-packages/py4j/java_gateway.py", line 1304, in __call__
    return_value = get_return_value(

  File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 117, in deco
    raise converted from None

pyspark.sql.utils.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException

I am however able to write dataframe as delta lake at destination using - new_target_df.write.format('delta').mode('append').option("overwriteSchema", "true").save(delta_table_path) so I don't think it is a permission issue. I am Synapse Administrator, Synapse workspace has Storage Blob Data Contributor on ADLS.

It seems to be an error with Hive Metastore maybe but I dont understand clearly. I recreated workspace to no avail. Please help

Answer 1

Hi @Yash Tamakuwala ,
Thankyou for using Microsoft Q&A platform and posting your queries.
In Azure Synapse workspace, you need to go to Develop tab, and create a new notebook in order to run these queries . The notebook should be attached to Spark pool . You can create Apache spark pool in Manage tab of Synapse Workspace and attach your notebook with the spark pool.

Note: As Spark pools are a provisioned service, you pay for the resources provisioned. You can go with small node size and keep Max number of nodes as 3 for getting the least charge

1. SHOW TABLES

2. CREATE DATABASE AdventureWorksLT2019

3. DROP TABLE IF EXISTS table_name

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you.
  Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators