"append" statement doesn't work while appending the data stating "ClassCastException"

Question

We have recently upgraded our cluster from Databricks version 6.3 to 7.3 . Databricks version 7.3 doesn't support implicit conversion of data types while appending the data using pyspark into existing table in sql server. What can we do to automate the data type conversion? Is there any configuration setting for this?

Answer 1

Hello @Manoj Biswal (Harman Connected Services Inc) ,
Thanks for the ask and using Microsoft Q&A platform .

I think setting the option spark.sql.storeAssignmentPolicy to "Legacy" is help . Please read about this here

Please do let me know how it goes .
Thanks
Himanshu

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
  • Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of [Q&A Volunteer Moderators][50]