I am generating parquet files using Data Flow in ADF. The files have some fields with decimal data types.
When I try to create a Hive external table on top of those parquet files, I get the below error while reading the data through Hive.
org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in block.
I think this issue is caused because of different parquet conventions used for decimal fields in Hive and Spark.(Data Flow essentially uses Spark only).
I think the issue gets resolved if you are generating parquet using Spark in Databricks. Then you can set Spark.sql.parquet.writeLegacyFormat=true but how to handle same thing in Azure Data Factory Data Flows?