question

YarmerChad-7721 avatar image
0 Votes"
YarmerChad-7721 asked ShaikMaheer-MSFT commented

Why does dynamic datatype in ADX fails when loading in data factory?

ADX supports dynamic datatypes but when I am loading the table through data factory it errors out and states that the dataype is not supported. But I can create adx tables with dynamic types no problem. It is just the loading from data factory that is the issue. Is there a fix for this?

azure-data-factory
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @YarmerChad-7721 ,

Welcome to Microsoft Q&A Platform. Thank you for posting query here.

Could you please help with below asks to understand issue and repro.

  • Could you please share which activity gives error and error details or screenshot? More details on your activity configuration helps here.

  • Are you able to create dataset for your table with dynamic types in ADF? Also, Could you please check is schema is getting imported inside the dataset?

0 Votes 0 ·



Parquet file contained column 'error_msg', which is of a non-primitive, unsupported type. Activity ID: f6a85951-5e26-4f3d-bbe2-04d0c479e177

The column in question is an array column from hive. which from docs says the dynamic datatype would work.

I can create the table in adx with no problem but the parquet file with array type is where the error comes in.
132390-error.png


0 Votes 0 ·
error.png (66.7 KiB)

Hi @YarmerChad-7721 ,

Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

0 Votes 0 ·

Hi @YarmerChad-7721 ,

Following to see if the below answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

0 Votes 0 ·

1 Answer

ShaikMaheer-MSFT avatar image
0 Votes"
ShaikMaheer-MSFT answered

Hi @YarmerChad-7721 ,

Thank you for posting query on Microsoft Q&A. Thank you for sharing screenshot of error.

Error is happening when creating Dataset for parquet file. You may receive this error message, when you have columns which are unsupported data type in parquet files.

You may checkout the file contained column “error_msg” data type and make sure you are using the supported data type.

Supported data type mappings for parquet files.

132549-image.png

Click here to check documentation for same.

Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators


image.png (231.7 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.