Dynamic Schema / ADF

Question

question

raffycomendador-3425 asked Nov 29, '21 Raffy-7546 answered Dec 11, '21

Dynamic Schema / ADF

Hi, am trying to ingest text files going to parquet table dynamically from different source files (column names and count differ per each file). Thus, am not defining the table schema anywhere (source, sink, dataset) to make this work. but due to this, the parquet table have Prop_0, prop_1 etc. as column names as a result (as expected :-) ).

How can I make the ADF use the column names included in the source file to have the schema of both dataset, source, sink get updated accordingly so that the target parquet table will have proper column names?

Thank you in advance.

azure-data-factory

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 1 · 2021-11-30T23:01:44.9Z

KranthiPakala-MSFT answered Nov 30, '21 KranthiPakala-MSFT commented Dec 3, '21

Hi @raffycomendador-3425,

Thanks for your query and using Microsoft Q&A forum.

Have you tried enabling the first row as header property in your source dataset? If not could you please try it and see if that helps.

In case if that is not helpful, the alternate would be to go with parameterized mapping for each source file. To explore more about dynamic column mapping please refer to this video - Dynamic Column mapping in Copy Activity in Azure Data Factory

Here is the public doc related to parameterized mapping: Parameterize mapping

Hope this info helps. Do let us know if you have further query.

Please don't forget to click on and upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

image.png (78.0 KiB)

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

KranthiPakala-MSFT · Dec 03, 2021 at 08:18 PM

Hi there,

Just checking in to see if the above suggestion was helpful. If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

0 ·

Answer 2 · 2021-12-05T23:24:59.797Z

Raffy-7546 answered Dec 5, '21 KranthiPakala-MSFT commented Dec 6, '21

Hi, I have yet to apply the resolution provided. will try to revert within the week. thank you and God bless

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

KranthiPakala-MSFT · Dec 06, 2021 at 07:14 PM

Thanks for the update @Raffy-7546. Please feel free to reach out if you have further query.

Have a good day!

1 ·

Answer 3 · 2021-12-11T09:15:32.023Z

Raffy-7546 answered Dec 11, '21

Hi, cant find the accept button :) but this is helpful and is accepted. thanks much!

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

question