question

xyzchatter-0237 avatar image
0 Votes"
xyzchatter-0237 asked KranthiPakala-MSFT commented

Column mapping problem in Azure data factory

Hi all,

My source is HIVE and Target is Azure SQL Server. I am trying to use a query source from hive. But in column mapping page instead of column names it shows as tablename.columnname, hence it can not do the column mapping automatically. Any idea why this is happening? example if source table is T1 in the column mapping page it looks like this after manual mapping

T1.C1----->C1
T1.C2----->C2

Have huge tables with 200+ columns, it would be a pain if I do the manual mapping.

Thanks in advance

azure-data-factory
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @xyzchatter-0237 ,

Thanks for reaching out. Could you please confirm if your source is Hive tables in Azure HDInsight? And also what is the connector being used - Is it HIVE connector or ODBC?

0 Votes 0 ·

Have you tried the Hive connector in dataflow? You can do complex column mapping techniques after.

0 Votes 0 ·

@Kiran-MSFT - Cant use HIVE connector as the source is having kerberos authentication. Had to use ODBC connector only.

0 Votes 0 ·

1 Answer

xyzchatter-0237 avatar image
1 Vote"
xyzchatter-0237 answered KranthiPakala-MSFT commented

hi @KranthiPakala-MSFT - It is Hive tables with ODBC connection. I found a temporary solution to provide alias like this, in the source:

SELECT C1 AS C1, C2 AS C2, C3 AS C3.........FROM TAB

But just an ugly way to handle that for all the pipelines....

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @xyzchatter-0237, Thanks for your response and sharing the workaround here. We will reach out to product team to double check on this behavior and will get back to you as soon as I have an update from them.

Thank you for your patience.

0 Votes 0 ·

Hi @xyzchatter-0237 ,

Just heard back from internal team. As you are using ODBC connector, the issue could be caused by your driver itself. The column names ADF showed out are just the results returned by the driver. If you still need further assistance on this issue. I would suggest to file a support case so that a support engineer can take a deeper look into it.

Do let us know if you have further query.

Thank you

0 Votes 0 ·

Hi @xyzchatter-0237 ,

Just checking to see if you have any further query regarding this issue?

Thanks

0 Votes 0 ·