question

53848354 avatar image
0 Votes"
53848354 asked ShaikMaheer-MSFT commented

Table Storage Dynamic lookup query from ADF

Hello everybody.

I'm trying to setup an ADF Pipeline that 'explodes' data from an Azure Table Storage to a file system creating csv files with dynamic names based on the Partition Key value of the table.

I have on the left side a list of devices taken from a SQL Azure DB (Now they're 16 but in real-life will be thousands) and for each device I need to create every day a CSV.

I've configured a copy pipeline like this
127974-image.png


and set a TableStorage Source inside the FOR EACH cycle
127870-image.png


and everything seems to work well except the fact that the filter expression of the TABLESTORAGE provide a strange error:

The function 'length' expects its parameter to be an array or a string. The provided value is of type 'Object'.

127983-image.png

I've tried several ways to format the input string, all using or not the Dynamic Content to parse it but I always obtain the same error.

PartitionKey eq '{@item().deviceserial}' and Timestamp gt datetime'2021-02-01T00:00:00' and Timestamp le datetime'2021-02-02T00:00:00'
or
PartitionKey eq @item().deviceserial and Timestamp gt datetime'2021-02-01T00:00:00' and Timestamp le datetime'2021-02-02T00:00:00'

but nothing changes.

Where am I wrong ?
Thanks
Max


azure-data-factoryazure-table-storage
image.png (42.0 KiB)
image.png (38.3 KiB)
image.png (101.3 KiB)
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @53848354,

Welcome to Microsoft Q&A forum and thanks for posting your query.

As per the error message, it seems like the issue points to a LENGTH function input value but you have shared the Query for your Table storage which is not relevant as my understanding.

128050-image.png

Could you please confirm if you are using a LENGTH function anywhere in your FOREACH activity? If so could you please share the complete dynamic expression where it is being used and the variable/parameter used in it and a sample input value so that we can better assist on this query.

We look forward to your response.




0 Votes 0 ·
image.png (131.1 KiB)
53848354 avatar image 53848354 KranthiPakala-MSFT ·

AS I Know we're not using it in the code.
In attach you find the Complete JSON of the Pipeline.
If I have to watch in other places let me know, I'm new to ADF.
Thank you very much!
Max
128239-pipeline-json.txt


0 Votes 0 ·
pipeline-json.txt (11.1 KiB)

Hi @53848354,

Thanks for your response and details. I checked the pipeline JSON payload and didn't notice any LENGTH function used anywhere. This is strange. If that is the case then I suspect the error message is not relevant.

Have you got a chance to try changing the query for your Table Storage ? Instead of two dates, try with one date field and see if that makes a difference.

If that didn't help, I would recommend filing a support ticket for deeper investigation as the error message is irrelevant to the issue so that a support engineer can collect more details and involve relevant product team to further troubleshoot and resolve the issue.

Do let us know how it goes.

0 Votes 0 ·
Show more comments

1 Answer

ShaikMaheer-MSFT avatar image
1 Vote"
ShaikMaheer-MSFT answered ShaikMaheer-MSFT commented

Hi @53848354,

Thank you for posting query on Microsoft Q&A and thank you for sharing your pipeline json.

I see small errors in your pipeline json.

It seems you are passing @activity('lkDeviceList').output to items field in ForEach activity. You should pass @activity('lkDeviceList').output.value there.

Could you please update your configurations accordingly and see if that helps.

Hope this will help. Please let us know if any further queries. Thank you.


  • Please don't forget to click on Accept Answer button 130286-image.png whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators



image.png (2.5 KiB)
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @53848354,

Following up to check is above answer helps you? If yes, Please Accept Answer. Accepting answer helps community as well. Please let us know if any further queries. Thank you.

0 Votes 0 ·

Hi @53848354,

Just checking is above answer helps you? If yes, Please Accept Answer. Accepting answer helps community as well. Please let us know if any further queries. Thank you.

0 Votes 0 ·

Thank you! Now it works (with other errors that I'll debug soon but ... works)

Max

0 Votes 0 ·

Hi @53848354 ,

Thank you for marking as accepted answer.

0 Votes 0 ·