question

arkiboys avatar image
0 Votes"
arkiboys asked AnnuKumari-MSFT answered

@range for building Ids in a string

ADF pipeline reads .csv file daily which is populated by calling api url
The .csv file has different number of rows for each day, could be 10, may be 50, may be 300 rows, etc. for each day...

I am using @Range() function inside foreach activity so that batches of Ids are built...
@Range(0, 10)
The above goes through the iteration 10 times but what if there are say 243 rows and I want to build the string for 10 Ids at a time and process each batch of 10 Ids and then get the next 10 Ids and process them...
for example:
The first 10 Ids to be processed, then the next 10 and then the next 10, etc. untill the iteration gets to the final loop which could be 243
How can I use the @Range function in a loop to do what I am after?
Perhaps this is related to offset parameter in calling the api url and the pagination functionality?

Thank you

azure-data-factory
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Is there a metadata or lookup as an input to your ForEach?

0 Votes 0 ·

yes, the .csv is read by lookup activity which is just before the foreach.
Thanks

0 Votes 0 ·
AnnuKumari-MSFT avatar image
0 Votes"
AnnuKumari-MSFT answered

Hi @arkiboys ,

I hope the query has been answered here : https://docs.microsoft.com/en-us/answers/questions/824597/index.html

Please let us know if it's resolved. Thanks !

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PratikSomaiya avatar image
0 Votes"
PratikSomaiya answered arkiboys edited

Hello @arkiboys

If the ForEach activity's input is a lookup, you can pass the Range to loop using lookup

In ForEach activity you can mention number of batches to run in parallel as below

195926-image.png



image.png (25.1 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

not quite following what you are suggesting.
Basically as I mentioned previously, the .csv rows changes daily.
Let's say today there are 153 rows in the .csv which is read by lookup.
The next activity is foreach.
My goal is to get all the IdNames in the company field in .csv
At present, in the foreach activity, using array and variable I build the url string using all the dNames from the .csv
example:
http://xyz in ('dName1', 'dName2', 'dName3', ..., 'dName153')
This is fine but the IN clause can get too long and so the api does not accept that long string so I am trying to process the http in batches like:
http://xyz in ('dName1', 'dName2', 'dName3', ..., 'dName10')
then
http://xyz in ('dName11', 'dName12', 'dName13', ..., 'dName20')
then
http://xyz in (dName21', 'dName22', 'dName23', ..., 'dName30')
...

I have started to use @range(0, 10) but not sure how to do the next @range and the one after ...
Hope you see what I mean
Thank you

0 Votes 0 ·