question

AdityaRaj-8613 avatar image
0 Votes"
AdityaRaj-8613 asked ShaikMaheer-MSFT commented

pass select query in filter adf

I am trying to pass select query in my filter .I am getting {"StatusCode":"DF-Executor-StoreIsNotDefined","Message":"Job failed due to reason: The store configuration is not defined. This error is potentially caused by invalid parameter assignement in the pipeline.","Details":""}


parameters{
srcTable as string,
trgTable as string,
filter as string,
unique_columns as string,
cdc_columns as string,
partition_columns as string,
load_type as string,
max_cdc_column as timestamp
}
source(output(
PERSONID as decimal(38,0),
NAME as string,
LASTMODIFYTIME as timestamp
),
allowSchemaDrift: true,
validateSchema: false,
query: (concat('select * from ',$srcTable,' where ',$filter)),
format: 'query') ~> source
Filter1 alterRow(insertIf(equalsIgnoreCase($load_type,'insert')),
updateIf(equalsIgnoreCase($load_type,'update')),
upsertIf(equalsIgnoreCase($load_type,'merge'))) ~> AlterRow1
source filter($cdc_columns> 'select WATERMARKVALUE from watermarktable where TABLENAME = $srcTable') ~> Filter1
AlterRow1 sink(allowSchemaDrift: true,
validateSchema: false,
deletable:false,
insertable:true,
updateable:true,
upsertable:false,
keys:(split($unique_columns,',')),
format: 'table',
skipDuplicateMapInputs: true,
skipDuplicateMapOutputs: true,
saveOrder: 1,
stageInsert: true) ~> sink1

azure-data-factory
· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @AdityaRaj-8613 ,

Thank you for posting your query on Microsoft Q&A Platform.

Did you tired passing hard coded values in to your parameters and performing data preview at each transformation level to find exactly at which transformation your data flow is breaking?

Please Note: If your $filter parameter going to contain single quotes in it then you need escape them with \ symbol. Please check below link to understand better.
https://docs.microsoft.com/en-us/answers/questions/440686/index.html

Thank you.

0 Votes 0 ·

Hi @AdityaRaj-8613 ,

Did you get chance to try above comment mentioned ask? Kindly share updates, so that we can try to repro and help in resolution. Thank you.

0 Votes 0 ·

Hi @AdityaRaj-8613 ,

Tried above comment mentioned ask? Kindly share updates, so that we can try to repro and help in resolution. Thank you.

0 Votes 0 ·

I tried it and its working

0 Votes 0 ·

Hi @AdityaRaj-8613 Thank you for confirming and accepting answer.

0 Votes 0 ·

1 Answer

MarkKromer-MSFT avatar image
0 Votes"
MarkKromer-MSFT answered

Have you tried testing this by hardcoding the query string in the source?

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.