question

TheShimmyShams-4331 avatar image
0 Votes"
TheShimmyShams-4331 asked ShaikMaheer-MSFT commented

Azure Data Factory: Job failed due to reason Expression cannot be parsed

I am attempting to use a pipeline parameter in a ForEach activity, however I am receiving a "DF-Executor-ParseError" error with no details:
114648-error-message.png


In checking the activities within the ForEach we can see the incoming parameters evaluate as expected:
114722-image.png


And if I plug in the values directly into the Debug Settings the underlying flow runs just fine:
114696-image.png

114609-image.png


I've attached copies of the Pipeline definition as well as the Data Flow script. Does anyone have an idea on what I might be missing?

Thank you!

114705-pipeline-manually-run-items.txt
114717-data-flow-script-test-generics.txt


azure-data-factory
· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Afraid I haven't used the regex component but are you sure you need the single quotes or the escaping of the backslash? I would expect the field to take the regex string and handle any escaping if required

0 Votes 0 ·

Thank you Ryan. In testing the Data Flow directly, the single quotes and escaping does appear to be required by ADF. The value of the field is evaluated by the expression builder which need the quotes for identifying strings.
115166-image.png

The string without the escaping does return a rather familiar error message.
115222-image.png
An "Unable to Parse" with no details.

In checking if the single quotes are causing problems in the Pipeline itself, I can check the Expression box for the parameter on the Data Flow call.
115184-image.png

This removes the single quotes when the Pipeline passes the parameter down.
115214-image.png

However the same error is returned.
115233-image.png

Thank you for the suggestion though!

0 Votes 0 ·
image.png (10.5 KiB)
image.png (10.8 KiB)
image.png (67.7 KiB)
image.png (24.0 KiB)
image.png (46.9 KiB)

Sorry I couldn't be more help

What I would suggest is start small, rather than passing a variable, pass a small regex string and validate that it results in what you want then gradually build up to your output

One observation I do make, in your screenshot showing "no single quotes", I wouldn't expect the double backslash here too for a standard regex string, a double backslash is effectively saying this is a literal backslash whereas you want to say the \d represents a number

0 Votes 0 ·
Show more comments

1 Answer

ShaikMaheer-MSFT avatar image
2 Votes"
ShaikMaheer-MSFT answered ShaikMaheer-MSFT commented

Hi @TheShimmyShams-4331 ,

Welcome to Microsoft Q&A Platform. Thank you for posting your query here.

I tried to repro your scenario by creating parameters and passing similar values in it. I haven't got any issues too. Please check below gifs for same.

Data flow Activity:
115145-dataflowactivity.gif

Data Flow Execution:
115162-dataflowexecution.gif

Hence, It seems values what you are passing from your expressions in to parameters may some where causing data flow to break. It would be great if you can check the values what your execution is passing in to parameters.

Kindly feel free to share the exact values which causing your execution to break during run time. So, that we can help you to fix that as well. Thank you.


  • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification.


· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you for your response. Here are more details about the Pipeline and the values it passes down to the Data Flow:

The Pipeline starts with a lookup to a table that stores the Regular Expressions.
115110-image.png

The row of the table contains the pattern, the target table, the target column, and the name/label for the data being returned by the pattern.
115213-image.png
115221-image.png

The Regular Expressions are fed into a ForEach activity, referencing the results of the Lookup.
115178-image.png
115231-image.png

IThe activity in the ForEach calls the Data Flow and provides the parameters.
115232-image.png

Currently I feed the iterator value through a variable.
115193-image.png
115070-image.png

I've tried using @item().Pattern directly in the DF call, but got the same error.

I've attached the input JSON for one iteration of the loop. Is there any other information I can provide?

Thank you again!

115241-adf-foreach-input-json.txt

0 Votes 0 ·
image.png (12.2 KiB)
image.png (51.7 KiB)
image.png (28.6 KiB)
image.png (26.9 KiB)
image.png (37.4 KiB)
image.png (11.8 KiB)
image.png (18.0 KiB)
image.png (30.5 KiB)

Hi @TheShimmyShams-4331 ,

We reached product team internally on this issue. We will share updates soon once we see response from product team. Thanks for being patience.

0 Votes 0 ·

@TheShimmyShams-4331 - This is because dataflow requires regex to be escaped like \s+ needs to be \\s+.

Dataflow's expression editor will force you to do this like for example:

ORDER OF:\s+\d+(.?).AP would need to passed in as ORDER OF:\\s+\\d+(.?).AP

It appears that the Lookup you are using has the pattern without escaping as this 'ORDER OF:\s+\d+(.*?).AP'
That's why it is failing when passed into dataflow.

Two ways to fix this :
1) Adding an escape slash like above in the table you are looking up from so that the column value is ORDER OF:\\s+\\d+(.*?).AP?
2) Updating the Set Variable value to @replace(item().Pattern,'\','\\') so that slash is escaped properly

1 Vote 1 ·