question

ConnorHazen-9254 avatar image
0 Votes"
ConnorHazen-9254 asked ChristianSchreuder-4912 commented

ADF calling Azure Function: '404 not found' exception once out of every 10 runs (after 2 seconds)

I am, seemingly at random, receiving 404 not found errors on azure function calls from within ADF pipeline. They occur once every 10 runs, roughly, and come in two forms:

The first has this HTML as its error:

Call to provided Azure function 'Parse' failed with status-'NotFound' and message - '<html> <head><title>404 Not Found</title></head> <body bgcolor="white"> <center><h1>404 Not Found</h1></center> <hr><center>nginx</center> </body> </html> '.

The second is:

Call to provided Azure function 'Parse' failed with status-'NotFound' and message - 'Invoking Azure function failed with HttpStatusCode - NotFound.'.



These are interspersed within successful runs and rerunning the pipeline with the same triggered file (this a blob upload trigger) will run with no problems suggesting that it is not an input issue. Furthermore, the fact that they occur 2 seconds means it is not a timeout issue? Any thoughts?


azure-data-factoryazure-functions
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @ConnorHazen-9254 and welcome to Microsoft Q&A.

I did a little digging and found a similar case from around a year ago. In that case the cause was an overly long URL / query string (>2000 characters).

Could you give me an idea of how long the URL / query string is, and how frequently you are calling the function?

Azure function is not in my area of expertise, so I also wanted to consider possibility like overwhelming the endpoint.

0 Votes 0 ·

Hi @MartinJaffer-MSFT thank you for helping me debug this issue.

We are running the function currently 20-50 times a day. However that is going to scale up in the near future to a 1000+ a day.

As far as the URL string, it is simply the function app url, key, and three body params that are between 10 to 100 characters. There are no headers. So it is not an overlong URL string.

Again, thank you.

0 Votes 0 ·

Thank you for letting me know length isn't the problem, @ConnorHazen-9254 .

I would like to pass your case into the care of my colleague @ChaitanyaNaykodiMSFT-9638 . Azure Functions falls under his area of expertise.

0 Votes 0 ·
ChaitanyaNaykodiMSFT-9638 avatar image
0 Votes"
ChaitanyaNaykodiMSFT-9638 answered MitchellBennett-5751 commented

Hello @ConnorHazen-9254, Thank you for providing the details above. While looking at the backend, I think the issue behind intermittent 404 errors might be related to the synchronous implementation of the code. As Python is a single-threaded runtime, a host instance for Python can process only one function invocation at a time by default. For applications that process a large number of I/O events and/or is I/O bound, you can improve performance significantly by running functions asynchronously. It will be helpful you could go through this Async section of the Scale and Performance documentation for Python Function Apps and see if making the code Async helps in getting rid of the issue.
Let me know if there are any additional concerns. Thank you!


· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you for the advice, this is certainly a change we will implement. However, I am doubtful as to it being the only issue. For example, we yesterday had an invocation fail where the previous run was over an hour before. How would the single threaded nature of python cause this issue when there is only one run occurring, or it is the first run in a cluster.

0 Votes 0 ·

From my own digging, this seems quite similar to this https://github.com/Azure/azure-functions-host/issues/5149


There are no concurrent active runs for some of the failures, there is no timeout issue here, and it does not appear to be a scale issue.

Furthermore, I have made the function asynchronous now, this did not solve the issue.

Finally, it's also important to note that these failures do not show up on the monitor panel of the the function app. They are not failed runs or internal errors, they simply do not make it to the function app. Hence why the error link above seems related.

0 Votes 0 ·

Did you find a solution to this?

Just wanted to say that I am experiencing this issue also. The issue description is basically a perfect match with the problem I am having except I'm using the HTTP trigger.

0 Votes 0 ·
ConnorHazen-9254 avatar image
0 Votes"
ConnorHazen-9254 answered ConnorHazen-9254 edited

I solved this problem, I believe, in a roundabout manner. Still not sure what was causing it. However, I moved all of my code to a new function, meaning I made a new function app, made a new function in visual studio code, and then copied over the scripts(I left the new .vscode directory etc. - essentially everything expect the python files I had written was replaced by the new function I had made).

UPDATE: spoke too soon. still experiencing issues.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MitchellBennett-5751 avatar image
0 Votes"
MitchellBennett-5751 answered TaylorTeske-8548 commented

I have created a support ticket for this issue and have been told that the problem has been identified by Microsoft engineers and they are working on an "enhancement" for it.

In the mean time what I have done to work around this issue is to set the maximum number of retry attempts and the retry interval on each Azure Function activity in my pipeline.

Hopefully this helps and the issue gets resolved soon!

· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Are you by chance using python 3.9? I have a suspicion this may be the cause.

0 Votes 0 ·

I have tried using both Python 3.9 and 3.6 and I still had the same problem.

0 Votes 0 ·

I have received feedback from Microsoft employees about this issue and their position, at time of writing, is that the 404 issue (outlined in the problem description) is expected due to the cold-start limitation of the Consumption Plan for Azure Functions. They report that switching to the Premium Plan, which contains dedicated resources, is supposed to prevent this issue from happening. If the Premium Plan is undesirable then they recommend setting the number of retry attempts and the retry interval on the Azure Function activity - as I mentioned in my comment above.

0 Votes 0 ·

Out of curiosity, with setting the retry attempts and interval, have you still encountered any 404s? I stumbled upon this recent thread as we seem to be experiencing intermittent 404s and tried implementing a max retry to 3 and delay interval to 10 seconds.

0 Votes 0 ·
ChristianSchreuder-4912 avatar image
0 Votes"
ChristianSchreuder-4912 answered

Hi, if you restart your Function App does the error go away for a few hours before it comes back?

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

dthunn-2704 avatar image
0 Votes"
dthunn-2704 answered ChristianSchreuder-4912 commented

At my work we are experiencing the exact same issue, about 1 / 10 requests results in a 404, there is no clear way to fix this problem.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Upgrading the function app to premier tier, where it doesn't have this cold start issue, works as a temporary fix.

0 Votes 0 ·