question

PrajwalRam-0051 avatar image
0 Votes"
PrajwalRam-0051 asked PrajwalRam-0051 commented

Azure Function, Timer Trigger suddenly stops firing

Context:
I have set up a Function App on Azure under an App Service Plan (Premium functions - ae north).
The function app misbehaves after being live for sometime.

App Service Plan:
"kind": "elastic"
"location": "UAE North"
"name": "EP1",
"tier": "ElasticPremium",
"size": "EP1",
"family": "EP",
"capacity": 1


App Characteristics
- There are a total of 13 functions within the app that are created to be fired at different intervals (two times/three times/six times every 24 hours).
- The Functions app has been developed, debugged and deployed via VScode and work as intended initially when deployed.
- The basic use case of the funtion app is to call relevant API endpoints, pre-process and transform data and push it to an external cloud data warehouse. Everytime an app is fired, an email is triggered via sendgrid to inform the status of the ETL activity to the monitoring user.
- The first time the functions are triggered, the app is designed to ingest the historic data based on certain parameters configured during deployment
- The apps are also designed to ensure that at any given point in time, they run for not longer than 60 min
- There are some temporary files being staged in /tmp folder, but, the last time i checked they are minimal and are designedto be overwritten during consequtive every run

Issue

  • The Funtion app works perfectly fine and suddenly stops firing.

  • This has happened in two instances and both the times, a simple act of restarting the app gets it back up and running as intended.

  • There are two ways i get to know if the app have stopped firing, the emails on the status is not triggered and i also ensure to cross validate the latest injestion dates on the cloud data warehouse to cross off email client issue.
    I have noticed that this happens during every for some reason i dont understand. Although its happend only twice now.

  • I have checked the logs to see if there are any programming errors or execptions that has caused the trigger of mails or update of Data warehouse, there is no evidence of that. As mentioned, a simple act of restarting the app get everything up and running without any issues

I am trying to understand why this happens and this is not a scalable option to move forward with the solution.

Note:
Self taught programmer here with little development experience currently familiarizing myself with the azure ecosytem. Please go easy on the comments

azure-functions
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

MayankBargali-MSFT avatar image
0 Votes"
MayankBargali-MSFT answered MayankBargali-MSFT commented

@PrajwalRam-0051 As you have mentioned that you have mutiple timer function so it looks like it is missing the run for some of the timer trigger functions. There could be different reasons for this issue and I will suggest you to review the below document to troubleshoot the issue and see if you are able to find the root cause.

Timer triggered function app uses TimerTriggerAttribute. This attribute consists of the Singleton Lock feature which ensures that only a single instance of the function is running at any given time. If any process runs longer than the scheduled timer, the new incoming process waits for the older process to finish and then uses the same instance. If you are using the same storage account across different timer trigger functions then this could be one of the reasons as mentioned here.
You can disable that behavior if you wish by setting UseMonitor = false on your TimerTrigger attribute and see if resolves the issue.

You can also run the Diagnose and solve problems on your function app and search for "Timer Trigger Issue Analysis" to get the insights and recommendation.

The other reason could be a restart and I will suggest you to check the Web App Restart detection section.
https://github.com/Azure/azure-functions-host/wiki/Investigating-and-reporting-issues-with-timer-triggered-functions-not-firing
https://github.com/Azure/azure-webjobs-sdk-extensions/wiki/TimerTrigger#troubleshooting

· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HI Mayank,

Thank you for the suggestion. I have made necessary changes as per your suggestion. Will test it for another week or so before confirming if it works.

1 Vote 1 ·

@PrajwalRam-0051 Sure. Feel free to get back to me if you need any assistance.

0 Votes 0 ·

Hi Mayank,

Unfortunately, the same issue recurred again.
It's so strange that it happens only over the weekends. Over the weekdays, the functions app performs as intended.
I double checked the
- Cron Expression
- RunOnStartup option for the individual apps (Set to false)
- App Configuration TimeZone (Set to operating timezone)
But, The issue stills seems to persist

I noted from one of the links that the always on feature needs to be checked in case of an App Service plan, however, not table to find this in general settings tab of configuration section.

It looks like in the current scenario, will just have to restart the Function App every weekend....

Let me know your thoughts..


0 Votes 0 ·
Show more comments