YAML pipeline sometimes fails with ERROR_COULD_NOT_CONNECT_TO_REMOTESVC when deploying Azure functions to an app service

Bill Wolohan 41 Reputation points
2022-03-03T14:48:58.253+00:00

This is most likely not a configuration issue since the pipeline works most of the time.
I would guess our pipeline fails less than 20% of the time.

Rerun failed jobs will sometimes work correctly but usually not.
Sometimes restarting the app service then rerunning failed jobs works.
We are deploying to a slot. Deleting and recreating the slot then rerunning works every time so far.

These functions are deployed for 3 different environments in 2 different subscriptions. All have intermittent failures.
We have 3 app services hosting functions in each environment. This has happened on different app services in different environments.

We are using a self hosted agent for our pipelines.

Because we have multiple app services we use a template in our pipelines. Here is the template

parameters:

  • name: webAppName
    type: string
  • name: azureSubscription
    type: string
  • name: resourceGroupName
    type: string
  • name: packagePath
    type: string
  • name: swapSlots
    type: boolean
    default: 'False'

steps:

  • task: AzureRmWebAppDeployment@4
    displayName: 'Deploy to ${{parameters.webAppName}} slot (with -verbose)'
    inputs:
    ConnectionType: 'AzureRM'
    azureSubscription: ${{parameters.azureSubscription}}
    appType: 'webApp'
    WebAppName: '${{parameters.webAppName}}'
    deployToSlotOrASE: true
    ResourceGroupName: '${{parameters.resourceGroupName}}'
    SlotName: 'deploy'
    packageForLinux: '${{parameters.packagePath}}'
    AppSettings: '-BuildNumber $(Build.BuildNumber) -APPCONFIG_CONNECTIONSTRING $(APPCONFIG_CONNECTIONSTRING) -AppConfigConnectionString $(APPCONFIG_CONNECTIONSTRING) -AppConfigEnvironment $(ASPNETCORE_ENVIRONMENT)'
    enableCustomDeployment: true
    DeploymentType: 'webDeploy'
    RemoveAdditionalFilesFlag: true
    ExcludeFilesFromAppDataFlag: true
    RenameFilesFlag: true
    TakeAppOfflineFlag: true
    errorActionPreference: 'continue'
    AdditionalArguments: '-retryAttempts:6 -retryInterval:5000 -verbose'
  • ${{ if eq(parameters.swapSlots, True) }}:
    • template: swap-app-service.yml
      parameters:
      webAppName: '${{parameters.webAppName}}'
      azureSubscription: '${{parameters.azureSubscription}}'
      resourceGroupName: '${{parameters.resourceGroupName}}'
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,249 questions
Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
6,865 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Bill Wolohan 41 Reputation points
    2022-03-08T13:57:01.567+00:00

    Unfortunately, I don't get a lot of detail. This is run from a DevOps build/deploy agent using the Azure app service deploy task AzureRMWebAppDeployment@4. It calls msdeploy.exe with the verbose flag as seen below.

    Are there other flags or other ways to get more details from msdeploy?

    Here is the output I get.

    2022-03-08T13:14:50.6539832Z ##[section]Starting: Deploy to -metrics-fun slot (with -verbose)
    2022-03-08T13:14:50.6809948Z ==============================================================================
    2022-03-08T13:14:50.6810290Z Task : Azure App Service deploy
    2022-03-08T13:14:50.6810666Z Description : Deploy to Azure App Service a web, mobile, or API app using Docker, Java, .NET, .NET Core, Node.js, PHP, Python, or Ruby
    2022-03-08T13:14:50.6811016Z Version : 4.198.0
    2022-03-08T13:14:50.6811229Z Author : Microsoft Corporation
    2022-03-08T13:14:50.6811518Z Help : https://aka.ms/azureappservicetroubleshooting
    2022-03-08T13:14:50.6811815Z ==============================================================================
    2022-03-08T13:14:51.4369377Z Got service connection details for Azure App Service:'xxx-metrics-fun'
    2022-03-08T13:14:52.1222256Z Trying to update App Service Application settings. Data: null
    2022-03-08T13:14:52.3880753Z App Service Application settings are already present.
    2022-03-08T13:14:52.9403042Z [command]"C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -verb:sync -source:package='E:\azagent\01_work\3\drop\Descartes.Matchpoint.Metrics.zip' -dest:auto,ComputerName='https://xxx-metrics-fun-deploy.scm.azurewebsites.net:443/msdeploy.axd?site=xxx-metrics-fun',UserName='$xxx-metrics-fun__deploy',Password='    ',AuthType='Basic' -setParam:name='IIS Web Application Name',value='xxx-metrics-fun' -enableRule:AppOffline -skip:Directory=App_Data -retryAttempts:6 -retryInterval:5000 -verbose -userAgent:VSTS_af34c7f0-c49c-4dc0-b70b-278201c9de02_build_430_0
    2022-03-08T13:14:53.4476145Z Info: Using ID 'e57d45e0-300b-4141-a2f0-fc3d065f294d' for connections to the remote server.
    2022-03-08T13:14:53.4489726Z Verbose: Pre-authenticating to remote agent URL 'https://xxx-metrics-fun-deploy.scm.azurewebsites.net:443/msdeploy.axd?site=xxx-metrics-fun' as '$xxx-metrics-fun__deploy'.
    2022-03-08T13:15:03.7152523Z Verbose: Performing synchronization pass #1.
    2022-03-08T13:15:03.8332931Z Verbose: Pre-authenticating to remote agent URL 'https://xxx-metrics-fun-deploy.scm.azurewebsites.net:443/msdeploy.axd?site=xxx-metrics-fun' as '$xxx-metrics-fun__deploy'.
    2022-03-08T13:15:08.8335915Z Verbose: The HTTP connection (ID='e57d45e0-300b-4141-a2f0-fc3d065f294d', type ='GetTraceStatus') is being kept alive while the request is processed.
    2022-03-08T13:15:23.8641226Z Verbose: The HTTP connection (ID='e57d45e0-300b-4141-a2f0-fc3d065f294d', type ='GetTraceStatus') is being kept alive while the request is processed.
    2022-03-08T13:15:38.9291734Z Verbose: The HTTP connection (ID='e57d45e0-300b-4141-a2f0-fc3d065f294d', type ='GetTraceStatus') is being kept alive while the request is processed.
    2022-03-08T13:15:39.1423764Z [command]"C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -verb:sync -source:package='E:\azagent\01_work\3\drop\Descartes.Matchpoint.Metrics.zip' -dest:auto,ComputerName='https://xxx-metrics-fun-deploy.scm.azurewebsites.net:443/msdeploy.axd?site=xxx-metrics-fun',UserName='$xxx-metrics-fun__deploy',Password='',AuthType='Basic' -setParam:name='IIS Web Application Name',value='xxx-metrics-fun' -enableRule:AppOffline -skip:Directory=App_Data -retryAttempts:6 -retryInterval:5000 -verbose -userAgent:VSTS_af34c7f0-c49c-4dc0-b70b-278201c9de02_build_430_0
    2022-03-08T13:15:39.6250229Z Info: Using ID 'fa30c723-b447-41e9-865a-ba14d04ae7c7' for connections to the remote server.
    2022-03-08T13:15:39.6267844Z Verbose: Pre-authenticating to remote agent URL 'https://xxx-metrics-fun-deploy.scm.azurewebsites.net:443/msdeploy.axd?site=xxx-metrics-fun' as '$xxx-metrics-fun__deploy'.
    2022-03-08T13:15:50.6264681Z Verbose: Performing synchronization pass #1.
    2022-03-08T13:15:50.7253527Z Verbose: Pre-authenticating to remote agent URL 'https://xxx-metrics-fun-deploy.scm.azurewebsites.net:443/msdeploy.axd?site=xxx-metrics-fun' as '$xxx-metrics-fun__deploy'.
    2022-03-08T13:15:55.7329016Z Verbose: The HTTP connection (ID='fa30c723-b447-41e9-865a-ba14d04ae7c7', type ='GetTraceStatus') is being kept alive while the request is processed.
    2022-03-08T13:16:10.7611293Z Verbose: The HTTP connection (ID='fa30c723-b447-41e9-865a-ba14d04ae7c7', type ='GetTraceStatus') is being kept alive while the request is processed.
    2022-03-08T13:16:25.7789315Z Verbose: The HTTP connection (ID='fa30c723-b447-41e9-865a-ba14d04ae7c7', type ='GetTraceStatus') is being kept alive while the request is processed.
    2022-03-08T13:16:25.9265835Z [command]"C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -verb:sync -source:package='E:\azagent\01_work\3\drop\Descartes.Matchpoint.Metrics.zip' -dest:auto,ComputerName='https://xxx-metrics-fun-deploy.scm.azurewebsites.net:443/msdeploy.axd?site=xxx-metrics-fun',UserName='$xxx-metrics-fun__deploy',Password='    ',AuthType='Basic' -setParam:name='IIS Web Application Name',value='xxx-metrics-fun' -enableRule:AppOffline -skip:Directory=App_Data -retryAttempts:6 -retryInterval:5000 -verbose -userAgent:VSTS_af34c7f0-c49c-4dc0-b70b-278201c9de02_build_430_0
    2022-03-08T13:16:26.4512705Z Info: Using ID 'f4c38f7d-b64e-41c0-876c-d87e85fe2bae' for connections to the remote server.
    2022-03-08T13:16:26.4528202Z Verbose: Pre-authenticating to remote agent URL 'https://xxx-metrics-fun-deploy.scm.azurewebsites.net:443/msdeploy.axd?site=xxx-metrics-fun' as '$xxx-metrics-fun__deploy'.
    2022-03-08T13:16:39.4448536Z Verbose: Performing synchronization pass #1.
    2022-03-08T13:16:39.5359999Z Verbose: Pre-authenticating to remote agent URL 'https://xxx-metrics-fun-deploy.scm.azurewebsites.net:443/msdeploy.axd?site=xxx-metrics-fun' as '$xxx-metrics-fun__deploy'.
    2022-03-08T13:16:44.5392934Z Verbose: The HTTP connection (ID='f4c38f7d-b64e-41c0-876c-d87e85fe2bae', type ='GetTraceStatus') is being kept alive while the request is processed.
    2022-03-08T13:16:50.1221096Z Verbose: Received response from agent (HTTP status 'OK').
    2022-03-08T13:16:59.5749946Z Verbose: The HTTP connection (ID='f4c38f7d-b64e-41c0-876c-d87e85fe2bae', type ='GetTraceStatus') is being kept alive while the request is processed.
    2022-03-08T13:17:14.5973837Z Verbose: The HTTP connection (ID='f4c38f7d-b64e-41c0-876c-d87e85fe2bae', type ='GetTraceStatus') is being kept alive while the request is processed.
    2022-03-08T13:17:54.1945450Z ##[error]Error: Error Code: ERROR_CONNECTION_TERMINATED
    More Information: Web Deploy experienced a connection problem with the server and had to terminate the connection. Contact your server administrator if the problem persists. Learn more at: https://go.microsoft.com/fwlink/?LinkId=221672#ERROR_CONNECTION_TERMINATED.
    Error: Root element is missing.
    Error count: 1.

    2022-03-08T13:17:55.1331052Z Successfully added release annotation to the Application Insight : usspi-ain-dt-mcap-01
    2022-03-08T13:17:58.0597340Z Successfully updated deployment History at https://xxx-metrics-fun-deploy.scm.azurewebsites.net/api/deployments/974411646745475131
    2022-03-08T13:17:58.0605123Z App Service Application URL: http://xxx-metrics-fun-deploy.azurewebsites.net
    2022-03-08T13:17:58.0710966Z ##[section]Finishing: Deploy to xxx-metrics-fun slot (with -verbose)