question

SamkoYoon-2920 avatar image
0 Votes"
SamkoYoon-2920 asked MikeUrnun commented

Azure function container fail to start in AKS

I've deployed azure python function image(mcr.microsoft.com/azure-functions/python:3.0-python3.8) to AKS. It has one blob trigger function and it works fine locally but when it's deployed to AKS, it's throwing below error.

 info: Host.Startup[314]
       Loading functions metadata
 info: Host.Startup[315]
       1 functions loaded
 info: Host.Triggers.Warmup[0]
       Initializing Warmup Extension.
 info: Host.Startup[503]
       Initializing Host. OperationId: '96345dde-4fe4-44bc-9687-4f92a13abe67'.
 info: Host.Startup[504]
       Host initialization: ConsecutiveErrors=7, StartupCount=8, OperationId=96345dde-4fe4-44bc-9687-4f92a13abe67
 info: Microsoft.Azure.WebJobs.Hosting.OptionsLoggingService[0]
       LoggerFilterOptions
       {
         "MinLevel": "None",
         "Rules": [
           {
             "ProviderName": null,
             "CategoryName": null,
             "LogLevel": null,
             "Filter": "<AddFilter>b__0"
           },
           {
             "ProviderName": "Microsoft.Azure.WebJobs.Script.WebHost.Diagnostics.SystemLoggerProvider",
             "CategoryName": null,
             "LogLevel": "None",
             "Filter": null
           },
           {
             "ProviderName": "Microsoft.Azure.WebJobs.Script.WebHost.Diagnostics.SystemLoggerProvider",
             "CategoryName": null,
             "LogLevel": null,
             "Filter": "<AddFilter>b__0"
           }
         ]
       }
 info: Microsoft.Azure.WebJobs.Hosting.OptionsLoggingService[0]
       FunctionResultAggregatorOptions
       {
         "BatchSize": 1000,
         "FlushTimeout": "00:00:30",
         "IsEnabled": true
       }
 info: Microsoft.Azure.WebJobs.Hosting.OptionsLoggingService[0]
       SingletonOptions
       {
         "LockPeriod": "00:00:15",
         "ListenerLockPeriod": "00:01:00",
         "LockAcquisitionTimeout": "10675199.02:48:05.4775807",
         "LockAcquisitionPollingInterval": "00:00:05",
         "ListenerLockRecoveryPollingInterval": "00:01:00"
       }
 info: Microsoft.Azure.WebJobs.Hosting.OptionsLoggingService[0]
       QueuesOptions
       {
         "BatchSize": 16,
         "NewBatchThreshold": 8,
         "MaxPollingInterval": "00:01:00",
         "MaxDequeueCount": 5,
         "VisibilityTimeout": "00:00:00"
       }
 info: Microsoft.Azure.WebJobs.Hosting.OptionsLoggingService[0]
       BlobsOptions
       {
         "CentralizedPoisonQueue": false
       }
 info: Microsoft.Azure.WebJobs.Hosting.JobHostService[0]
       Starting JobHost
 info: Host.Startup[401]
       Starting Host (HostId=sandboxdeployment55b745968bxggd9, InstanceId=44dd9602-2325-49b0-9d79-a12cb0dbd034, Version=3.0.15725.0, ProcessId=1, AppDomainId=1, InDebugMode=False, InDiagnosticMode=False, FunctionsExtensionVersion=(null))
 info: Host.Startup[314]
       Loading functions metadata
 info: Host.Startup[0]
       Loading proxies metadata
 info: Host.Startup[0]
       Initializing Azure Function proxies
 fail: Host.Startup[515]
       A host error has occurred during startup operation '96345dde-4fe4-44bc-9687-4f92a13abe67'.
 Autofac.Core.DependencyResolutionException: An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = ProxyHandler (ReflectionActivator), Services = [Microsoft.Azure.AppService.Proxy.Runtime.ProxyHandler], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = ApiInspectorHandler (ReflectionActivator), Services = [Microsoft.Azure.AppService.Proxy.Runtime.Handlers.ApiInspectorHandler], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = DebugLogStorageService (DelegateActivator), Services = [Microsoft.Azure.AppService.Proxy.Runtime.Logging.IDebugLogService], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = FileSystemStorage (DelegateActivator), Services = [DetailedTrace (Microsoft.Azure.AppService.Proxy.Common.Infra.IBlobStorage)], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> Read-only file system (See inner exception for details.) (See inner exception for details.) (See inner exception for details.) (See inner exception for details.)
  ---> Autofac.Core.DependencyResolutionException: An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = ApiInspectorHandler (ReflectionActivator), Services = [Microsoft.Azure.AppService.Proxy.Runtime.Handlers.ApiInspectorHandler], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = DebugLogStorageService (DelegateActivator), Services = [Microsoft.Azure.AppService.Proxy.Runtime.Logging.IDebugLogService], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = FileSystemStorage (DelegateActivator), Services = [DetailedTrace (Microsoft.Azure.AppService.Proxy.Common.Infra.IBlobStorage)], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> Read-only file system (See inner exception for details.) (See inner exception for details.) (See inner exception for details.)
  ---> Autofac.Core.DependencyResolutionException: An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = DebugLogStorageService (DelegateActivator), Services = [Microsoft.Azure.AppService.Proxy.Runtime.Logging.IDebugLogService], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = FileSystemStorage (DelegateActivator), Services = [DetailedTrace (Microsoft.Azure.AppService.Proxy.Common.Infra.IBlobStorage)], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> Read-only file system (See inner exception for details.) (See inner exception for details.)
  ---> Autofac.Core.DependencyResolutionException: An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = FileSystemStorage (DelegateActivator), Services = [DetailedTrace (Microsoft.Azure.AppService.Proxy.Common.Infra.IBlobStorage)], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> Read-only file system (See inner exception for details.)
  ---> System.IO.IOException: Read-only file system
    at System.IO.FileSystem.CreateDirectory(String fullPath)
    at System.IO.Directory.CreateDirectory(String path)
    at Microsoft.Azure.AppService.Proxy.Client.IoC.PersistenceModule.GetLogLocation(String type) in C:\source\src\Microsoft.Azure.AppService.Proxy.Client\IoC\PersistenceModule.cs:line 57
    at Microsoft.Azure.AppService.Proxy.Client.IoC.PersistenceModule.<Load>b__0_0(IComponentContext ctx) in C:\source\src\Microsoft.Azure.AppService.Proxy.Client\IoC\PersistenceModule.cs:line 25
    at Autofac.RegistrationExtensions.<>c__DisplayClass5_0`1.<Register>b__0(IComponentContext c, IEnumerable`1 p)
    at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)
    at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
    at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
    --- End of inner exception stack trace ---
    at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
    at Autofac.Core.Resolving.InstanceLookup.<Execute>b__5_0()
    at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator)
    at Autofac.Core.Resolving.InstanceLookup.Execute()
    at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
    at Autofac.Core.Resolving.InstanceLookup.ResolveComponent(IComponentRegistration registration, IEnumerable`1 parameters)
    at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
    at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
    at Autofac.ResolutionExtensions.ResolveNamed[TService](IComponentContext context, String serviceName, IEnumerable`1 parameters)
    at Autofac.ResolutionExtensions.ResolveNamed[TService](IComponentContext context, String serviceName)
    at Microsoft.Azure.AppService.Proxy.Client.IoC.DiagnosticsModule.<>c.<Load>b__0_0(IComponentContext c) in C:\source\src\Microsoft.Azure.AppService.Proxy.Client\IoC\DiagnosticsModule.cs:line 19
    at Autofac.RegistrationExtensions.<>c__DisplayClass5_0`1.<Register>b__0(IComponentContext c, IEnumerable`1 p)
    at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)
    at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
    at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
    --- End of inner exception stack trace ---
    at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
    at Autofac.Core.Resolving.InstanceLookup.<Execute>b__5_0()
    at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator)
    at Autofac.Core.Resolving.InstanceLookup.Execute()
    at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
    at Autofac.Core.Resolving.InstanceLookup.ResolveComponent(IComponentRegistration registration, IEnumerable`1 parameters)
    at Autofac.Core.Activators.Reflection.AutowiringParameter.<>c__DisplayClass0_0.<CanSupplyValue>b__0()
    at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
    at Autofac.Core.Activators.Reflection.ReflectionActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
    at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
    --- End of inner exception stack trace ---
    at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
    at Autofac.Core.Resolving.InstanceLookup.<Execute>b__5_0()
    at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator)
    at Autofac.Core.Resolving.InstanceLookup.Execute()
    at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
    at Autofac.Core.Resolving.InstanceLookup.ResolveComponent(IComponentRegistration registration, IEnumerable`1 parameters)
    at Autofac.Core.Activators.Reflection.AutowiringParameter.<>c__DisplayClass0_0.<CanSupplyValue>b__0()
    at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
    at Autofac.Core.Activators.Reflection.ReflectionActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
    at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
    --- End of inner exception stack trace ---
    at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
    at Autofac.Core.Resolving.InstanceLookup.<Execute>b__5_0()
    at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator)
    at Autofac.Core.Resolving.InstanceLookup.Execute()
    at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
    at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable`1 parameters)
    at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(IComponentRegistration registration, IEnumerable`1 parameters)
    at Autofac.Core.Container.ResolveComponent(IComponentRegistration registration, IEnumerable`1 parameters)
    at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
    at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
    at Autofac.ResolutionExtensions.Resolve[TService](IComponentContext context, IEnumerable`1 parameters)
    at Autofac.ResolutionExtensions.Resolve[TService](IComponentContext context)
    at Microsoft.Azure.AppService.Proxy.Client.ProxyClientFactory.Create(String proxiesJson, ILogger logger) in C:\source\src\Microsoft.Azure.AppService.Proxy.Client\ProxyClientFactory.cs:line 66
    at Microsoft.Azure.WebJobs.Script.ProxyFunctionProvider.LoadProxyMetadata(String proxiesJson, Dictionary`2 functionErrors, ILogger logger) in /src/azure-functions-host/src/WebJobs.Script/Host/ProxyFunctionProvider.cs:line 123
    at Microsoft.Azure.WebJobs.Script.ProxyFunctionProvider.ReadProxyMetadata(String scriptPath, ILogger logger, Dictionary`2 functionErrors) in /src/azure-functions-host/src/WebJobs.Script/Host/ProxyFunctionProvider.cs:line 110
    at Microsoft.Azure.WebJobs.Script.ProxyFunctionProvider.LoadFunctionMetadata() in /src/azure-functions-host/src/WebJobs.Script/Host/ProxyFunctionProvider.cs:line 83
    at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
    at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
    at System.Lazy`1.CreateValue()
    at Microsoft.Azure.WebJobs.Script.ProxyFunctionProvider.GetFunctionMetadataAsync() in /src/azure-functions-host/src/WebJobs.Script/Host/ProxyFunctionProvider.cs:line 56
    at Microsoft.Azure.WebJobs.Script.FunctionMetadataManager.AddMetadataFromCustomProviders(IEnumerable`1 functionProviders, List`1 functionMetadataList) in /src/azure-functions-host/src/WebJobs.Script/Host/FunctionMetadataManager.cs:line 202
    at Microsoft.Azure.WebJobs.Script.FunctionMetadataManager.LoadCustomProviderFunctions(List`1 functionMetadataList) in /src/azure-functions-host/src/WebJobs.Script/Host/FunctionMetadataManager.cs:line 195
    at Microsoft.Azure.WebJobs.Script.FunctionMetadataManager.LoadFunctionMetadata(Boolean forceRefresh, Boolean includeCustomProviders) in /src/azure-functions-host/src/WebJobs.Script/Host/FunctionMetadataManager.cs:line 148
    at Microsoft.Azure.WebJobs.Script.FunctionMetadataManager.GetFunctionMetadata(Boolean forceRefresh, Boolean applyAllowList, Boolean includeCustomProviders) in /src/azure-functions-host/src/WebJobs.Script/Host/FunctionMetadataManager.cs:line 83
    at Microsoft.Azure.WebJobs.Script.ScriptHost.GetFunctionsMetadata() in /src/azure-functions-host/src/WebJobs.Script/Host/ScriptHost.cs:line 346
    at Microsoft.Azure.WebJobs.Script.ScriptHost.InitializeAsync(CancellationToken cancellationToken) in /src/azure-functions-host/src/WebJobs.Script/Host/ScriptHost.cs:line 277
    at Microsoft.Azure.WebJobs.Script.ScriptHost.StartAsyncCore(CancellationToken cancellationToken) in /src/azure-functions-host/src/WebJobs.Script/Host/ScriptHost.cs:line 257
    at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
    at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService.UnsynchronizedStartHostAsync(ScriptHostStartupOperation activeOperation, Int32 attemptCount, JobHostStartupMode startupMode) in /src/azure-functions-host/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs:line 278
 info: Microsoft.Azure.WebJobs.Hosting.JobHostService[0]
       Stopping JobHost


I think this has to do with lacking write access to /home/site/wwwroot but not sure how to resolve this...

Below is my dockerfile

 # To enable ssh & remote debugging on app service change the base image to the one below
 # FROM mcr.microsoft.com/azure-functions/python:3.0-python3.8-appservice
 FROM mcr.microsoft.com/azure-functions/python:3.0-python3.8
    
 ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
     AzureFunctionsJobHost__Logging__Console__IsEnabled=true
    
 COPY requirements.txt /
    
 # Install pip dependencies and remove them after the pip install
 RUN apt-get update \
     && apt-get install -y --no-install-recommends gcc \
     && rm -rf /var/lib/apt/lists/* \
     && pip install -r /requirements.txt \
     && apt-get purge -y --auto-remove gcc
    
 # Adding company PKI Certificates to System Trusted Store
 COPY Certificates/der/*.crt /build/pki/
 COPY Certificates/pem/*.crt /build/pki/pem/
 RUN cp /build/pki/pem/*.crt /usr/local/share/ca-certificates/ && update-ca-certificates
    
 # Microsoft.AspNetCore.Server.Kestrel issue
 EXPOSE 8080
 ENV ASPNETCORE_URLS=http://*:8080
 ENV DOTNET_CLI_TELEMETRY_OPTOUT=1
 ENV COMPlus_EnableDiagnostics=0
    
 COPY . /home/site/wwwroot


Any guide would be appreciated!

azure-functions
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@SamkoYoon-2920 I'm unsure about the exact directory trying to be created here but I believe using strace might shed some light. Also, based on the error, could you confirm if you are running your containers as read-only.


0 Votes 0 ·
MikeUrnun avatar image MikeUrnun PramodValavala-MSFT ·

Hi @SamkoYoon-2920 - Were you able to resolve this issue?

0 Votes 0 ·

0 Answers