question

SunilCATO-2514 avatar image
0 Votes"
SunilCATO-2514 asked GabrielSoudryStudentatCentraleSupe-3951 commented

Failed to initialize Integration Runtime (Self-hosted) node components

We recently configured our datafactory to collect data from 60 different branches of our business. We created 60 different integration runtime via the Data Factory UI and then installed IR on a local computer in each branch and copied over the registration key.

59 of these worked perfectly. On one single case though I get an error that says

"The integration Runtime (Self-hosted) node has encountered an error during registration.
Failed to initialize Integration Runtime (Self-hosted) node components. You may check event log for more detail."

I'm trying to debug this issue quickly if possible. I'm hoping it's something simple like a firewall configuration, I could not find this specific error anywhere on the internet. Has anyone seen this before?

The registration key is recognized on the runtime in question but then when I click "Finish" it just stays stuck in "initializing...." for a long time and then throws the error.
10211-accept1.png
10212-error1.png



I tried uploading the xml log but for some reason I am unable to upload a small 80kb file. I went through the log myself and found that there's a few errors that occur when I face the above issue:

 Authentication Key is not assigned yet.
 ---------------
 Pre-validate token failed. Error code: AuthSasEffectiveInFuture 
 ---------------
 DEBUG: Pre-validate token failed. Error code: AuthSasEffectiveInFuture Activity ID: c8d641ad-d4e0-4127-9d14-0f62e33bd98d ErrorCode : AuthSasEffectiveInFuture, InnerException : null, ServiceTrace : Exception detail: Microsoft.DataTransfer.AuthService.Models.KmsApiException: ErrorCode : AuthSasEffectiveInFuture, InnerException : null, ServiceTrace : at Microsoft.DataTransfer.AuthService.Models.ClientTokenValidator.Validate(String token) at Microsoft.DataProxy.MidgardCore.HighAvailability.TokenValidator.Validate(String token) at Microsoft.DataProxy.MidgardCore.StartupStage.StartWorkerProcessStage.PreValidateToken(ConfigVariableManager configManager)
 ---------------
 DEBUG: Failed in Retry Stage. ActivityID: c8d641ad-d4e0-4127-9d14-0f62e33bd98d Activity ID: c8d641ad-d4e0-4127-9d14-0f62e33bd98d ErrorCode : AuthSasEffectiveInFuture, InnerException : null, ServiceTrace : Exception detail: Microsoft.DataTransfer.AuthService.Models.KmsApiException: ErrorCode : AuthSasEffectiveInFuture, InnerException : null, ServiceTrace : at Microsoft.DataTransfer.AuthService.Models.ClientTokenValidator.Validate(String token) at Microsoft.DataProxy.MidgardCore.HighAvailability.TokenValidator.Validate(String token) at Microsoft.DataProxy.MidgardCore.StartupStage.StartWorkerProcessStage.PreValidateToken(ConfigVariableManager configManager) at Microsoft.DataProxy.MidgardCore.StartupStage.StartWorkerProcessStage.Run(ConfigVariableManager configManager, CancellationToken ct) at Microsoft.DataProxy.MidgardCore.StartupStage.StageRetry.<Run>d__18.MoveNext()
    
 -----------
    
 Create cloud endpoint listener failed. TransferMode: Streamed, 
    
 -----------
    
 DEBUG: Create cloud endpoint listener failed. TransferMode: Streamed, Endpoint Information: e35c1065-eac7-4295-9cd4-a0a8b86919d6. Activity ID: 56b00c61-3aad-4178-88b4-42bfc37dcd98 Generic: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' Exception detail: Microsoft.ServiceBus.AuthorizationFailedException: Generic: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' ---> Microsoft.ServiceBus.Messaging.Amqp.AmqpException: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.EndOpen(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.DuplexAmqpLink.OnOperationComplete(AmqpObject link, IAsyncResult result, Boolean isOpen) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.EndOpen(IAsyncResult result) at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.AmqpRelay.ConnectTask.<Start>d__4.MoveNext() --- End of inner exception stack trace --- at Microsoft.ServiceBus.AmqpRelay.ConnectTask.OnException(Exception exception) at Microsoft.ServiceBus.AmqpRelay.ConnectTask.<Start>d__4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.AmqpRelay.<OpenAsync>d__70.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.AmqpRelayedConnection.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.GetOrCreateConnection(Uri via, TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.Connect(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayTcpClient.EnsureConnected(TimeSpan timeout, Boolean isRetry) at Microsoft.ServiceBus.RelayedOnewayTcpClient.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.RefcountedCommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayChannelListener.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.WebSocketRelayedConnectionListener.Open(TimeSpan timeout) at Microsoft.ServiceBus.Channels.BufferedConnectionListener.Open(TimeSpan timeout) at Microsoft.ServiceBus.SocketConnectionTransportManager.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.TransportManager.Open(TimeSpan timeout, TransportChannelListener channelListener) at Microsoft.ServiceBus.Channels.TransportManagerContainer.Open(TimeSpan timeout, SelectTransportManagersCallback selectTransportManagerCallback) at Microsoft.ServiceBus.Channels.TransportChannelListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.ConnectionOrientedTransportChannelListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.SocketConnectionChannelListener`2.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.DataProxy.MidgardCore.ServiceBusBoundHostManager.CreateServiceBusListner(ServiceBusDataSU12CY13 sbData, IEnumerable`1 impContracts, TransferMode transferMode, ICollection`1 exList)
    
    
    
 -------------------
    
 DEBUG: Unable to create cloud enpdoint listener. Error message: One or more errors occurred.. Activity ID: 56b00c61-3aad-4178-88b4-42bfc37dcd98 Generic: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' Generic: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:59Z' ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:59Z' Exception detail: System.AggregateException: One or more errors occurred. ---> Microsoft.ServiceBus.AuthorizationFailedException: Generic: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' ---> Microsoft.ServiceBus.Messaging.Amqp.AmqpException: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.EndOpen(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.DuplexAmqpLink.OnOperationComplete(AmqpObject link, IAsyncResult result, Boolean isOpen) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.EndOpen(IAsyncResult result) at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.AmqpRelay.ConnectTask.<Start>d__4.MoveNext() --- End of inner exception stack trace --- at Microsoft.ServiceBus.AmqpRelay.ConnectTask.OnException(Exception exception) at Microsoft.ServiceBus.AmqpRelay.ConnectTask.<Start>d__4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.AmqpRelay.<OpenAsync>d__70.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.AmqpRelayedConnection.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.GetOrCreateConnection(Uri via, TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.Connect(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayTcpClient.EnsureConnected(TimeSpan timeout, Boolean isRetry) at Microsoft.ServiceBus.RelayedOnewayTcpClient.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.RefcountedCommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayChannelListener.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.WebSocketRelayedConnectionListener.Open(TimeSpan timeout) at Microsoft.ServiceBus.Channels.BufferedConnectionListener.Open(TimeSpan timeout) at Microsoft.ServiceBus.SocketConnectionTransportManager.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.TransportManager.Open(TimeSpan timeout, TransportChannelListener channelListener) at Microsoft.ServiceBus.Channels.TransportManagerContainer.Open(TimeSpan timeout, SelectTransportManagersCallback selectTransportManagerCallback) at Microsoft.ServiceBus.Channels.TransportChannelListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.ConnectionOrientedTransportChannelListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.SocketConnectionChannelListener`2.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.DataProxy.MidgardCore.ServiceBusBoundHostManager.CreateServiceBusListner(ServiceBusDataSU12CY13 sbData, IEnumerable`1 impContracts, TransferMode transferMode, ICollection`1 exList) --- End of inner exception stack trace --- at Microsoft.DataProxy.MidgardCore.ServiceBusBoundHostManager.SpecifyServiceBusAddressList(ReadOnlyCollection`1 sbDataList) at Microsoft.DataProxy.MidgardCore.ConnectToServiceBusStage.<Run>d__4.MoveNext() ---> (Inner Exception #0) Microsoft.ServiceBus.AuthorizationFailedException: Generic: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' ---> Microsoft.ServiceBus.Messaging.Amqp.AmqpException: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:57Z' at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.EndOpen(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.DuplexAmqpLink.OnOperationComplete(AmqpObject link, IAsyncResult result, Boolean isOpen) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.EndOpen(IAsyncResult result) at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.AmqpRelay.ConnectTask.<Start>d__4.MoveNext() --- End of inner exception stack trace --- at Microsoft.ServiceBus.AmqpRelay.ConnectTask.OnException(Exception exception) at Microsoft.ServiceBus.AmqpRelay.ConnectTask.<Start>d__4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.AmqpRelay.<OpenAsync>d__70.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.AmqpRelayedConnection.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.GetOrCreateConnection(Uri via, TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.Connect(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayTcpClient.EnsureConnected(TimeSpan timeout, Boolean isRetry) at Microsoft.ServiceBus.RelayedOnewayTcpClient.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.RefcountedCommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayChannelListener.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.WebSocketRelayedConnectionListener.Open(TimeSpan timeout) at Microsoft.ServiceBus.Channels.BufferedConnectionListener.Open(TimeSpan timeout) at Microsoft.ServiceBus.SocketConnectionTransportManager.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.TransportManager.Open(TimeSpan timeout, TransportChannelListener channelListener) at Microsoft.ServiceBus.Channels.TransportManagerContainer.Open(TimeSpan timeout, SelectTransportManagersCallback selectTransportManagerCallback) at Microsoft.ServiceBus.Channels.TransportChannelListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.ConnectionOrientedTransportChannelListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.SocketConnectionChannelListener`2.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.DataProxy.MidgardCore.ServiceBusBoundHostManager.CreateServiceBusListner(ServiceBusDataSU12CY13 sbData, IEnumerable`1 impContracts, TransferMode transferMode, ICollection`1 exList)<--- ---> (Inner Exception #1) Microsoft.ServiceBus.AuthorizationFailedException: Generic: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:59Z' ---> Microsoft.ServiceBus.Messaging.Amqp.AmqpException: ExpiredToken: The token is expired. Expiration time: '2020-06-15 21:46:59Z' at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.EndOpen(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.DuplexAmqpLink.OnOperationComplete(AmqpObject link, IAsyncResult result, Boolean isOpen) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.EndOpen(IAsyncResult result) at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.AmqpRelay.ConnectTask.<Start>d__4.MoveNext() --- End of inner exception stack trace --- at Microsoft.ServiceBus.AmqpRelay.ConnectTask.OnException(Exception exception) at Microsoft.ServiceBus.AmqpRelay.ConnectTask.<Start>d__4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.AmqpRelay.<OpenAsync>d__70.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.AmqpRelayedConnection.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.GetOrCreateConnection(Uri via, TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.RelayedOnewayAmqpListenerClient.Connect(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayTcpClient.EnsureConnected(TimeSpan timeout, Boolean isRetry) at Microsoft.ServiceBus.RelayedOnewayTcpClient.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.RefcountedCommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.RelayedOnewayChannelListener.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.WebSocketRelayedConnectionListener.Open(TimeSpan timeout) at Microsoft.ServiceBus.Channels.BufferedConnectionListener.Open(TimeSpan timeout) at Microsoft.ServiceBus.SocketConnectionTransportManager.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.TransportManager.Open(TimeSpan timeout, TransportChannelListener channelListener) at Microsoft.ServiceBus.Channels.TransportManagerContainer.Open(TimeSpan timeout, SelectTransportManagersCallback selectTransportManagerCallback) at Microsoft.ServiceBus.Channels.TransportChannelListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.ConnectionOrientedTransportChannelListener.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.SocketConnectionChannelListener`2.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.DataProxy.MidgardCore.ServiceBusBoundHostManager.CreateServiceBusListner(ServiceBusDataSU12CY13 sbData, IEnumerable`1 impContracts, TransferMode transferMode, ICollection`1 exList)<--- 


azure-data-factory
accept1.png (135.4 KiB)
error1.png (291.0 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

SunilCATO-2514 avatar image
2 Votes"
SunilCATO-2514 answered GabrielSoudryStudentatCentraleSupe-3951 commented

Hi again!

I fixed the issue, I tried making sense of those logs and "AuthSasEffectiveInFuture" this caught my eye. I followed a hunch and checked timezone settings on that computer and realized the branch had been set to an incorrect timezone (they are UTC -0700 but they were set to UTC -0500) so to correct that the branch manager and manually edited the time to 2 hours back.

This causes the authentication between Azure and the node to break.

I feel so bad to have wasted my entire day and your time on such a silly issue, had the error code from Azure been "Check your system time" life would have been a little easier.

Thanks so much for trying to help!

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @SunilCATO-2514 ,

Glad to know that you were able to figure it out and unblocked. Thank you for sharing the resolution details as it will be helpful for other community members who experience similar issue.

Please feel free to click "Accept Answer", so that others can find the answer quickly.



Thank you.

1 Vote 1 ·

Hi Sunil,

Can you please elaborate more on the resolution, even i am also getting the same error. If possible please provide me some screen shots where it needs to be modified.

Thanks in Advance!!

0 Votes 0 ·

I love you so much <3 Thank you it's working, my dual boot linux causing the same trouble with timezone

0 Votes 0 ·
KranthiPakala-MSFT avatar image
1 Vote"
KranthiPakala-MSFT answered KranthiPakala-MSFT converted comment to answer

Hi @SunilCATO-2514 ,

Welcome to Microsoft Q&A and thanks for reaching out.

Are you seeing this error consistently or a transient one? I would also recommend you to please check event logs for more information about the root cause of the issue. If possible please do attach the log file.

10121-items-1.png



Also, please make sure Remote access to intranet option is enabled on the node

10111-shir.png






I see that you have created 60 SHIR in your ADF. Ss there any specific reason you opted out to created multiple nodes for each Integration Runtime?
You can associate a self-hosted integration runtime with multiple on-premises machines or virtual machines in Azure. These machines are called nodes. You can have up to four nodes associated with a Single self-hosted integration runtime. Using node you can reduce your Integration Runtime count from 60 IR's/4 nodes = 15 IR's which will also help you maintain a High availability and scalability (to achieve higher performance). For more info please refer to :

Hope this helps.



Thank you

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members.






items-1.png (158.8 KiB)
shir.png (120.0 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you so much for attempting to help me.

I've updated my question to provide some screenshots and log info, I tried attaching XML logs but the website just won't let me upload them.



The reason we went with 60 different IRs was because we were trying to collect data from 60 different isolated networks with a combination of 6 different DB systems in place. We wanted to isolate every branch as much as possible. This is our first major Azure implementation, we're moving from AWS to Azure since it's easier to connect to the Power Platform where our BI resides. I will look at the links you provided about scalability & availability to see if that applies to us and tweak our plan accordingly.

0 Votes 0 ·

Hi @SunilCATO-2514 ,


Thanks for your response and additional details. While I continue to investigate on this, could you please try regenerating a new authentication key in ADF UI and try registering IR node with that key? Please let us know how it goes.




Thank you


0 Votes 0 ·