Considerations for NetTcpBinding/NetNamedPipeBinding you may not be aware

 

NetTcpBinding is a strange beast and chances are you will encounter several problems in production you never experienced in development or staging phases. The information you will see here will be either fragmented or hidden in the fine print throughout  MSDN documentation.

Considerations about net.tcp binding

 

Port Sharing

Net.tcp services using shared port needs to run under an account that is either SYSTEM, NETWORK or IIS_IUSR or be part of local Administrators. If part of local administrators the application hosting the service must run in elevated privileges.

To add new accounts that may use port sharing edit <allowAccounts> from SMSvcHost.exe.config (normally at C:\Windows\Microsoft.NET\Framework\v4.0.30319 for .NET 4.0+ or C:\Windows\Microsoft.NET\Framework\v3.0\Windows Communication Foundation\). Notice that the users/groups must be in SID format. You may use PowerShell to translate a user account into SID: https://technet.microsoft.com/en-us/library/ff730940.aspx

$objUser = New-Object System.Security.Principal.NTAccount("fabrikam", "kenmyer") $strSID = $obj
User.Translate([System.Security.Principal.SecurityIdentifier]) $strSID.Value

If you perform an in-place update from Windows 2008 R2 to Windows 2012 R21, net.tcp sharing services may stop working with a error similar to this one because there is as mismatch between .NET Framework and WCF:

Log Name: System
Source: SMSvcHost 4.0.0.0
Date: 10/22/2015 11:49:42 AM
Event ID: 7
Task Category: Sharing Service
Level: Error
Keywords: Classic
User: LOCAL SERVICE
Computer: SERVER1.contoso.local
Description:
A request to start the service failed. Error Code: System.TypeLoadException: Could not load type 'System.Runtime.Diagnostics.ITraceSourceStringProvider' from assembly 'System.ServiceModel.Internals, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'.
at System.ServiceModel.Channels.BinaryMessageEncoderFactory..ctor(MessageVersion messageVersion, Int32 maxReadPoolSize, Int32 maxWritePoolSize, Int32 maxSessionSize, XmlDictionaryReaderQuotas readerQuotas, Int64 maxReceivedMessageSize, BinaryVersion version, CompressionFormat compressionFormat)
at System.ServiceModel.Channels.BinaryMessageEncodingBindingElement.CreateMessageEncoderFactory()
at System.ServiceModel.Channels.ConnectionOrientedTransportChannelListener..ctor(ConnectionOrientedTransportBindingElement bindingElement, BindingContext context)
at System.ServiceModel.Channels.NamedPipeChannelListener..ctor(NamedPipeTransportBindingElement bindingElement, BindingContext context)
at System.ServiceModel.Channels.NamedPipeTransportBindingElement.BuildChannelListener[TChannel](BindingContext context)
at System.ServiceModel.Channels.Binding.BuildChannelListener[TChannel](Uri listenUriBaseAddress, String listenUriRelativeAddress, ListenUriMode listenUriMode, BindingParameterCollection parameters)
at System.ServiceModel.Description.DispatcherBuilder.MaybeCreateListener(Boolean actuallyCreate, Type[] supportedChannels, Binding binding, BindingParameterCollection parameters, Uri listenUriBaseAddress, String listenUriRelativeAddress, ListenUriMode listenUriMode, ServiceThrottle throttle, IChannelListener& result, Boolean supportContextSession)
at System.ServiceModel.Description.DispatcherBuilder.BuildChannelListener(StuffPerListenUriInfo stuff, ServiceHostBase serviceHost, Uri listenUri, ListenUriMode listenUriMode, Boolean supportContextSession, IChannelListener& result)
at System.ServiceModel.Description.DispatcherBuilder.InitializeServiceHost(ServiceDescription description, ServiceHostBase serviceHost)
at System.ServiceModel.ServiceHostBase.InitializeRuntime()
at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Activation.SharingService.StartControlService()
at System.ServiceModel.Activation.SharingService.Start()
at System.ServiceModel.Activation.TcpPortSharing.OnStart(String[] args) Process Name: SMSvcHost Process ID: 2208

To resolve this issue, install at least .NET 4.5.2 to bring .NET and WCF to the same page (http://www.microsoft.com/en-us/download/details.aspx?id=42642)

Net.tcp/net.pipe channels are sessionful and may leak sessions if configured differently

This problem is also common when using NetPipeBindind (net.pipe). Net.tcp channel is sessionful (i.e. it will establish a session between client and service) and if you configure instance mode differently (e.g. PerCall) it will still create sessions. If a misbehaving client is not closing the connection properly or reliable sessions is enabled (see reliable session later), the service will hold a zombie session for the time the receive time out is set (by default 10 minutes). Clients should instantiate a service proxy (or use the channel factory to get a proxy) so the operation it should do and close or abort the inner channel after that. Reuse a proxy for the lifetime of the client application is always a bad idea. If using load balance, never hold a proxy between calls unless you are indeed using session in your service. This is the snipped of the correct way to invoke a WCF service from a client application:

  
             var client = new Service1Client();
             try
             {
  
                 var response = client.WhoAmI(); // Call the service
  
                 Console.WriteLine("Response = {0}", response);
                 client.Close(); // Close afterwards
             }
             catch (Exception ex)
             {
                 Console.WriteLine("\n\nError: {0}", ex.Message);
                 var inner = ex.InnerException;
                 while (inner != null)
                 {
                     Console.WriteLine("Inner: {0}", inner.Message);
                     inner = inner.InnerException;
                 }
                 if (client.State != CommunicationState.Closed)
                 {
                     client.Abort(); // If service is not closed at this point, abort
                 }
             }

The common exceptions when WCF sessions are leaking is occasionally receiving are timeout error and communication exceptions on the client side that will only subside when the service is restarted or the 10 minutes receive timeout is elapsed without any new connection. Exceptions in the client side looks like these ones:

System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue.

System.TimeoutException: This request operation sent to net.tcp://contoso:9090/servicemodelsamples/nettcp did not receive a reply within the configured timeout

However, there will be times you cannot control all the client applications using the service, so if this is the case, make sure you set the receiveTimeout property of NetTcpBinding to a short time, as 30 seconds for instance, so the zombies sessions will be dead after 30 seconds, instead of the default 10 minutes.  Don’t worry because this setting is mostly a misname for “session time out” and it is not direct to the time a request may take to be received. This is a sample config settings to make receiveTimout 30 seconds:

 <bindings>
   <netTcpBinding>
     <binding 
              receiveTimeout="00:00:30">
       <security mode="Transport">
         <transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />
       </security>
     </binding>
   </netTcpBinding>
 </bindings>

If you have .NET 4.5+, it is easy to troubleshoot the issue using ETL traces. When you see clients stop responding, capture ETL/ETW traces on the service side (server) using a batch file as below (I like to call it get-traces.bat):

@echo off ECHO These commands will enable tracing: @echo on
logman create trace "redist_mspartners" -ow -o %temp%\redist_mspartners.etl -p "Microsoft-Windows-Application Server-Applications" 0xffffffffffffffff 0xff -nb 16 16 -bs 1024 -mode Circular -f bincirc -max 4096 -ets logman update trace "redist_mspartners" -p {7F3FE630-462B-47C5-AB07-67CA84934ABD} 0xffffffffffffffff 0xff -ets
@echo off echo ECHO Reproduce your issue and enter any key to stop tracing @echo on pause logman stop "redist_mspartners" -ets
@echo off echo Tracing has been captured and saved successfully at %temp%\redist_mspartners.etl pause

Run only when the problem is happening and for little time as this file grows fast and is circular. Open the resulting trace with Message Analyzer and look for “Concurrent sessions ratio” in the summary. In the example below there are 30/30 possible concurrent connections. While this was happening the client was throwing

image

If you have previous version of .NET you may consider using the custom service behavior I discuss here to monitor your service when necessary: http://blogs.msdn.com/b/rodneyviana/archive/2014/10/08/verifying-current-calls-and-sessions-during-runtime.aspx

Identifying session leak for Advanced Users (also valid for net.pipe)

If you are comfortable analyzing dump files in WinDBG, you may use NetExt to verify the runtime throttling counters as well as look for mismatches between the service session mode and the channel service mode. To learn how to have NetExt installed, see this: Getting started with NetExt

Open the dump file in WinDBG. Load netext and index the heap:

.load netext !windex

List all services

!wservice

 0:000>; .load netext
 netext version 2.1.0.5000 Oct  5 2015
 License and usage can be seen here: !whelp license
 Check Latest version: !wupdate
 (...)
  
 0:000>; !windex
 Starting indexing at 19:52:07 PM
 Indexing finished at 19:52:09 PM
 7,916,949 Bytes in 43,512 Objects
 Index took 00:00:01
 0:000>; !wservice
 Address             State   (...)  Calls/Max   Sessions/Max    ConfigName,.NET Type
 00000016d97fd998    Opened  (...)  0n31/0n100     0n30/0n30    "VanillaService.DataContractSample",VanillaService.DataContractSample
  
 1 ServiceHost object(s) found

 

Notice that you see a very similar number of calls and sessions and the service should increase only one of them. If there are calls and sessions instances it is indication that there is a link. Clicking on the object link, it will show the details.

 0:000>; !wservice 00000016d97fd998
  
 Service Info
 ================================
 Address            : 00000016D97FD998
 Configuration Name : VanillaService.DataContractSample
 State              : Opened
 (...)
 Calls/Max Calls    : 0n31/0n100
 Sessions/Max       : 0n30/0n30 <;-- Max session reached
 (...)
 Session Mode       : False  <;-- Service level session mode is FALSE
  
 Service Behaviors
 ================================
 Concurrency Mode   : Multiple
 Instance Mode      : PerCall <;-- Instancing is not PerSession
 Add Error in Faults: false
 (...)
  
 Service Base Addresses
 ================================
 net.tcp://localhost:9090/servicemodelsamples/
  
 Channels
 ================================
 Address            : 00000016D9889AA8
 Listener URI       : net.tcp://localhost:9090/servicemodelsamples/nettcp
 Binding Name       : http://tempuri.org/:NetTcpBinding
 Aborted            : No
 State              : Opened
 Transaction Type   : No transaction
 Listener State     : Opened
 Timeout settings   : Open [00:01:00] Close [00:01:00] Receive: [00:10:00] Send: [00:01:00]
 Server Capabilities: SupportsServerAuth [Yes] SupportsClientAuth [Yes] SupportsClientWinIdent [Yes]
 Request Prot Level : EncryptAndSign
 Response Prot Level: EncryptAndSign
 Events Raised      : No Event raised
 Handles Called     : OnOpeningHandle OnOpenedHandle 
 Session Mode       : True <;-- Tcp Channel is Session oriented by design
 (...)
  
  
 Endpoints
  
 ================================
  
 Address            : 00000016D98770F8
 URI                : net.tcp://localhost:9090/servicemodelsamples/nettcp
 Is Anonymous       : False
 Configuration Name : VanillaService.IDataContractSample
 Type Name          : VanillaService.IDataContractSample
 Listening Mode     : Explicit
 Class Definition   : 00007ffebf71c160 VanillaService.IDataContractSample
 Behaviors          : 00000016d98773c8
 Binding            : 00000016d9871b18
 (...)

The details are very important because we will compare the service session mode with the net.tcp channel session mode (which is by design PerSession). The service session mode is defined by instancing context mode. By default it is PerCall. To change it you should declare it as an attribute of the service class. See this: https://msdn.microsoft.com/en-us/library/system.servicemodel.servicebehaviorattribute.instancecontextmode(v=vs.110).aspx

Verifying further down in the net.tcp channel session mode setting, it is defined as true. This is by channel design: it is sessionful. There is no way to change this. So, since there is a mismatch between the service session mode (declared by the developer) and the channel session mode (defined by .NET WCF), there will be a leak. The client in this scenario is closing the proxy properly after making the request.

For this situation the only remedy is to decrease the receive timeout setting in the net.tcp binding configuration as mentioned previously. So, if you are using net.tcp and do not mean to leverage sessions, set receive timeout value to 15 or 30 seconds. If you don’t know if your application requires session is because it does not. If you control the WCF Service server side, use PerSession instancing mode (and still keep receive timeout low).

Bottleneck on client side

By default, a net.tcp or net.pipe client will be limited to 10 concurrent outbound connections. It is a good number if the client is a standalone application. If the WCF client is a Web Application, the concurrent outbound connections limited to 10 may become a bottleneck. You may increase that value by changing maxConnections attribute in net.tcp binding in the client side configuration. This set has no effect on the server side. See: https://msdn.microsoft.com/en-us/library/ms731343(v=vs.110).aspx

If you are an advanced user, use this commands in a dump file to see the outbound calls (requires NetExt):

!wfrom -implement System.ServiceModel.Channels.TransportOutputChannel where((!$implement("*preamble*"))&&($enumname(state)!="Closed")) $a("Address",$addr()),$a("Url",to.uri.m_String),$a("State",$enumname(state)),$a("Open",channelManager.connectionPool.openCount),$a("Max",channelManager.connectionPool.maxCount)

 0:000>; !wfrom -implement System.ServiceModel.Channels.TransportOutputChannel where((!$implement("*preamble*"))&&($enumname(state)!="Closed")) $a("Address",$addr()),$a("Url",to.uri.m_String),$a("State",$enumname(state)),$a("Open",channelManager.connectionPool.openCount),$a("Max",channelManager.connectionPool.maxCount)
 Address: 0000000002C29590
 Url: net.tcp://localhost:9090/servicemodelsamples/nettcp
 State: Opened
 Open: 0n30
 Max: 0n30
 Address: 0000000002C98238
 Url: net.tcp://localhost:9090/servicemodelsamples/nettcp
 State: Opened
 Open: 0n30
 Max: 0n30
 Address: 0000000002C9DCC8
 Url: net.tcp://localhost:9090/servicemodelsamples/nettcp
 State: Opened
 Open: 0n30
 Max: 0n30
 (...)
  
 30 Object(s) listed
 36 Object(s) skipped by filter
  

 

Reliable Session

Reliable Session is the WCF implementation of Oasis WS-RealiableMessaging (RM). Reliable session is an overkill for net.tcp binding because net.tcp already implements everything implemented by reliable sessions except for the reconnection. However, reliable session uses a different and complex code path and at a high performance cost. It really brings no added benefit but on the contrary, it opens a myriad of potential issues. I heard this from WCF product team when working a case where reliable session was being used.

The problem is having channel leak because of session mode mismatch (as explained before) with the extras hassle of reliable messaging. So, please use reconnection logic instead of reliable session if you do need reliable sessions.

Below is the advanced troubleshoot technique of searching for reliable session in a dump file and checking if the inner channel can be reopened (not aborted). This is THE NUMBER ONE source of problem with performance when using WCF with net.tcp or net.pipe. NetExt query:

!wfrom -type System.ServiceModel.Channels.ReliableChannelBinder?ChannelSynchronizer* where (($contains($typename(),"_ChannelSynchronizer<"))&&($enumname(state)!="Closed")) $a("Address",$addr()), $a("State",$enumname(state)), $a("Inner Channel State",$enumname(currentChannel.state)),$a("Fault Mode",$enumname(faultMode)), $a("Is Aborted?",currentChannel.aborted)

 0:000>; !wfrom -type System.ServiceModel.Channels.ReliableChannelBinder?ChannelSynchronizer* where (($contains($typename(),"_ChannelSynchronizer<"))&&($enumname(state)!="Closed")) $a("Address",$addr()), $a("State",$enumname(state)), $a("Inner Channel State",$enumname(currentChannel.state)),$a("Fault Mode",$enumname(faultMode)), $a("Is Aborted?",currentChannel.aborted)
 Address: 01102AF4
 State: ChannelOpening
 Inner Channel State: Closed
 Fault Mode: IfNotSecuritySession
 Is Aborted?: 1 <;--- Leak: Inner channel aborted, Reliable Session will timeout with receiveTimeout
 Address: 01123844
 State: ChannelOpening
 Inner Channel State: Closed
 Fault Mode: IfNotSecuritySession
 Is Aborted?: 1
 Address: 01138E74
 State: ChannelOpening
 Inner Channel State: Closed
 Fault Mode: IfNotSecuritySession
 Is Aborted?: 1
 Address: 01139F30
 State: ChannelOpening
 Inner Channel State: Closed
 Fault Mode: IfNotSecuritySession
 Is Aborted?: 1
 (...)
 Address: 18262A8C
 State: ChannelOpened
 Inner Channel State: Opened
 Fault Mode: IfNotSecuritySession
 Is Aborted?: 0
 Address: 182EFD90
 State: ChannelOpened
 Inner Channel State: Opened
 Fault Mode: IfNotSecuritySession
 Is Aborted?: 0
 Address: 184818C4
 State: ChannelOpened
 Inner Channel State: Opened
 Fault Mode: IfNotSecuritySession
 Is Aborted?: 0
 Address: 185EBB28
 State: ChannelOpened
 Inner Channel State: Opened
 Fault Mode: IfNotSecuritySession
 Is Aborted?: 0
 Address: 18615B84
 State: ChannelOpened
 Inner Channel State: Opened
 Fault Mode: IfNotSecuritySession
 Is Aborted?: 0
  
 100 Object(s) listed
 208 Object(s) skipped by filter

 

Load balancing net.tcp

Preventing load balance idle timeout is the only justified reason to use reliable session with net.tcp if you are really leveraging sessions. But, again, the same result can be achieved without reliable session if WCF Service receive timeout matches load balancer session idle timeout. Don’t feel encouraged to follow this path.

The use of load balancer is to provide scalability, so using sessions will require the configuration of stick sessions in the load balancer, most of the modern load balancers offers this type of sticky session that is different from HTTP’s. This sticky sessions will always match a client connection to a server. This defeats the purpose of scalability.