Secure a standalone cluster on Windows using X.509 certificates

This article describes how to secure the communication between the various nodes of your standalone Windows cluster, as well as how to authenticate clients connecting to this cluster, using X.509 certificates. This ensures that only authorized users can access the cluster, the deployed applications and perform management tasks. Certificate security should be enabled on the cluster when the cluster is created.

For more information on cluster security such as node-to-node security, client-to-node security, and role-based access control, see Cluster security scenarios.

Which certificates will you need?

To start with, download the standalone cluster package to one of the nodes in your cluster. In the downloaded package, you will find a ClusterConfig.X509.MultiMachine.json file. Open the file and review the section for security under the properties section:

"security": {
    "metadata": "The Credential type X509 indicates this is cluster is secured using X509 Certificates. The thumbprint format is - d5 ec 42 3b 79 cb e5 07 fd 83 59 3c 56 b9 d5 31 24 25 42 64.",
    "ClusterCredentialType": "X509",
    "ServerCredentialType": "X509",
    "CertificateInformation": {
        "ClusterCertificate": {
            "Thumbprint": "[Thumbprint]",
            "ThumbprintSecondary": "[Thumbprint]",
            "X509StoreName": "My"
        },        
        "ClusterCertificateCommonNames": {
            "CommonNames": [
            {
                "CertificateCommonName": "[CertificateCommonName]",
                "CertificateIssuerThumbprint": "[Thumbprint1,Thumbprint2,Thumbprint3,...]"
            }
            ],
            "X509StoreName": "My"
        },
        "ServerCertificate": {
            "Thumbprint": "[Thumbprint]",
            "ThumbprintSecondary": "[Thumbprint]",
            "X509StoreName": "My"
        },
        "ServerCertificateCommonNames": {
            "CommonNames": [
            {
                "CertificateCommonName": "[CertificateCommonName]",
                "CertificateIssuerThumbprint": "[Thumbprint1,Thumbprint2,Thumbprint3,...]"
            }
            ],
            "X509StoreName": "My"
        },
        "ClientCertificateThumbprints": [
            {
                "CertificateThumbprint": "[Thumbprint]",
                "IsAdmin": false
            },
            {
                "CertificateThumbprint": "[Thumbprint]",
                "IsAdmin": true
            }
        ],
        "ClientCertificateCommonNames": [
            {
                "CertificateCommonName": "[CertificateCommonName]",
                "CertificateIssuerThumbprint": "[Thumbprint]",
                "IsAdmin": true
            }
        ],
        "ReverseProxyCertificate": {
            "Thumbprint": "[Thumbprint]",
            "ThumbprintSecondary": "[Thumbprint]",
            "X509StoreName": "My"
        },
        "ReverseProxyCertificateCommonNames": {
            "CommonNames": [
                {
                "CertificateCommonName": "[CertificateCommonName]"
                }
            ],
            "X509StoreName": "My"
        }
    }
},

This section describes the certificates that you need for securing your standalone Windows cluster. If you are specifying a cluster certificate, set the value of ClusterCredentialType to X509. If you are specifying server certificate for outside connections, set the ServerCredentialType to X509. Although not mandatory, we recommend to have both these certificates for a properly secured cluster. If you set these values to X509 then you must also specify the corresponding certificates or Service Fabric will throw an exception. In some scenarios, you may only want to specify the ClientCertificateThumbprints or ReverseProxyCertificate. In those scenarios, you need not set ClusterCredentialType or ServerCredentialType to X509.

Note

A thumbprint is the primary identity of a certificate. Read How to retrieve thumbprint of a certificate to find out the thumbprint of the certificates that you create.

The following table lists the certificates that you will need on your cluster setup:

CertificateInformation Setting Description
ClusterCertificate Recommended for test environment. This certificate is required to secure the communication between the nodes on a cluster. You can use two different certificates, a primary and a secondary for upgrade. Set the thumbprint of the primary certificate in the Thumbprint section and that of the secondary in the ThumbprintSecondary variables.
ClusterCertificateCommonNames Recommended for production environment. This certificate is required to secure the communication between the nodes on a cluster. You can use one or two cluster certificate common names. The CertificateIssuerThumbprint corresponds to the thumbprint of the issuer of this certificate. You may specify multiple issuer thumbprints if more than 1 certificate with the same common name is being used.
ServerCertificate Recommended for test environment. This certificate is presented to the client when it tries to connect to this cluster. For convenience, you can choose to use the same certificate for ClusterCertificate and ServerCertificate. You can use two different server certificates, a primary and a secondary for upgrade. Set the thumbprint of the primary certificate in the Thumbprint section and that of the secondary in the ThumbprintSecondary variables.
ServerCertificateCommonNames Recommended for production environment. This certificate is presented to the client when it tries to connect to this cluster. The CertificateIssuerThumbprint corresponds to the thumbprint of the issuer of this certificate. You may specify multiple issuer thumbprints if more than 1 certificate with the same common name is being used. For convenience, you can choose to use the same certificate for ClusterCertificateCommonNames and ServerCertificateCommonNames. You can use one or two server certificate common names.
ClientCertificateThumbprints This is a set of certificates that you want to install on the authenticated clients. You can have a number of different client certificates installed on the machines that you want to allow access to the cluster. Set the thumbprint of each certificate in the CertificateThumbprint variable. If you set the IsAdmin to true, then the client with this certificate installed on it can do administrator management activities on the cluster. If the IsAdmin is false, the client with this certificate can only perform the actions allowed for user access rights, typically read-only. For more information on roles read Role based access control (RBAC)
ClientCertificateCommonNames Set the common name of the first client certificate for the CertificateCommonName. The CertificateIssuerThumbprint is the thumbprint for the issuer of this certificate. Read Working with certificates to know more about common names and the issuer.
ReverseProxyCertificate Recommended for test environment. This is an optional certificate that can be specified if you want to secure your Reverse Proxy. Make sure reverseProxyEndpointPort is set in nodeTypes if you are using this certificate.
ReverseProxyCertificateCommonNames Recommended for production environment. This is an optional certificate that can be specified if you want to secure your Reverse Proxy. Make sure reverseProxyEndpointPort is set in nodeTypes if you are using this certificate.

Here is example cluster configuration where the Cluster, Server, and Client certificates have been provided. Please note that for cluster/ server/ reverseProxy certificates, thumbprint and common name are not allowed to be configured together for the same cert type.

{
   "name": "SampleCluster",
   "clusterConfigurationVersion": "1.0.0",
   "apiVersion": "2016-09-26",
   "nodes": [{
       "nodeName": "vm0",
       "metadata": "Replace the localhost below with valid IP address or FQDN",
       "iPAddress": "10.7.0.5",
       "nodeTypeRef": "NodeType0",
       "faultDomain": "fd:/dc1/r0",
       "upgradeDomain": "UD0"
   }, {
       "nodeName": "vm1",
       "metadata": "Replace the localhost with valid IP address or FQDN",
       "iPAddress": "10.7.0.4",
       "nodeTypeRef": "NodeType0",
       "faultDomain": "fd:/dc1/r1",
       "upgradeDomain": "UD1"
   }, {
       "nodeName": "vm2",
       "iPAddress": "10.7.0.6",
       "metadata": "Replace the localhost with valid IP address or FQDN",
       "nodeTypeRef": "NodeType0",
       "faultDomain": "fd:/dc1/r2",
       "upgradeDomain": "UD2"
   }],
   "properties": {
       "diagnosticsStore": {
       "metadata":  "Please replace the diagnostics store with an actual file share accessible from all cluster machines.",
       "dataDeletionAgeInDays": "7",
       "storeType": "FileShare",
       "IsEncrypted": "false",
       "connectionstring": "c:\\ProgramData\\SF\\DiagnosticsStore"
       }
       "security": {
           "metadata": "The Credential type X509 indicates this is cluster is secured using X509 Certificates. The thumbprint format is - d5 ec 42 3b 79 cb e5 07 fd 83 59 3c 56 b9 d5 31 24 25 42 64.",
           "ClusterCredentialType": "X509",
           "ServerCredentialType": "X509",
           "CertificateInformation": {
               "ClusterCertificateCommonNames": {
                 "CommonNames": [
                   {
                     "CertificateCommonName": "myClusterCertCommonName",
                     "CertificateIssuerThumbprint": "7c fc 91 97 13 66 8d 9f a8 ee 71 2b a2 f4 37 62 00 03 49 0d"
                   }
                 ],
                 "X509StoreName": "My"
               },
               "ServerCertificateCommonNames": {
                 "CommonNames": [
                   {
                     "CertificateCommonName": "myServerCertCommonName",
                     "CertificateIssuerThumbprint": "7c fc 91 97 13 16 8d ff a8 ee 71 2b a2 f4 62 62 00 03 49 0d"
                   }
                 ],
                 "X509StoreName": "My"
               },
               "ClientCertificateThumbprints": [{
                   "CertificateThumbprint": "c4 c18 8e aa a8 58 77 98 65 f8 61 4a 0d da 4c 13 c5 a1 37 6e",
                   "IsAdmin": false
               }, {
                   "CertificateThumbprint": "71 de 04 46 7c 9e d0 54 4d 02 10 98 bc d4 4c 71 e1 83 41 4e",
                   "IsAdmin": true
               }]
           }
       },
       "reliabilityLevel": "Bronze",
       "nodeTypes": [{
           "name": "NodeType0",
           "clientConnectionEndpointPort": "19000",
           "clusterConnectionEndpointPort": "19001",
           "leaseDriverEndpointPort": "19002",
           "serviceConnectionEndpointPort": "19003",
           "httpGatewayEndpointPort": "19080",
           "applicationPorts": {
               "startPort": "20001",
               "endPort": "20031"
           },
           "ephemeralPorts": {
               "startPort": "20032",
               "endPort": "20062"
           },
           "isPrimary": true
       }
        ],
       "fabricSettings": [{
           "name": "Setup",
           "parameters": [{
               "name": "FabricDataRoot",
               "value": "C:\\ProgramData\\SF"
           }, {
               "name": "FabricLogRoot",
               "value": "C:\\ProgramData\\SF\\Log"
           }]
       }]
   }
}

Certificate roll over

When using certificate common name instead of thumbprint, certificate roll over doesn't require cluster configuration upgrade. For issuer thumbprint upgrades, make sure the new thumbprint list has intersection with the old list. You will first have to do a config upgrade with the new issuer thumbprints and then install the new certificates (both cluster/server certificate and issuer certificates) in the store. Please keep the old issuer cert in the cert store at least 2 hours after installing the new issuer cert.

Acquire the X.509 certificates

To secure communication within the cluster, you will first need to obtain X.509 certificates for your cluster nodes. Additionally, to limit connection to this cluster to authorized machines/users, you will need to obtain and install certificates for the client machines.

For clusters that are running production workloads, you should use a Certificate Authority (CA) signed X.509 certificate to secure the cluster. For details on obtaining these certificates, go to How to: Obtain a Certificate.

For clusters that you use for test purposes, you can choose to use a self-signed certificate.

Optional: Create a self-signed certificate

One way to create a self-signed cert that can be secured correctly is to use the CertSetup.ps1 script in the Service Fabric SDK folder in the directory C:\Program Files\Microsoft SDKs\Service Fabric\ClusterSetup\Secure. Edit this file to change the default name of the certificate (look for the value CN=ServiceFabricDevClusterCert). Run this script as .\CertSetup.ps1 -Install.

Now export the certificate to a PFX file with a protected password. First get the thumbprint of the certificate. From the Start menu, run the Manage computer certificates. Navigate to the Local Computer\Personal folder and find the certificate you just created. Double-click the certificate to open it, select the Details tab and scroll down to the Thumbprint field. Copy the thumbprint value into the PowerShell command below, after removing the spaces. Change the String value to a suitable secure password to protect it and run the following in PowerShell:

$pswd = ConvertTo-SecureString -String "1234" -Force –AsPlainText
Get-ChildItem -Path cert:\localMachine\my\<Thumbprint> | Export-PfxCertificate -FilePath C:\mypfx.pfx -Password $pswd

To see the details of a certificate installed on the machine you can run the following PowerShell command:

$cert = Get-Item Cert:\LocalMachine\My\<Thumbprint>
Write-Host $cert.ToString($true)

Alternatively, if you have an Azure subscription, follow the section Add certificates to Key Vault.

Install the certificates

Once you have certificate(s), you can install them on the cluster nodes. Your nodes need to have the latest Windows PowerShell 3.x installed on them. You will need to repeat these steps on each node, for both Cluster and Server certificates and any secondary certificates.

  1. Copy the .pfx file(s) to the node.
  2. Open a PowerShell window as an administrator and enter the following commands. Replace the $pswd with the password that you used to create this certificate. Replace the $PfxFilePath with the full path of the .pfx copied to this node.

    $pswd = "1234"
    $PfxFilePath ="C:\mypfx.pfx"
    Import-PfxCertificate -Exportable -CertStoreLocation Cert:\LocalMachine\My -FilePath $PfxFilePath -Password (ConvertTo-SecureString -String $pswd -AsPlainText -Force)
    
  3. Now set the access control on this certificate so that the Service Fabric process, which runs under the Network Service account, can use it by running the following script. Provide the thumbprint of the certificate and "NETWORK SERVICE" for the service account. You can check that the ACLs on the certificate are correct by opening the certificate in Start > Manage computer certificates and looking at All Tasks > Manage Private Keys.

    param
    (
    [Parameter(Position=1, Mandatory=$true)]
    [ValidateNotNullOrEmpty()]
    [string]$pfxThumbPrint,
    
    [Parameter(Position=2, Mandatory=$true)]
    [ValidateNotNullOrEmpty()]
    [string]$serviceAccount
    )
    
    $cert = Get-ChildItem -Path cert:\LocalMachine\My | Where-Object -FilterScript { $PSItem.ThumbPrint -eq $pfxThumbPrint; }
    
    # Specify the user, the permissions and the permission type
    $permission = "$($serviceAccount)","FullControl","Allow"
    $accessRule = New-Object -TypeName System.Security.AccessControl.FileSystemAccessRule -ArgumentList $permission
    
    # Location of the machine related keys
    $keyPath = Join-Path -Path $env:ProgramData -ChildPath "\Microsoft\Crypto\RSA\MachineKeys"
    $keyName = $cert.PrivateKey.CspKeyContainerInfo.UniqueKeyContainerName
    $keyFullPath = Join-Path -Path $keyPath -ChildPath $keyName
    
    # Get the current acl of the private key
    $acl = (Get-Item $keyFullPath).GetAccessControl('Access')
    
    # Add the new ace to the acl of the private key
    $acl.SetAccessRule($accessRule)
    
    # Write back the new acl
    Set-Acl -Path $keyFullPath -AclObject $acl -ErrorAction Stop
    
    # Observe the access rights currently assigned to this certificate.
    get-acl $keyFullPath| fl
    
  4. Repeat the steps above for each server certificate. You can also use these steps to install the client certificates on the machines that you want to allow access to the cluster.

Create the secure cluster

After configuring the security section of the ClusterConfig.X509.MultiMachine.json file, you can proceed to Create your cluster section to configure the nodes and create the standalone cluster. Remember to use the ClusterConfig.X509.MultiMachine.json file while creating the cluster. For example, your command might look like the following:

.\CreateServiceFabricCluster.ps1 -ClusterConfigFilePath .\ClusterConfig.X509.MultiMachine.json

Once you have the secure standalone Windows cluster successfully running, and have setup the authenticated clients to connect to it, follow the section Connect to a secure cluster using PowerShell to connect to it. For example:

$ConnectArgs = @{  ConnectionEndpoint = '10.7.0.5:19000';  X509Credential = $True;  StoreLocation = 'LocalMachine';  StoreName = "MY";  ServerCertThumbprint = "057b9544a6f2733e0c8d3a60013a58948213f551";  FindType = 'FindByThumbprint';  FindValue = "057b9544a6f2733e0c8d3a60013a58948213f551"   }
Connect-ServiceFabricCluster $ConnectArgs

You can then run other PowerShell commands to work with this cluster. For example, Get-ServiceFabricNode to show a list of nodes on this secure cluster.

To remove the cluster, connect to the node on the cluster where you downloaded the Service Fabric package, open a command line and navigate to the package folder. Now run the following command:

.\RemoveServiceFabricCluster.ps1 -ClusterConfigFilePath .\ClusterConfig.X509.MultiMachine.json

Note

Incorrect certificate configuration may prevent the cluster from coming up during deployment. To self-diagnose security issues, please look in event viewer group Applications and Services Logs > Microsoft-Service Fabric.