Edit

Production readiness checklist

  • Article

Is your application and cluster ready to take production traffic? Running and testing your application and your cluster doesn't necessarily mean it's ready to go into production. Keep your application and cluster running smoothly by going through the following checklist. We strongly recommend all these items to be checked off. Obviously, you can choose to use alternative solutions for a particular line item (for example, your own diagnostics frameworks).

Prerequisites for production

  1. Azure Service Fabric best practices: Application Design, Security, Networking, Capacity planning and scaling, Infrastructure as Code, and Monitoring and Diagnostics.
  2. Configure FabricTransport settings if you're using the Reliable Actors programming model and require secure inter-service communication.
  3. For clusters with more than 20 cores or 10 nodes, create a dedicated primary node type for system services. Add placement constraints to reserve the primary node type for system services.
  4. Use a D2v2 or higher SKU for the primary node type. It is recommended to pick a SKU with at least 50 GB hard disk capacity.
  5. Production clusters must be secure. For an example of setting up a secure cluster, see this cluster template. Use common names for certificates and avoid using self signed certs.
  6. Add resource constraints on containers and services, so that they don't consume more than 75% of node resources.
  7. Understand and set the durability level. Silver or higher durability level is recommended for node types running stateful workloads, and required for production.
  8. Understand and pick the reliability level of the node type. Silver or higher reliability is recommended, and required for production.
  9. Load and scale test your workloads to identify capacity requirements for your cluster.
  10. Your services and applications are monitored and application logs are being generated and stored, with alerting. For example, see Add logging to your Service Fabric application and Monitor containers with Azure Monitor logs.
  11. The cluster is monitored with alerting (for example, with Azure Monitor logs).
  12. The underlying virtual machine scale set infrastructure is monitored with alerting (for example, with Azure Monitor logs.
  13. The cluster has primary and secondary certificates always (so you don't get locked out).
  14. Maintain separate clusters for development, staging, and production.
  15. Application upgrades and cluster upgrades are tested in development and staging clusters first.
  16. Turn off automatic upgrades in production clusters, and turn it on for development and staging clusters (rollback as needed).
  17. Establish a Recovery Point Objective (RPO) for your service, and set up a disaster recovery process and test it out.
  18. Plan for scaling your cluster manually or programmatically.
  19. Plan for patching your cluster nodes.
  20. Establish a CI/CD pipeline so that your latest changes are being continually tested. For example, using Azure DevOps or Jenkins
  21. Test your development & staging clusters under load with the Fault Analysis Service and induce controlled chaos.
  22. Plan for scaling your applications.

If you're using the Service Fabric Reliable Services or Reliable Actors programming model, the following items need to be checked off:

  1. Upgrade applications during local development to check that your service code is honoring the cancellation token in the RunAsync method and closing custom communication listeners.
  2. Avoid common pitfalls when using Reliable Collections.
  3. Monitor the .NET CLR memory performance counters when running load tests and check for high rates of Garbage Collection or runaway heap growth.
  4. Maintain offline backup of Reliable Services and Reliable Actors and test the restoration process.
  5. Your Primary NodeType Virtual Machine instance count should ideally be equal to the minimum for your Clusters Reliability tier; conditions when appropriate to exceed the Tier minimum includes: temporarily when vertically scaling your Primary NodeTypes Virtual Machine Scale Set SKU.

Optional best practices

While the above lists are pre-requisites to go into production, the following items should also be considered:

  1. Plug into the Service Fabric health model for extending the built-in health evaluation and reporting.
  2. Deploy a custom watchdog that is monitoring your application and reports load for resource balancing.

Next steps