Validate Azure Stack Hub system state

As an Azure Stack Hub operator, being able to determine the health and status of your system on demand is essential. The Azure Stack Hub validation tool (Test-AzureStack)) is a PowerShell cmdlet that lets you run a series of tests on your system to identify failures if present. You'll typically be asked to run this tool through the privileged end point (PEP) when you contact Microsoft Customer Services Support (Microsoft Support) with an issue. With the system-wide health and status information at hand, Microsoft Support can collect and analyze detailed logs, focus on the area where the error occurred, and work with you to fix the issue.

Running the validation tool and accessing results

You can use the PEP to run the validation tool. The tool can take a while to run. The length of the time depends on the number of virtual machines in your system. Each test returns a PASS/FAIL status in the PowerShell window.

Here's an outline of the end-to-end validation testing process:

  1. Establish the trust. On an integrated system, run the following command from an elevated Windows PowerShell session to add the PEP as a trusted host on the hardened VM running on the hardware lifecycle host or the Privileged Access Workstation.

    winrm s winrm/config/client '@{TrustedHosts="<IP Address of Privileged Endpoint>"}'
    

    If you're running the Azure Stack Development Kit (ASDK), sign in to the development kit host.

  2. Access the PEP. Run the following commands to establish a PEP session:

    Enter-PSSession -ComputerName "<ERCS VM-name/IP address>" -ConfigurationName PrivilegedEndpoint -Credential $localcred 
    

    Tip

    To access the PEP on an Azure Stack Development Kit (ASDK) host computer, use AzS-ERCS01 for -ComputerName.

  3. Once you're in the PEP, run:

    Test-AzureStack
    

    For more information, see Parameter considerations and Use case examples.

  4. If any tests report FAIL, run Get-AzureStackLog. For instructions on an integrated system, see how to run Get-AzureStackLog on Azure Stack Hub integrated systems.

    The cmdlet gathers logs generated by Test-AzureStack. We recommend you don't collect logs and contact Microsoft Support instead if tests report WARN.

  5. If you're instructed to run the validation tool by the Microsoft Support, the Microsoft Support representative will request the logs you collected to continue troubleshooting your issue.

Tests available

The validation tool lets you run a series of system-level tests and basic cloud scenarios that provide you with insight to the current state, allowing you to fix issues in your system.

Cloud infrastructure tests

These low impact tests work on an infrastructure level and provide you with information on various system components and functions. Currently, tests are grouped into the following categories:

Test Category Argument for -Include and -Ignore
Azure Stack Hub ACS Summary AzsAcsSummary
Azure Stack Hub Action Plan Status AzsActionPlanStatus
Azure Stack Hub Agent Health AzsAgentHealth
Azure Stack AKS Arc Preview Summary AzsAksArcPreviewSummary
Azure Stack Hub Active Directory Summary AzsAdSummary
Azure Stack Hub Alert Summary AzsAlertSummary
Azure Stack Hub Application Crash Summary AzsApplicationCrashSummary
Azure Stack Hub Backup Share Accessibility Summary AzsBackupShareAccessibility
Azure Stack Hub Control Plane Summary AzsControlPlane
Azure Stack Hub DCB Validation AzsDCBValidation
Azure Stack Hub Defender Summary AzsDefenderSummary
Azure Stack Hub ECE Provisioning Status AzsEceProvisioningStatus
Azure Stack Hub External Certificates Summary AzsExternalCertificates
Azure Stack Hub GPU Configuration Summary AzsGpuConfigurationSummary
Azure Stack Hub Hardware Monitor Enable AzsGpuConfigurationSummary
Azure Stack Hub Hosting Infrastructure Firmware Summary AzsHostingInfraFWSummary
Azure Stack Hub Hosting Infrastructure Summary AzsHostingInfraSummary
Azure Stack Hub Hosting Infrastructure Utilization AzsHostingInfraUtilization
Azure Stack Hub Identity Provider Connectivity AzsIdentityProviderConnectivity
Azure Stack Hub Identity Provider Status AzsIdentityProviderStatus
Azure Stack Hub Infrastructure Capacity AzsInfraCapacity
Azure Stack Hub Infrastructure File Validation AzsInfraFileValidation
Azure Stack Hub Infrastructure Performance AzsInfraPerformance
Azure Stack Hub Infrastructure Role Summary AzsInfraRoleSummary
Azure Stack Hub Infrastructure Update Memory AzsInfraUpdateMemory
Azure Stack Hub Infrastructure Update Summary AzsInfraUpdateSummary
Azure Stack Hub NCImos Validation AzsNCImosValidation
Azure Stack Hub Network Infra AzsNetworkInfra
Azure Stack Hub OEM Hardware AzsOemHardware
Azure Stack Hub OEM Update Prerequisites AzsOemUpdatePrereqs
Azure Stack Hub Portal and API Summary AzsPortalAPISummary
Azure Stack Hub RDMA Operational Status AzsRdmaOperationalStatus
Azure Stack Hub Resource Provider Authentication Status AzsResourceProviderAuthStatus
Azure Stack Hub Scale Unit VM Events AzsScaleUnitEvents
Azure Stack Hub Scale Unit VM Resources AzsScaleUnitResources
Azure Stack Hub Scenarios AzsScenarios
Azure Stack Hub Service Fabric Role Summary AzsSFRoleSummary
Azure Stack Hub SQL Diagnostics AzsSQLDiagnostics
Azure Stack Hub SQL Summary AzsSQLSummary
Azure Stack Hub Stamp BMC Summary AzsStampBMCSummary
Azure Stack Hub Storage Data Plane AzsStorageDataPlane
Azure Stack Hub Storage Services Summary AzsStorageSvcsSummary
Azure Stack Hub SQL Store Summary AzsStoreSummary
Azure Stack Hub Usage Summary AzsUsage
Azure Stack Hub Usage Bridge Summary AzsUsageBridgeSummary
Azure Stack Hub VM Placement Summary AzsVmPlacement

Cloud scenario tests

In addition to the infrastructure tests in the previous section, you can also run cloud scenario tests to check functionality across infrastructure components. Cloud admin credentials are required to run these tests because they involve resource deployment.

Note

Currently you can't run cloud scenario tests using Active Directory Federated Services (AD FS) credentials.

The following cloud scenarios are tested by the validation tool:

  • Resource group creation
  • Plan creation
  • Offer creation
  • Storage account creation
  • Virtual machine creation (VM)
  • Blob storage operation
  • Queue storage operation
  • Table storage operation

Parameter considerations

  • The parameter List can be used to display all available test categories.

  • The parameters Include and Ignore can be used to include or exclude test categories. For more information about these arguments, see the following section.

    Test-AzureStack -Include AzsSFRoleSummary, AzsInfraCapacity
    
    Test-AzureStack -Ignore AzsInfraPerformance
    
  • A tenant VM is deployed as part of the cloud scenario tests. You can use DoNotDeployTenantVm to disable this VM deployment.

  • You need to supply the ServiceAdminCredential parameter to run cloud scenario tests as described in the Use case examples section.

  • BackupSharePath and BackupShareCredential are used when testing infrastructure backup settings as shown in the Use case examples section.

  • DetailedResults can be used to get pass/fail/warning information for each test, as well as the overall run. When not specified, Test-AzureStack returns $true if there are no failures, and $false if there are failures.

  • TimeoutSeconds can be used to set a specific time for each group to complete.

  • The validation tool also supports common PowerShell parameters: Verbose, Debug, ErrorAction, ErrorVariable, WarningAction, WarningVariable, OutBuffer, PipelineVariable, and OutVariable. For more information, see About Common Parameters.

Use case examples

Run validation without cloud scenarios

Run the validation tool without the ServiceAdminCredential parameter to skip running cloud scenario tests:

New-PSSession -ComputerName "<ERCS VM-name/IP address>" -ConfigurationName PrivilegedEndpoint -Credential $localcred -SessionOption (New-PSSessionOption -Culture en-US -UICulture en-US)
Test-AzureStack

Run validation with cloud scenarios

Supplying the validation tool with the ServiceAdminCredentials parameter runs the cloud scenario tests by default:

Enter-PSSession -ComputerName "<ERCS VM-name/IP address>" -ConfigurationName PrivilegedEndpoint -Credential $localcred 
Test-AzureStack -ServiceAdminCredential "<Cloud administrator user name>" 

If you wish to run ONLY cloud scenarios without running the rest of the tests, you can use the Include parameter to do so:

Enter-PSSession -ComputerName "<ERCS VM-name/IP address>" -ConfigurationName PrivilegedEndpoint -Credential $localcred 
Test-AzureStack -ServiceAdminCredential "<Cloud administrator user name>" -Include AzsScenarios   

The cloud admin user name must be typed in the UPN format: serviceadmin@contoso.onmicrosoft.com (Microsoft Entra ID). When prompted, type the password to the cloud admin account.

Groups

To improve the operator experience, a Group parameter has been enabled to run multiple test categories at the same time. Currently, there are three groups defined: Default, UpdateReadiness, and SecretRotationReadiness.

  • Default: Considered to be a standard run of Test-AzureStack. This group is run by default if no other groups are selected.

  • UpdateReadiness: A check to see if the Azure Stack Hub instance can be updated. When the UpdateReadiness group is run, warnings are displayed as errors in the console output, and they should be considered as blockers for the update. The following categories are part of the UpdateReadiness group:

    • AzsInfraFileValidation
    • AzsActionPlanStatus
    • AzsStampBMCSummary
  • SecretRotationReadiness: A check to see if the Azure Stack Hub instance is in a state in which secret rotation can be run. When the SecretRotationReadiness group is run, warnings are displayed as errors in the console output and they should be considered as blockers for secret rotation. The following categories are part of the SecretRotationReadiness Group:

    • AzsAcsSummary
    • AzsDefenderSummary
    • AzsHostingInfraSummary
    • AzsInfraCapacity
    • AzsInfraRoleSummary
    • AzsPortalAPISummary
    • AzsSFRoleSummary
    • AzsStorageSvcsSummary
    • AzsStoreSummary

Group parameter example

The following example runs Test-AzureStack to test system readiness before installing an update or hotfix using Group. Before you start the installation of an update or hotfix, run Test-AzureStack to check the status of your Azure Stack Hub:

Test-AzureStack -Group UpdateReadiness

Run validation tool to test infrastructure backup settings

Before configuring infrastructure backup, you can test the backup share path and credential using the AzsBackupShareAccessibility test:

Enter-PSSession -ComputerName "<ERCS VM-name/IP address>" -ConfigurationName PrivilegedEndpoint -Credential $localcred 
Test-AzureStack -Include AzsBackupShareAccessibility -BackupSharePath "\\<fileserver>\<fileshare>" -BackupShareCredential $using:backupcred

After configuring backup, you can run AzsBackupShareAccessibility to validate the share is accessible from the ERCS:

Enter-PSSession -ComputerName "<ERCS VM-name/IP address>" -ConfigurationName PrivilegedEndpoint -Credential $localcred 
Test-AzureStack -Include AzsBackupShareAccessibility

To test new credentials with the configured backup share, run:

Enter-PSSession -ComputerName "<ERCS VM-name/IP address>" -ConfigurationName PrivilegedEndpoint -Credential $localcred 
Test-AzureStack -Include AzsBackupShareAccessibility -BackupShareCredential "<PSCredential for backup share>"

Run validation tool to test network infrastructure

This test checks the connectivity of the network infrastructure bypassing the Azure Stack Hub software defined network (SDN). It demonstrates connectivity from a Public VIP to the configured DNS forwarders, NTP servers, and authentication endpoints. This includes connectivity to Azure when using Microsoft Entra ID as identity provider or the federated server when using AD FS as identity provider.

Include the debug parameter to get a detailed output of the command:

Test-AzureStack -Include AzsNetworkInfra -Debug

Next steps

To learn more about Azure Stack Hub diagnostics tools and issue logging, see Azure Stack Hub Diagnostic log collection.

To learn more about troubleshooting, see Microsoft Azure Stack Hub troubleshooting.