Tutorial: Simulate a failure in accessing read-access redundant storage

This tutorial is part two of a series. In this tutorial, you can use either Fiddler or Static Routing to simulate failure for requests to the primary endpoint of your read-access geo-redundant (RA-GRS) storage account, and have the application read from the secondary endpoint.

Scenario app

To complete this tutorial, you must have completed the previous storage tutorial: Make your application data highly available with Azure storage.

In part two of the series, you learn how to:

Prerequisites

To simulate a failure using Fiddler:

If you don't have an Azure subscription, create a free account before you begin.

Simulate a failure with Fiddler

To simulate failure with Fiddler, you inject a failed response for requests to the primary endpoint of your RA-GRS storage account to simulate a failure.

Follow the following steps to simulate a failure, and primary endpoint restoration with fiddler.

Launch fiddler

Open Fiddler, select Rules and Customize Rules.

Customize Fiddler rules

The Fiddler ScriptEditor launches showing the SampleRules.js file. This file is used to customize Fiddler. Paste the following code sample in the OnBeforeResponse function. The new code is commented out to ensure that the logic it creates is not implemented immediately. Once complete select File and Save to save your changes.

    /*
        // Simulate data center failure
        // After it is successfully downloading the blob, pause the code in the sample,
        // uncomment these lines of script, and save the script.
        // It will intercept the (probably successful) responses and send back a 503 error. 
        // When you're ready to stop sending back errors, comment these lines of script out again 
        //     and save the changes.

        if ((oSession.hostname == "contosoragrs.blob.core.windows.net") 
            && (oSession.PathAndQuery.Contains("HelloWorld"))) {
            oSession.responseCode = 503;  
        }
    */

Paste customized rule

Start and pause the application

Run the application in your IDE or text editor. Once the application begins reading from the primary endpoint, press any key in the console window to pause the application.

Simulate failure

With the application paused you can now uncomment the custom rule we saved in Fiddler in a preceding step. The code sample looks for requests to the RA-GRS storage account and if the path contains the name of the image, HelloWorld, it returns a response code of 503 - Service Unavailable.

Navigate to Fiddler and select Rules -> Customize Rules.... Uncomment out the following lines, replace STORAGEACCOUNTNAME with the name of your storage account. Select File -> Save to save your changes.

Note

If you are running the sample application on Linux, you need to restart Fiddler whenever you edit the CustomRule.js file, in order for Fiddler to install the custom logic.

         if ((oSession.hostname == "STORAGEACCOUNTNAME.blob.core.windows.net")
         && (oSession.PathAndQuery.Contains("HelloWorld"))) {
         oSession.responseCode = 503;
         }

To resume the application, press any key.

Once the application starts running again, the requests to the primary endpoint begin to fail. The application attempts to reconnect to the primary endpoint 5 times. After the failure threshold of five attempts, it requests the image from the secondary read-only endpoint. After the application successfully retrieves the image 20 times from the secondary endpoint, the application attempts to connect to the primary endpoint. If the primary endpoint is still unreachable, the application resumes reading from the secondary endpoint. This pattern is the Circuit Breaker pattern described in the previous tutorial.

Paste customized rule

Simulate primary endpoint restoration

With the Fiddler custom rule set in the preceding step, requests to the primary endpoint fail. In order to simulate the primary endpoint functioning again, you remove the logic to inject the 503 error.

To pause the application, press any key.

Navigate to Fiddler and select Rules and Customize Rules.... Comment or remove the custom logic in the OnBeforeResponse function, leaving the default function. Select File and Save to save the changes.

Remove customized rule

When complete, press any key to resume the application. The application continues reading from the primary endpoint until it hits 999 reads.

Resume application

Simulate a failure with an invalid static route

You can create an invalid static route for all requests to the primary endpoint of your read-access geo-redundant (RA-GRS) storage account. In this tutorial, the local host is used as the gateway for routing requests to the storage account. Using the local host as the gateway causes all requests to your storage account primary endpoint to loop back inside the host, which subsequently leads to failure. Follow the following steps to simulate a failure, and primary endpoint restoration with an invalid static route.

Start and pause the application

Run the application in your IDE or text editor. Once the application begins reading from the primary endpoint, press any key in the console window to pause the application.

Simulate failure

With the application paused, start command prompt on Windows as an administrator or run terminal as root on Linux. To get information about the storage account primary endpoint domain, enter the following command on a command prompt or terminal.

nslookup STORAGEACCOUNTNAME.blob.core.windows.net

Replace STORAGEACCOUNTNAME with the name of your storage account. Copy to the IP address of your storage account to a text editor for later use. To get the IP address of your local host, type ipconfig on the Windows command prompt, or ifconfig on the Linux terminal.

To add a static route for a destination host, type the following command on a Windows command prompt or Linux terminal.

route add <destination_ip> gw <gateway_ip>

Replace <destination_ip> with your storage account IP address, and <gateway_ip> with your local host IP address. To resume the application, press any key.

Once the application starts running again, the requests to the primary endpoint begin to fail. The application attempts to reconnect to the primary endpoint 5 times. After the failure threshold of five attempts, it requests the image from the secondary read-only endpoint. After the application successfully retrieves the image 20 times from the secondary endpoint, the application attempts to connect to the primary endpoint. If the primary endpoint is still unreachable, the application resumes reading from the secondary endpoint. This pattern is the Circuit Breaker pattern described in the previous tutorial.

Simulate primary endpoint restoration

To simulate the primary endpoint functioning again, delete the static route of the primary endpoint from the routing table. This allows all requests to the primary endpoint to be routed through the default gateway.

To delete the static route of a destination host, the storage account, type the following command on a Windows command prompt or linux terminal.

route del <destination_ip> gw <gateway_ip>

Press any key to resume the application. The application continues reading from the primary endpoint until it hits 999 reads.

Resume application

Next steps

In part two of the series, you learned about simulating a failure to test read access geo-redundant storage, such as how to:

Read the following article to learn more about how RA-GRS storage works (and its associated risks).