Write directly to storage

APPLIES TO: yesSDK v4 no SDK v3

You can read and write directly to your storage object without using middleware or context object. This can be appropriate for data your bot uses to preserve a conversation, or data that comes from a source outside your bot's conversation flow. In this data storage model, data is read in directly from storage instead of using a state manager. The code examples in this article show you how to read and write data to storage using memory, Cosmos DB, Azure Blob, and Azure Blob transcript storage.

Prerequisites

Note

Both .NET Core 2.1 and .NET Core 3.1 versions of the C# VSIX templates are available in Visual Studio. When creating new bots in Visual Studio 2019, you should use the .NET Core 3.1 templates. The current bot samples use .NET Core 3.1 templates. You can find the samples that use .NET Core 2.1 templates in the 4.7-archive branch of the BotBuilder-Samples repository. For information about deploying .NET Core 3.1 bots to Azure, see Deploy your bot.

About this sample

The sample code in this article begins with the structure of a basic echo bot, then extends that bot's functionality by adding additional code (provided below). This extended code creates a list to preserve user inputs as they are received. Each turn, the full list of user inputs is echoed back to the user. The data structure containing this list of inputs is then saved to storage at the end of that turn. Various types of storage are explored as additional functionality is added to this sample code.

Memory storage

The Bot Framework SDK allows you to store user inputs using in-memory storage. Memory storage is used for testing purposes only and is not intended for production use. In-memory storage is volatile and temporary since the data is cleared each time the bot is restarted. Persistent storage types, such as database storage, are best for production bots. Be sure to set storage to Cosmos DB, Blob Storage, or Azure Table storage before publishing your bot.

Build a basic bot

The rest of this topic builds off of an Echo bot. The Echo bot sample code can be locally built by following the Quickstart instructions for building either a C# EchoBot, JS EchoBot or Python EchoBot.

EchoBot.cs

using System;
using System.Threading.Tasks;
using Microsoft.Bot.Builder;
using Microsoft.Bot.Schema;
using System.Collections.Generic;
using System.Linq;
using System.Threading;

// Represents a bot saves and echoes back user input.
public class EchoBot : ActivityHandler
{
   // Create local Memory Storage.
   private static readonly MemoryStorage _myStorage = new MemoryStorage();

   // Create cancellation token (used by Async Write operation).
   public CancellationToken cancellationToken { get; private set; }

   // Class for storing a log of utterances (text of messages) as a list.
   public class UtteranceLog : IStoreItem
   {
      // A list of things that users have said to the bot
      public List<string> UtteranceList { get; } = new List<string>();

      // The number of conversational turns that have occurred
      public int TurnNumber { get; set; } = 0;

      // Create concurrency control where this is used.
      public string ETag { get; set; } = "*";
   }

   // Echo back user input.
   protected override async Task OnMessageActivityAsync(ITurnContext<IMessageActivity> turnContext, CancellationToken cancellationToken)
   {
      // preserve user input.
      var utterance = turnContext.Activity.Text;

      // Make empty local log-items list.
      UtteranceLog logItems = null;

      // See if there are previous messages saved in storage.
      try
      {
         string[] utteranceList = { "UtteranceLog" };
         logItems = _myStorage.ReadAsync<UtteranceLog>(utteranceList).Result?.FirstOrDefault().Value;
      }
      catch
      {
         // Inform the user an error occurred.
         await turnContext.SendActivityAsync("Sorry, something went wrong reading your stored messages!");
      }

      // If no stored messages were found, create and store a new entry.
      if (logItems is null)
      {
         // Add the current utterance to a new object.
         logItems = new UtteranceLog();
         logItems.UtteranceList.Add(utterance);

         // Set initial turn counter to 1.
         logItems.TurnNumber++;

         // Show user new user message.
         await turnContext.SendActivityAsync($"{logItems.TurnNumber}: The list is now: {string.Join(", ", logItems.UtteranceList)}");

         // Create dictionary object to hold received user messages.
         var changes = new Dictionary<string, object>();
         {
            changes.Add("UtteranceLog", logItems);
         }
         try
         {
            // Save the user message to your Storage.
            await _myStorage.WriteAsync(changes, cancellationToken);
         }
         catch
         {
            // Inform the user an error occurred.
            await turnContext.SendActivityAsync("Sorry, something went wrong storing your message!");
         }
      }
      // Else, our storage already contained saved user messages, add new one to the list.
      else
      {
         // add new message to list of messages to display.
         logItems.UtteranceList.Add(utterance);
         // increment turn counter.
         logItems.TurnNumber++;

         // show user new list of saved messages.
         await turnContext.SendActivityAsync($"{logItems.TurnNumber}: The list is now: {string.Join(", ", logItems.UtteranceList)}");

         // Create Dictionary object to hold new list of messages.
         var changes = new Dictionary<string, object>();
         {
            changes.Add("UtteranceLog", logItems);
         };

         try
         {
            // Save new list to your Storage.
            await _myStorage.WriteAsync(changes,cancellationToken);
         }
         catch
         {
            // Inform the user an error occurred.
            await turnContext.SendActivityAsync("Sorry, something went wrong storing your message!");
         }
      }
      ...  // OnMessageActivityAsync( )
   }
}

Start your bot

Run your bot locally.

Start the emulator and connect your bot

Install the Bot Framework Emulator Next, start the emulator and then connect to your bot in the emulator:

  1. Click the Create new bot configuration link in the emulator "Welcome" tab.
  2. Fill in fields to connect to your bot, given the information on the webpage displayed when you started your bot.

Interact with your bot

Send a message to your bot. The bot will list the messages it has received.

Emulator test storage bot

Using Cosmos DB

Important

The Cosmos DB storage class has been deprecated. Containers originally created with CosmosDbStorage had no partition key set, and were given the default partition key of _/partitionKey.

Containers created with Cosmos DB storage can be used with Cosmos DB partitioned storage. Read Partitioning in Azure Cosmos DB for more information.

Also note that, unlike the legacy Cosmos DB storage, the Cosmos DB partitioned storage does not automatically create a database within your Cosmos DB account. You need to create a new database manually, but skip manually creating a container since CosmosDbPartitionedStorage will create the container for you.

Now that you've used memory storage, we'll update the code to use Azure Cosmos DB. Cosmos DB is Microsoft's globally distributed, multi-model database. Azure Cosmos DB enables you to elastically and independently scale throughput and storage across any number of Azure's geographic regions. It offers throughput, latency, availability, and consistency guarantees with comprehensive service level agreements (SLAs).

Set up a Cosmos DB resource

To use Cosmos DB in your bot, you'll need to create a database resource before getting into the code. For an in-depth description of Cosmos DB database and app creation access the documentation here for Cosmos DB dotnet or Cosmos DB nodejs.

Create your database account

  1. In a new browser window, sign in to the Azure portal.

    Create Cosmos DB database account

  2. Click Create a resource > Databases > Azure Cosmos DB

    Cosmos DB new account page

  3. On the New account page, provide Subscription, Resource group information. Create a unique name for your Account Name field - this eventually becomes part of your data access URL name. For API, select Core(SQL), and provide a nearby Location to improve data access times.

  4. Then click Review + Create.

  5. Once validation has been successful, click Create.

The account creation takes a few minutes. Wait for the portal to display the Congratulations! Your Azure Cosmos DB account was created page.

Add a database

Important

Unlike the legacy Cosmos DB storage, which has now been deprecated, the Cosmos DB partitioned storage does not automatically create a database within your Cosmos DB account.

  1. Navigate to the Data Explorer page within your newly created Cosmos DB account, then choose Create Database from the drop-down box next to the Create Container button. A panel will then open on the right hand side of the window, where you can enter the details for the new database.

    Create cosmosdb databas resource image

  2. Enter an ID for your new database and, optionally, set the throughput (you can change this later) and finally click OK to create your database. Make a note of this database ID for use later on when configuring your bot.

    Cosmos cosmosdb database resource details image

  3. Now that you have created a Cosmos DB account and a database, you need to copy over some of the values for integrating your new database into your bot. To retrieve these, navigate to the Keys tab within the database settings section of your Cosmos DB account. From this page you will need your Cosmos DB endpoint (URI) and your authorization key (PRIMARY KEY).

    Cosmos DB Keys

You should now have a Cosmos DB account, containing a database and have the following details ready to configure your bot.

  • Cosmos DB Endpoint
  • Authorization Key
  • Database ID

Add Cosmos DB configuration information

Our configuration data to add Cosmos DB storage is short and simple. Use the details you made a note of in the previous part of this article to set your endpoint, authorization key and database ID. Finally, you should choose an appropriate name for the container that will be created within your database to store your bot state. In the example below the container will be called "bot-storage".

Note

You should not create the container yourself. Your bot will create it for you when creating its internal Cosmos DB client, ensuring it is configured correctly for storing bot state.

Add the following information to your configuration file.

appsettings.json

"CosmosDbEndpoint": "<your-cosmosdb-uri>",
"CosmosDbAuthKey": "<your-authorization-key>",
"CosmosDbDatabaseId": "<your-database-id>",
"CosmosDbContainerId": "<your-container-id>"

Installing Cosmos DB packages

Make sure you have the packages necessary for Cosmos DB.

Install-Package Microsoft.Bot.Builder.Azure

Cosmos DB implementation

Note

Version 4.6 introduced a new Cosmos DB storage provider, the Cosmos DB partitioned storage class, and the original Cosmos DB storage class is deprecated. Containers created with Cosmos DB storage can be used with Cosmos DB partitioned storage. Read Partitioning in Azure Cosmos DB for more information.

Also note that, unlike the legacy Cosmos DB storage, the Cosmos DB partitioned storage does not automatically create a database within your Cosmos DB account. You need to create a new database manually, but skip manually creating a container since CosmosDbPartitionedStorage will create the container for you.

The following sample code runs using the same bot code as the memory storage sample provided above. The code snippet below shows an implementation of Cosmos DB storage for 'myStorage' that replaces local Memory storage. Memory Storage is commented out and replaced with a reference to Cosmos DB.

Startup.cs

using Microsoft.Bot.Builder.Azure;

Within ConfigureServices, create the storage instance for CosmosDB partitioned storage.

// Use partitioned CosmosDB for storage, instead of in-memory storage.
services.AddSingleton<IStorage>(
    new CosmosDbPartitionedStorage(
        new CosmosDbPartitionedStorageOptions
        {
            CosmosDbEndpoint = Configuration.GetValue<string>("CosmosDbEndpoint"),
            AuthKey = Configuration.GetValue<string>("CosmosDbAuthKey"),
            DatabaseId = Configuration.GetValue<string>("CosmosDbDatabaseId"),
            ContainerId = Configuration.GetValue<string>("CosmosDbContainerId"),
            CompatibilityMode = false,
        }));

Start your Cosmos DB bot

Run your bot locally.

Test your Cosmos DB bot with bot framework emulator

Now start your bot framework emulator and connect to your bot:

  1. Click the Create new bot configuration link in the emulator "Welcome" tab.
  2. Fill in fields to connect to your bot, given the information on the webpage displayed when you started your bot.

Interact with your Cosmos DB bot

Send a message to your bot, and the bot will list the messages it received. Emulator running

View your Cosmos DB data

After you have run your bot and saved your information, we can view the data stored in the Azure portal under the Data Explorer tab.

Data Explorer example

Using Blob storage

Azure Blob storage is Microsoft's object storage solution for the cloud. Blob storage is optimized for storing massive amounts of unstructured data, such as text or binary data.

Create your Blob storage account

To use Blob storage in your bot, you'll need to get a few things set up before getting into the code.

  1. In a new browser window, sign in to the Azure portal.

    Create Blob storage

  2. Click Create a resource > Storage > Storage account - blob, file, table, queue

    Blob storage new account page

  3. In the New account page, enter Name for the storage account, select Blob storage for Account kind, provide Location, Resource group and Subscription information.

  4. Then click Review + Create.

  5. Once validation has been successful, click Create.

Create Blob storage container

Once your Blob storage account is created, open this account by

  1. Selecting the resource.

  2. Now "Open" using Storage Explorer (preview)

    Create Blob storage container

  3. Right click BLOB CONTAINERS, select Create blob container.

  4. Add a name. You will use this name for the value "your-blob-storage-container-name" to provide access to your Blob Storage account.

Add Blob storage configuration information

Find the Blob Storage keys you need to configure Blob Storage for your bot as shown above:

  1. In the Azure portal, open your Blob Storage account and select Settings > Access keys.

    Find Blob Storage Keys

We will use key1 Connection string as the value "your-blob-storage-account-string" to provide access to your Blob Storage account.

Installing Blob storage packages

If not previously installed, install the following packages.

Install-Package Microsoft.Bot.Builder.Azure.Blobs

Blob storage implementation

Blob storage is designed to store bot state.

Note

As of version 4.10, Microsoft.Bot.Builder.Azure.AzureBlobStorage is deprecated. Use the new Microsoft.Bot.Builder.Azure.Blobs.BlobsStorage in its place.

EchoBot.cs

using Microsoft.Bot.Builder.Azure.Blobs;

Update the line of code that points "myStorage" to your existing Blob Storage account.

EchoBot.cs

private static readonly BlobsStorage _myStorage = new BlobsStorage("<your-azure-storage-connection-string>", "<your-blob-storage-container-name>");

Once your storage is set to point to your Blob Storage account, your bot code will now store and retrieve data from Blob Storage.

Start your Blob storage bot

Run your bot locally.

Start the emulator and connect your Blob storage bot

Next, start the emulator and then connect to your bot in the emulator:

  1. Click the Create new bot configuration link in the emulator "Welcome" tab.
  2. Fill in fields to connect to your bot, given the information on the webpage displayed when you started your bot.

Interact with your Blob storage bot

Send a message to your bot, and the bot will list the messages it receives.

Emulator test storage bot

View your Blob storage data

After you have run your bot and saved your information, we can view it in under the Storage Explorer tab in the Azure portal.

Blob transcript storage

Azure blob transcript storage provides a specialized storage option that allows you to easily save and retrieve user conversations in the form of a recorded transcript. Azure blob transcript storage is particularly useful for automatically capturing user inputs to examine while debugging your bot's performance.

Note

Python does not currently support Azure Blob transcript storage. While JavaScript supports Blob transcript storage, the following directions are for C# only.

Set up a Blob transcript storage container

Azure blob transcript storage can use the same blob storage account created following the steps detailed in sections "Create your blob storage account" and "Add configuration information" above. We now add a container to hold our transcripts

Create transcript container

  1. Open your Azure blob storage account.
  2. Click on Storage Explorer.
  3. Right click on BLOB CONTAINERS and select create blob container.
  4. enter a name for your transcript container and then select OK. (We entered mybottranscripts)

Blob transcript storage implementation

The following code connects transcript storage pointer _myTranscripts to your new Azure blob transcript storage account. To create this link with a new container name, <your-blob-transcript-container-name>, creates a new container within Blob storage to hold your transcript files.

Blob transcript storage is designed to store bot transcripts.

Note

As of version 4.10, Microsoft.Bot.Builder.Azure.AzureBlobTranscriptStore is deprecated. Use the new Microsoft.Bot.Builder.Azure.Blobs.BlobsTranscriptStore in its place.

echoBot.cs

using Microsoft.Bot.Builder.Azure.Blobs;

public class EchoBot : ActivityHandler
{
   ...

   private readonly BlobsTranscriptStore _myTranscripts = new BlobsTranscriptStore("<your-azure-storage-connection-string>", "<your-blob-transcript-container-name>");

   ...
}

Store user conversations in azure blob transcripts

After a blob container is available to store transcripts you can begin to preserve your users' conversations with your bot. These conversations can later be used as a debugging tool to see how users interact with your bot. Each emulator Restart conversation initiates the creation of a new transcript conversation list. The following code preserves user conversation inputs within a stored transcript file.

  • The current transcript is saved using LogActivityAsync.
  • Saved transcripts are retrieved using ListTranscriptsAsync. In this sample code the Id of each stored transcript is saved into a list named "storedTranscripts". This list is later used to manage the number of stored blob transcripts we retain.

echoBot.cs


protected override async Task OnMessageActivityAsync(ITurnContext<IMessageActivity> turnContext, CancellationToken cancellationToken)
{
    await _myTranscripts.LogActivityAsync(turnContext.Activity);

    List<string> storedTranscripts = new List<string>();
    PagedResult<Microsoft.Bot.Builder.TranscriptInfo> pagedResult = null;
    var pageSize = 0;
    do
    {
       pagedResult = await _myTranscripts.ListTranscriptsAsync("emulator", pagedResult?.ContinuationToken);
       pageSize = pagedResult.Items.Count();

       // transcript item contains ChannelId, Created, Id.
       // save the channelIds found by "ListTranscriptsAsync" to a local list.
       foreach (var item in pagedResult.Items)
       {
          storedTranscripts.Add(item.Id);
       }
    } while (pagedResult.ContinuationToken != null);

    ...
}

Manage stored blob transcripts

While stored transcripts can be used as a debugging tool, over time the number of stored transcripts can grow larger than you care to preserve. The additional code included below uses DeleteTranscriptAsync to remove all but the last three retrieved transcript items from your blob transcript store.

echoBot.cs


protected override async Task OnMessageActivityAsync(ITurnContext<IMessageActivity> turnContext, CancellationToken cancellationToken)
{
    await _myTranscripts.LogActivityAsync(turnContext.Activity);

    List<string> storedTranscripts = new List<string>();
    PagedResult<Microsoft.Bot.Builder.TranscriptInfo> pagedResult = null;
    var pageSize = 0;
    do
    {
       pagedResult = await _myTranscripts.ListTranscriptsAsync("emulator", pagedResult?.ContinuationToken);
       pageSize = pagedResult.Items.Count();

       // transcript item contains ChannelId, Created, Id.
       // save the channelIds found by "ListTranscriptsAsync" to a local list.
       foreach (var item in pagedResult.Items)
       {
          storedTranscripts.Add(item.Id);
       }
    } while (pagedResult.ContinuationToken != null);

    // Manage the size of your transcript storage.
    for (int i = 0; i < pageSize; i++)
    {
       // Remove older stored transcripts, save just the last three.
       if (i < pageSize - 3)
       {
          string thisTranscriptId = storedTranscripts[i];
          try
          {
             await _myTranscripts.DeleteTranscriptAsync("emulator", thisTranscriptId);
           }
           catch (System.Exception ex)
           {
              await turnContext.SendActivityAsync("Debug Out: DeleteTranscriptAsync had a problem!");
              await turnContext.SendActivityAsync("exception: " + ex.Message);
           }
       }
    }
    ...
}

The following link provides more information concerning Azure Blob Transcript Storage

Additional Information

Manage concurrency using eTags

In our bot code example we set the eTag property of each IStoreItem to *. The eTag (entity tag) member of your store object is used within Cosmos DB to manage concurrency. The eTag tells your database what to do if another instance of the bot has changed the object in the same storage that your bot is writing to.

Last write wins - allow overwrites

An eTag property value of asterisk (*) indicates that the last writer wins. When creating a new data store, you can set eTag of a property to * to indicate that you have not previously saved the data that you are writing, or that you want the last writer to overwrite any previously saved property. If concurrency is not an issue for your bot, setting the eTag property to * for any data that you are writing enables overwrites.

Maintain concurrency and prevent overwrites

When storing your data into Cosmos DB, use a value other than * for the eTag if you want to prevent concurrent access to a property and avoid overwriting changes from another instance of the bot. The bot receives an error response with the message etag conflict key= when it attempts to save state data and the eTag is not the same value as the eTag in storage.

By default, the Cosmos DB store checks the eTag property of a storage object for equality every time a bot writes to that item, and then updates it to a new unique value after each write. If the eTag property on write doesn't match the eTag in storage, it means another bot or thread changed the data.

For example, let's say you want your bot to edit a saved note, but you don't want your bot to overwrite changes that another instance of the bot has done. If another instance of the bot has made edits, you want the user to edit the version with the latest updates.

First, create a class that implements IStoreItem.

EchoBot.cs

public class Note : IStoreItem
{
    public string Name { get; set; }
    public string Contents { get; set; }
    public string ETag { get; set; }
}

Next, create an initial note by creating a storage object, and add the object to your store.

EchoBot.cs

// create a note for the first time, with a non-null, non-* ETag.
var note = new Note { Name = "Shopping List", Contents = "eggs", ETag = "x" };

var changes = Dictionary<string, object>();
{
    changes.Add("Note", note);
};
await NoteStore.WriteAsync(changes, cancellationToken);

Then, access and update the note later, keeping its eTag that you read from the store.

EchoBot.cs

var note = NoteStore.ReadAsync<Note>("Note").Result?.FirstOrDefault().Value;

if (note != null)
{
    note.Contents += ", bread";
    var changes = new Dictionary<string, object>();
    {
         changes.Add("Note1", note);
    };
    await NoteStore.WriteAsync(changes, cancellationToken);
}

If the note was updated in the store before you write your changes, the call to Write will throw an exception.

To maintain concurrency, always read a property from storage, then modify the property you read, so that the eTag is maintained. If you read user data from the store, the response will contain the eTag property. If you change the data and write updated data to the store, your request should include the eTag property that specifies the same value as you read earlier. However, writing an object with its eTag set to * will allow the write to overwrite any other changes.

Next steps

Now that you know how to read read and write directly from storage, lets take a look at how you can use the state manager to do that for you.