How to handle changes from different containers in Cosmos DB through Change Feed?

Question

I have some amount of containers in Cosmos DB that changes all the time. I need to provide some mechanism for reading all the changes from those containers.

I'm trying to implement builder/factory for Change Feed Processor (CFP). In my case, I have to create CFP instances dynamically for the different container. How I see the solution right now - I need a WebJob/Console Application that listens to the queue. When another application creates a new container in Cosmos DB it also sends a new message to the queue. Message in the queue contains all information for creating new CFP (connection string, collection name, lease container name, etc). The application creates new CFP and runs it in a new thread in the background forever.

Here is the code how I'm creating a new CFP

private void StartNewProcessor()
{
    new List<Task>().Add(Task.Run(async () =>
    {
        var container = Database.GetContainer(ContainerName);
        var lease = Database.GetContainer(LeaseName);

        var changeFeedProcessor = container.GetChangeFeedProcessorBuilder<Item>(ProcessorName, ProcessData)
            .WithLeaseContainer(lease)
            .WithInstanceName(InstanceName)
            .Build();

        await changeFeedProcessor.StartAsync();

        Console.WriteLine($"Change Feed Processor: {ProcessorName} have been started");

        Console.ReadKey(true);

        await changeFeedProcessor.StopAsync();
    }));
}

The problem is that it's a bad approach since there can be 100 and more collections in the future, so I'll need to create 100 extra threads in background. I'm looking for some ideas regarding the architecture application and how to do all that in the right way. It will be great if it is possible to handle changes for all containers in one application.

Answer 1

Unfortunately, there is no way to work around this. Each container will require its own Change Feed Processor to listen for changes. Feel free to raise a feature request to include support for your scenario.

But though you have many threads, most of them would be waiting on network for changes rather than performing any CPU intensive tasks. Based on that, one improvement would be to use a pull-based model for some containers that don't see lots of changes or don't require real time processing.