question

RuneLarsen-9318 avatar image
0 Votes"
RuneLarsen-9318 asked ·

How to order documents in a Cosmos DB collection by the time inserted

We are building a large platform where we want to expose events to many subscribers.

We have decided we want to expose the events through a REST-API to be able to support all our use cases. We are not able to use event hub or Kafka since we need millions of topics. We also need to authorize access to the different events based on type of event and which person/business is related to.

One option is to use Cosmos DB as storage, but I am not able to see how we could get the order of the elements correctly. I understand that _ts property contains the epoch time since last changed, but that is only in seconds and would cause us to have hundreds of events at the same time.

Is there a way? I guess we could use the change feed to trigger something but does the change feed have some index we could set back on the document?

azure-cosmos-db
10 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

FelixP-2774 avatar image
0 Votes"
FelixP-2774 answered ·

Hi - just starting answer questions on this platform so bear with me.

Are you looking to order event reads or writes? It might be worth considering what consistency level you need, as this could affect the ordering you see.

This page of docs might be helpful:

If you need stricter consistency guarantees than the ones provided by session consistency and single-digit-millisecond latency for writes, it is recommended that you use bounded staleness consistency level.

https://docs.microsoft.com/en-us/azure/cosmos-db/consistency-levels-choosing

The cosmos dB change feed might also be what you’re looking for:

T

here is guaranteed order in the change feed within a partition key value but not across partition key values. You should select a partition key that gives you a meaningful order guarantee.

https://docs.microsoft.com/en-us/azure/cosmos-db/change-feed-design-patterns

Please let me know if this helps.

10 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MarkBrownMSFT avatar image
0 Votes"
MarkBrownMSFT answered ·

Cosmos only has second granularity with _ts. If you need a finer grain you will need to create your own timestamp property. Data is always stored in the order it is committed. However I would look to use a composite index with the partition key and order by on your timestamp property as queries will be run from this index and will be less expensive than without where you use order by in your queries.

· 1
10 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

This would be a platform with potensial thousands of applications writing the a API to publish events. We will have a created timestamp on the event when, but we cant trust that timestamp for the ordering when subscribers reads from the event. They could risk that a earlier event is inserted after a later event. So we need something for when it is written to cosmos in addition. We could of course write a timestamp when the event component receives, but that again would be multiple pods that also could cause that earlier event is written after a later event.

0 Votes 0 ·