Plan your Azure Time Series Insights GA environment
This article describes how to plan your Azure Time Series Insights general availability (GA) environment based on your expected ingress rate and your data retention requirements.
Watch this video to learn more about data retention in Azure Time Series Insights and how to plan for it:
To get started with Azure Time Series Insights, it’s best if you know how much data you expect to push by the minute and how long you need to store your data.
For more information about capacity and retention for both Time Series Insights SKUs, see Time Series Insights pricing.
To best plan your Time Series Insights environment for long-term success, consider the following attributes:
- Storage capacity
- Data retention period
- Ingress capacity
- Shaping your events
- Ensuring that you have reference data in place
By default, Time Series Insights retains data based on the amount of storage you provision (units × the amount of storage per unit) and ingress.
You can change the Data retention time setting in your Azure Time Series Insights environment. You can enable up to 400 days of retention.
Azure Time Series Insights has two modes:
- One mode optimizes for the most up-to-date data. It enforces a policy to Purge old data leaving recent data available with the instance. This mode is on, by default.
- The other optimizes data to remain below the configured retention limits. Pause ingress prevents new data from being ingressed when its selected as the Storage limit exceeded behavior.
You can adjust retention and toggle between the two modes on the environment’s configuration page in the Azure portal.
You can configure a maximum of 400 days of data retention in your Azure Time Series Insights GA environment.
Configure data retention
In the Azure portal, select your Time Series Insights environment.
In the Time Series Insights environment pane, under Settings, select Configure.
In the Data retention time (in days) box, enter a value between 1 and 400.
To learn more about how to implement an appropriate data retention policy, see How to configure retention.
The second area to focus on when planning your Time Series Insights environment is ingress capacity. Ingress capacity is a derivative of the per-minute allocation.
From a throttling perspective, an ingressed data packet that has a packet size of 32 KB is treated as 32 events, each 1 KB in size. The maximum allowed event size is 32 KB. Data packets larger than 32 KB are truncated.
The following table summarizes the ingress capacity per unit for each Time Series Insights SKU:
|SKU||Event count per month||Event size per month||Event count per minute||Event size per minute|
|S1||30 million||30 GB||720||720 KB|
|S2||300 million||300 GB||7,200||7,200 KB|
You can increase the capacity of an S1 or S2 SKU to 10 units in a single environment. You can't migrate from an S1 environment to an S2. You can't migrate from an S2 environment to an S1.
For ingress capacity, first determine the total ingress you require on a per-month basis. Next, determine what your per-minute needs are.
Throttling and latency play a role in per-minute capacity. If you have a spike in your data ingress that lasts less than 24 hours, Time Series Insights can "catch up" at an ingress rate of two times the rates listed in the preceding table.
For example, if you have a single S1 SKU, you ingress data at a rate of 720 events per minute, and the data rate spikes for less than one hour at a rate of 1,440 events or less, there's no noticeable latency in your environment. However, if you exceed 1,440 events per minute for more than one hour, you likely will experience latency in data that is visualized and available for query in your environment.
You might not know in advance how much data you expect to push. In this case, you can find data telemetry for Azure IoT Hub and Azure Event Hubs in your Azure portal subscription. The telemetry can help you determine how to provision your environment. Use the Metrics pane in the Azure portal for the respective event source to view its telemetry. If you understand your event source metrics, you can more effectively plan and provision your Time Series Insights environment.
Calculate ingress requirements
To calculate your ingress requirements:
Verify that your ingress capacity is above your average per-minute rate and that your environment is large enough to handle your anticipated ingress equivalent to two times your capacity for less than one hour.
If ingress spikes occur that last for longer than 1 hour, use the spike rate as your average. Provision an environment with the capacity to handle the spike rate.
Mitigate throttling and latency
For information about how to prevent throttling and latency, see Mitigate latency and throttling.
Shape your events
It's important to ensure that the way you send events to Time Series Insights supports the size of the environment you are provisioning. (Conversely, you can map the size of the environment to how many events Time Series Insights reads and the size of each event.) It's also important to think about the attributes that you might want to use to slice and filter by when you query your data.
Review the JSON shaping documentation in Sending events.
Ensure that you have reference data
A reference dataset is a collection of items that augment the events from your event source. The Time Series Insights ingress engine joins each event from your event source with the corresponding data row in your reference dataset. The augmented event is then available for query. The join is based on the Primary Key columns that are defined in your reference dataset.
Reference data isn't joined retroactively. Only current and future ingress data is matched and joined to the reference dataset after it's configured and uploaded. If you plan to send a large amount of historical data to Time Series Insights and don't first upload or create reference data in Time Series Insights, you might have to redo your work (hint: not fun).
To learn more about how to create, upload, and manage your reference data in Time Series Insights, see our Reference dataset documentation.
Business disaster recovery
This section describes features of Azure Time Series Insights that keep apps and services running, even if a disaster occurs (known as business disaster recovery).
As an Azure service, Time Series Insights provides certain high availability features by using redundancies at the Azure region level. For example, Azure supports disaster recovery capabilities through Azure's cross-region availability feature.
Additional high-availability features provided through Azure (and also available to any Time Series Insights instance) include:
- Failover: Azure provides geo-replication and load balancing.
- Data restoration and storage recovery: Azure provides several options to preserve and recover data.
- Site recovery: Azure provides site recovery features through Azure Site Recovery.
- Azure Backup: Azure Backup supports both on-premises and in-cloud backup of Azure VM's.
Make sure you enable the relevant Azure features to provide global, cross-region high availability for your devices and users.
If Azure is configured to enable cross-region availability, no additional cross-region availability configuration is required in Azure Time Series Insights.
IoT and event hubs
Some Azure IoT services also include built-in business disaster recovery features:
- IoT Hub high-availability disaster recovery, including intra-region redundancy
- Event Hubs policies
- Azure Storage redundancy
Integrating Time Series Insights with the other services provides additional disaster recovery opportunities. For example, telemetry sent to your event hub might be persisted to a backup Azure Blob storage database.
Time Series Insights
There are several ways to keep your Time Series Insights data, apps, and services running, even if they're disrupted.
However, you might determine that a complete backup copy of your Azure Time Series environment also is required, for the following purposes:
- As a failover instance specifically for Time Series Insights to redirect data and traffic to
- To preserve data and auditing information
In general, the best way to duplicate a Time Series Insights environment is to create a second Time Series Insights environment in a backup Azure region. Events are also sent to this secondary environment from your primary event source. Make sure that you use a second, dedicated consumer group. Follow that source's business disaster recovery guidelines, as described earlier.
To create a duplicate environment:
- Create an environment in a second region. For more information, see Create a new Time Series Insights environment in the Azure portal.
- Create a second dedicated consumer group for your event source.
- Connect that event source to the new environment. Make sure that you designate the second, dedicated consumer group.
- Review the Time Series Insights IoT Hub and Event Hubs documentation.
If an event occurs:
- If your primary region is affected during a disaster incident, reroute operations to the backup Time Series Insights environment.
- Use your second region to back up and recover all Time Series Insights telemetry and query data.
If a failover occurs:
- A delay might also occur.
- A momentary spike in message processing might occur, as operations are rerouted.
For more information, see Mitigate latency in Time Series Insights.
Get started by creating a new Time Series Insights environment in the Azure portal.
Learn how to add an Event Hubs event source to Time Series Insights.
Read about how to configure an IoT Hub event source.