Is it possible to use Azure Time Series Insight for historical data if I don't want to touch IoT hub ? If not, what would be the recommended way to store and query existing time-series data in Azure?
Is it possible to use Azure Time Series Insight for historical data if I don't want to touch IoT hub ? If not, what would be the recommended way to store and query existing time-series data in Azure?
Hello @basalganglia,
Azure Time Series Insights can be seen as a data lake with data stored in Apache Parquet format and with a visual explorer on top of it.
It is capable of storing years of data and makes it queriable to you using the explorer (or via API) in seconds.
Yes, you can use Azure Time Series Insights with almost any data source (via an EventHub as event source) which matches a qualified Time-series-ID and optionally an event-source-specific timestamp property name.
There is even a usecase describing ingesting data from eg Stream Analytics of Azure Functions (thus the warm path output).
Time Series Insights (gen2) has its own storage, based on a regular Azure Storage data lake (gen 2). So you do not have to store data yourself, this is done in that data lake. Technically, you even have access to the raw Apache Parquet so you could do additional querying outside TSI. Keep in mind that data is mended to be readonly, you can not alter it or delete it.
You can start querying using the Visual Explorer and save and share certain views on the data. Of course, this is done by hand.
If you want to query via some programming, most of the time, the TSI Rest API is powerful already for custom queries.
There are some small costs involved, every time you access this 'cold store'. If you query really a lot of time, this can add up. TSI also offers a 'warm store' (see it as data kept in memory, the data will always arrive in the cold store). You pay upfront for this but querying is free (up to 31 days).
Personally, I like it a lot because TSI gives me direct visual insights into my data lake thus the long-term stored raw data.
An introduction to TSI can be found here.
--If a reply is helpful, please Upvote and Accept it as the answer. This will help others with the same question--
Thank you for the thorough answer! However, as I wrote in the original question, I was wondering can I use the TSI with raw data without IoT hub, Event Hub or any other hub or Azure streaming service (I have terabytes of raw historical time-series to explore) which I take it is not possible. So this clarified it for me, I'm looking at influxdb next!
Thanks again!
Hello @tyvitumake-0199,
in that case, you need to ingest the historical data, via an eventhub (with data coming from logic near the database and perhaps an Azure Function in between).
Based on your case, also check out other services first before you make a decision.
Another solution is Azure Data Explorer. This offers even more data analytics capabilities.
Or check out Azure Data Bricks if you are interested in Open Source solutions.
5 people are following this question.
Is there a limit to how many types there can be in a TSI environment?
Granting data access to groups or multiple users for Time Series Insights
Time Series Insight API with MSAL and Angular 9
Application Insights Snippet based setup Error Mismatched anonymous define() module
How to group events by an event variable/property in Aggregate Series API (Gen2) ?