Data collection, retention and storage in Application Insights
When you install Azure Application Insights SDK in your app, it sends telemetry about your app to the Cloud. Naturally, responsible developers want to know exactly what data is sent, what happens to the data, and how they can keep control of it. In particular, could sensitive data be sent, where is it stored, and how secure is it?
First, the short answer:
- The standard telemetry modules that run "out of the box" are unlikely to send sensitive data to the service. The telemetry is concerned with load, performance and usage metrics, exception reports, and other diagnostic data. The main user data visible in the diagnostic reports are URLs; but your app shouldn't in any case put sensitive data in plain text in a URL.
- You can write code that sends additional custom telemetry to help you with diagnostics and monitoring usage. (This extensibility is a great feature of Application Insights.) It would be possible, by mistake, to write this code so that it includes personal and other sensitive data. If your application works with such data, you should apply a thorough review processes to all the code you write.
- While developing and testing your app, it's easy to inspect what's being sent by the SDK. The data appears in the debugging output windows of the IDE and browser.
- The data is held in Microsoft Azure servers in the USA or Europe. (But your app can run anywhere.) Azure has strong security processes and meets a broad range of compliance standards. Only you and your designated team have access to your data. Microsoft staff can have restricted access to it only under specific limited circumstances with your knowledge. It's encrypted in transit, though not in the servers.
The rest of this article elaborates more fully on these answers. It's designed to be self-contained, so that you can show it to colleagues who aren't part of your immediate team.
What is Application Insights?
Azure Application Insights is a service provided by Microsoft that helps you improve the performance and usability of your live application. It monitors your application all the time it's running, both during testing and after you've published or deployed it. Application Insights creates charts and tables that show you, for example, what times of day you get most users, how responsive the app is, and how well it is served by any external services that it depends on. If there are crashes, failures or performance issues, you can search through the telemetry data in detail to diagnose the cause. And the service will send you emails if there are any changes in the availability and performance of your app.
In order to get this functionality, you install an Application Insights SDK in your application, which becomes part of its code. When your app is running, the SDK monitors its operation and sends telemetry to the Application Insights service. This is a cloud service hosted by Microsoft Azure. (But Application Insights works for any applications, not just those that are hosted in Azure.)
The Application Insights service stores and analyzes the telemetry. To see the analysis or search through the stored telemetry, you sign in to your Azure account and open the Application Insights resource for your application. You can also share access to the data with other members of your team, or with specified Azure subscribers.
You can have data exported from the Application Insights service, for example to a database or to external tools. You provide each tool with a special key that you obtain from the service. The key can be revoked if necessary.
Application Insights SDKs are available for a range of application types: web services hosted in your own J2EE or ASP.NET servers, or in Azure; web clients - that is, the code running in a web page; desktop apps and services; device apps such as Windows Phone, iOS, and Android. They all send telemetry to the same service.
What data does it collect?
How is the data is collected?
There are three sources of data:
The SDK, which you integrate with your app either in development or at run time. There are different SDKs for different application types. There's also an SDK for web pages, which loads into the end-user's browser along with the page.
- Each SDK has a number of modules, which use different techniques to collect different types of telemetry.
- If you install the SDK in development, you can use its API to send your own telemetry, in addition to the standard modules. This custom telemetry can include any data you want to send.
- In some web servers, there are also agents that run alongside the app and send telemetry about CPU, memory, and network occupancy. For example, Azure VMs, Docker hosts, and J2EE servers can have such agents.
- Availability tests are processes run by Microsoft that send requests to your web app at regular intervals. The results are sent to the Application Insights service.
What kinds of data are collected?
The main categories are:
- Web server telemetry - HTTP requests. Uri, time taken to process the request, response code, client IP address. Session id.
- Web pages - Page, user and session counts. Page load times. Exceptions. Ajax calls.
- Performance counters - Memory, CPU, IO, Network occupancy.
- Client and server context - OS, locale, device type, browser, screen resolution.
- Exceptions and crashes - stack dumps, build id, CPU type.
- Dependencies - calls to external services such as REST, SQL, AJAX. URI or connection string, duration, success, command.
- Availability tests - duration of test and steps, responses.
- Trace logs and custom telemetry - anything you code into your logs or telemetry.
How can I verify what's being collected?
If you're developing the app using Visual Studio, run the app in debug mode (F5). The telemetry appears in the Output window. From there, you can copy it and format it as JSON for easy inspection.
There's also a more readable view in the Diagnostics window.
For web pages, open your browser's debugging window.
Can I write code to filter the telemetry before it is sent?
This would be possible by writing a telemetry processor plugin.
How long is the data kept?
Raw data points (that is, items that you can query in Analytics and inspect in Search) are kept for up to 90 days. If you need to keep data longer than that, you can use continuous export to copy it to a storage account.
Aggregated data (that is, counts, averages and other statistical data that you see in Metric Explorer) are retained at a grain of 1 minute for 90 days.
Who can access the data?
The data is visible to you and, if you have an organization account, your team members.
It can be exported by you and your team members and could be copied to other locations and passed on to other people.
What does Microsoft do with the information my app sends to Application Insights?
Microsoft uses the data only in order to provide the service to you.
Where is the data held?
- In the USA, Europe or Southeast Asia. You can select the location when you create a new Application Insights resource.
Does that mean my app has to be hosted in the USA, Europe or Southeast Asia?
- No. Your application can run anywhere, either in your own on-premises hosts or in the Cloud.
How secure is my data?
Application Insights is an Azure Service. Security policies are described in the Azure Security, Privacy, and Compliance white paper.
The data is stored in Microsoft Azure servers. For accounts in the Azure Portal, account restrictions are described in the Azure Security, Privacy, and Compliance document.
Access to your data by Microsoft personnel is restricted. We access your data only with your permission and if it is necessary to support your use of Application Insights.
Data in aggregate across all our customers' applications (such as data rates and average size of traces) is used to improve Application Insights.
Could someone else's telemetry interfere with my Application Insights data?
They could send additional telemetry to your account by using the instrumentation key, which can be found in the code of your web pages. With enough additional data, your metrics would not correctly represent your app's performance and usage.
If you share code with other projects, remember to remove your instrumentation key.
Is the data encrypted?
Not inside the servers at present.
All data is encrypted as it moves between data centers.
Is the data encrypted in transit from my application to Application Insights servers?
Yes, we use https to send data to the portal from nearly all SDKs, including web servers, devices and HTTPS web pages. The only exception is data sent from plain HTTP web pages.
How do I send data to Application Insights using TLS 1.2?
To insure the security of data in transit to the Application Insights endpoints, we strongly encourage customers to configure their application to use at least Transport Layer Security (TLS) 1.2. Older versions of TLS/Secure Sockets Layer (SSL) have been found to be vulnerable and while they still currently work to allow backwards compatibility, they are not recommended, and the industry is quickly moving to abandon support for these older protocols.
The PCI Security Standards Council has set a deadline of June 30th, 2018 to disable older versions of TLS/SSL and upgrade to more secure protocols. Once Azure drops legacy support, if your application/clients cannot communicate over at least TLS 1.2 you would not be able to send data to Application Insights. The approach you take to test and validate your application's TLS support will vary depending on the operating system/platform as well as the language/framework your application uses.
We do not recommend explicitly setting your application to only use TLS 1.2 unless absolutely necessary as this can break platform level security features that allow you to automatically detect and take advantage of newer more secure protocols as they become available such as TLS 1.3. We recommend performing a thorough audit of your application's code to check for hardcoding of specific TLS/SSL versions.
Platform/Language specific guidance
|Azure App Services||Supported, configuration may be required.||Support was announced in April 2018. Read the announcement for configuration details.|
|Azure Function Apps||Supported, configuration may be required.||Support was announced in April 2018. Read the announcement for configuration details.|
|.NET||Supported, configuration varies by version.||For detailed configuration info for .NET 4.7 and earlier versions refer to these instructions.|
|Status Monitor||Supported, configuration required||Status Monitor relies on OS Configuration + .NET Configuration to support TLS 1.2.|
|Node.js||Supported, in v10.5.0, configuration may be required.||Use the official Node.js TLS/SSL documentation for any application specific configuration.|
|Java||Supported, JDK support for TLS 1.2 was added in JDK 6 update 121 and JDK 7.||JDK 8 uses TLS 1.2 by default.|
|Linux||Linux distributions tend to rely on OpenSSL for TLS 1.2 support.||Check the OpenSSL Changelog to confirm your version of OpenSSL is supported.|
|Windows 8.0 - 10||Supported, and enabled by default.||To confirm that you are still using the default settings.|
|Windows Server 2012 - 2016||Supported, and enabled by default.||To confirm that you are still using the default settings|
|Windows 7 SP1 and Windows Server 2008 R2 SP1||Supported, but not enabled by default.||See the Transport Layer Security (TLS) registry settings page for details on how to enable.|
|Windows Server 2008 SP2||Support for TLS 1.2 requires an update.||See Update to add support for TLS 1.2 in Windows Server 2008 SP2.|
|Windows Vista||Not Supported.||N/A|
Check what version of OpenSSL your Linux distribution is running
To check what version of OpenSSL you have installed, open the terminal and run:
openssl version -a
Run a test TLS 1.2 transaction on Linux
To run a basic preliminary test to see if your Linux system can communicate over TLS 1.2. Open the terminal and run:
openssl s_client -connect bing.com:443 -tls1_2
Personal data stored in Application Insights
Our Application Insights personal data article discusses this issue in-depth.
Can my users turn off Application Insights?
Not directly. We don't provide a switch that your users can operate to turn off Application Insights.
However, you can implement such a feature in your application. All the SDKs include an API setting that turns off telemetry collection.
Data sent by Application Insights
The SDKs vary between platforms, and there are several components that you can install. (Refer to Application Insights - overview.) Each component sends different data.
Classes of data sent in different scenarios
|Your action||Data classes collected (see next table)|
|Add Application Insights SDK to a .NET web project||ServerContext
|Install Status Monitor on IIS||Dependencies
|Add Application Insights SDK to a Java web app||ServerContext
|Define default properties||Properties on all standard and custom events|
|Call TrackMetric||Numeric values
|Call Track*||Event name
|SDK can't collect data. For example:
- can't access perf counters
- exception in telemetry initializer
For SDKs for other platforms, see their documents.
The classes of collected data
|Collected data class||Includes (not an exhaustive list)|
|Properties||Any data - determined by your code|
|DeviceContext||Id, IP, Locale, Device model, network, network type, OEM name, screen resolution, Role Instance, Role Name, Device Type|
|ClientContext||OS, locale, language, network, window resolution|
|ServerContext||Machine name, locale, OS, device, user session, user context, operation|
|Inferred||geo location from IP address, timestamp, OS, browser|
|Metrics||Metric name and value|
|Events||Event name and value|
|PageViews||URL and page name or screen name|
|Client perf||URL/page name, browser load time|
|Ajax||HTTP calls from web page to server|
|Requests||URL, duration, response code|
|Dependencies||Type(SQL, HTTP, ...), connection string or URI, sync/async, duration, success, SQL statement (with Status Monitor)|
|Exceptions||Type, message, call stacks, source file and line number, thread id|
|Crashes||Process id, parent process id, crash thread id; application patch, id, build; exception type, address, reason; obfuscated symbols and registers, binary start and end addresses, binary name and path, cpu type|
|Trace||Message and severity level|
|Perf counters||Processor time, available memory, request rate, exception rate, process private bytes, IO rate, request duration, request queue length|
|Availability||Web test response code, duration of each test step, test name, timestamp, success, response time, test location|
|SDK diagnostics||Trace message or Exception|
This product includes GeoLite2 data created by MaxMind, available from http://www.maxmind.com.