Build your first Microsoft Graph Data Connect application
Microsoft Graph Data Connect augments Microsoft Graph’s transactional model with an intelligent way to access rich data at scale. The data covers how workers communicate, collaborate, and manage their time across all the applications and services in Microsoft 365. Ideal for big data and machine learning, Data Connect allows you to develop applications for analytics, intelligence, and business process optimization by extending Microsoft 365 data into Azure. By integrating in this way, you'll be able to take advantage of the vast suite of compute, storage in Azure while staying compliant with industry standards and keeping your data secure.
Microsoft Graph Data Connect uses Azure Data Factory to copy Microsoft 365 data to your application’s storage at configurable intervals. It also provides a set of tools to streamline the delivery of this data to Microsoft Azure, letting you access the most applicable development and hosting tools available. Data Connect also grants a more granular control and consent model: you can manage data, see who is accessing it, and request specific properties of an entity. This enhances the Microsoft Graph model, which grants or denies applications access to entire entities.
You can use Data Connect to enable machine learning scenarios for your organization.. In these scenarios, you can create applications that provide valuable information to your stakeholders, train machine learning models, and even perform forecasting based on large amounts of acquired data.
Get started
In this tutorial, you will be creating your first Microsoft Graph Data Connect application. Exciting, right? We think so too! To get started, you'll need to set up a few things first.
Prerequisites
To complete this lab, you will need the following subscriptions or licenses.
Microsoft 365 tenancy
- If you do not have one, you get one (for free) by signing up to the Microsoft 365 Developer Program.
- Multiple Microsoft 365 users with emails sent and received.
- Access to at least two accounts that meet the following requirements:
- Must have the Global administrator role assigned.
- Must have access to the Microsoft 365 Admin Center.
Microsoft Azure subscription
- If you do not have one, you can get one (for free) in our Azure website.
- The account used to sign in must have the Global administrator role granted to it.
- The Azure subscription must be in the same tenant as the Microsoft 365 tenant, as Graph Data Connect will only export data to an Azure subscription in the same tenant, not across tenants.
- Your Microsoft 365 and Azure tenants must be in the same Microsoft Azure Active Directory tenancy.
Make sure you have Visual Studio installed on your development machine.
Note
The screenshots and examples used in this lab are from an Microsoft 365 test tenant with sample email from test users. You can use your own Microsoft 365 tenant to perform the same steps. No data is written to Microsoft 365. A copy of email data is extracted from all users in an Microsoft 365 tenant and copied to an Azure Blob Storage account that you maintain control over who has access to the data within the Azure Blob Storage.
Set up your Microsoft 365 tenant and enable Microsoft Graph Data Connect
Prior to using Microsoft Graph Data Connect for the first time, you need to configure your Microsoft 365 tenant. This involves turning on the service and configuring a security group with permissions to approve data extraction requests.
Grant Azure AD users the Global administrator role
In this step, you will ensure that two users in your Microsoft 365 tenant have the Global administrator role enabled.
Configure Microsoft Graph Data Connect consent request approver group
In this step, you will setup your Microsoft 365 tenant to enable usage of Microsoft Graph Data Connect.
Open a browser and go to your Microsoft 365 Admin Portal.
On the sidebar navigation, select Active Groups.
Select the Add a group button.
Use the following to create the new mail-enabled security group and select the Add button.
- Type: Mail-enabled security
- Name: Consent Request Approvers
- Email Prefix: consentrequestapprovers
It can take up to an hour before the newly created group shows up in the list. When the group has been created, select it.
Go to the Active groups option again and search for the group you just created.
Select the group and in the Members tab, select View all and manage members.
Add the two users that you enabled the Global administrator role to this new group.
Enable Microsoft Graph Data Connect in your Microsoft 365 tenant
In this step, you will enable the Microsoft Graph Data Connect service on your Microsoft 365 tenant.
While you are still signed in to the Microsoft 365 Admin Portal, select the Settings > Org settings menu item.
Select the Microsoft Graph Data Connect service.
Select the checkbox that says turn Microsoft Graph Data Connect on or off for your entire organization to enable Data Connect.
Enter Consent Request Approvers (or the name of the group you created previously) in the group of users to make approval decisions and select Save.
Set up your Azure Active Directory app registration
In this exercise you will create, run, and approve an Azure Data Factory pipeline to extract data from Microsoft 365 to an Azure Storage Blob for additional processing.
Create a Microsoft Azure Active Directory application registration
The first step is to create an Azure AD application that will be used as the security principal to run the data extraction process.
Open a browser and go to your Azure Portal.
Sign in using an account with Global administrator rights to your Azure and Microsoft 365 tenants.
On the sidebar navigation, select Azure Active Directory (Azure AD).
On the Azure AD Overview page, select App registrations from the Manage section of the menu.
Select the New registration button.
Use the following values to create a new Azure AD application and select Register.
- Name: Microsoft Graph Data Connect Data Transfer
- Supported account types: Accounts in this organizational directory only.
- Redirect URI: Leave the default values.
Locate the Application (client) ID and copy it as you will need it later in this tutorial. This will be referred to as the service principal ID.
Locate the Directory (tenant) ID and copy it as you will need it later in this tutorial. This will be referred to as the tenant ID.
On the sidebar navigation, select Certificates and secrets under Manage.
Select the New client secret button. Set Description to any name, set Expires to any value in the dropdown and choose Add.
- After the client secret is created, make sure you save the Value somewhere safe, as it will no longer be available later, and you will need to create a new one.
- This will be referenced as the service principal key.
On the sidebar navigation for the application, select Owners.
Verify that your account is listed as an owner for the application. If it isn't listed as an owner, add it.
Set up your Azure Storage resource
In this step you will create an Azure Storage account where Microsoft Graph data connect will store the data extracted from Microsoft 365 for further processing.
Open a browser and go to your Azure Portal.
Sign in using an account with Global administrator rights to your Azure and Microsoft 365 tenants.
On the sidebar navigation, select Create a resource.
Find the Storage Account resource type and use the following values to create it, then select Review + create.
- Subscription: select your Azure subscription
- Resource group: GraphDataConnect (or select an existing resource group)
- Storage account name: mgdcm365datastore
- Region: pick an Azure region in the same region as your Microsoft 365 region
- Performance: Standard
- Redundancy: Geo-redundant storage (GRS)
- Advanced tab:
- Access tier: Hot
Review that the settings match those shown in the previous step and select Create.
After the Azure Storage account has been created, grant the Azure AD application previously created the proper access to it.
Select the Azure Storage account.
On the sidebar menu, select Access control (IAM).
Select the Add button in the Add a role assignment block.
Use the following values to find the application you previously selected to grant it the Storage Blob Data Contributor role, then select Save.
- Role: Storage Blob Data Contributor
- Assign access to: User, group or service principal
- Select: Microsoft Graph data connect Data Transfer (the name of the Azure AD application you created previously)
Create a new container in the mgdcm365datastore Azure Storage account.
Select the mgdcm365datastore Azure Storage account.
On the sidebar menu, select Containers under the Blob service section.
Select the +Container button at the top of the page and use the following values and then select Create.
- Name: m365mails
- Public access level: Private (no anonymous access)
Set up your Azure Data Factory resource
The next step is to use the Azure Data Factory to create a pipeline to extract the data from Microsoft 365 to the Azure Storage account using Microsoft Graph data connect.
Create an Azure Data Factory pipeline
Open a browser and go to your Azure Portal.
Sign in using an account with Global administrator rights to your Azure and Microsoft 365 tenants.
On the sidebar navigation, select Create a resource.
Find the Data Factory resource type and use the following values to create it, then select Create.
Subscription: select your Azure subscription
Resource group: GraphDataConnect
Region: pick an Azure region in the same region as your Microsoft 365 region
Name: dfM365toBlobStorage
Version: V2
In the Git configuration tab, make sure you either configure Git or select the option Configure Git later.
After the Azure Data Factory resource is created, select the Author and Monitor tile to launch the Azure Data Factory full screen editor.
Switch from the Overview to the Manage experience by selecting it from the left-hand navigation.
By default, the Azure Data Factory will use an integration runtime that is auto-resolving the region. Because Data Connect requires that your source and destination, and integration runtime to exist in the same Microsoft 365 region, we recommend that you create a new integration runtime with a fixed region.
Select Integration runtimes > New.
Select Azure, Self-Hosted and select Continue.
Select Azure for network environment and select Continue.
Use the following details to complete the form on the final screen and then select Create.
- Name: name of your integration runtime
- Description: enter a description
- Region: select the region that matches your Microsoft 365 region
- Virtual network configuration (preview): Disabled
Switch from the Manage to the Author experience by selecting it from the left-hand navigation.
Create a new pipeline by selecting the plus icon, then pipeline.
Drag the Copy Data activity from the Move and Transform section onto the design surface.
Select the activity in the designer.
Select the General tab and give it a name and description.
- Name: CopyFromM365toBlobStorage
- Description: A description you want.
In the activity editor pane below the designer, select the Source tab, then select New.
Locate the dataset Office 365, select it and then select the Continue button.
The designer will update the Source tab with the Microsoft 365 connector settings.
Select the Open option next to the Source dataset field.
In the table settings, select the Connection tab, then the New button.
In the dialog that appears, enter the previously created Azure AD application's Application ID and Secret ID in the Service principal ID and Service principal key fields respectively, then select Create.
Select the integration runtime you previously created in the Connect via integration runtime dropdown.
After creating the Microsoft 365 connection, for the Table field, select BasicDataSet_v0.Message_v0.
Switch from Office365Table to Pipeline > Source. Use the following values for the Date filter.
- Column name: CreatedDateTime
- Start time (UTC): select a date sometime prior to the current date
- End time (UTC): select the current date
- Select Import schema in the Output columns section.
Select the Copy data activity in the pipeline tab, then select the Sink tab.
- Select the New button, select Azure Blob Storage, and then select the Continue button.
- Select Binary as the format for the data and then select the Continue button.
- Give the dataset the name M365JsonFile and create new linked service if it does not exist already.
In the table select the Connection tab, then select New.
Set the following values in the dialog, then select Finish.
- Authentication method: Service principal
- Azure subscription: Select all
- Storage account name: mgdcm365datastore
- This is the storage account created earlier in this exercise.
- Tenant: enter the ID of your Azure tenant
- Service principal ID: enter the ID of the Azure AD application you previously created
- Service principal key: enter the hashed key of the Azure AD application you previously created
Next to the File path field, select Browse.
Select the name of the storage container you created previously.
With the pipeline created, select the Validate All button at the top of the designer.
After validating (and fixing any issues that were found), select the Publish All button at the top of the designer.
Run the Azure Data Factory Pipeline
With the pipeline created, now it is time to run it.
Note
It can take several minutes for the consent request to appear and it is not uncommon for the entire process (start, requesting consent and after approving the consent completing the pipeline run) to take over 40 minutes.
In the Azure Data Factory designer, with the pipeline open, select Add trigger > Trigger Now.
After starting the job, from the sidebar menu, select Monitor to view current running jobs.
On the left-side navigation bar, locate the Pipeline runs tab and select it. Select the pipeline under the Pipeline name column to view the Activity runs. This pipeline will show as In progress.
After you are in the Activity runs view, go to the Activity runs section, which is located in the bottom side of the page.
Hover over the Activity name and select the goggles option. This will bring up the Details tab.
In the Details screen, look for the status of the pipeline activity as highlighted in the following image. In this case you can see it is in a state of RequestingConsent.
At this point, the activity run is internally paused until someone manually approves the consent request via the Microsoft 365 admin center or via PowerShell.
Monitor data consent requests with Microsoft 365 Admin Center and PowerShell
A Microsoft 365 administrator has the ability to approve or deny consent requests. This can be done via the Microsoft 365 Admin Center or programmatically via PowerShell.
Approve consent requests
Open a browser and go to your Microsoft 365 Admin Portal.
To approve or deny consent requests, go to Privileged Access.
Select a pending Data Access Request.
In the Data Access Request call out, select the Approve button.
Verify extracted data from Microsoft 365 to Azure Storage Blob
Open a browser and go to your Azure Portal.
Sign in using an account with Global administrator rights to your Azure and Microsoft 365 tenants.
On the sidebar navigation, select the All resources menu item.
In the list of resources, select the Azure Storage account you created previously in this tutorial.
On the sidebar navigation menu, select Blobs from the Azure Storage account blade.
Select the container created previously in this tutorial that you configured the Azure Data Factory pipeline as the sink for the extracted data. You should see data in this container now.
Create your ASP.NET web application to process the exported data
In this section we will be building your first ASP.NET project application for to process the Microsoft Graph Data Connect data that was exported.
Create a new ASP.NET project
Open Visual Studio and select File > New > Project.
In the New Project dialog, do the following.
Search ASP.NET Web Application in the search box and select the ASP.NET Web Application (.NET Framework) option.
Click on Next.
Enter EmailMetrics for the name of the project.
Select .NET Framework 4.7.2 for the framework option.
Select Create.
Important
Ensure that you enter the exact same name for the Visual Studio Project that is specified in this quick start instructions. The Visual Studio Project name becomes part of the namespace in the code. The code inside these instructions depends on the namespace matching the Visual Studio Project name specified in these instructions. If you use a different project name the code will not compile unless you adjust all the namespaces to match the Visual Studio Project name you enter when you create the project.
- In the new ASP.NET Web Application project dialog, select MVC.
- Select Create.
Add and configure your Azure Storage as a Connected Service
In the Solution Explorer tool window, right-click the Connected Services node and select Add Connected Service.
On the Connected Services dialog, select the green + sign which is located in the upper-right corner of the dialog.
In the Add dependency dialog, select Azure Storage and select Next.
In the Azure Storage dialog, select the subscription and storage account where you exported the data in the previous exercise, select Next.
Provide the Azure Storage connection a name of AzureStorageConnectionString and select Next.
Select Finish.
Create a new model class that will be used to store the email metrics
In the Solution Explorer tool window, right-click the Models folder and select Add > Class.
In the Add New Item dialog, select Class, set the name of the file to EmailMetric.cs and select Add.
Add the following code to the class EmailMetric you just created.
public string Email; public double RecipientsToEmail;
Create a new controller that will calculate and display the results
Right-click the Controllers folder and select Add > Controller.
In the Add Scaffold dialog, select MVC 5 Controller - Empty and select Add.
When prompted, name the controller EmailMetricsController and select OK.
Add the following using statements after the existing using statements at the top of the file containing the EmailMetricsController class.
using System.Collections.Generic; using System.Configuration; using System.IO; using System.Linq; using System.Threading.Tasks; using System.Web.Mvc; using Azure.Storage.Blobs; using Azure.Storage.Blobs.Models; using Newtonsoft.Json.Linq;Add the following code to the EmailMetricsController class. These will be used to connect to the Azure Storage Account that contains the exported data.
private const string connectionStringName = "AzureStorageConnectionString"; private const string emailBlobName = "m365mails";Add the following method to the EmailMetricsController class. This will process an Azure Blob and update a collection representing the email accounts and how many recipients there were combined across all emails found for the extracted accounts.
private async Task ProcessBlobEmails(List<Models.EmailMetric> emailMetrics, BlobClient emailBlob) { using (var stream = new MemoryStream()) { var response = await emailBlob.DownloadToAsync(stream); var pos = stream.Seek(0, SeekOrigin.Begin); using (var reader = new StreamReader(stream)) { string line; while ((line = reader.ReadLine()) != null) { var jsonObj = JObject.Parse(line); // extract sender var sender = jsonObj.SelectToken("Sender.EmailAddress.Address")?.ToString(); // No sender - skip this one if (string.IsNullOrEmpty(sender)) continue; // extract and count up recipients var totalRecipients = 0; totalRecipients += jsonObj.SelectToken("ToRecipients")?.Children().Count() ?? 0; totalRecipients += jsonObj.SelectToken("CcRecipients")?.Children().Count() ?? 0; totalRecipients += jsonObj.SelectToken("BccRecipients")?.Children().Count() ?? 0; var emailMetric = new Models.EmailMetric(); emailMetric.Email = sender; emailMetric.RecipientsToEmail = totalRecipients; // if already have this sender... var existingMetric = emailMetrics.FirstOrDefault(metric => metric.Email == emailMetric.Email); if (existingMetric != null) { existingMetric.RecipientsToEmail += emailMetric.RecipientsToEmail; } else { emailMetrics.Add(emailMetric); } } } } }Add the following method to the EmailMetricsController class. This will enumerate through all blobs in the specified Azure Storage account's specified container and send each one to
ProcessBlobEmails()method added in the last step.private async Task<List<Models.EmailMetric>> ProcessBlobFiles() { var emailMetrics = new List<Models.EmailMetric>(); var connectionString = ConfigurationManager.ConnectionStrings[connectionStringName]; // Connect to the storage account var containerClient = new BlobContainerClient(connectionString.ConnectionString, emailBlobName); foreach (var blob in containerClient.GetBlobs()) { if (blob.Properties.BlobType == BlobType.Block && // Don't process blobs in the metadata folder !blob.Name.StartsWith("metadata/")) { var blobClient = containerClient.GetBlobClient(blob.Name); await ProcessBlobEmails(emailMetrics, blobClient); } } return emailMetrics; }Add the following action to the EmailMetricsController that will use the methods added this class to process the emails and send the results to the view.
[HttpPost, ActionName("ShowMetrics")] [ValidateAntiForgeryToken] public async Task<ActionResult> ShowMetrics() { var emailMetrics = await ProcessBlobFiles(); return View(emailMetrics); }
Create a new view for the EmailMetrics index action
In the Solution Explorer tool window, right-click the Views > EmailMetrics folder and select Add > View.
In the Add New Scaffolded Item dialog box, select MVC 5 View, then select Add.
In the Add View dialog, set the View name to Index, leave the remaining input controls to their default values, and select Add.
Update the markup in the new Views > EmailMetrics > Index.cshtml to the following. This will add a form with a single button that will submit an HTTP POST to the custom controller action added in the last step.
@{ ViewBag.Title = "Index"; } <h2>Email Metrics</h2>This application will look at the email data for emails extracted to the Azure Blob Storage account and display the total number of recipients from each sender.
@using (Html.BeginForm("ShowMetrics", "EmailMetrics", FormMethod.Post)) { @Html.AntiForgeryToken() <div> <button type="submit">View email metrics</button> </div> <div> <em>Please be patient as this can take a few moments to calculate depending on the size of the exported data...</em> </div> }
Create a new view for the EmailMetrics ShowMetrics action
In the Solution Explorer tool window, right-click the Views > EmailMetrics folder and select Add > View.
In the Add View dialog, set the following values and leave the remaining input controls to their default values and select Add.
View name: ShowMetrics
Template: List
Model class: EmailMetric (EmailMetric.Models)
Tip
In case you can't see the EmailMetric model in the dropdown box, please build the solution.
Update the markup in the new Views > EmailMetrics > ShowMetrics.cshtml to the following. This will display the results of the calculations.
@model IEnumerable<EmailMetrics.Models.EmailMetric> @{ ViewBag.Title = "ShowMetrics"; } <h2>Email Metrics</h2> <table class="table"> <tr> <th>Sender</th> <th>Number of Recipients</th> </tr> @foreach (var item in Model) { <tr> <td>@Html.DisplayFor(modelItem => item.Email)</td> <td>@Html.DisplayFor(modelItem => item.RecipientsToEmail)</td> </tr> } </table>
Update the navigation to have a way to get to the new controller
In the Solution Explorer tool window, locate and open the file Views > Shared > Layout.cshtml.
Replace the contents with the following code.
<!-- new code --> <li>@Html.ActionLink("Email Metrics", "Index", "EmailMetrics")</li>
Test the application
In Visual Studio, select Debug > Start Debugging.
When the application is built and loads in a new browser window, select the Email Metrics item in the top navigation bar.
On the Email Metrics page, select the View email metrics button.
When the page loads, you will see a list of emails addresses that were found among all emails with a sum of all the recipients sent between them, as shown from a small sample set in a test email extract in the following figure.
Congratulations!
You've completed the Microsoft Graph Data Connect tutorial. Now you have a starting point to experiment further with Data Connect. You can start by checking out our documentation.
Have an issue with this section? If so, please give us some feedback so we can improve this section.
Feedback
Issottometti u ara feedback għal