Set up Event hubs
The first step toward using Spark Structured Streaming is setting up Azure Event Hubs in your Azure subscription.
Create an Event Hubs namespace
In the Azure portal, select + Create a resource. Enter event hubs into the Search the Marketplace box, select Event Hubs from the results, and then select Create.
In the Create Namespace pane, enter the following information:
- Name: Enter a unique name, such as databricksdemoeventhubs. Uniqueness will be indicated by a green check mark.
- Pricing tier: Select Basic.
- Subscription: Select the subscription group you're using for this module.
- Resource group: Choose your module resource group.
- Location: Select the location you're using for this module.
Create an event hub
After your Event Hubs namespace is provisioned, browse to it and add a new event hub by selecting the + Event Hub button on the toolbar.
On the Create Event Hub pane, enter:
- Name: Enter databricks-demo-eventhub.
- Partition Count: Enter 2.
Create a shared access policy
From the left menu in your Event Hubs namespace, select Event Hubs, and then select databricks-demo-eventhubs from the list.
On the pane for your event hub, select Shared access policies under Settings in the left menu.
Select the Add button on the toolbar.
On the Add SAS Policy pane, enter the following information:
- Policy name: Enter both.
- Manage: Leave unselected.
- Send: Select this box.
- Listen: Select this box.
Capture the connection string's primary key for the shared access policy
After you create your shared access policy, select it from the list of shared access policies. Then copy the connection string's primary key by selecting the copy button.
Save the copied primary key to Notepad.exe or another text editor for later reference.