Deploying Infrastructure on Azure with Azure Resource Manager & Python SDK.

Once we have gone through how to authenticate the application with Azure AD through the multi-tenant approach and single tenant approach, in this blog we will look at deploying a VM on a VNet, linked to a storage account. Typically when you are looking at automating workloads, you would like to simplify user input as much as possible.

This application loads a simple form @ http://127.0.0.1:8000/deploy that takes inputs from the user. This is a simplified sample of course, but in the back-end it could be either creating an ARM template and deploying, which is a preferred route for large scale deployments or deploy single components at a time for small scale/adhoc requirements.

We will look at the latter today, and in the coming few days we could look at the template based deployment.

You can download and install the latest version of Azure Python SDK with the pip installer:

pip install azure

This is the link to view the SDK in detail.

https://github.com/Azure/azure-sdk-for-python

Coming back to our application, following is the loading screen, assuming that the user has logged in with their application user name & password.

 

DeploymentScreen1

Figure 1: Loading screen

If we take a look at the deployment option module, views.py. The def deployment method loads the admin settings (which is the client_id & secret of the application registered with the AD it in subscription it is hosted in).

When the screen is loaded, we pre-load it with list of regions available today, storage account types and the various VM sizes available. The code has a hard coded else part, which could be replaced by reading from local DB settings.

To call any Python SDK function, we need to send two parameters for all the functsions:

1. Access token

2. Subscription IDs

Since this simply retrieving a standard list we will use the application’s own client_id and secret using the following method.

We first extract it from the default Django storage:

if AuthSettings_Admin.objects.filter(user_id="Admin").exists():

            admin_authsettings = AuthSettings_Admin.objects.filter(user_id="Admin")

            for admin_auth in admin_authsettings:

                client_id = admin_auth.client_id

                tenant_id=admin_auth.tenant_id

                subscription_id=admin_auth.subscription_id

                client_secret =admin_auth.client_secret

Using these we get the access token generated calling the method:

token = get_access_token(tenant_id=tenant_id,client_id=client_id,client_secret=client_secret)

The method get_access_token described below is based on raw REST call, we now have an inbuilt ADL library for Python just released, and will update the article to add that as well in some time:

Make sure you urlencode the client secret in the body.

def get_access_token(tenant_id,client_id,client_secret):

   

    url = "https://login.microsoftonline.com/" + tenant_id + "/oauth2/token"       

       body_data = "&grant_type=client_credentials&resource=https://management.core.windows.net/&

client_id="+ client_id + "&client_secret="+ urllib.quote_plus(client_secret)

    #body_data = urllib.urlencode(body_data_json)

    headers = {"Content-Type":"application/x-www-form-urlencoded"}

    req = Request(method="POST",url=url,data=body_data)

    req_prepped = req.prepare()

    s = Session()

    res = Response()

    res = s.send(req_prepped)

    access_token_det = {}

    if (res.status_code == 200):

        responseJSON = json.loads(res.content)

        access_token_det["details"]= responseJSON["access_token"]

        access_token_det["status"]="1"

        access_token_det["exp_time"]=responseJSON["expires_in"]

        access_token_det["exp_date"]=responseJSON["expires_on"]

        access_token_det["accessDetails"]=responseJSON

    else:

        access_token_det["details"]= str(res.status_code) + str(res.json())

        access_token_det["status"]="0"

    return access_token_det

Once you get the access token, we call the following functions of the SDK to get the list of regions, vm sizes and storage account types:

We have added a layer of abstraction between the SDK functions and the Django application. This is to avoid changing the application if the function’s return types change.

Please refer to the module arm_framework on this link.

loc_list = arm_fw.get_list_regions(token["details"],subscription_id)

pricing_tier = arm_fw.get_list_vm_sizes(token["details"],subscription_id)

stor_acc_types = arm_fw.get_list_storage_types(token["details"],subscription_id)

We send all of these as template parameters to the html page. Please look at the template file for details on how it is loaded, it’s typical Django & and javascript code.

Now that we have the form loaded, the user can enter the deployment name and everything gets populated.

Next step is to Get VM Image list. The Azure Resource Manager provides VM list based on location, and in multi-Step:

1. Get Publisher

2. Get Offer for the publisher

3. Get SKUs for the offer, publisher

4. Get Versions for the offer, publisher and SKUs

The form first loads the list of publisher, it is loading as a raw list which is not a very good way of loading, ideally one must provide a search box and return search results based on that.

We use the following function from SDK to load the list of publishers (getvmimagepublishers):

pub_list = arm_fw.get_list_vm_publishers(access_token=acc_token, subscription_id=subscription_id,region_name=region_name)

Based on the publishers, we call the rest of the functions:

Offers:

offer_list = arm_fw.get_list_vm_offers(access_token=acc_token, subscription_id=subscription_id,region_name=region_name,publisher_name=publisher_name)

DeploymentSelectOffer

SKUs:

skus_list = arm_fw.get_list_vm_skus(access_token=acc_token, subscription_id=subscription_id,region_name=region_name,publisher_name=publisher_name,

offer_name=offer)

DeploymentSelectSkU

Versions:

versions_list = arm_fw.get_list_vm_versions(access_token=acc_token, subscription_id=subscription_id,region_name=region_name,publisher_name=publisher_name,

offer_name=offer,sku_name=skus)

Note: You will notice, that for some publishers the version list returns empty. That means that VM image does not exist. While this application doesn’t handle it, you must ideally cater to it by providing appropriate messages.

Fetch Subscriptions:

The last bit is to fetch the list of subscriptions that the user initially had entered into settings manually or through the OpenID connect methodology explained earlier. This is done through list stored in the local DB.

Submit Deployment

This is the last step of the deployment method. First thing the client determines is whether it is an app-only authentication or user-based authentication. This it does by calling the web method on Django server as shown below in the html template file:

If it’s app-only it is very simple, the client sends all the deployment details in a JSON object directly to the server:

           url: '../authdetails/',

            type: 'POST',

            data: deploymentDetails,

            dataType: 'json',

            headers: { "X-CSRFToken": getCookie('csrftoken') },

            error: function(e){console.log(e.message)},

            success: function (data) {

                deploymentDetails["auth_type"] = data["auth_type"]

                //console.log(data["auth_type"])

                if (data["auth_type"] == "A") {

                    $.ajax({

                        url: '../deploy/',

                        type: 'post',

                        headers: { "X-CSRFToken": getCookie('csrftoken') },

                        data: deploymentDetails,

                        dataType: 'json',

                        success: function (data) {

                            $("#deploy_results").html(JSON.stringify(data))

                            $("#dep_spinner").hide()

                        }

                    })

However, if it’s the user-based authentication, the client opens a pop-up with the auth_url that the server returns:

var url = data["auth_code_url"]

                    popup = window.open(url, "Login", width=300,height=200);

This is to get the authorization code that will be used to generate the token.

The views.py constructs the auth_code_url using the client_if & tenant_id saved by the user as shown below.

authorize_url = "https://login.microsoftonline.com/" + tenant_id +"/oauth2/authorize?client_id="+client_id+"&response_type=code"

            auth_response["auth_type"] = auth_type

            auth_response["auth_code_url"] = authorize_url

            auth_response["auth_token"]= None

The popup window loads the user Azure AD login page. Once the user successfully logs in, it returns the control back to the Django Server as provided in the redirect URI and passes the access code as a query string.

This function immediately returns the control back to the template calling it’s get_Access_code function.

def get_access_code(request):

    query_string = request.META["QUERY_STRING"]

    code_det = {"code":query_string}

    htmlstring = "<html><head></title><script type='text/javascript'>window.opener.get_access_code("+json.dumps(code_det)+");window.close();</script></head><body></body></html>"

    return HttpResponse(htmlstring)

The javascript function get_access_code is called as described below:

code_str = code["code"]

        var code_arr = ""

        code_arr = code_str.toString().split('&')

        code_val = code_arr[0].replace("code=", "")

        deploymentDetails["auth_code"] = code_val

        $.ajax({

            url: '../deploy/',

            type: 'post',

            headers: { "X-CSRFToken": getCookie('csrftoken') },

            data:deploymentDetails,

            dataType:'json',

            success: function (data) {

                $("#dep_spinner").hide()

                $("#deploy_results").html(JSON.stringify(data))

            }

The function sends the details to the same Django application function create_deployment() as described below:

The method gets the access token either directly through the app-only route or using the authorization code:

           if auth_type == "U":

                        auth_code = request.POST["auth_code"]

                        token_url = "https://login.microsoftonline.com/" + tenant_id+"/oauth2/token"

        

                        body="grant_type=authorization_code&code="+auth_code+"&redirect_uri="+redirect_uri+"

&client_id="+client_id+"&client_secret="+ urllib.quote_plus(client_secret)+"&resource=https://management.core.windows.net/"

headers ={"Content-Type":"application/x-www-form-urlencoded"}

                        req = Request(method="POST",url=token_url,data=body)

                        req_prepped = req.prepare()

                        s = Session()

                        res = Response()

                        res = s.send(req_prepped)

                       

                        responseJSON = json.loads(res.content)

                        token = responseJSON["access_token"]

                        token_result = {"POSTURL":token_url,"body":body,"headers":headers,"Response":responseJSON}

                    else:

                        res = get_access_token(tenant_id=tenant_id,client_id=client_id,client_secret=client_secret)

                        token_result = {"Response":res}

                        if(res["status"]=="1"):

                            token = res["details"]

                        else:

                            token=None

                    deploymentDetails = request.POST

                    subscription_id=deploymentDetails["subscriptionID"]

               

Create Resource group:

res_grp_created_status = arm_fw.create_resource_group(access_token=token,subscription_id=subscription_id,resource_group_name=

deploymentDetails["resourceGroupName"],region=deploymentDetails["location"])

This sends the details to our abstract layer which in turn calls the Create Vnet function in the Python SDK:

def create_resource_group(access_token, subscription_id,resource_group_name, region):

    cred = SubscriptionCloudCredentials(subscription_id, access_token)

    resource_client = ResourceManagementClient(cred, user_agent='SDKSample/1.0')

    resource_group_params = ResourceGroup(

                location=region,

                tags={

                    'RGID': subscription_id + resource_group_name,

                },

            )

    result_create = resource_client.resource_groups.client.resource_groups.create_or_update(resource_group_name, resource_group_params)

    success=False

    if result_create.status_code == 200:

        success = True

    elif result_create.status_code == 201:

        success = True

    else:

        success = False

    result_json = {"success":success, "resource_created":result_create.resource_group.name}

    return result_json

Create Virtual Network

if the resource group gets created successfully, we then create the Virtual Network

if res_grp_created:

                        vnet_created_status = arm_fw.create_virtual_network(access_token=token,subscription_id=subscription_id,

resource_group_name=res_grp_name_created,virtualnetwork_name=deploymentDetails["vnetName"],region=deploymentDetails["location"], subnets=None,addresses=None,dns_servers=None)

As described in the function below, it is creating a single address space, subnet and dns server

def create_virtual_network(access_token, subscription_id,resource_group_name, virtualnetwork_name, region, subnets=None,

                           addresses=None, dns_servers=None):

    cred = SubscriptionCloudCredentials(subscription_id, access_token)

    resource_client = ResourceManagementClient(cred, user_agent='SDKSample/1.0')

    resource_client.providers.register('Microsoft.Network')

    network_client = NetworkResourceProviderClient(cred)

    network_parameters = None

    if (subnets != None and addresses != None and dns_servers != None):

        addr_space = AddressSpace(address_prefixes=addresses)

        dhcp_opts = DhcpOptions(dns_servers=dns_servers)

        subnet_det = subnets

        network_parameters = VirtualNetwork(location=region, virtualnetwork_name=virtualnetwork_name,

                    address_space= addr_space,dhcp_options=dhcp_opts,subnets=subnet_det)

    else:

        if DefaultNetworkSettings.objects.filter(setting_type_id="default").exists():

            def_settings = DefaultNetworkSettings.objects.filter(setting_type_id="default")

    add_sp=None

            sb_net_prefix=None

            sb_net_name=None

            for ds in def_settings:

                add_sp = ds.default_address_space

                sb_net_prefix = ds.default_address_range

                sb_net_name = ds.default_subnet_name

            addr_space = AddressSpace(address_prefixes=[add_sp])

            subnet_det = [Subnet(name=sb_net_name,address_prefix=sb_net_prefix)]

        network_parameters = VirtualNetwork(location=region, virtualnetwork_name=virtualnetwork_name,address_space=addr_space,dhcp_options=None,

subnets=subnet_det)

    result_create=network_client.virtual_networks.create_or_update(resource_group_name,virtualnetwork_name,network_parameters)

    success=False

    if result_create.status_code == 200:

        success = True

    elif result_create.status_code == 201:

        success = True

    else:

        success = False

    result_json = {"resource_created": virtualnetwork_name, "success":success,"subnet_name":sb_net_name}

    return result_json

Create Storage Account

stor_acc_created_status = arm_fw.create_storage_account(access_token=token,subscription_id=subscription_id,

resource_group_name=res_grp_name_created,storage_account_name=deploymentDetails["storageAccountName"],region=deploymentDetails["location"],storage_type=deploymentDetails["storageAccountType"])

Following is the underlying function for creating storage account:

def create_storage_account(access_token, subscription_id,resource_group_name, storage_account_name, region, storage_type):

    cred = SubscriptionCloudCredentials(subscription_id, access_token)

    resource_client = ResourceManagementClient(cred, user_agent='SDKSample/1.0')

    resource_client.providers.register('Microsoft.Storage')

    storage_client = StorageManagementClient(cred, user_agent='SDKSample/1.0')

    storage_params = StorageAccountCreateParameters(

        location=region,

        account_type=storage_type,

        )

    result_create = storage_client.storage_accounts.create(resource_group_name,storage_account_name,storage_params)

    success=False

    if result_create.status_code == 200:

        success = True

    elif result_create.status_code == 201:

        success = True

    else:

        success = False

    result_json = {"success":success, "resource_created":storage_account_name}

    return result_json

Create Virtual Machine

Finally we call the create Virtual Machine sending the created details.

vm_created_status = arm_fw.create_virtual_machine(access_token=token,subscription_id=subscription_id,resource_group_name=res_grp_name_

created,vm_name=deploymentDetails["vmName"],vnet_name=vnet_name_created,subnet_name=subnet_name_created,

vm_size=deploymentDetails["vmPricingTier"],storage_name=stor_acc_created_name,region=deploymentDetails["location"],vm_username=deploymentDetails["vmUserName"],vm_password=deploymentDetails["vmPassword"],publisher=deploymentDetails["vmPublisher"],offer=deploymentDetails["vmOffer"],sku=deploymentDetails["vmSKU"],version=deploymentDetails["vmVersion"])

With underlying function as:

def create_virtual_machine(access_token, subscription_id,resource_group_name, vm_name, vnet_name,

                           subnet_name, vm_size, storage_name, region, vm_username, vm_password,

                           publisher, offer, sku, version):

    interface_name=vm_name + 'nic'

    cred = SubscriptionCloudCredentials(subscription_id, access_token)

    resource_client = ResourceManagementClient(cred)

    resource_client.providers.register('Microsoft.Compute')

    compute_client = ComputeManagementClient(cred)

    hardware_profile = HardwareProfile(virtual_machine_size=vm_size)

    os_profile = OSProfile(computer_name = vm_name, admin_username = vm_username, admin_password = vm_password)

    image_reference = ImageReference(publisher=publisher, offer=offer, sku=sku, version=version)

    storage_profile = StorageProfile(os_disk=OSDisk(caching=CachingTypes.none, create_option=DiskCreateOptionTypes.from_image,

  name=vm_name, virtual_hard_disk=VirtualHardDisk(

uri='https://{0}.blob.core.windows.net/vhds/{1}.vhd'.format( torage_name,vm_name)

)), image_reference=image_reference   )

    nic_reference = create_network_interface(access_token,subscription_id,resource_group_name,interface_name, vnet_name, subnet_name, region)

    nw_profile = NetworkProfile(

            network_interfaces=[

                                NetworkInterfaceReference(reference_uri=nic_reference)

                                ]

                                )

    vm_parameters = VirtualMachine(location=region, name=vm_name, os_profile=os_profile, hardware_profile= hardware_profile,

                                   network_profile=nw_profile, storage_profile=storage_profile, image_reference=image_reference)

    result_create = compute_client.virtual_machines.create_or_update(resource_group_name, vm_parameters)

    success=False

    if result_create.status_code == 200:

        success = True

    elif result_create.status_code == 201:

        success = True

    else:

        success = False

    result_json = {"resource_created": vm_name, "success":success}

    return result_json

The application right now returns the names of the created resources as is, one could modify this to show a user friendly message.

Refresh Token

Please note, ideally you should be adding a check to ensure the access token does not expire, and to call the refresh token before it expires.