Microsoft Cloud OS Network Platform (1/2)

In a previous post I've described what is Hybrid Cloud in terms of Microsoft. I've mentioned one part of Cloud OS called "Cloud OS Network". Today I will describe what is "Microsoft Cloud OS Network platform" (or shortly - COSN Platform) - a stack of Microsoft products and technologies, on which all Cloud OS Network services are based on.

In few words - COSN Platform is a complex solution, that service providers can install in their environments to offer Hyper-V based VMs with network and storage resources as IaaS. Also it includes some PaaS services like databases, web sites and service bus. It can be used by big enterprises to build their own highly automated private cloud with Azure-like principals.

Sometimes COSN Platform is also called "Azure Pack platform", because COSN Platform uses Windows Azure Pack portal as a tenant portal, but this is only the top of the iceberg. Let's see what it COSN Platform in details - from the bottom to the top.

What is COSN Platform for a customer

COSN Platform allows customers (tenants) to use cloud services on a platform, that is built using Microsoft Azure cloud principals and similar technologies. It is highly available, scalable and cost efficient. It provides similar to Microsoft Azure experience, but from a local service provider datacenter. It supports hybrid cloud deployments, that use customers private cloud, Microsoft Azure and service provider's datacenter resources.

If the customer expects to use Microsoft Azure in the future (because it is the unavoidable future), than COSN Platform and local service provider can be a middle step between private cloud and public cloud paradigm.

What is COSN Platform for a service provider

COSN Platform allows service providers to offer IaaS and PaaS services, that looks similar to Microsoft Azure services from a customer perspective. This is an almost end-to-end solution, that is available from one vendor. It is like a little part of a big Microsoft cloud that you can deploy in your own datacenter. It is reliable and cost efficient, that is very important if you want to provide services on a competitive market.

Compute stack

COSN Platform uses Hyper-V 2012 R2 for compute. You can also use Hyper-V 2012 if you see any reason for that. With a release of Hyper-V 2016, its support also will be added, with cool new features like Shielded VMs.

Customers can create VMs up to 64 vCPU, 2 Tb RAM and with up to 256 virtual disks up to 64 Tb each (for VHDX format). For more details see Hyper-V 2012 R2 limits on Technet.

Customers can use Generation 1 or Generation 2 VMs, and they can install any supported guest OS into this VM.

Generation 1 VMs, based on VHD (not VHDX disks) can be easily migrated from COSN Platform to Microsoft Azure, because this type of VM is supported in both environments. That's where hybrid cloud starts :)

Storage stack

This is pretty simple - service provider can provide customers any storage that can be connected to Hyper-V hosts. It can be FC/FCoE/iSCSI or Windows Server 2012 R2 Storage Spaces.

Recommended approach for service providers is to connect all available storages to a cluster of Scale-Out File Servers (SOFS) and make it available as Cluster Shared Volumes to all hosts. Main benefit of this approach - simplified storage management, because all storages will be visible like traditional file folders.

Optimal way for cheap and fast cloud storage, that we usually recommend, is to use 2-3 big JBOD enclosures with 60-80 disks, with 20% of SSD disks (400-800 Gb) and 80% NL SAS (3-4 Tb).

You create Mirrored Storage Spaces on these JBODs with Storage Tiering and connect such storage to SOFS. After that you'll achieve fast and huge storage, resistance to 1 JBOD failure, several disks failure and to SOFS server failure.

Networking stack

The main idea of networking in COSN Platform is that must have an ability to easily connect your existing On-Premise LAN with a virtual network in service provider environment. To achieve this goal, COSN Platform uses NVGRE technology and special gateways, which allow tenant to create and manage its own network configuration - IP address pools, NAT rules, DNS settings and site-to-site VPN settings.

This is called "Virtual network" and it is a little bit different from traditionally used VLANs. Key differences:

  1. You can have as many networks as you want. No 4096 network limit like with VLANs
  2. Tenants can configure their network settings by themselves, without asking techsupport to do this
  3. All networks are securely isolated from each other
  4. This functionality is network hardware independent.

Tenants don't need to bother about IP settings inside your VMs - just configure it one time for virtual network, and management stack will configure network interfaces every time a new VM is created.

Another cool thing that I want to mention is a site-to-site VPN functionality. Currently COSN Platform fully supports only one method - IPSec IKEv2. It uses a special RRaS-based highly available cluster of multi-tenant gateways, which run inside VMs. Management stack configures this gateways for tenants. Same gateways are used for NAT, and tenants can manage their own NAT rules from their portal. With this site-to-site functionality tenants can connect their virtual networks in service provider cloud with their virtual networks in Azure just in several clicks. That's where hybrid cloud continues. I'll publish a how-to article about this soon.

But COSN Platform is flexible - you can use traditional VLANs too if you wish. Or you can use your own hardware or software gateways for VPNs. The main limitation of this approach is that tenants won't be able to control everything from the tenant portal, and they will need to ask technical support to create network configurations for them. Also I want to mention that COSN Platform networking stack is extensible - you can extend base functionality by such cool things like Cisco ACI and 5nine Cloud Security software.

Management stack

So, we're on the main complex and interesting topic. Management is the most critical part in the modern cloud, because cloud must be elastic, automated and cost-effective. COSN Platform management stack consists of 3 Microsoft products - Windows Server 2012 R2 (non-R2 is also supported, but not recommended), System Center 2012 R2 and Windows Azure Pack. Also several components require Microsoft SQL Server. You can use 2012 or 2014, Standard or Enterprise - see the exact requirements for every component. All components that are not marked as optional are required in COSN Platform.

Windows Server 2012 R2

COSN Platform uses these Windows Server 2012 R2 components:

  1. Hyper-V Server. Because compute stack and networking stack are built upon Hyper-V functionality.
  2. File Services. Optional component for storage stack if you want to use Storage Spaces, Scale-out File Server or Windows Server-based iSCSI Target, but it is required for VMM Library.
  3. Active Directory Domain Services (and DNS Server). AD is the core of platform security and rights management. Windows Server Failover clusters require AD to work, so no high availability is possible without AD in Microsoft world.
  4. Active Directory Certificate Services. Optional component, that will help you to make internal certificate authority environments, that is required for COSN Platform. But you can use other types of certificate authority if you wish, it is also supported.
  5. DHCP Server. Optional component. I hope everybody understands what DHCP is needed for.
  6. IIS. All web server based roles run on IIS. Apache is not supported :)
  7. Server Manager - nice GUI-based management tool, that allows you to manage local or remote Windows Servers. Very helpful for multiple role management and troubleshooting in one window.
  8. Routing and Remote Services - multi-teant NVGRE gateway is built upon RRaS.
  9. WSUS. Optional component. If you wish to install all updated manually - then maybe you don't need it. If you're crazy, you can install all updated from Microsoft Update automatically, but this is not recommended, because don't gives you so much needed control as WSUS does.
  10. WDS. Optional component, that is needed for Bare-metal deployment mechanism in VMM.
  11. Active Directory Federation Services - optional component if you wish to configure federated authentication for Azure Pack.
  12. Web Application Proxy or Appication Request Routing services can be used as reverse proxies to publish Azure Pack to the internet.

System Center Virtual Machine Manager (VMM)

COSN Platform uses VMM to manage fabric, library and build clouds. It communicates with Hyper-V hosts, NVGRE Gateways and Scale-out File Servers directly. It simplifies storage and network management by configuring network on hosts and guest VMs and by connecting hosts to proper types of storage if needed (including SAN device configuration if a proper SMI-S connector is available for the device). Networking stack is being managed in VMM by using Logical and Virtual Networks. More details can be found here.

VMM Library consists of virtual disk images, VM templates, hardware profiles and OS Profiles. It stores all the data on file shares. When tenant creates a new VM, VMM library is being used to provide a list of available VM templates and to deploy this template to a proper host or storage device. You can use several different libraries for internal admin-use and for tenants. Windows Server deduplication functionality is very recommended for VMM Libraries, because it reduces free storage usage dramatically.

VMM uses a term "cloud" to describe a set of Hyper-V hosts, logical networks, storage devices, libraries and users, that can access these resources. You can have multiple clouds in your environment - Management Cloud for internal services, Compute Cloud for tenant VMs etc.

VMM is responsible for Hyper-V hosts update installation. Details are here. Also VMM allows you to prepare new hosts and reinstall current hosts in several clicks. This is called "Bare-metal deployment". VMM connects to the management interface of the hardware server (HP iLO, Dell iDRACK etc), powers it on, deploys a proper OS image, installs OS, enabled and configures Hyper-V. The whole process is very automated and reduces time to deploy or redeploy OS on a host. Is this a kind of routine that always wanted to eliminate? :)

To be honestly said, VMM is the most critical component in the management stack, because it ties up all other stacks and other management stack components rely on it. COSN Platform is not possible without VMM. Pay the most attention to it while deploying and configuring COSN Platform.

System Center Operations Manager (SCOM)

SCOM is used for fabric monitoring and for collecting performance data. It is also required for service reporting. Although it is not a 100% required component, I recommend you to use it. It allows out of the box monitoring of all COSN Platform components, with built-in remediation actions and knowledge base. You can use any other monitoring system if you with in a parallel, but dedicate some time to deploy and configure SCOM, because it really helps to understand what's gone wrong in cases if failure.

System Center Service Provider Framework (SPF)

SPF is used in the middle of VMM and Azure Pack to connect them with each other. Almost all IaaS-related tasks Azure Pack sends to SPF, and SPF does some internal magic by processing these requests and sends it to VMM. SPF is something you usually don't see or manage, it just works. But it is absolutely required for COSN Platform, because Azure Pack is useless without SPF. Tricky thing - SPF is a part of System Center Orchestrator distributive, but it is totally separated and has different purpose.

System Center Service Management Automation (SMA)

SMA is a part of System Center Orchestrator distributive, it solves similar tasks, but it's a separate product. SMA is required if you want to automate tasks in your cloud. You can skip it's installation if you don't need automation, but we recommend to use it. SMA can provide you reach mechanics to create and manage complex workflows, but unlike traditional Orchestrator, workflows are based on PowerShell scripts instead of a cute GUI interface. SMA don't have its own management console, the whole management and workflow authoring is being made through a special tab in Azure Pack admin portal.

Some examples of SMA-based automations, that service providers develop:

  1. Send an e-mail with details to a customer after user account creation in Azure Pack
  2. Configure backup in DPM after new VM creation, and Protection Group in DPM is being chosen by a custom property in the VM template in VMM
  3. Deploy complex multi-VM services, like highly-available RDS farm or multi-tier SharePoint installation (one VM with DB, another VM with Front-end)

So it you have a frequently repeatable task in COSN Platform, it will be a good idea to automate it with SMA.

System Center Data Protection Manager (DPM)

DPM is used to backup all hosts and VMs in the fabric, also you can use it to automatically backup all tenant backups. The most modern version of DPM - 2012 R2 - can backup Hyper-V VMs, SQL Server databases, SharePoint farms, Exchange databases, Windows and Windows Server OSs (full OS backup or just System State). This is mostly a Microsoft-oriented backup, but you can backup Linux if it runs inside a Hyper-V VM, this is fully supported.

Cool thing of DPM is that it uses local disks for short-term (1-90 days), and Azure storage for long-term  (up to 99 years!). Also DPM can use tapes for long-term, but this functionality looks outdated comparing to the price, stability and extensibility comparing to Azure storage. On Ignite was announced, that DPM extension for Windows Azure Pack will be launched soon. This extension could add backup management functionality for Azure Pack tenants, but unfortunately this project was closed.

DPM is the recommended, but not the only solution that can be used with COSN Platform. You can successfully use Symantec, Acronis, Veeam of other vendor solutions to achieve the same goal. Some solutions even have an extension for Azure Pack for tenant backup management.

System Center Configuration Manager

ConfigMgr is an optional component that can be used to collect inventory data, deploy updates and new software, deploy new servers and make desired state configuration management. But honestly - service providers usually don't use ConfigMgr. New hosts can be deployed by Bare Metal Deployment in VMM, updated management also can by managed by VMM via WSUS, additional software is usually not installed in the fabric, and there are much lighter and cloud-specific inventory tools like SIL. But you can use it if you wish, I just don't see so much effort from it in COSN Platform.

P.S. People usually call Configuration Manager as "SCCM", although this is not an acronym that Microsoft loves. Official short name of Configuration Manager is "ConfigMgr" or "CM".

System Center Service Reporting

Service Reporting is used with Azure Pack Usage service to collect and analyze cloud usage data for tenants - RAM consumed by tenants, CPU load, Storage IOPS etc. This data can be accessed by Excel or other analytic tools via OLAP cube. It is usually used to provide consumption reports and billing to an external billing system.

It is an optional component. You can use it, or deploy a commercial billing tool which will solve the same task.

Windows Azure Pack (WAP)

Azure Pack or WAP is the most interesting component of COSN Platform, and it is the component that makes COSN Platform a finished solution. If a few worlds - Azure Pack is a complex product, which in installed on top of Windows Server, SQL Server and System Center, and which allows to provide IaaS and PaaS services to tenants, with an interface similar to Microsoft Azure. And please, don't say "Azure" when you mean WAP. Azure is Azure, and Azure Pack is a totally different product, which solves similar tasks differently. Usually we don't say "Windows" when we talk about Azure Pack, just like we don't say "Windows Azure" anymore. Azure Pack is not Windows-centric. Microsoft loves Linux, and Azure Pack is a great environment for IaaS with Linux inside guest VMs.

WAP was developed with Azure-consistency in mind. Front-end of WAP uses some code from Microsoft Azure, and when you look on it first time, you can see that its UI looks very similar to Azure. But remember - Azure Pack is very different from Microsoft Azure inside. Microsoft Azure don’t use System Center, but Azure Pack is built on top of several System Center components and can't work without System Center at all. Azure Pack represents a hybrid cloud strategy of Microsoft - it uses similar cloud idea as Microsoft Azure, it looks similar, virtual networks can be integrated with Microsoft Azure, and VMs use the same format, that allows to migrate them between Azure Pack environment and Microsoft Azure environment. Azure Pack is important for Microsoft to advertise hybrid cloud idea, so it was made a free product.

Azure Pack is a complex thing, it consists of 10+ components, which are described here. Besides some components that you don't usually see (public tenant API, admin API etc), there are 2 main components - admin portal and tenant portal. Admin portal is used to manage users, configure plans and configure integration with SPF, SCOM, Web Sites and Database servers (these services are called "clouds" in the UI).

Tenant portal is being used by tenants, and it is the most interesting thing in Azure Pack. It provides following functionality:

  1. User management - user registration, password reset, plan assignment, adding of co-administrators.
  2. Network management - virtual network creation, NAT rules management, site-to-site VPN configuration.
  3. Standalone VMs - management of virtual machines, created from VMM-based VM templates. Change size of VMs, checkpoint management, console access, adding of ISOs from the library and additional data disks, virtual disk size extension, live graphics of CPU/RAM/Network/Storage usage.
  4. Virtual Machine Roles - similar functionality to provide new VM creation functionality for tenants, but it uses special mechanism using VM Role Authoring Tool instead of VMM VM Templates. Advantages of VM Roles comparing to Standalone VMs are horizontal scaling ability, advanced scripting and advanced mechanics to install software (Exchange Server, SQL Server etc) into VM using the inputs, provided by tenant via the VM creation UI. I'll write another article about this, because this is a hot topic.
  5. Web Sites - ability to create simple IIS-based websites or websites based with some preinstalled software like popular CMS, CRM and trade systems like Drupal, WordPress, SugarCRM or OpenCart. There is a cool function from Microsoft Azure called "WebJobs", which allows you to run advanced scripts in your web site environment.
  6. SQL Server databases - provides tenant with an easy way to create Microsoft SQL Server database. Tenant just provides user and password for his DB and receives a connection string that can be used for a web site or any other application that requires SQL Server database to function.
  7. MySQL databases - similar database-as-a-service, but based on MySQL.
  8. Service Bus - service, similar to Azure Service Bus. Not so frequently used.

Azure Pack Tenant Portal is currently available in 12 languages:

  1. English
  2. German
  3. Spanish
  4. French
  5. Italian
  6. Japanese
  7. Korean
  8. Portuguese
  9. Russian
  10. Turkish
  11. Chinese traditional
  12. Chinese simplified.

 Tenants also can install Azure Powershell in their PCs to connect to their Azure Pack resources using PowerShell. Tenant Public API is used for that on a service provider side. I'll provide the instruction how to use this in the nearest future.

 High-level view

So, in high level, these are main Microsoft Cloud OS Network components:






Hyper-V 2012 or Hyper-V 2012 R2

Storage for hosts

Windows Server 2012 R2 Storage Spaces and Scale-out File Server or traditional SAN


Hyper-V Extensible Switch, isolation based on NVGRE or VLANs

Security and Access Control core

Active Directory

Fabric Management

System Center 2012 R2 Virtual Machine Manager

Library Management

System Center 2012 R2 Virtual Machine Manager

Cloud and Tenant Management

System Center 2012 R2 Virtual Machine Manager, Windows Azure Pack Admin Portal

Interface for tenants

Windows Azure Pack Tenant Portal


System Center 2012 R2 Service Provider Framework

Management components Back-End

SQL Server 2012 or SQL Server 2014

Reverse Proxy

Web Application Proxy, Application Request Routing or similar


System Center 2012 R2 Operations Manager


System Center 2012 R2 Data Protection Manager or similar

Optional / Nice to have


Process Automation

System Center 2012 R2 Service Management Automation

Inventory and configuration management

System Center 2012 R2 Configuration Manager

Usage and chargeback

System Center 2012 R2 Service Reporting and Windows Azure Pack Usage

Web sites hosting

IIS and Windows Azure Pack


SQL Server 2012 or 2014, MySQL and Windows Azure Pack

Service Bus

Windows Azure Pack

That's all for today. In the nearest future I'll provide some interesting info about these topics:

  1. High availability of COSN platform components
  2. COSN Platform deployment options
  3. Reference architecture of COSN Platform - CPS Premium.
  4. Tenant experience through portal and PowerShell
  5. Integration of COSN Platform and Microsoft Azure

UPD: My second post about COSN Platform is already available here.