MS Multicast IP & Windows Media Technologies Deployment
A History of Windows® Media Services and the Microsoft Digital Nervous System
On This Page
The State of Information Management
The Microsoft Network Upgrade
Planning for Multicast IP
Microsoft's ATM Initiative
Implementing Windows MEDIA technologies
1998: The Microsoft Multicast Network
Media Events' Services
For More Information
"Virtually everything in business today is an undifferentiated commodity except how a company manages its information. How you manage information determines whether you win or lose." - Bill Gates
For years, Microsoft has known that how a company uses information can ultimately determine its success or failure. And the efficiency of a company's communication technology determines how useful such information will be.
Microsoft's "digital nervous system" is comprised of inter-connected PCs and integrated software providing a rapid, accurate, global information flow. Instant data availability provides faster, better-informed business decisions and narrows the gap between a company and its market. This data stream can also enhance organizational "preparedness" during the first days or even hours of a challenge response – often the most critical period for decisive action.
Creating a true "digital nervous system" takes commitment, time, imagination, resources and persistence that not every enterprise will be able to manage. The reward for those who succeed is a powerful advantage over those who can not, or will not, evolve to this next level of commerce.
Years ago, Microsoft made the commitment to develop innovative technology for its own use and, through the resultant products, for the world.
This technical white paper examines in detail the evolution of Microsoft's ATM backbone, the deployment of Multicast IP, the development of Windows Media Technologies and the role of streaming media in driving these advancements.
This document is not intended to be a project model, project planning tool or training aid for deploying ATM, IP Multicast or streamed media applications. The plans, actions and decisions in this white paper are being presented to foster new ideas, generate further discussion and identify potential planning issues when considering network re-design.
As early as 1992, Microsoft began an interactive TV research and development initiative known as Microsoft Interactive TV (MITV). Recognizing the emerging role of the Internet in global communications, the MITV team immediately reconfigured their video server to embrace it. This response lead to today's current Window's Media Technologies.
Due to the popularity of Windows Media Services, Microsoft decided to reinvent its communication infrastructure to take full advantage of the emerging streaming media technologies. In addition, this network would have to handle the business of testing enterprise level products, providing a research platform, hosting Intranet / Internet content and delivering millions of e-mail messages everyday.
The purpose of the ATM upgrade was to prepare the network for future applications that required multicasting functionality to stream audio, video and data across the corporate backbone. Microsoft undertook deploying ATM, enabling Multicast IP and implementing Microsoft Windows Media Technologies.
As the digital nervous system grew, the Windows Media Events team owned the production and global delivery of streamed content across Microsoft corporation, coordinating and delivering timely and cost effective streaming media events. Today this system delivers more than 300 events per month.
One of foremost tasks of the Windows Media Events team was introducing the NetShow Network web site where Microsoft employees throughout the world can access live or on-demand streaming media with the click of a mouse.
Although Microsoft built most of its network with available technologies, few other companies ever combined them into one integrated network. By upgrading to ATM, Microsoft laid the foundation for a high performance, reliable, and scalable multicast platform to distribute multimedia content to its employees.
The implementation and continuing upgrade of Microsoft's corporate network – and the corresponding advances in Windows Media Technologies – is an extremely ambitious undertaking.
The lessons learned from these inter-related projects are important for Microsoft as it moves forward with subsequent deployments of Windows Media. Other companies, armed with the knowledge and lessons learned gained through this experience, can apply some of these initiatives to plan an introduction of Windows Media Technologies and its extensive services in their own organizations. Future technology trends are important considerations, because being positioned to adapt to a new technology – and integrating that technology quickly – provides a distinct competitive advantage.
The State of Information Management
Within Microsoft, the company-wide practice of using new technologies as they are being developed is known as "Eating your own dogfood." This corporate custom drives Microsoft employees to consistently exceed limitations while striving to create better, faster, easier-to-use processes and products to manage digital information.
Microsoft's current Intranet is highly visible when delivering primary business initiatives, such as Product Development, Marketing and Support. But behind the scenes, in an equally strategic capacity, common business initiatives (Finance, Human Resources, Sales, etc.) are responsible for the less "glamorous" task of running the business smoothly and efficiently. These groups also benefit greatly from their integration into the "digital nervous system" where well-informed business decisions can be made quickly, accurately and globally.
Microsoft has reduced the "reaction time" of data gathering, consolidation, response and delivery by constructing one of the worlds largest switched, distributed ATM backbones and the largest comparable IP Multicast implementation.
Microsoft interacts within this environment through its Windows® Media Technologies: Windows Media Player, Windows NT® Server NetShow™ Services, and NetShow Theater Server. In fact, streaming media applications, such as online training, are consistently the most heavily implemented multicast-enabled applications at Microsoft.
Since the introduction of Microsoft NetShow streaming technologies in 1996, corporate services for live and on-demand audio and video content have increased dramatically as the company embraces and extends the services available with each new product version.
Today, Microsoft delivers Windows Media-related services to virtually all its corporate desktops throughout in the world. These services include:
Distance education training
Work group collaboration
Video / audio conferencing
Digital music audio
Broadcast, Cable and Corporate TV
The introduction of the Windows Media Technologies was directly responsible for redesigning the corporate Intranet plan to include the current ATM backbone and Multicast IP environment. To best appreciate Microsoft's innovative development process, it is necessary to understand the events that led to the deployment of Microsoft's IP Multicast and Windows Media services.
In The Beginning: Interactive Information at Your Fingertips
In 1992, Microsoft began an interactive TV research and development initiative known as Microsoft Interactive TV (MITV). Cable television companies were promising to deliver super fast bandwidth into the home "very soon." With more than 60% of all US households subscribing to cable TV, the potential viewing audience was enormous compared to the relatively small number of households that owned a computer. For nearly a half century, television has been the reigning central source of information and entertainment supplied to American homes and most of world.
Microsoft recognized that inexpensive, fast bandwidth cable distribution was the beginning of a "direct home access" era in the digital information revolution.
Microsoft's Interactive Television initiative was a beta test for the Windows NT-based video server. This server, code-named "Tiger" was a broad band solution to streaming on-demand audio and video content, and was designed to "multicast" streams of data in a one-to-many configuration. The goal in this specific initiative was to demonstrate Tiger's ability to deliver pay-per-view motion pictures to multiple homes via a single server.
As the pilot program evolved, Microsoft formed a partnership with a local cable service provider to deliver content to several hundred homes within a 3-mile radius of its corporate campus. These beta test participants were provided with a specially modified cable selector box that delivered the standard 40+ channels of cable programming plus a special "on-demand movie channel" (MITV.) A telephone line "feedback" connection allowed viewers to access an interactive on-screen program guide.
The MITV channel offered viewers VCR-style options – stop, play, rewind, fast-forward and pause– to eliminate viewing timetables and personal schedule conflicts.
As the cable industry struggled with the technology and management needed to "re-wire America", Microsoft realized that the promise of cable-based high bandwidth for the home market was remaining merely a "promise." Despite having developed a very desirable product that could deliver rich content such as feature-length movies over high-bandwidth systems, Microsoft found no ready market for its technology because the necessary distribution infrastructure did not yet exist.
As with many other visionary innovations, it was a related technological development that brought Microsoft a new, albeit different, solution. The Internet blossomed, presenting a nascent global distribution option.
In December of 1995, Microsoft changed their primary corporate initiative to include the Internet into some aspect of the Internet in every Microsoft product — from Windows to Word to Publisher and beyond. By adopting this initiative, Microsoft would extend the use of the Internet into most, if not all, of it's products by embracing the Internet and making it an integral part of everyday corporate information sharing.
The MITV team immediately reconfigured their video server technology to create a product capable of embracing the Internet. The first step was deconstructing "Tiger" to determine how they could adapt components of the video server to the Internet and which features would be the most viable to include in the 1.0 product.
Those components were:
Video pump (streaming audio / video)
These "stand-alone" components evolved into features found in the current Microsoft BackOffice® family of servers: data encryption, commerce services, data security and systems management.
The Windows Media Technologies development team focused on the video pump – the component that streamed audio and video – as two separate projects. One, code-named "Cougar," was designed to deliver on-demand (stored) content and the other, code-named "ActiveRadio," was designed to deliver live content. Just before its release-to-manufacturing (RTM) the product group decided that version 1.0 of the recombined pump would stream audio only.
The media streaming product that evolved made it necessary to rebuild Microsoft's corporate network and laid the groundwork for the "digital nervous system." That product was NetShow 1.0, the predecessor of the current Windows Media Technologies.
The Microsoft Network Upgrade
The corporate culture at Microsoft is based on rapid information exchange, so word of the exciting new Windows Media technologies spread quickly. Many of Microsoft's employees who enjoyed beta testing and trailblazing new technologies were anxious to try it.
By June of 1996, the beta NetShow 1.0 had audio-only functionality and employees were able to download the software and install it onto their server-based office PCs. These "rogue servers" proliferated across the corporate network and lead quickly to numerous bandwidth issues that interfered with daily network operations. This, along with a number of scattered employee-hosted "private" networks, prompted Microsoft IT to consider revamping the corporate network. The network was falling behind the technology curve and would have difficulty delivering necessary services as newer technologies and applications became available.
In mid-1996, the network infrastructure was not adequately designed to handle private network issues, nor was it prepared for the introduction of the new audio streaming technologies introduced by the new Windows Media. While Microsoft IT researched the feasibility of a network upgrade, the Windows Media Team (WMT) launched an initiative to increase public awareness for Windows Media and expose Microsoft employees to the new technology. The goal was to demonstrate how this tool could enhance corporate communications.
To assist with their plan, WMT enlisted the aid of the Microsoft Investor Relations group to produce a live streamed audio first-quarter earnings report – a company first. The Investor Relations group were enthusiastic because this event would deliver the information faster and easier than ever before. In addition to receiving the report by email, employees would be able to listen in, live, on the analysts phone conference immediately following the public announcement of corporate earnings.
Microsoft's corporate network then had limited multicasting capabilities of about 300 Kb/sec. The "audio streamed only" earnings report broadcast was expected to require only about 80 Kb/sec., so there was not much concern that it would overload the network. WMT worked closely with the IT Network Engineering to tunnel between four of the buildings on campus using Unix-based multicast routed (mrouted) daemons running on each segment. Tunneling is a unicast encapsulation performed by DVMRP routers to move multicast packets through unicast-only networks. This was necessary to move multicast packets from one building to another because, at the time, Microsoft's network was unicast-only.
A NetShow Live Server, located in the product group's building, was connected by a telephone line to the conference call of the earnings report and streamed live to more than 500 listeners — proving that the technology to be viable and popular.
By November, the product was finalizing and quickly approaching its RTM date. A second streaming event presented Bill Gates' Comdex '96 keynote speech in Las Vegas – another proof of the Windows Media Technologies' ability to stream live content over the Internet.
Following up on the past two successful events and the NetShow 1.0 product ship, WMT decided to take on an even larger test – the "Microsoft Technical Briefing" (MTB) scheduled for January 1997. This weeklong event allowed Microsoft sales force employees, consultants and system engineers to transfer competencies on emerging product features, functions, and deployment environments that facilitate the applied development of technical solutions for our customers. The MTB would prove the use of Windows Media Technologies' ability to deliver distance learning and save the corporation money by allowing any and all members of the field to view the training sessions regardless of their geographic location on the corporate network. The event would showcase on-demand training; both time zone and geographic shifted.
The Windows Media Technologies product group quickly realized they didn't have the expertise to manage the network responsible for delivering the streaming content and they didn't want to get into the business of managing content or the development of it either. By working with other groups who could specialize in creating and managing the expected increase in requests for content, as well as organizations that had the power and the know-how to manage network issues, the product group could go back to focusing on the development of the product. Thus, three teams came together, creating a triumvirate that would forever change the face of the corporate network and the delivery of live audio and video data across the corporation – creating the first recognition of the digital nervous system.
Microsoft Windows Media Technologies Product Group (WMT)
As mentioned before, WMT began as the Interactive TV project came to an end. WMT was responsible for developing the product, and like many other product groups across Microsoft, they believed in being the first "production environment" for their product. Therefore, they provided not only the development resources in creating the Windows Media code, but they were also expected to own and manage the initiative of testing the product in both a closed and a "real-time" open network environment.
Windows Media Events Group (WME)
WME began in late 1996. Originally known as Microsoft Staging before inheriting Windows Media content development and management, this team owned the corporate staging rooms. They provided technical support for all large conference rooms on the Microsoft campus of which much of the Windows Media content creation originated. This made them a logical choice for the product group to approach for content development and management. When Microsoft Staging took on Windows Media content management, they took on the new group name of Windows Media Events.
IT Global Networking & Systems (GN)
IT Global Networking & Systems (GN) was also established in 1996. Up until that time, separate teams existed with a focus on function: Network Operations (NetOps) focused on the daily operations of the corporate network. This included network accounts, administration and maintaining the overall pulse of the network. Network Engineering (NetEng) focused on testing and planning of future upgrades to the network and troubleshooting network problems. A third organization existed, called Infrastructure Technologies (Infratek). Though this team did not report into the same management infrastructure as NetOps and NetEng, they provided technical research and analysis on all of the corporate enterprise level components – the network being one of them. A couple of team members focused solely on network technologies. When GN was created Microsoft IT combined all of the networking groups into one. This made it easier to maintain a sole network focus, while providing all aspects of network management and deployment all in one organization, including delivering corporate and Internet networking services to all Microsoft entities, subsidiaries, and specific joint ventures and businesses worldwide.
If WMT wanted to popularize NetShow they knew they would need the skills and expertise of GN and Media Events. GN handling network issues – including any Local Area Network (LAN), Metropolitan Area Network (MAN), and Wide Area Network (WAN) configuration issues and changes to the network infrastructure. Media Events would handle the content development and administration, thus leaving WMT to focus solely on the product development.
In January of 1997, the three teams worked together and helped produce over 150 Technical Briefing events across the US and in Johannesburg, Sydney and London. Participants who could not physically attend a briefing or missed a session were able to review the event via on-demand access.
The audience feedback was invaluable, and managers saw the potential in how they could deliver information efficiently to anyone in the company. Naturally, the request for NetShow events increased quickly. To better manage the request process, Media Events created a content distribution Web site known as The NetShow Network. This web-based application had a SQL Server™-based database backend to simplify content management and access. After the Technical Briefing, they were hosting a few shows a month.
With an ever-increasing popularity for streaming media, Microsoft's Global Networks team moved forward with their network upgrade initiative. Because Microsoft's Windows Media Technologies deployment is so heavily intertwined and dependant upon the corporate network upgrade, a prerequisite to this case study is understanding the distinctive corporate network environment itself. Thus, an examination of how the network upgrade was accomplished is necessary.
Planning for Multicast IP
When the Windows Media development group sought assistance presenting Microsoft's first audio-streamed event, several IT engineers suggested that NetShow might "flood the network" with data. As recently as 1996, Microsoft's corporate network was unicast-only, prompting concern that NetShow might also occupy all the available bandwidth and interfere with Microsoft's core business functions on the network, including its extremely high volume of email. Concurrent broadcast problems and projected additional demands of new multicast applications prompted similar fears of system overload.
Infratek, chartered to examine emerging technologies, identified several issues related directly to the design of the initial multicast enabling – protecting the GIGAswitch from excessive multicast rates, choosing routing / topology options, and developing efficient implementation plans. Infratek began by accepting the network infrastructure's limitations.
Preliminary Multicast IP Risks
Both Multicast and non-Multicast (unicast or broadcast) IP use a simple extension of the Internet Protocol, routing each packet by destination and / or source address. They differ in application method and potential risks when Multicast IP is implemented as an infrastructure technology / application. During initial evaluation of various Multicast IP technologies, Infratek defined the following risks:
Unrealistic Customer Expectations
Multicast IP's limitations must be fully understood when implemented within a network infrastructure and as an application.
Multicast IP is commonly perceived to be a way distribute several types of data to a divergent base of users – from 3-way collaborative conferences to live audio transmissions with thousands of clients. Although Multicast IP's flexibility and adaptability does allow for digital voice, video and distribution in various combinations, it relies upon random, non-deterministic, inherently unreliable packet-switched transmission.
The strength of circuit-switched media, whether POTS ("plain-old telephone service") or ISDN video, is in their single application dedication. When a switched circuit is operating, no other application can infringe upon its allocated bandwidth.
Conversely, when transmitting digitized content over a packet-switched network, such as the Microsoft Intranet or the Internet, there is no guarantee of packet sequence, jitter-free reception, data integrity, packet arrival time or even that the packet will arrive. This can cause problems ranging from minor, momentary interruptions of multicast video to serious disruptions that prevent the client application from displaying content.
A low-bandwidth network can be effectively saturated, especially long-distance circuits such as multiple or fractional DS-1. A multicast source (e.g. a NetShow server) can easily create session content (a "show") at 56Kb/s to 1.5Mb/s. for any network free of Multicast IP filters. Effective bandwidth during the multicast is determined by subtracting session bandwidth from original circuit bandwidth.
Under certain conditions, routers fail to route multicast traffic as expected. This often occurs on multi-access networks with resident multicast-enabled and multicast-disabled routers, but other topologies and network components can also fail to route. Mixed Open Shortest Path First (OSPF) / Multicast Extended OSPF (MOSPF) networks are prone to problems where the OSPF Designated Router (DR) – and by extension the Backup Designated Router (BDR) – must have the multicast extensions to OSPF enabled for any given multicast-enabled network.
The DR is solely responsible for generating multicast link-state announcements. If the DR cannot generate these announcements, multicast routing fails for that, and all downstream networks. Enabling all affected routers to understand multicast extensions to OSPF can help remedy this deficiency. Routers that cannot run the MOSPF extensions should be configured to reject DR/BDR status by lowering its OSPF Router Priority to zero (0).
Unintentional Interruption by User Application
No limit or governing factor exists to prevent allocation of all available bandwidth for non-multicast use. Transmission rates for multicast session packets are relatively constant and predictable whereas packets generated by unicast data-transfer typically proceed in "bursts" at the maximum possible rate, which can vary. While transmitting a constant rate multicast session, another session (such as a file download using up to 100% of circuit bandwidth) can be opened at a variable fps rate. This can cause multicast reception interruptions because, in any given second, the traffic carried over that network may be alternately composed of entirely non-multicast or mixed-multicast plus non-multicast traffic.
Intentional Interruption by Unauthorized Parties
Currently, there is no system to authenticate group-member eligibility or content additions. A basic security mechanism would ensure that multicast group sessions are comprised of legitimate members and help prevent the introduction of false or incorrect content.
For example, any user joining an existing multicast group is able to contribute content to that session. A NetShow server transmitting an audio session (live radio) is merely a member of the group. Every client machine is a potential content source, capable of inputting live audio (CD, microphone, etc.) into the session. This intermixing of two or more content sources can create unintelligible crosstalk. Session interruption due to content corruption also continues to be an inherent risk in the duplex capability of Multicast IP.
No Distribution Security
Multicast session reception cannot be limited to specific devices or user accounts. This deficiency presents a similar, but more passive, form of intentional interception / interruption risk. If a LAN PC is receiving a session, every PC running Multicast IP software on that LAN can receive the same content, even if users have no 'legal' or 'license' session rights. This is a growing concern when distributing potentially sensitive information over any media. Confidential communications during a multicast session require encryption to limit effective content reception, because selective content reception is not currently possible.
Infratek recognized Multicast IP as an efficient method for distributing a variety of data across the corporate network. This efficacy was offset by new challenges maintaining network, data, and information integrity within a corporation as complex as Microsoft. Fortunately, these were logical extrapolations of well-understood risks inherent in every network and multicast content integrity could be protected by adopting existing data routing and encryption technologies.
Multicast IP Routing Options
A limited pilot project enabled Multicast IP services on the corporate network without compromising the existing routed IP network's integrity and stability. Infratek proposed two options based on the existing network configuration:
Option #1: Dynamic Distance Vector Multicast Routing Protocol (DVMRP) / MOSPF Multicast Routing with rate limiting controls
Option #2: Tunneled DVMRP Multicast Routing with rate limiting controls
Option #1: Dynamic DVMRP / MOSPF Multicast Routing
A rate-limit feature of the DVMRP service in 3Com NETBuilder II software allowed Multicast IP routing across existing IP network facilities by controlling the number of frames forwarded each second to a particular interface. When transmitting to a network sensitive to multicast frame rates, setting each router's limit to a known value controls the aggregate number of frames per second.
The corporate data center Switched FDDI Backbone was the core of Microsoft's global corporate network, so there was concern that Multicast IP frames could cause network congestion and interrupt the normal forwarding of Unicast IP traffic. The DEC GIGAswitch's Switch Control Processor has a functional multicast limit of 2700, 4470 byte frames per second. Exceeding this rate results in discarding some multicast frames as they are sent to output buffers.
Option #2: Tunneled DVMRP Multicast Routing
This plan proposed "tunneling" to avoid using multicast service on the corporate data center Switched FDDI Backbone. Tunneling is a process that encapsulates all multicast traffic in unicast frames to create an engineered multicast distribution tree. This option would require additional router infrastructure to serve as a 'virtual hub' for Multicast IP routing services and a tunnel endpoint.
Figure 1: Tunneled DVMRP Multicast Routing with Rate Limiting Controls
Connecting a specially designed interface IP router to the backbone network and a separate segment would create a multicast hub network. Backbone routers in each campus building would have a DVMRP tunnel established to an interface on that hub router, forming a tunneled, dynamic neighbor DVMRP network.
The major benefit of this model was maintaining existing multicast levels across the corporate data center Backbone DEC GIGAswitch. Multicast frames would be encapsulated in unicast IP and delivered to the 'proxy backbone' for routing and distribution. All dynamic routing maintenance and monitoring would take place on this routed network instead of on the FDDI backbone network.
This strategy required supplementary network hardware, increasing configuration costs and adding to router processing overhead.
Recommendation: Option 1
After review by the Network Engineering team, Dynamic DVMRP / MOSPF Multicast Routing with Rate Limiting Controls was chosen for implementation. The advantages of this configuration were:
Using the existing infrastructure to eliminate modification costs
Limited risk shown by proof-of-concept (GIGAswitch)
Ease of router administration by eliminating tunnel configuration
Initially, six to eight site networks would be added to the multicast routing network by phased migration. A main source building would be selected and one network in that building would be enabled per day for multicast routing to quickly establish a base level of observable multicast traffic. Each successive, incremental load could then be monitored under normal networking conditions. This configuration was a temporary solution until the Asynchronous Transfer Mode (ATM) upgrade.
Microsoft Multicast BackBone Deployment: Phase One
To alleviate the development community's growing concern about rogue NetShow content providers on the corporate network, Network Engineering chose the Puget Sound area MAN for the first phase of its Multicast Backbone deployment. Introducing multicast services to the corporate campus first also provided high performance for development and testing while protecting the network infrastructure from increased multicast level side effects.
Proving the Strategy: A Test Scenario
The initial multicast test plan connected development, engineering, and operational employees in several buildings to evaluate the performance impact of a DVMRP tunnel configuration on a 3Com NETBuilder II bridge / router platform using the existing infrastructure. The DVMRP tunnel was a common configuration used throughout the Internet Mbone (Multicast backBONE) that protected the GIGAswitch by controlling excessive multicast rates. An unmodified Windows 95-based platform and Microsoft NetShow software served as the multicast-enabled client.
A 3Com router with an assigned IP address replaced a network computer as the tunnel endpoint to measure the processing load incurred by termination. When the router was configured with the duplex endpoint, the tunnel began functioning as expected.
The textual user interface of the 3Com router interpreted data, using the Statistics (e.g. Show STATistics –DVMRP) service and selected 'Super User' commands to view process runtimes. The UI readings were captured in text files, parsed and imported into a Microsoft Word document. Metrics were then calculated using a Microsoft Excel spreadsheet.
Microsoft Network Monitor captured traffic routed to and generated by the 3Com router. The 'print to file' function generated frame rate lists that Microsoft Excel converted into frame and byte rate data.
This measured the processing load imposed on a single router configured with a single tunnel endpoint. The 3Com router became a multicast router with a simple, dual-interface configuration. Measuring the DVMRP service was essential because it was responsible for de-encapsulating tunneled traffic and collecting routing information. It used the tunneling and routing functions of the DVMRP service on a 3Com NETBuilder that represented a "worst-case" maximized processing environment. The lesson learned was that a 3Com NETBuilder router using DVMRP only to route Multicast IP should never exceed the processing overhead level established by this initial test.
The test provided Network Engineering metrics to create multicast traffic models:
Multicast audio traffic rate was low, about 3KB/s per group, which was easily accommodated and scalable within all LAN technologies at Microsoft. 300KB/s at up to 400 fps of multicast audio traffic did not place an unacceptable load on the router.
DVMRP service allowed for an interface "throttle" that could rate-limit multicast traffic forwarded to a given network. This proved useful in LANs that became saturated or on serial WAN networks. Study continues in this area because the throttle function may itself place additional processing demands on the router:
Multicast Enabling the Microsoft MAN
An early Multicast design for the Microsoft Puget Sound area MAN was unnecessarily complex (see diagram, next page).
Figure 2: Proposed Multicast on the Microsoft MAN
All backbone routers would run DVMRP and MOSPF configured for route bleeding, rate limiting and scoping (limiting Multicast IP based on the destination address)
All downstream, intra-area routers would run only MOSPF
All backbone routers would be Multicast Autonomous System Boundary Routers (MASBR), forcing all multicast packets to the backbone and adding additional processing overhead
In addition to the testing phase, Network Engineering wanted training resources made available to provide basic multicast skills to the operations staff and specialty training for operational troubleshooters.
Microsoft Multicast Backbone Deployment: Phase Two
As multicast enabling for the North American WAN began, the ATM initiative (defined in the next section) was also progressing smoothly. The ATM upgrade introduced Cisco routers that did not use existing 3Com multicast-routing protocol. NetShow and Network Engineering worked with 3Com engineers to ensure efficient data movement between the Cisco and 3Com routers that linked Microsoft's MAN and WAN.
The North American region was divided into groups and implemented on five successive nights:
Group 1 – New York City, Atlanta, Dallas, Oakbrook (Chicago), Foster City (California), Santa Monica (California), Mississauga (Toronto), and Waltham (Boston).
Group 2 – Southfield (Detroit), Bloomington (Minneapolis), Wilmington (Philadelphia), Houston, Phoenix, Denver, Irvine (California), and St. Louis.
Group 3 – Chicago (Downtown), Farmington (Hartford), Edison Charlotte (PSS site), Los Colinas, TX (PSS Site), and Washington DC.
Group 4 – Smallest Offices – Tampa, Portland, Vancouver, Ottawa, Sacramento, Kansas City, Rochester, Pittsburgh, Montreal, Cincinnati, Calgary, Richmond, Indianapolis, Cupertino (California), and Salt Lake City.
Group 5 – Salt Lake City, and Raleigh
By leveraging the same OSPF structure used on the MAN, 40 sites were multicast enabled within a week. The sites ranged in size from 7 to 1000 people and had bandwidth capabilities ranging from 256Kb/s to 20Mb/s.
Microsoft's ATM Initiative
Bandwidth issues compounded for Microsoft IT as the NetShow product gained popularity throughout the corporation. By early 1997, the Infratek group had completed feasibility studies of ATM as an end-to-end network solution and concluded it was not a viable solution. Microsoft had recently reorganized, combining the corporate IT Network Engineering team and the MSN™ Operations Engineering, which also analyzed ATM's potential. Where Infratek evaluated ATM as end-to-end solution; MSN engineers evaluated it as a network backbone.
About this time, a growing number of rogue NetShow servers and unauthorized "private networks," configured by employees for testing purposes, were causing problems. After evaluating ATM as a backbone and, in response to growing network problems, the newly-formed Global Networks organization offered these issues and reasons for a network upgrade:
Centralized and non-scalable architecture
Microsoft's network was designed primarily to serve the Redmond area due to its corporate / geographic growth. This highly centralized topology and non-scalable architecture delivered poor performance at remote locations.
The network relied on five year old LAN and WAN hardware. - Fiber Distributed Data Interface (FDDI) LAN network hardware. Switched FDDI in Redmond, FDDI to switched Ethernet, and shared Ethernet to buildings and offices. The original equipment vendor no longer offered service for this fully amortized hardware. - Pre-IP protocol WAN network hardware such as HDLC bridges and Ethernet repeaters, unable to scale up to Multicast IP, multimedia, or higher bandwidth applications.
Increased Development / Lab resources
Microsoft's product groups were requesting additional development and lab resources as the complexity of their products – multicast, streaming, and quality of service – taxed the network's capabilities.
Increased Quality of Service (QoS) Applications Use
NetShow provided the strongest argument for considering Multicast IP functionality, but other applications would benefit immensely from various combinations of voice, video and data transmission.
Global Networks found that many of Microsoft's line-of-business applications were originally created to serve only the local corporate campus environments and were incapable of addressing issues that could affect the efficiency of an increasingly busy network. These products, used by a growing employee base around the world, exacerbated the degradation of the network performance over the WAN.
The graph below illustrates GN's network evaluation by showing how applications and processes affect the network. The original network design minimized most Redmond processes, but their impact increased exponentially as the topology expanded to other regions.
Figure 3: : Application Performance on the old network – RT = Round Trips
Planning for ATM Deployment
After initial analysis, GN studied several potential backbone technologies capable of delivering the services needed to run Microsoft's business:
Asynchronous Transfer Mode (ATM)
The Ethernet technologies were found to be unsuitable because neither platform provided the capability of operating multiple logical network layering within a single physical network. This would mean constructing multiple parallel physical networks to provide the product development and research, Internet functionality and the complex corporate communications that Microsoft required.
ATM technology presented the most viable solution for the corporate network initiative and a number of advantages. ATM was a connection-oriented, cell-based technology that relays traffic across a Broadband Integrated Services Digital Network (B-ISDN). This cost-effective way of transmitting voice, video, and data across a network was precisely what Microsoft IT was looking for. Equally important was ATM's ability to provide the necessary scale of bandwidth and flexibility to allow multiple logical network layering on a single common physical network.
Microsoft needed its network to run business, test enterprise level products, perform research, host intranet / Internet content and deliver millions of e-mail messages everyday. This ATM layering capability was crucial to IT's goal of delivering broad functionality needed by the corporate network. Using Ethernet and Fast Ethernet to link desktop to backbone completed the solution.
IT engineers also considered the following industry related ATM issues:
ATM was the choice of most major carrier and WAN providers. This would lower the total cost of ownership (TCO) for maintenance, operation and future proofing of investments.
ATM allowed for bandwidth flexibility and incremental bandwidth increases. This made it more scalable and cost effective in cost per megabit-per-second (Mbps) than the common point-to-point method.
ATM logical network layering provided for multiple services on a single managed transport; reducing maintenance and operational support responsibilities.
Establishing Worldwide Networking Initiatives
The ATM deployment was "green-lighted" when the Office of the President gave its unconditional backing to provide funding, which ensured the ultimate success of the project. With this powerful mandate, GN was able to make swift strategic and budgetary decsions. Given Microsoft's crucial reliance on the corporate network –legacy applications and business systems tied directly into the network – research and planning decisions were supported unilaterally. GN began with four project initiatives:
Standardizing and replacing networking equipment
To standardize on a scalable supplier that could meet any global service need with high quality routers and ATM equipment.
To replace outdated, unsupported field equipment and upgrade the WAN to support new streaming, multicast and other services in development.
Using carrier ATM services
GN would take advantage of competitive bandwidth offerings by replacing private WAN circuits with carrier ATM service. This would ease service extension across multiple carriers for worldwide coverage and connectivity. GN could adapt efficiently to new and changing business requirements by providing ATM services within and across multiple carriers. (see diagram below)
Standardizing the Architecture
GN planned to improve worldwide network support processes and procedures by designing a topology model compatible with the Redmond-focused LAN design. Using ATM with Fast Ethernet switching and routing worldwide would simplify maintenance and operations management, provide easier upgrade paths for future core/backbone network initiatives and protect the overall network investment.
Figure 4 How carrier ATMs work together efficiently
Leveraging Internet Virtual Private Networks (VPN) as a WAN
GN could access the Internet through multiple regional sites, rather than only through Redmond, by migrating or integrating the use of VPN's over the Internet. This was a cost advantage.
Figure 5 Internet Virtual Private Networks as Transport
Migration to an Internet VPN model would present a further integration on an IP transport for both intranet and Internet / extranet (external business) connectivity. Internet connectivity between Microsoft and Microsoft sites, businesses and Microsoft customers would improve the economy of scale for cost, performance and support.
The ATM Topology
GN created the following regional topologies for the network upgrade to ATM:
Corporate Campus (Distributed and Switched) Architecture
The diagram below illustrates 2 of 3 tiers proposed for the Redmond backbone.
Tier 1 - Synchronized Optical NETwork (SONET) - Campus and regional Puget Sound (MAN)
Tier 2 - ATM building and data center switches
Tier 3 - (not shown) IP and multi-protocol routers in buildings and data center.
This configuration contains no single point of failure.
Figure 6: Distributed and switched corporate campus ATM architecture.
Integrated Lab Architecture
The Redmond campus ATM backbone would use virtual network layering to carry multiple networks. This arrangement overlaid multiple developer, lab and private physically separated networks (Human Resources, Finance, etc.) on the backbone and inter-connected with routers or gateways. In this way, a single transport (ATM) could support all needed services.
Figure 7: Integrated lab architecture on Microsoft corporate campus.
Regional ATM Architectures
The architecture beyond Redmond follows a two-tiered approach:
Tier 1 - Backbone (center cloud) - Regional hubs with redundant backbone links
Tier 2 - Planned subsidiary links (remaining clouds)
This regional hub architecture provides redundancy and traffic management flexibility throughout the enterprise and allows IT to take advantage of non-uniform services to various remote sites.
The diagrams on the following pages show the planned ATM architecture in North America, Europe and the Rest of the World (ROW).
Figure 8: North American ATM Architecture
Figure 9: European ATM Architecture
Figure 10: Rest of World ATM Architecture
The purpose of the ATM upgrade was to prepare the network backbone for future applications that required multicasting functionality to stream audio, video and data across the corporate backbone.
With the Microsoft ATM backbone deployment underway, IT Global Networks returned to its Multicast IP initiative.
Implementing Windows MEDIA technologies
Microsoft undertook deploying ATM, adding network Multicast IP and planning Microsoft Windows Media Technologies implementation virtually simultaneously. The Microsoft Windows Media Technologies implementation needed only a defined basic network infrastructure to proceed and the resultant inter-development created a particularly innovative environment.
Microsoft Windows Media Technologies was popular from the first beta versions of the 1.0 application (audio-only), but the corporate network lacked the performance to meet the increasing demands of new streaming technologies and the proliferation of private networks. The Windows Media Technologies development team (WMT) promoted the use of Windows Media Technologies by producing corporate communication events, which spurred the demand for further content development.
By early 1997:
The Global Networks and Infratek IT teams were implementing the ATM and Multicast IP initiative.
The Media Events team coordinated development and administration of media streaming and began marketing their services to Microsoft employees.
The Windows Media Technologies team produced a working beta of NetShow Services version 2.0.
These teams worked collaboratively to create a Windows Media platform that could meet the demand for live and on-demand streamed audio / video content both on the corporate Intranet and the Internet.
Global Network's Impact Assessment
During the assessment, GN worked with WMT to estimate the impact of these new technologies on network performance, the Windows Media Events team supported the GN and WMT team's role as content administrators and a Media Events engineer worked directly with the product development team to ease this transition by learning the technology first hand.
Windows Media Technologies operates in two unique modes:
Real-time - used for broadcasting live events and select local radio sources over the corporate data network. The video and/or audio sources are compressed during capture, encoded into a proprietary format, "chopped" into IP frames, then broadcast on a specific User Datagram Protocol (UDP) port from the NetShow server.
Stored Playback - used for capturing, compressing and storing video and/or audio sources for fixed disk storage and retrieval. The NetShow server retrieves stored content in blocks, "chops" these blocks into frames and streams it via unicast UDP, TCP or HTTP.
In both processes, the editor or creator of the content designates a unicast or multicast packet format when combining and compressing the audio and/or video streams, then assigns a playback rate, which determines the required streaming bandwidth.
Microsoft's corporate network broadcast standards are:
Radio stations (audio only) 28Kbps
MSNBC (audio / video) 112Kbps
Live video events (audio / video) 56Kbps.
Stored and live content is created with the same playback considerations, with the exception of unicast/multicast frame designations. Consequently, a stored audio stream is configured for 28.8 Kbps and a video stream at either 56 or 128 Kbps when designating unicast playback.
When connecting the Windows Media Player to a NetShow server, software "negotiation" determines if the selected stream can be viewed at the current connection rate. If there is insufficient bandwidth to support the stream, the connection fails. Connections over low bandwidth analog RAS lines are typically capable of audio only playback, while ISDN based RAS connections can support the higher bandwidth stored video streams, or view live video events.
Remote Office Issues and Concerns
For Microsoft, the primary issues for supporting remote offices were limited bandwidth WAN connections, non-multicast routers and shared media Ethernet networks. Most remote offices had data connections of 512 Kbps or less and shared 10 Mbps of Ethernet for all client and server connections. With typical video streams using 56 Kbps to 128 Kbps of the available bandwidth, as few as 4 Windows Media clients running simultaneously could fully occupy the available WAN pipe resources. Although Global Networking is addressing these limitations, WAN connections are likely to remain limited, especially with the Internet's presence as a WAN connectivity resource.
Multicast support allows several clients to receive a broadcast with only one stream sent through the WAN pipe, rather than unicasting separate streams for each viewer. While multicast protocols reduce the WAN pipe impact, ultimately the choice of designating unicast or multicast formats resides with the content author.
Most remote offices access WAN routers and connections that do not support multicast and, among those that do, multicast requires additional overhead that limits the total number of multicast streams a router can handle.
Additionally, many remote offices have a shared Ethernet network, which means that the stream, whether multicast or unicast, will be presented to every linked computer in the office. This increases the probability of a negative impact on other traffic at that location, so ITG has established an administrative guideline that only one Advanced Streaming Format (ASF) stream of up to 128 Kbps may be received by any remote office.
Windows Media 3.0 Resolved Some Concerns
Release 3.0 of the Windows Media Technologies allowed users to select a lower playback bandwidth setting to accommodate a variety of network conditions. This would allow a RAS user to view a 56 Kbps video stream at 28.8 Kbps, or an ISDN user to view a 128 Kbps stream at 64 Kbps. Reduced bandwidth playback results only in reduced video quality by presenting fewer frames per second (fps).
WMT and GN also worked together to create Generic Quality of Service (GQoS) on all connections through routers that support Resource Reservation Protocol (RSVP). This standard managed the bandwidth available to applications or users across any network connection, including remote offices. GqoS, combined with the addition of multicast support to WAN routers and converting the remote offices to switched 100BaseT, reduced the constraints on Windows Media Technologies.
Although Global Networks and the Windows Media Technologies development group defined the project from the ground up, they faced many familiar enterprise issues:
Developing administrative models and processes
Managing migration complexities inherent in the new technologies
Estimating end-user training requirements
Determining hardware specifications
Assessing capacity planning and management issues
In addition to the expected considerations, there were a number of requirements that Global Networks felt necessary to define the project and increase the chances of its success. These fell into three categories:
Identifying the "ultimate application" – Since the predominant application would drive the multicast initiative, identifying this and other applications would gauge the cost of benefits. The application's functionality, determined by real-time company communication and more efficient data distribution, would drive the implementation rather than the underlying technology.
For Microsoft, the "ultimate application" was the Windows Media Technologies. By focusing on Windows Media Technologies client applications, rather than the back-end structure and delivery mechanism, deployment success became easier to measure.
Targeted experts – Experts within the Global Networks group needed to understand the way Multicast IP effects the architecture by requiring a different set of routing protocols than a unicast-only network.
Cross-technology experts – Windows Media Services engineers responsible for troubleshooting both client and server needed to become Multicast technology experts. Familiarity with group addresses, time to live (TTL) and Codec variants are essential in understanding how ASF can affect a multicast network infrastructure. Also, a support staff for the web interface for the NetShow Network is necessary to ensure an easy access for all users.
Operating System support – Enterprise operating system platforms need to support multicast. Microsoft Windows® and Microsoft Windows NT® operating systems support multicast upon conventional installation. UNIX is also able to support multicast IP.
Scalable multicast client/server software – Windows Media Technologies is a scalable technology that can be integrated into existing support structures. The integration of the Windows Media Player and server with Windows NT Server and its built-in Web server, Microsoft Internet Information Server (IIS), provides an underlying scalable technology for other applications.
Sufficient LAN/WAN infrastructure and end-point capabilities – During the initial network upgrade assessment and testing phase, Global Networks required that final implementation would support the desired multicast streams. Switched Ethernet (10Mb/s) which easily handles up to 110Kb/s streams and the use of L2 snooping reduced extraneous traffic on heavily loaded networks – Internet Group Management Protocol (IGMP) snooping, and Cisco Group Management Protocol (CGMP). It was also necessary to consider the power of the standard desktop PC at Microsoft with regard to the network bit-rate. Pentium-class PCs can decode at 28Kbps - 110Kbps, but higher rates would require a more powerful chipset. So, recognizing the end-point processing power provided an upper limit for necessary network performance.
Network hardware with multicast capability – Although most routers support some form of multicast routing protocol, Global Networks needed to determine if upgrades of the software code/firmware or the network hardware were necessary during the project's life cycle and beyond.
Consistent hardware and software versions – It was necessary to maintain common and supported network hardware for all network equipment that ensured consistent, expected behavior across the network. This would simplify familiarization with hardware, software and their inherent problems.
Windows Media guidelines – Windows Media Services created guidelines for the Remote Encoder / Multicast Channel Manager (REX/MCM) pair.http://www.microsoft.com/netshow. This gave Global Networks and the Windows Media Events team the responsibility of ensuring that the encoder system and supporting media hardware could perform an acceptable level of bit delivery across the network to the MCM as a prerequisite to providing multicast performance.
Matching circuit bandwidth & media types – The Windows Media Events group needed to understand the general bandwidth schema for the intended streamed media audience. Requests for content development clearly identified these criteria. If the majority of viewers were dialing in with 33.6Kb/s modems, it was unnecessary to Codec with a 56Kb/s bit-rate. If the video quality is crucial to the content's subject matter (i.e. a graphic software application demo rather than a "talking head" financial report), it is better to prevent connection by low-rate sites across the WAN.
Channel configurations – Planning for locally administered group addresses and scheduling allowable TTLs could be the key to limiting, controlling, and shaping multicast sessions.
Multicast IP addressing scheme – Multicast IP addressing, along with TTL configuration and delivery mode would all affect functionality and ease of operation. Communicating this when setting up routers and when configuring the Windows Media servers or clients would be essential. Policies needed to be instituted so that all engineers would adhere to the same rules.
Support procedures – Helpdesk personnel should be able to understand how the Windows Media client works, how the server delivers the content and how client and server communicate. Training Helpdesk personnel would be necessary so that they could recognize the difference between media delivery problems and a bad or unsupported video card failure.
1998: The Microsoft Multicast Network
Although Microsoft built most of its network with available technologies, it is likely one of few, if any companies that had ever combined them into one integrated network. By upgrading to ATM in 1997, Microsoft laid the foundation for developing a high performance, reliable, and scalable multicast platform to distribute multimedia content to its employees.
Microsoft's ATM initiative "wove" integral, state-of-the-art communication technologies – ATM, SONET, and Fast Ethernet – into a high performance, scaleable, manageable, flexible network. This hybrid meta-technology allowed Global Networks to convert an "anarchy" of sub-networks into a rich, responsive environment. The original centralized star topology was transformed into a distributable ATM mesh topology and an innovative addressing plan allowed a further evolutionary metamorphosis by allowing business and development to coexist in different layers on the same network. The result was one of the largest, most innovative global enterprise networks in the world.
Microsoft's ATM: A Technical Examination
Layer 1 - The primary transport infrastructure of the new Microsoft Corporate network is a series of SONET rings. Buildings are connected to a core site by either a combination of single mode fiber or leased SONET links, depending on the location and available access to the building.
Layer 2 - FORE ATM switches comprise a distributed, switched ATM backbone.
Layer 3 - High-performance, multi-protocol routers provide the IP, multicast and multi-protocol connectivity across the entire campus. Unique aspects of each of the first three layers of the OSI reference model as it pertains to the Microsoft network are described further in the following sections.
Layer 1 Transport Architecture
The proliferation of SONET and Digital Cross-connect System (DCS) transport technology – collectively known as STM or Synchronous Transfer Mode – as the Layer 1 network architecture had its origin in Microsoft's growth and associated intranet and Internet communications requirements. In Microsoft's deeply meshed, high-density corporate backbone / developmental laboratory environment, SONET rings and DCSs provide the reliability, manageability and scalability to allow moves, adds, and changes. When implemented effectively, these transmission technologies provide reliable network availability and broadband traffic fidelity. Strategic deployment of SONET ring transmission systems reduces the implementation and operational complexity of a large campus network.
SONET's nearly error-free delivery and 100% availability are a direct result of decades of Telecommunications Operations research, manifest in the Section, Line and Path Overhead bytes. Approximately 13% of the overhead bandwidth is a series of data communications channels embedded in the SONET overhead that communicates the alarms, performance monitoring and self-healing protection switching algorithms between all optically-connected nodes in the system. These physically in-band data communications channels are logically out of band since no payload capacity is sacrificed.
In the early 1990's, Microsoft's legacy PBX systems were expanded to accommodate more employees, Product Support Services and Sales Offices in the North American WAN, and eventually connections to Internet Service Providers (ISPs.)
To meet this growth, asynchronous circuits (i.e., DS1s and DS3s) increased exponentially, until economy of scale demanded moving to the next generation TDM hierarchy, Synchronous Optical Networks (SONET). This rapidly maturing technology was a cost-effective basis for voice and data communications between the Microsoft campus and Carrier Points of Presence for WAN, and various campus sites and the local MAN.
Microsoft benefited by having two progressive Local Exchange Carriers, GTE and US West, who provided dedicated SONET ring services to interconnect offsite Microsoft buildings, Inter-Exchange Carriers and ISPs to the Redmond campus.
In 1994, the first SONET ring, a Fujitsu FLM-600 OC-12 Unidirectional Path Switched Ring (UPSR), was installed between the main core sites on the Redmond Campus. Additional OC-48 rings, provided by both GTE and US West, facilitated Metro connectivity between the Redmond campus Corporate Data Center and Microsoft facilities or carrier POPs in the Metro area. Enterprise bandwidth growth within 6 months of the OC-12 ring turn-up made upgrading to the next highest TDM aggregate rate (OC-48) a necessity.
Four carrier-supplied Metro OC-48 rings, installed in mid 1995, quickly reached 75% of capacity. Two additional campus OC-48 ring systems were added to support heavy PBX trunking and laboratory network connections to various telecom resources. In mid1997, two additional OC-48 2-Fiber BLSR rings were dedicated to Internet ingress and egress traffic
Until January of 1997, the preferred SONET ring topology was UPSR because most network services were centralized within the Corporate Data Center and UPSR is optimized for hubbing topologies. The "Web" changed organizational traffic patterns from centralized to distributed and increased traffic load on the enterprise and the industry by two orders of magnitude. This shift in topology and scale required an appropriate modification of the underlying transport architecture, one that would support distributed Broadband (45Mbps or greater) traffic patterns and Wideband (1.5 to 45 Mbps) legacy traffic.
The upgrade and evolution of the corporate network drove the Broadband shift from a shared-Ethernet / switched-FDDI backbone to a switched-Ethernet / switched-ATM backbone. The need to partially mesh the campus "Core" sites with OC-12c ATM trunks and replace the Metro DS3 router links with OC-3c ATM trunks made a series of transport system overhauls and expansions necessary. These increased capacity, flexibility, and reliability and a 4-fiber BLSR OC-48 ring system was required to handle the needed capacity and connectivity. The service now includes DS1 voice and data, DS3 ATM cells and IP frames, OC-3c/12c ATM cells and IP frames.
To support "IP over ATM" on SONET and "IP over SONET", the Alcatel 4-Fiber Bi-directional Line-Switched Ring (BLSR) OC-48 system was chosen as the broadband transport topology. Maximum connectivity between campus core sites was achieved by constructing a series of three overlapping rings of 3-4 nodes each. Additional, strategically placed, nodes could unlock additional Bandwidth if the topology changes or new core sites are constructed.
An Alcatel Broadband DCS at the Corporate Data Center provides a critical component of service survivability. The B-DCS, at the intersection of three OC-12 and ten OC-48 SONET rings, provides segregation and aggregation functions on traffic passing between rings or dropping at the Corporate Data Center. The B-DCS connects the SONET UPSR and BLSR rings at OC-12 tributary rates passing Wideband services to a collocated Alcatel Wideband DCS via OC-3 tributaries for DS1 grooming and filling.
Future upgrades to the B-DCS will permit OC-48 STM and ATM tributaries, and eventually OC-192 integrated ring functions. The SONET/ATM integration with the B-DCS and similar highly integrated and massively scalable bandwidth management systems form the cornerstone of the future Microsoft Corporate Network. These systems will need to discriminate between Cell and Frame payloads, segregating and aggregating data flows to maximize trunk efficiency.
Figure 11: Microsoft Corporate Network Core Architecture
Layer 2 Architecture
There are 7 primary core sites within the main campus: six buildings across the corporate campus and the primary campus data center. Each hub contains two 10 GB Fore ASX-1000 ATM switches, interconnected with a mesh of OC-12 trunks. The primary OC-12 connection between the core sites is established by the underlying SONET infrastructure and a secondary OC-12 interconnection between the core sites is established by single mode fiber (see diagram, below).
ATM switches are interconnected in a hierarchical configuration using PNNI as the Layer 2 routing protocol. The ATM PNNI topology is shown below:
Figure 12: Microsoft ATN Network PNNI Topology
Layer 3 Architecture (IP)
Microsoft uses a classless IPv4 address architecture, with variable-length network prefixes using both private (RFC-1918) and public (IANA-allocated) addresses.
A functional distinction is made between 'corporate' and 'lab' environments for address allocation. Typically, a corporate network environment (hosting ITG-supported clients) is addressed from Microsoft public space. Laboratory and experimental networks (those hosting unsupported devices) are addressed using private network space.
Network prefix aggregation is noted at a number of scales. The largest is campus-wide prefix blocks – Redmond corporate campus network falls within four blocks, each corresponding to 16-bit prefixes (/16, equivalent to 'Class B' address spaces.) The next scale encompasses continental and regional corporate networks – North America, Europe, and the Pacific Rim are each allocated another 16-bit prefix.
The corporate network address blocks are not announced to the global Internet, but a similar addressing architecture is followed in each of Microsoft's Internet Data Center locations. Typically, a large (16-bit or 18-bit) prefix is used and announced as an aggregated route from these sites.
The unicast architecture is primarily comprised of IP, IPX and Appletalk connectivity. The underlying design goal was to optimize the new IP infrastructure while continuing to support legacy applications and IPX / Appletalk connectivity. This was achieved by configuring two ATM Emulated LANS (ELANs) across the entire backbone to specifically provide all campus buildings with IP connectivity.
All data center routers are configured to be members of these two IP ELANs and another ELAN was designed specifically for multi-protocol (IPX and Appletalk) connectivity. Within the data center, two additional IP ELANs were configured to allow intra data center transit for backups, server replication and other functions.
Each ELAN has a dedicated LES/BUS and LECS provided by a Catalyst 5002 with an ATM card.
The logical connectivity provided via these ELANs is illustrated below:
Figure 13: Microsoft Unicast Architecture
The diagram below shows the logical topology of the multicast network at Layer 3. The network consists of several multipoint signaling networks (one per core site, and one backbone). Multipoint signaling allows us to distribute multicast traffic using the ATM fabric by point to multipoint virtual circuits. In this way, multicast core routers are not burdened with packet level replication and, since ATM switches forwarding in hardware, possess better scaling potential as multicast traffic grows. Also shown is the legacy network connection (running MOSPF) where DVMRP is the common multicast protocol between the Cisco and 3COM platforms.
Figure 14: Microsoft Multicast Architecture
The migration to a new backbone presented many unique challenges. Entire user populations were moved from the FDDI backbone to the ATM backbone in one night, but their common resources (file and print servers) often remained behind, awaiting their own scheduled date.
A scaled approach was used to solve the problem of a high-speed, reliable interconnection between the ATM and FDDI network backbones. Two Cisco 7507 routers served as load-balanced, redundant interconnection paths for unicast IP. A Cisco 7505 router was configured as a multi-protocol interconnection path, serving protocols other than IP. Each of these routers were attached to the FDDI network by a switched, full-duplex FDDI interface and had at least one OC-3 interface to the ATM backbone.
To take advantage of multiple unicast IP paths scalability, multicast IP was transmitted between backbone networks by unicast encapsulation. A 3Com NETBuilder II router was installed on the FDDI backbone and configured as a multicast boundary router for DVMRP and MOSPF routing domains. The DVMRP configuration included a virtual interface as a unicast 'tunnel' endpoint.
The other endpoint was a router configured to act as a multicast core node in the ATM network and as a DVMRP virtual 'tunnel' endpoint to make DVMRP and PIM translation possible between routing domains.
Data Center Design
IP sub-netting coupled with the Virtual LAN features of the Catalyst switch provides high performance connectivity for nearly 1500 servers located In the data center. This is accomplished by Cisco 7513 routers, each having seven fast-Ethernet interfaces, connected to seven VLANs on a Catalyst switch. Each VLAN on the Catalyst is comprised of twelve 10/100 Ethernet ports – one for uplink router connection, eleven connecting servers – and a connection to the ATM backbone by four IP ELANS and the multi-protocol ELAN described above.
Each data center router is also configured as part of the associated multipoint signaling network described above to allow high performance multicast connectivity. The data center design is illustrated below:
Figure 15: Microsoft Corporate Data Center Architecture
The figure below illustrates a simple cable room design for a building with a single cable room. The building router is connected to the building's ATM switch through multiple ATM interfaces, logically configured into the building IP and multi-protocol ELANs and the associated multipoint signaling network. The router connects to the hub building Catalyst switch through an Etherchannel connection. In the case of multiple Catalysts in a cable room, each subsequent Catalyst is connected to the hub catalyst through an Etherchannel connection. Each cable room comprises a single IP sub-net as well as a single Appletalk and IPX network. The Catalyst switches provide a combination of switched Ethernet and switched fast-Ethernet connections to each data jack throughout the building.
Figure 16: Microsoft Corporate Building Network Architecture
Protocols and Applications
PIM-DM (Dense Mode)
PIM Dense Mode was the multicast routing protocol chosen to begin deployment. This protocol received strong support from Cisco and offered simple configuration and troubleshooting. PIM Dense Mode does have flood and prune characteristics similar to DVMRP, making it a short-term protocol choice as Wide Area multicast deployment continues. Currently, PIM Sparse Mode is the replacement candidate for the long-term backbone conversion plan.
The network unicast routing protocol selected was OSPF for its standards and large multi-vendor support. The initial design is a non-congruent multicast and unicast topology, requiring implementation of a separate unicast routing protocol (RIPv2) to support RPF checks within PIM. The long-term plan calls for congruent unicast and multicast protocols, eliminating the need for RIPv2 in the network and allowing PIM to use the primary unicast routing protocol.
In a distributed Layer 3 framework, there is a well-defined mechanism for distributing IP multicast traffic by the Class D addressing scheme, IGMP and PIM. This Layer 3 configuration provides propagation control of IP multicast traffic. Where IP multicast traffic crosses a Layer 2 switch, it is propagated to all ports on that switch. Cisco provides an intelligent method to limit the propagation of this multicast traffic across a Layer 2 switch using Cisco Group Management Protocol (CGMP) between the Cisco router and a Cisco Catalyst switch. CGMP allows the Catalyst switch to leverage off the IGMP information in the Cisco router and intelligently prune multicast traffic off ports that have not specifically joined a multicast session.
The Microsoft network uses CGMP to restrict the propagation of multicast traffic on the Catalyst switches and enhance the performance and scalability of the multicast infrastructure.
Distance-Vector Multicast Routing Protocol (DVMRP), an offshoot of the RIP protocol, was the first widely-deployed multicast routing protocol and used to build the Internet multicast backbone (MBONE.)
It began as a research tool, possessing many special features not typically found in other routing protocols. DVMRP's ability to encapsulate multicast routing and data packets within unicast packets lead to the rise of 'tunneling' multicast IP through unicast networks. It also provides for bit rate limiting of multicast traffic to a routing interface ('rate-limiting') and distribution control of multicast datagrams based on destination address ('address-scoping'.) DVMRP extended the use of the Time-To-Live (TTL) field in the IPv4 header to rudimentary propagation control over multicast traffic ('TTL thresholds').
Microsoft used DVMRP in a limited capacity at the outset of its multicast-enabled network. Its first use was a grass-roots effort by the development community to build an environment where multicast applications could be created and tested. By using DVMRP tunnels, a multicast structure was laid over the unicast-only corporate campus network, but the result was an unsupportable non-standard routing platform without a central administrative authority.
DVMRP was a 'boundary' protocol on corporate-supported routers for a short time, where its rate-limiting and address-scoping functions were used to provide control over datagram traffic heading into the switched FDDI backbone. DVMRP was eventually replaced by MOSPF except in multicast routing domains external to Microsoft corporate OSPF structures.
The multicast extensions to the Open Shortest-Path First protocol (MOSPF) are data types and events that allow multicast group and routing information to be contained within OSPF structures. MOSPF adds multicast group data types, flags to indicate underlying network multicast ability and new router functions to the OSPF foundation. MOSPF allows for inter-operation between unicast-only and multicast-enabled routers, but does not include tunneling or rate-limiting functions.
MOSPF was chosen for Microsoft's legacy MAN and WAN environment because it was supportable, scalable and available across routing platforms in use at that time. It was deployed across the Puget Sound MAN and North American WAN networks, displacing DVMRP as the single multicast routing protocol. As Microsoft moves away from MOSPF-capable routing platforms, PIM multicast protocols are replacing MOSPF as the standard multicast routing protocol.
Static Multicast Routes
Static multicast routing was employed where multicast and unicast routing domains diverged.
Media Events' Services
As the digital nervous system grew, the WME team virtually owned the global delivery of streamed content across Microsoft corporation, coordinating and delivering appropriate, timely and cost effective Windows Media events.
The NetShow Network Website
One of foremost tasks of the WME team was introducing the NetShow Network web site. This web-based content delivery mechanism allowed Microsoft employees to select either on-demand or live streamed content using a broadcast calendar interface.
Currently, the NetShow Network is shipped (full source) on the NetShow SDK. Several groups within Microsoft — including WME and the Microsoft Research organizations — used this product as part of their own corporate presence portal.
The first WebServer configuration was an Intel Pentium 90 processor with 32MB RAM. A second Pentium based machine with 64MB RAM ran the Microsoft SQL Server 6.5 database. Since the encoder performs most of the processing, these machines were isolated to their own dual-Pentium processor with 64MB RAM. The WebServer and SQL Server-based machines were installed in the corporate data center, while the encoders were distributed across the corporate campus in the conference building, IMG buildings and at Microsoft Studios.
The NetShow Network is composed of two components:
The Guide helps find and view events using HTML, Visual Basic® Scripting Edition, and a Java language-based calendar running on Microsoft Internet Explorer. Other installation requirements are the Windows Media Player control and the NetShow FTS control.
The Administrator helps event coordinators to manage events using Visual Basic Document Object Control that takes control of the entire Web page. For Microsoft's corporate implementation, the Visual Basic Setup utility was used to create an installation program for Visual Basic runtime that allowed Administrator to communicate with a SQL Server via ODBC.
With this system, Microsoft employees around the world are able to access either live or on-demand streaming content with a mouse click. The NetShow Network's popular "always live" select radio stations and the MSNBC video streams coupled with its easy to use features, has made it possible for Windows Media Events to deliver more than 300 streaming events per month.
In August of 1997, the WME team made a strategic move to the new Microsoft corporate studios, situated within walking distance (0.5 miles) from both the corporate campus and the Interactive Media Group (IMG) on the Microsoft Redmond West campus.
Microsoft Studios was designed to be a state-of-the-art digital studio, complete with sound stages and recording studios. Media professionals, understanding that Microsoft's core business was software, owned the primary services and operational duties of the studio. WME installed a Windows Media Technologies front end with Channel Manager, on-demand storage, multiple encoders, and associated signal enhancing equipment in the Microsoft Studios facility.
By the RTM of NetShow Services version 3.0, not many of the requirements for NetShow Network had changed, but the NetShow Network source available on the SDK today is designed to work with Microsoft Access upon installation. The option to upgrade to SQL Server was available, but the Administrator, upgraded from a Document Object to a regular OLE Control wrapped within a Visual Basic executable component, required Microsoft Access.
Windows Media Technologies Configuration & Topologies
The configuration of the servers that make up the corporate Windows Media Technologies platform have increasingly grown as hardware technologies have advanced. WME uses high-end PC configurations for all content streaming, allowing them to deliver corporate network broadcasts at a minimum of 100 Kbps for video and audio streaming and 56 Kbps for audio only. Some content such as MSNBC and the Washington Dept. of Transportation (WADOT) stream at 300 Kbps and 500 Kbps respectively. Here is a quick rundown of the hardware changes over the last couple of years:
Encoders: Dual Intel P6/200, 32MB RAM, Winnov video capture cards, and Sound Blaster 16 sound card
Channel Manager: P6/200, 64MB RAM
No Windows Media Server
Encoders: Dual Intel P6/200, 32MB RAM, Winnov video capture cards, and Sound Blaster 16 sound card
Channel Manager: Quad P6/200, 128MB RAM
No Windows Media Server
Encoders: Intel Pentium II/300, 64MB RAM, Intel Smart Video Recorder III, Sound Blaster 64 AWE sound card and Antex SX-36 sound cards
Channel Manager: Quad P6/200, 128MB RAM
Windows Media Server: Quad P6/200, 128MB RAM, 62GB HD space
Encoders: Intel Pentium II/400, 64MB RAM, Intel Smart Video Recorder III, Sound Blaster 64 AWE sound card and Antex SX-36 sound cards
Channel Manager: Quad P6/200, 128MB RAM
Windows Media Server: Quad P6/200, 128MB RAM, 62GB HD space
The following diagram represents Microsoft's basic multicast configuration. Note that the numbers represented are standard Windows Media Technologies requirements:
Figure 17: Basic Multicast Corporate Configuration
The following diagram represents Microsoft's basic on-demand configuration:
Figure 18: On-Demand Corporate Configuration
Microsoft has benefited from having an extensive, well-established conference room structure in place before developing the corporate Windows Media Technologies topology. These facility resources were the basis for a streamed media content development platform that stretched across the corporate campus.
Figure 19: Microsoft Corporate Campus Topology
A large component of ASF content is interactivity with slides created with the Microsoft PowerPoint® presentation graphics program. Microsoft employees frequently use this feature with distance learning media streams where slides can help maintain a constant content flow and simplify the topic. The following diagram represents Microsoft's PowerPoint streamed media platform design:
Figure 20: Microsoft PowerPoint Slide / Windows Media Topology
This diagram represents Microsoft's use of the Internet to receive and deliver streamed media:
Figure 21: Corporate Streaming Media Internet Service
WME's Support Policies
Having such an extremely popular web front end in place, WME needed an immediate definition of support policies to guide the growing variety of requests for content production. The general guidelines established were:
WME would be responsible for coordinating streaming live and recorded audio/video content to the Intranet /Internet via the NetShow Network.
WME would use only proven (RTM) Windows Media Technologies for event support. Beta testing of new Windows Media Technologies was acceptable only if time and resources permitted.
Clients (Microsoft employees) would be responsible for obtaining all NDA and copyright permissions.
Scheduling and cost issues for all audio / video equipment would be the responsibility of the client.
All encoders in Media Services staging rooms would be free of charge and scheduled for use by reservation.
All live events would be monitored for quality at the WME office location. In addition, the WME team would escalate any client problems and issues to the proper support groups for resolution.
Cumulative hit information for each event would be recorded and sent to the client within 48 hours of the event and other reports provided if requested by the client.
Other policies: minimum client request notifications, client and NetShow presenter requests and on-demand posting requests were also defined. WME eased support burdens and effectively educated NetShow Services users by using simple text and graphic representations to outline support scenarios specific to the type of content requested. Topics included:
Key players involved in complete delivery of content
Service request procedures presented as a step-by-step "how to" list
Responsibilities of the client, WME and other involved support groups such as Helpdesk
Escalation procedures with detailed contact information
Feedback gathering procedures to ensure appropriate post-event analysis
WME's Support Scenarios
Microsoft employees are encouraged to assume responsibility for reserving conference rooms, arranging for catering and scheduling Windows Media Technologies events. To make the process easier, the Windows Media Events organization has created support scenarios for every type of request for Intranet or Internet streamed content. This eliminates confusion or duplication of efforts concerning issues of contacts, responsibilities and final deliverables.
The following is a list of NetShow Network support scenarios offered by WME:
Corporate Intranet Requests:
Intranet Onsite Live Audio & Video Request
This request scenario delineates the delivery of real time streamed audio/video content originating in Microsoft Staging Rooms for the corporate Intranet.
Intranet Onsite Live Audio Request
This request scenario delineates the delivery of real time streamed audio only content originating in Microsoft Staging Rooms for the corporate Intranet.
Intranet Onsite Live NetShow Presenter Request
This request scenario delineates the delivery of real time streamed audio/video content and Microsoft PowerPoint slides originating in Microsoft Staging Rooms for the corporate Intranet.
Intranet Onsite Live Audio & Video / On-Demand ASF Capture to MCM Request
This request scenario delineates the delivery of real time streamed audio/video content originating in Microsoft Staging Rooms for the corporate Intranet with capture to ASF for later on-demand use.
Intranet Offsite Live Audio / Video Request
This request scenario delineates the delivery of real time streamed audio/video content originating from an off-site location for the corporate Intranet.
Intranet Offsite Live Audio Request
This request scenario delineates the delivery of real time streamed audio only content originating from an offsite location for the corporate Intranet.
Intranet Offsite Live NetShow Presenter Request
This request scenario delineates the delivery of real time streamed audio/video content, and PowerPoint slides originating from an offsite location for the corporate Intranet.
Internet Live and On-Demand Audio & Video Request
This request scenario delineates the delivery of real time streamed audio/video content originating from any location to the Internet, including on-demand streaming after the live event.
With this extensive definition of support, responsibilities and contacts, the WME team has minimized support issues. This is essential, given that the WME team is comprised of only six people – 1 supervisor, 2 event coordinators and 3 operations managers. With this small, efficient team, they have successfully grown the digital nervous system into Microsoft's current world-class communication environment.
The implementation and continuing upgrade of Microsoft's corporate network – and the corresponding advances in Windows Media technologies – was an extremely ambitious undertaking. NetShow Services currently delivers more than 300 live and on-demand streaming events per month over one of the largest ATM implementations in the world.
The lessons learned from these inter-related projects are important for Microsoft as it moves forward with subsequent deployments of Windows Media. Other companies, armed with the knowledge gained through this experience, can plan an introduction of Windows Media Technologies and its extensive services with confidence proven by Microsoft's success. Future technology trends are important considerations, because being positioned to adapt to a new technology – and integrating that technology quickly – provides a distinct competitive advantage.
During the life cycle of the network and Windows Media projects, Global Networks maintained three goals:
To upgrade the network to an ATM backbone.
To multicast-enable the network
To support both planned and unexpected products and technologies that followed the ATM initiative by using the simultaneous build-up of the Windows Media platform to optimize a unique co-development environment
The lessons learned from these projects are:
Network Upgrade (ATM) & Multicast IP Implementation
- The infrastructure was not as delicate as the original architecture designers had believed. Global Networks modified the original designs to reduce the level of complexity in the placement of routers. This allowed MOSPF to work across the backbone without using DVMRP for rate limitation.
Protocol details were important in understanding the risks of a mixed environment, such as using MOSPF/OSPF, dependencies on designated routers (DRs) and "echo" problems with DRs on a misconfigured network.
Scoping tools were useful. It was important to recruit users who could participate in the initiative's success by providing the information necessary for them to make appropriate choices.
Creating schedules for TTL and providing locally scoped multicast addresses was a necessity.
During the implementations, bugs were found in the router multicast code, but a specifically assigned team resolved these multicast issues and helped moved the project forward at a faster pace.
Training and open discussions helped identify problems and non-problems, such as broadcast/multicast monitors in need of calibration. Sharing the extensive Global Networks and Infratek knowledge base was a huge asset in maintaining the high level of expertise needed to deliver the project by deadline.
- Most users didn't understand how multicast/broadcast, security, and quality of service issues could affect the network. Access to this information and/or focused training explaining these concerns may have cut down on the proliferation of rogue servers and other private network issues.
Windows Media Implementation
Network bandwidth issues are the foremost concern. Microsoft employees have unrestricted 24 x 7 access to the network, which presents an extremely challenging environment in which to implement a technology such as Windows Media Technologies. Evaluating existing corporate access needs during the research and design phase can help prevent future adaptability problems.
Use PowerPoint slides with streamed audio for general meeting and/or distance learning events. If this configuration can be used in place of video, it can lessen the overall bandwidth needed to deliver content. This can also produce a higher usage rate due to its simplicity as compared to a more technically demanding video.
Benefits & Return on Investments
Microsoft's Corporate Network is one of the largest routed corporate ATM Networks in the world. The scale (number of switches, connections in the ATM fabric, etc.) is unprecedented. Flexibility, scalability and future growth have been planned for and built in. Manageability and finding or creating the tools needed to do that management was also an integral part of the project.
Listed here are some of the tangible enhancements that illustrate Microsoft's return on investment:
Large reach, small investment – This ability to reach large number of people with very little resources using the accessibility of broadcast TV with the interactivity of Intranet technology has saved the company the expense of videotape production and physical distribution. By using of streamed media, Microsoft has shortened the content development production cycle and can deliver that content either during or at any time after the event, to virtually every corporate desktop around the world.
Timesavings for end user – Windows Media Player users can now access either live or on-demand content with the click of a mouse. Previously, users often needed to find and schedule a VCR or waited weeks for a requested videotape to be sent to them.
Easier to create distance learning materials – With distance training, instructors no longer need to establish an expert level in the subject as was necessary in the past for the development of computer based training (CBT). Using Window Media Technologies, the instructor can design a streamed media event that includes synchronized PowerPoint slides and other training aids. For conferences or multi-layered training events, a user can now watch all of the events at their leisure.
Controlling corporate distractions – Microsoft has found that streamed media technologies allow for a greater control over corporate rumors and myths. During a major company event, such as a reorganization or a public announcement, Microsoft executives have appeared live and conducted interactive interviews to explain what will change and related issues. The company saves not only time and money by not having to send employees to an in-person event, but it also makes the event accessible to employees worldwide on its secure corporate network.
Microsoft has also saved time and money presenting many of their regular events with Windows Media Technologies:
Quarterly Earnings Call & Analysts Meeting – Microsoft reaches more than 13,000 employees worldwide during the live discussion portion of its quarterly earnings report. Of that number, 5,000 are connected directly to the corporate Intranet, and 8,000 are linked through the Internet. This greatly simplifies participating in the event and delivers vital financial information to its shareholders much sooner than in the past.
Executive Chats – Microsoft conducts a number of "executive chats" using Windows Media Technologies, that allow employees to get to know an executive better. Microsoft averages approximately 600 participants via live and on-demand logins. This is a far larger number than a typical "conference room discussion" could accommodate.
Field Sales Quarterly CDs – In the past Microsoft spent more than $450,000 to produce the paper-based new-hire training materials. By using Windows Media Technologies, they now produce a CD containing streamed media content that has a production cost of only $20,000 – a savings of more than 95%!
As Microsoft IT looks to the future of computer communications – and all communications – it is obvious that the tools that they have today will fall short. Therefore, they are working on the communications tools and software of tomorrow. In order to meet that future need, their research and development teams need to work in an environment that matches the computing environment of the future as closely as possible whether it's over the intranet or the Internet.
In addition, Microsoft IT has a worldwide business to run. The data and applications used to run the business are critical to the corporation's success. The computer network is the digital nervous system and truly at the heart of the business.
Microsoft's computer network serves as both the company's development laboratory and its business operations center. The goal of Microsoft IT was to architect and build a computer network that would be state-of-the-art for both. Up to today, that project has been successful.
Microsoft IT has built an intelligent, scaleable network that is capable of integrating voice, video and data over a common infrastructure worldwide. This new network enables next generation applications such as Windows Media Technologies and the NetMeeting® conferencing software to empower employees to collaborate on projects with colleagues around the world.
Research and development now has a leading-edge computing arena upon which they can test applications and other software just as if it were on the Internet but in a secure and safe environment. On the same network, at the same time, our business can also enjoy the advantages of that same network. End users have a multi-media environment at their disposal with all the advantages of a high-end development network. On top of that, they can all receive consistent, interactive training direct to their desktops.
The worldwide creation of the Microsoft Digital Nervous System involved members throughout Microsoft's Global Networks and other IT Operations team members around the world. It serves as a model for expansion that will tie Microsoft employees together around the world.
Work is already underway to upgrade Microsoft's network in Europe. The rest of North America will follow and then the rest of the world.
The Windows Media Technologies development team is working on the next version of NetShow that will deliver new technologies including plans for a new HTML and ASP version of the Windows Media Server Admin application. This version eliminates the need for the Visual Basic runtime and supports many SQL & DHTML procedures that will enhance its management capabilities.
The Windows Media Events team has maintained its relationship with both the development team and the IT network organization. As the Windows Media Technologies phenomenon spreads throughout the corporation, considerations for the global network include better end-user support and regional content development deliverable worldwide to any other region.
Together these teams have worked to create the first "production environment" for all company products. This invaluable test platform builds a solid user experience base and demonstrates the new technology for others considering implementation.
A company's ability to manage its information can determine whether that company wins or loses. Today, Windows Media Technologies are the backbone of the Microsoft corporate Digital Nervous System. As future developments in audio and video streaming emerge, Microsoft is dedicated to be a trailblazer by incorporating these new technologies into its employees daily work environment.
For More Information
Latest information on Microsoft Windows Media Technologies can be found at:
To view additional IT Showcase material, please visit http://www.microsoft.com/technet/itshowcase/.
For any questions, comments or suggestions on this document, or to obtain additional information about Microsoft IT Showcase, please send email to email@example.com.
|1||These 2 cities weren't implemented until July 1997, with dedicated circuits.|
|2||The Windows Media Technologies guidelines can be further explained at|