Setting Up a Development Environment for LINQ to HPC

  • Article

Important

This will be the final preview of LINQ to HPC and we do not plan to move forward with a production release. In line with our announcement in October at the PASS conference, we will focus our effort on bringing Apache Hadoop to both Windows Server and Windows Azure. For more information, see Microsoft to develop Hadoop distributions for Windows Server and Azure and Microsoft Expands Data Platform to Help Customers Manage the ‘New Currency of the Cloud’.

To begin developing a LINQ to HPC application, you must set up a small Windows HPC cluster that has Microsoft HPC Pack 2008 R2 with Service Pack 3 installed. You must also install the HPC SDK and client utilities on a development computer. The topics in this section include the requirements and steps to help you set up your development environment and get started.

This section includes the following topics:

Overview of the requirements for a LINQ to HPC development environment

This section summarizes the basic requirements for setting up a development environment.

Development computer requirements

  • Microsoft Visual Studio 2010 with SP1 installed.

  • Microsoft HPC Pack 2008 R2 with SP3 SDK installed. Download SDK here.

  • Microsoft HPC Pack 2008 R2 with SP3 client utilities installed (to submit your application to an HPC cluster from a client computer and to remotely manage your cluster). Download client utilities here.

  • Your development computer should be joined to the same Active Directory domain as the cluster head node.

HPC cluster requirements

  • The minimum cluster size is based on the DSC replication factor which by default is set to three. In this case, you would need at least three nodes in your cluster. For more information, see Set Up a Development Cluster with Windows HPC Server.

  • Compute nodes that will run LINQ to HPC jobs should have a minimum of 4-8 GB of RAM, and a minimum of 200 GB available disk space. The appropriate amount of hard disk drive storage depends on the amount of data that you expect to process.

  • The nodes in your cluster must have a 64-bit version of the Windows Server 2008 R2 operating system installed. You can download an evaluation version of the operating system with the Windows HPC Server 2008 R2 Evaluation Suite here.

  • The nodes in your cluster must be part of an Active Directory domain. For more information about this and other general requirements, see one of the deployment guides referenced in Set Up a Development Cluster with Windows HPC Server.

Note

While LINQ to HPC jobs are not supported on Azure Nodes that are joined to an on-premises cluster, some types of LINQ to HPC applications are suitable to run in a standalone Windows Azure deployment. For more information, see the Window Azure HPC Scheduler. If you do want to develop LINQ to HPC applications for Windows Azure, it is helpful to have an on-premises cluster for initial development and debugging.