Wenming's Big Data and Big Compute Blog

All about running HPC and Big Data workload on the Microsoft Windows Azure platform. I have been and will continue to cover a variety of topics on Hadoop, HPC, Migration from / Interops with Unix, Tips and Tricks for running HPC and Big Data.

Data Science in a Box using IPython: Scipy and Scikit-Learn (3/4)

In the first two blogs of this series, we installed the IPython notebook using the minimum...

Author: HPC Trekker Date: 04/17/2013

Data Science in a Box using IPython: Installing IPython notebook (2/4)

In the previous blog, we demonstrated how to create a Windows Azure Linux VM in detail. We will...

Author: HPC Trekker Date: 04/09/2013

Data Science in a Box using IPython: Creating a Linux VM on Windows Azure (1/4)

I just returned from the Python in Finance Conference in New York, I would like to thank Bank of...

Author: HPC Trekker Date: 04/09/2013

Enter the Big Data Matrix: analyzing meanings and relations of everything (2/2)

Running the Python example step by step: We explained the basic idea behind LSA or latent semantic...

Author: HPC Trekker Date: 04/07/2013

Enter the Big Data Matrix: analyzing meanings and relations of everything (1/2)

Data Science is compute and labor intensive In the previous blogs, we showed you how to find a...

Author: HPC Trekker Date: 04/07/2013

New Breakthrough in Big Data Technologies: the NullSQL Paradigm shift

Mammoth the NullSQL tool Most of us by now understand the properties of big data. Many of us are...

Author: HPC Trekker Date: 04/01/2013

Make another small step, with the JavaScript Console Pig in HDInsight

Our previous blog, MapReduce on 27,000 books using multiple storage accounts and HDInsight showed...

Author: HPC Trekker Date: 03/31/2013

MapReduce on 27,000 books using multiple storage accounts and HDInsight

In our previous blog, Preparing and uploading datasets for HDInsight, we showed you some of the...

Author: HPC Trekker Date: 03/30/2013

Preparing and uploading datasets for HDInsight

In the previous blog ...

Author: HPC Trekker Date: 03/30/2013

Finding and pre-processing datasets for use with HDInsight

  Free datasets There are many difficult aspects associated with Big Data, getting a good,...

Author: HPC Trekker Date: 03/30/2013

Let there be Windows Azure HDInsight

Windows Azure HDInsight Service, formerly known as Hadoop on Windows Azure, is now available inside...

Author: HPC Trekker Date: 03/19/2013

Windows HPC Pack 2012 Tutorial (1/5): Installing Server 2012 on an IaaS VM

In a previous blog, I explained Big Compute and the newly released HPC Pack 2012 from...

Author: HPC Trekker Date: 12/19/2012

Windows HPC Pack 2012 released, HPC/Big Compute explained for Windows Azure developers

While most of you that have been using the Microsoft HPC Server product know the history and...

Author: HPC Trekker Date: 12/19/2012

Running Weather Research Forecast as a Service on Windows Azure

About 9 months ago, I deployed a Weather forecast demo at an internal Microsoft event, Techfest. The...

Author: HPC Trekker Date: 11/26/2012

Super computing 2012 and new Windows Azure HPC Hardware Announcement

The Super Computing conference attracts some of biggest names in the industry, academia, and...

Author: HPC Trekker Date: 11/21/2012

Hadoop On WindowsAzure Updated

HadoopOnAzure allows a user to run Hadoop on Microsoft Windows Azure as a service. It is currently...

Author: HPC Trekker Date: 08/22/2012

Deploying an HPC Cluster using just PowerShell (Part I)

Last month my colleague Don Pattee at the HPC team announced the availability of the Windows Azure...

Author: HPC Trekker Date: 12/06/2011

Just Published: Five learning samples for Windows HPC with Burst to Windows Azure covering Parametric Sweep, Cluster SOA, and Excel UDF offloading Programming Models

  If you have CPU intensive jobs that require 100s of Cores on Azure or on HPC on-premise...

Author: HPC Trekker Date: 05/06/2011

Windows HPC2008R2 SP2 Azure Burst Beta Documentation Available

  I will update here with more information as they become available for this exciting...

Author: HPC Trekker Date: 05/03/2011

Windows HPC Server 2008 R2 SP2 Beta is out!

I will be covering some of the new, and more complete Azure burst features on my blog, please stay...

Author: HPC Trekker Date: 05/02/2011

Don’t trust the weatherman? Running my own weather forecast on Windows HPC Server (Rain is coming back to Seattle Monday)

One of the interesting things living in Boulder Colorado for almost 20 years is that I bump into...

Author: HPC Trekker Date: 05/01/2011

Having a BLAST on Windows HPC with Windows Azure burst (1/3)

  My previous blog showed you how to make a movie using ray tracing software and the powerful...

Author: HPC Trekker Date: 05/01/2011

Got a powerful GPU? Run my World Wide Telescope 3D Seismic Simulation Tour

  For the last two years, I’ve been using my 6 core 3.3 ghz Intel Nehalem X 980 processor...

Author: HPC Trekker Date: 04/30/2011

HPC Server SP1: Burst to Azure with little or no programming

  We all know that Scientists and Engineers run Computationally intensive jobs. Many of us...

Author: HPC Trekker Date: 04/30/2011

Notes on Chemistry codes

I attended and gave a tutorial at the 11th LCI International conference last year at the Pittsburgh...

Author: HPC Trekker Date: 04/30/2011

Windows HPC Server 2008 R2 Launch Event in New York City

Windows HPC Server 2008 R2 or version 3 of the Windows HPC Server product has RTMed in July. The...

Author: HPC Trekker Date: 09/11/2010

New version: HPC Debugger Enhancements for MPI and Cluster-SOA

This package includes two enhancements to ease debugging HPC applications. The first feature...

Author: HPC Trekker Date: 01/13/2010

Windows HPC Server and the game of Go: Challenging 10^170 search space.

In the past year, the Microsoft HPC team has demonstrated the ManyFacesOfGo application running on...

Author: HPC Trekker Date: 01/06/2010

Windows HPC Server 2008R2 Beta1 Channel 9 Videos

At SC09 Microsoft Introduced Windows HPC Server 2008R2 Beta1.  Channel 9 videos for Beta1...

Author: HPC Trekker Date: 12/16/2009

Unix to Windows API dictionary project

As an HPC cross platform developer, I have spent a good part of my 20s working on scientific codes...

Author: HPC Trekker Date: 11/14/2009

Super computing 2009 next week

Quite a few folks from the Microsoft HPC team will be heading down to Portland next week to attend...

Author: HPC Trekker Date: 11/14/2009

China's new fastest supercomputer

China has apparently become the third country to build a petaflop supercomputer. Xinhua news agency...

Author: HPC Trekker Date: 10/31/2009

Microsoft buys Interactive Supercomputing (Star-P maker)

Kyril Faenov, the General Manager, HPC and Parallel Computing Technologies, announced last week that...

Author: HPC Trekker Date: 09/28/2009

Installing SUA--- a Walk through

I have explained what SUA is and how it differs from Cygwin in previous blogs. Basically, SUA stands...

Author: HPC Trekker Date: 09/28/2009

5 minute tutorial on the amazingly easy to use MPI Cluster Debugger

The MPI Cluster Debugger (with C/C++ project templates) is now available via the Microsoft Download...

Author: HPC Trekker Date: 07/18/2009

Examples of linking in MSMPI library.

MSMPI is Microsoft’s implementation of MPI (Message passing interface) library.  I’ve explained...

Author: HPC Trekker Date: 06/26/2009

LIZARD released into the HPC wild

    One of my previous blog explains what linpack is:   LAPACK, LINPACK, ATLAS,...

Author: HPC Trekker Date: 06/26/2009

Windows HPC resource Kit version 2 available

resourcekit.windowshpc.net Finally version 2 is updated and online. Changes include: New Parallel...

Author: HPC Trekker Date: 05/01/2009

Linux Cluster International Conference

The 10th LCI (Linux Cluster Institute) international Conference on High Performance Clustered...

Author: HPC Trekker Date: 03/21/2009

HPC MATH ATTACKS: BLAS/LAPACK, LINPACK, ATLAS, DGEMM, ACML, MKL, CUDA

When it comes to HPC development, linear algebra libraries are the essential building blocks. ...

Author: HPC Trekker Date: 02/24/2009

mpiexec not working on windows XP? We got a patch.

Overview This update fixes an issue where the HPC Pack 2008 SDK was attempting to use a function...

Author: HPC Trekker Date: 02/13/2009

HPC Web cast Marathon into 2009!

High Performance Computing (HPC) Clusters: Reduce Complexity with IBM and Microsoft Please join us...

Author: HPC Trekker Date: 01/02/2009

Say goodbye to 2008 with 10 HPC web casts

It's the last day of 2008, and I am uploading some of the web casts that we have done for HPC in the...

Author: HPC Trekker Date: 12/31/2008

We are HPCs @ PDC

I am at Microsoft PDC this week, demonstrating the Microsoft HPCS Server 2008. Please come by the...

Author: HPC Trekker Date: 10/27/2008

HPC Resource Kit

Yes, it's ready! I have compiled a 70 item kit full of white papers, presentations, videos, and...

Author: HPC Trekker Date: 10/23/2008

No scientist left behind with CRAY Supercomputer running Windows HPC Server 2008

Many people are still wondering exactly what Windows HPC Server 2008 is. On the launch day of the...

Author: HPC Trekker Date: 09/22/2008

Running PETSC on windows

PETSc, stands for portable, extensible toolkit for scientific computation, it is a suite of data...

Author: HPC Trekker Date: 09/08/2008

Porting your HPC Code to Windows

One of the most frequent questions I get when I talk to customers is, what is my best option for...

Author: HPC Trekker Date: 09/06/2008

Windows HPC Server 2008 Management, Monitoring and Diagnostics

I uploaded two more videos on Technet/Edge. They are worth watching if you are interesting in an...

Author: HPC Trekker Date: 09/01/2008

HPC Basic Profile Videos

HPC Basic Profile Web Service or the HPCBP for short, is a web service, built using the Windows...

Author: HPC Trekker Date: 08/24/2008

Next>