Data Science in a Box using IPython: Scipy and Scikit-Learn (3/4)

In the first two blogs of this series, we installed the IPython notebook using the minimum requirement.   

The third blog post will walk you through some of the common packages used for Data Science. 

SciPy/NumPy packages are usually mentioned together.  At this point, we have not installed SciPy, SciPy includes a collection of numerical packages, that includes Linear solvers that we used in a previous post.  Enter the Big Data Matrix: analyzing meanings and relations of everything (2/2)

To install the package type: sudo apt-get install python-scipy

Scikit Learn is a fantastic python-based machine learning package, it includes algorithms for both supervised and unsupervised learning.  Moreover, it includes support for sample datasets, data import tools, and model evaluation.

Scikit Learn is included with your Ubuntu distribution, but the default is about 2 versions behind.  The best way to install Scikit Learn is to use PIP.

type: pip install scikit-learn

The installation process includes building many of the packages from scratch; much of the code base is written in C. Check the installation for errors. You can verify by checking new files in /usr/local/lib/python2.7/dist-packages for sklearn.


Getting samples

It is easy to find samples and run them in IPython Notebook.  You can easily get them from various websites and even tutorials.  To save you time, I’ve make a small collection at: 

Get the package by typing:  wget 

On your Ubuntu box, you might have to install unzip by typing:  sudo apt-get install unzip 

Unzip; then copy content from BigDataSamples-master/ipythonMLsamples into your Ipython dir. A sample command may look like:

cp /home/azureuser/samples/BigDataSamples-master/ipythonMLsamples/* /home/azureuser/.ipython/

Check to make sure the files have been copied.


Running the samples

Go back to the website for IPython, log in, and the files listed should show up in the root directory.  Click on K-Means clustering on the handwritten digits data.




Click on the Play button to run the machine learning sample.



The code uses the K-means algorithm with 3 different types of initialization, then plots the results.




The code for making the color scattered plot.



Additional samples to explore

These samples are also includes, feel free to explore them on your own.

  • A demo of K-Means clustering on the handwritten digits data
    A demo of structured Ward hierarchical clustering on Lena image
    Faces dataset decompositions
    Gaussian Processes regression
    Manifold learning
    Non-linear SVM
    Hand writing recognition using SVM
    Hierarchical clustering-structured vs unstructured ward
    demo2 of the K Means clustering algorithm
    Weighted SVM
    Visualizing the stock market structure




IPython Notebook gives us a quick and easy way to share compute resources through the web-based IPython notebook interface.  Scikit-Learn, NumPy, and Scipy all simply work out of the box for IPython notebook.  The simple, yet powerful combination lets users focus on learning and getting the data analysis done.

In the next blog, we will introduce additional packages in Python that can be used for Data analysis including scaling out using clustering.