Overview: Numerical Libraries for F# and .NET Framework

Article
09/12/2011

Applies to: Functional Programming

Authors: Yin Zhu

Referenced Image

Get this book in Print, PDF, ePub and Kindle at manning.com. Use code “MSDN37b” to save 37%.

Summary: Numerical applications usually rely on existing libraries. For example, a job scheduling application may be reduced to a linear programming problem, which can be solved by many stable software packages. This overview surveys the popular numerical libraries available on .NET Framework.

This topic contains the following sections.

Introducing Numerical Libraries
Libraries
Other Numerical Libraries
Numerical Libraries Overview
Evaluating .NET Performance for Math
Summary
See Also

This article is associated with Real World Functional Programming: With Examples in F# and C# by Tomas Petricek with Jon Skeet from Manning Publications (ISBN 9781933988924, copyright Manning Publications 2009, all rights reserved). No part of these chapters may be reproduced, stored in a retrieval system, or transmitted in any form or by any means—electronic, electrostatic, mechanical, photocopying, recording, or otherwise—without the prior written permission of the publisher, except in the case of brief quotations embodied in critical articles or reviews.

Introducing Numerical Libraries

F# is a member of the .NET family of languages, which means that it can easily access any numerical library that was designed for the .NET Framework. There are also a few math libraries designed particularly for F#, but they generally do not offer as wide a range of functionality as do the general libraries for .NET Framework that can be used from C# and Visual Basic.

In general, most of the libraries targeting .NET Framework use a common subset of types and functionality that is understood by all languages. The libraries do not use language-specific types, such as an F# list, but prefer types that are understood by all languages, such as arrays or IEnumerable<T> (called seq<T> in F#).

Most of the libraries discussed in this article were not primarily designed for F#, but there are some exceptions. However, some of the .NET libraries provide additional library or samples for F# that makes them easier to use from F#.

Libraries discussed in this overview that provide some support for F# include:

**Libraries designed for F#—**The F# PowerPack with the F# MathProvider and FSharpChart are open-source libraries that provide types for matrices, wrappers for the native linear algebra implementation, and charting libraries for F#. The F# for Numerics and F# for Visualization commercial libraries implement linear algebra, numerical integration and differentiation, statistics, Fast Fourier Transform (FFT), and both 2D and 3D visualization.
**Libraries with F# interfaces—**The Math.NET Numerics open source library and the Extreme Optimization commercial library are aimed at the .NET Framework in general, but they both provide friendly interfaces and wrappers for the F# language.

Libraries

This section gives a brief overview of the F# PowerPack and the F# MathProvider, Microsoft Sho, and Math.NET Numerics. This chapter contains more information as well as examples for each of the libraries discussed in this section.

F# PowerPack and F# Math Provider

The F# PowerPack is a collection of F# libraries developed by the F# team. It contains a basic numerical library for working with vectors and matrices as well as several other features (including lexing, parsing, and LINQ support). The PowerPack library contains a managed implementation of basic operations with matrices.

The F# MathProvider extends the F# matrix and vector types with more complete support for linear algebra operations. It can be used as a wrapper for BLAS and LAPACK (using Netlib and Intel MKL implementations). The library also provides a managed implementation of many of the operations.

The F# PowerPack and the F# MathProvider are both released under Apache 2.0 license (allowing unlimited commercial and noncommercial use). For more information, see the following links and documents:

F# PowerPack and F# MathProvider (at CodePlex)
Tutorial: Using the F# PowerPack and the F# MathProvider

Math.NET Numerics

Math.NET Numerics aims to be the standard open-source math library for the .NET Framework. It provides the methods and algorithms for numerical computations in science, engineering, and everyday use. The functionality covered by Math.NET Numerics includes special functions, linear algebra, probability models, statistics, random numbers, interpolation, and integral transforms (FFT).

Math.NET Numerics provides a fully managed implementation that runs on .NET 4.0, Silverlight 4, and Mono (but can be compiled for other platforms). It also provides a parallelized managed implementation and supports optimization using native BLAS/LAPACK libraries (GotoBLAS, Intel MKL, and AMD ACML).

Math.NET Numerics is released under the MIT license (allowing unlimited commercial and noncommercial use). For more information, see the following resources:

Math.NET Numerics for .NET and Mono
Math.NET Numerics (at CodePlex)
Tutorial: Using Math.NET Numerics in F#.

Microsoft Sho

Microsoft Sho is a set of libraries and tools for numerical computing for the .NET Framework. Sho provides a programming interface reminiscent of Matlab or R that is based on the IronPython language. However, the comprehensive libraries provided by Sho can also be accessed from other .NET languages, including F#. The libraries implemented by Microsoft Sho include support for linear algebra, statistics, optimization, signal processing, as well as visualization.

Microsoft Sho is a stable research project developed by Microsoft Research. The source code for this library is not made available and the library can be used under a noncommercial license. For more information about Sho and using it from F#, see the following resources:

Other Numerical Libraries

The previous section presented libraries that are discussed in detail in this chapter and provided the links to articles that demonstrate how to use them from F#. There are a large number of other numerical libraries for the .NET Framework, both open-source and commercial. This section gives a brief (and necessarily incomplete) overview of some of them.

Alglib

Alglib is a cross-platform numerical analysis and data processing library. It is implemented in several different languages, including a managed C# version. The library supports linear algebra with matrix and vector operations, solving differential equations, linear and nonlinear equations, numerical integration, optimization, FFT, statistics, and data analysis. The library uses a lightweight programming interface. Most of the algorithms are implemented as static methods and their parameters use basic data types (such as arrays).

Alglib uses a dual licensing model. It can be used under the GPL license (for open source and research projects) and it is also available with a commercial license. More information is available on the project website:

Alglib project homepage

DotNumerics

DotNumerics is a managed numerical library written in C#. The main features of DotNumerics include linear algebra, solving differential equations, and optimization. The functionality is based on optimized FORTRAN code translated into C#.

The part for linear algebra includes code based on three libraries: BLAS, LAPACK, and Eispack. The optimization library supports unconstrained and bounded-constrained minimization using three methods (L-BFGS-B, Truncated Newton, and the Simplex method of Nelder and Mead).

DotNumerics is licensed under the GPL license. More information can be found on the project website:

DotNumerics: Numerical Library for .NET

Extreme Optimization Library

The Extreme Optimization Numerical Libraries for .NET are a collection of general-purpose mathematical and statistical classes built for the .NET Framework. They provide additional numerical types (complex, arbitrary-precision rational and floating point), sparse and dense matrices, vectors, support for generic arithmetic and libraries for linear algebra and statistics.

The linear algebra library supports both dense and sparse matrices and implements algorithms such as matrix factorization, BLAS and LAPACK routines, and others. The statistics library includes statistical models, distributions, hypothesis tests, as well as multivariate statistics.

Extreme Optimization Library is designed for the .NET Framework in general, but it contains a large number of F# samples. It is licensed under a commercial license and it provides a 60-day trial period. More information can be found on the library homepage:

Extreme Optimization Numerical Libraries for .NET

F# for Numerics and F# for Visualization

F# for Numerics and F# for Visualization are two libraries developed for the F# language that provide an elegant interface based on functional programming. The first library provides a wide range of numerical methods and the second adds visualization using 2D and 3D vector graphics.

The numerical methods implemented by F# for Numerics include linear algebra algorithms such as matrix decomposition, descriptive statistics, curve fitting and regression, as well as spectral methods such as FFT. The visualization library provides an easy-to-use API that can be used interactively from F# in Visual Studio.

Both F# for Numerics and F# for Visualization are available under a commercial license. More information can be found on their website:

F# for Numerics and F# for Visualization.

ILNumerics.Net

The ILNumerics.Net library provides numerical algorithms, linear algebra, sorting routines, and a library for visualization. The implementation uses processor-specific optimized native libraries such as Intel MKL, AMD ACML, and Netlib. It also wraps to a highly optimized Fast Fourier Transform library, FFTW3. The visualization and plotting features support both 2D and 3D charts based on OpenGL.

The ILNumerics.Net library is licensed under the LGPL license and the visualization library is available under the GPL license. More information can be found on the project homepage:

ILNumerics.Net - numeric computing for .NET

Microsoft Solver Foundation

Microsoft Solver Foundation (MSF) is a fully supported Microsoft product that provides a .NET solution for mathematical modeling and optimization. MSF provides dedicated solvers for a wide range of optimization problems including linear programming, quadratic programming, constraint satisfaction problem, and a general nonlinear optimization. MSF can be programmed using an optimization modeling language (OML), imperatively in C#, and using an F# domain-specific language.

Microsoft Solver Foundation can be easily integrated with other tools. It supports Excel integration as well as wrappers for other commercial optimization products, such as Gurobi, Ziena Knitro, Mosek, and others.

Microsoft Software Foundation is available in several versions. The Express edition is freely available, Standard edition is available as part of MSDN Subscription and Enterprise/Academic version can be obtained either commercially or as part of MSDN Academic Alliance. For more information see the following resources:

Microsoft Solver Foundation
Support Vector Machines in F# Using MSF (Yin Zhu's blog)
Logistic Regression in F# U (Yin Zhu's blog)

NMath .NET Math and Statistics

NMath is an object-oriented .NET math library on the .NET platform that supports linear algebra and function optimization. NMath Stats is a .NET statistics library that can be used for data manipulation, statistical computation, as well as biostatistics.

The NMath library provides functionality for linear algebra (including both sparse and dense matrices and routines such as matrix decomposition), optimization (including linear programming, quadratic programming and non-linear programming) and others. The NMath Stats library implements a data frame class and provides standard descriptive statistics, probability distributions, basic hypothesis tests, and others.

NMath and NMath Stats are available under a commercial license and they both provide a 30-day trial period. For more information, see the product website:

NMath .NET and Statistics

Numerical Libraries Overview

The following table gives a brief summary of the features available in the nine libraries introduced in the previous two sections. The information is based on the versions available at the time of writing and may change in the future.

Library	License	Linear Algebra	Optimization	FFT	Statistics	Designed for F#
Alglib	Dual (GPL, Commercial)	Managed	Yes	Yes	Yes	No
Math.NET Numerics	MIT	Managed, Native	No	No	Yes	Yes
DotNumerics	GPL	Managed	Yes	No	No	No
F# PowerPack and F# MathProvider	Apache 2.0	Managed, Native	No	No	No	Yes
Microsoft Sho	Noncommercial	Native	Yes	Yes	Yes	No
Microsoft Solver Foundation	Free, Commercial, Academic	N/A	Yes	No	No	No
Extreme Optimization	Commercial	Managed, Native	Yes	Yes	No	Samples
F# for Numerics, F# for Visualization	Commercial	Managed	No	Yes	Yes	Yes
NMath	Commercial	Native	Yes	Yes	No	No

Evaluating .NET Performance for Math

Math libraries for .NET can be implemented in several different ways. This section presents a general survey of the various techniques and discusses their advantages, disadvantages, and performance. Understanding the options is useful for choosing the right math library for F#.

This section presents a simple performance benchmark run on a single computer to give some performance estimates. It doesn't aim to provide complete information. When measuring performance, it is usually necessary to measure the performance in a concrete case. This section may offer some approximate background. Most importantly, it is often believed that a managed environment such as .NET must necessarily provide significantly worse performance than the native implementations. This is often not true, and there are also situations where a managed implementation is the only choice (for example, when using platforms such as Silverlight).

In addition to F# and C#, we also consider dynamic languages such as IronPython. IronPython is a managed implementation of Python for the .NET Framework. It provides similarly succinct syntax as F#. Unlike F#, IronPython is a dynamically typed language, which means that the performance of managed IronPython code is considerably slower (in the current implementation, at least). The usual way to write numerical computations in IronPython is to use optimized native libraries such as Microsoft Sho and Numpy/Scipy for IronPython.

The following list shows five common ways to implement numerical algorithms for the .NET Framework. The math libraries introduced in this article generally use one or more of the options:

Calling processor-optimized native code via P/Invoke. Managed code can use Platform Invoke to call a highly-optimized math library that uses CPU specific optimizations and instructions (for example, Intel MKL).
Calling non-optimized native code via P/Invoke. Many C, C++ or FORTRAN libraries for math are compiled to native code, but are not highly optimized. In many cases, they can still provide good performance.
Calling optimized native code from dynamic languages. Languages like IronPython use P/Invoke to call native libraries for numerical computing. However, they introduce additional overhead because of dynamic typing.
Managed code using C++/CLI. Code that was originally written using C/C++ can be compiled into .NET using the C++/CLI language. The potentially unsafe native code can be exposed by writing a managed .NET wrapper that can be used from F#.
Using unsafe C# code. The C# language makes it possible to write unsafe code that manipulates pointers directly and that does not perform array bounds checking. This alternative can yield a performance benefit over the standard managed implementation.
Managed C# code. The next option is to implement the math algorithm using fully portable and managed C# code that can run on any platform based on the .NET Framework.
Managed F# code. An algorithm implemented in F# will also be fully managed and portable. As the results show, the performance profile of F# is very similar to the performance of C#.
Managed IronPython code. A numerical computation can be also implemented in a dynamic language such as IronPython. The test used to measure the performance did not finish in a reasonable time, which shows that this approach is significantly less efficient than managed C# or F# code.

Table 1 compares the performance of the individual methods using matrix multiplication as an example. The test multiplies a random matrix of an arbitrarily chosen size 280×10304 with its transpose containing randomly generated floating point numbers between 0 and 1. The details about implementation and configuration used for testing are explained below.

Table 1. Performance of matrix multiplication

Method	Average time	Method	Average time
P/Invoke optimized (1)	0.41 s	Unsafe C# (5)	6.28s
P/Invoke native (2)	3.96 s	Managed C# (6)	13.16s
IronPython/NumPy (3)	5.11 s	Managed F# (7)	13.08s
Managed C++/CLI (4)	5.50 s	Managed IronPython (8)	did not finish

As expected, using P/Invoke (1) provides the best performance because it runs a highly optimized native code. The naive native implementation (2) is still faster than a managed implementation, but this difference would be less significant when performing more than a single operation, because there would be a larger overhead for the interoperability between the managed and the native code. Using a native implementation such as the one provided by NumPy from a dynamically typed language such as IronPython (3) gives somewhat worse performance, because of additional boxing and unboxing.

The implementations in C++/CLI (4) and in unsafe C# (5) are comparably efficient to the naive native implementation. Finally, fully managed code has lower performance (6, 7) but is still reasonable for many practical tasks (the example is about four times slower). The performance of managed code can be tuned in many ways and the tests presented in this article do not perform any sophisticated optimizations and measure naive code. For example, it is possible to improve the performance by running Native Image Generator (ngen) that generates native code before the application is used. Nevertheless, many managed libraries provide parallel implementation that can easily compensate for this slowdown (this is not measured in the above table). It is also worth noting that the performance of F# implementation is essentially the same as the C# code.

The managed IronPython implementation (8) is substantially slower than F# or C#. In IronPython, all values including numbers are boxed heap objects. This introduces significant overhead. The performed tests show that matrix multiplication for much smaller inputs (280×10 and 10×280) costs about 8 seconds. In practice, this means that dynamic languages always need to rely on efficient implementation of numeric computations in another language (such as native NumPy or managed Math.NET Numerics).

Finally, it is worth adding that only managed C# or managed F# code is available on all platforms based on the .NET Framework including Silverlight and Windows Phone 7. The managed IronPython implementation relies on the availability of additional Dynamic Language Runtime (DLR).

Configuration and Implementation Details

The tests were executed repeatedly and the results are averaged over 10 executions. The tests were performed using .NET Framework 4.0 on the 32-bit version of Windows 7. The hardware used was a standard business laptop with 2.0 GHz Dual Core CPU and 3 GB of memory. The specific configuration for individual methods was following:

Methods 1 and 2 are based on F# PowerPack and use F# MathProvider to call the native implementation. Method 1 uses optimized Intel MKL library for BLAS and method 2 uses unoptimized Natlib implementation.
The implementation in method 3 is written in unmanaged C++ and compiled to .NET using /clr option of the C++/CLI compiler.
Code for method 4 is implemented using unsafe C# and compiled with the /unsafe option.
Code for methods 5 and 6 uses standard mutable .NET array and imperative coding style based on for loops.

The code used to measure the performance of managed F# implementation used the code shown below. The implementation used in methods 4, 6, 7, and 8 followed the same style:

let matrixMultiply (a:float[,]) (b:float[,]) (result:float[,]) 
                   (row:int) (col:int) (inner:int) = 
    for i = 0 to row - 1 do
        for j = 0 to col - 1 do
            let mutable sum = 0.0
            for k = 0 to inner - 1 do
                sum <- sum + a.[i, k] * b.[k, j]
            result.[i, j] <- sum

Summary

This article presented a survey of numerical libraries for the .NET Framework that can be used when writing math in F#. This chapter includes more information about three of the discussed libraries, but there is a large number of other libraries, both open source and commercial.

The libraries that are discussed in detail in other articles include the F# PowerPack with the F# MathProvider (which provides a basic implementation of the matrix type for F# with native wrappers for BLAS and LAPACK libraries), Math.NET Numerics (which aims to be a standard open-source math library for .NET with a wide range of algorithms), and Microsoft Sho (which is a noncommercial numerical library from Microsoft Research).

The last part of the article focused on various options for implementing math on .NET. It compared the performances of several implementations, including optimized native implementation, naive native implementation, and fully managed code. Understanding the differences is important when choosing the right numerical library.