Model Evaluation on Windows
Currently, CNTK provides libraries in C++, C#/.NET, Python and Java for loading and evaluating models on Windows. Starting from v2.1, CNTK also supports Universal Windows Platform (UWP).
Using C#/.NET Managed API
The CNTK Library provides a managed (.NET) library for evaluation of CNTK models on CPU and GPU using C# and other .NET languages. The NuGet package CNTK.CPUOnly is designed for execution on CPU, and the CNTK.GPU is designed for execution on devices with NVIDIA GPU. Please refer to the NuGet Package page about details how to install CNTK Library NuGet Packages. Please note that Visual Studio 2017 is required and the target platform in the project property should be X64.
The CNTK Library Managed API allows you to load and evaluate pre-trained models on specified devices (CPU or GPU). It also provides different ways to help you to prepare input and output data in the dense, one-hot vector or sparse CSC format. The input data can be a single sample, batch of multiple samples, a sequence of samples, or a batch of sequences with variable lengths. A detailed description about batch and sequence format can be found at the page CNTK Concepts. After evaluation, the API provides convenient methods to get results.
C#/.NET Programming Guide
The usual steps for model evaluation using CNTK Library Managed API include:
- Install the appropriate NuGet package (CPU or GPU depending on usage) to your. Details are described in the CNTK NuGet Package.
- Load the model by Function.Load(). The returned value is a
Functioninstance referring to the loaded model.
- If needed, get inputs and outputs of the loaded model by calling the properties
Outputsof the Function instance returned by Load(). The inputs and outputs are represented as
Variable. Further information about each individual input or output, e.g.
Shapecan be found by calling corresponding properties of the Variable.
- For each input, prepare input data for evaluation. Depending on usage, input data might consist of a single sample, batch of multiple samples, a sequence of samples, or a batch of sequences with variable length. Regarding storage, input data can be in the dense format, one-hot vector, or sparse CSC format. The
Valueobject is used to represent the input data with different contents and formats. The CNTK API provides different methods to easily create the Value object from plain user data.
CreateBatch()creates a Value object containing a batch of single or multiple samples.
CreateSequence()creates a Value object containing a sequence with multiple samples. It is possible to specify whether this sequence is a new sequence or continuation of a previous sequence at the same index in the sequences list from a previous call to this method.
CreateBatchOfSequences()creates a Value object containing a batch of sequences. Each sequence has a variable length of samples. All the methods above support both the dense, one-hot vector or sparse CSC format.
- Call the
Evaluate()method to execute evaluation. The parameter
argumentsdefines input Variables and their associated Values, and
outputsdefines output Variables. If the Value object of a output variable is
null, CNTK Library allocates the actual storage for the output Value. On return, Evaluate() saves the evaluation results into the Value objects associated the output variables.
- Call the
GetDenseData()/GetOneHotData()method of the output Value to get the results in plain data format. It copies the data stored in the Value object into the buffer as a list of sequences with variable length samples, either in dense format or one-hot vector format (support for the sparse format will be added soon).
The CNTK Library Managed API is described in the CNTK Library C#/.NET Managed API page.
Evaluation of multiple requests in parallel
CNTK supports evaluating multiple requests in parallel. Because running evaluation on the same model instance is not thread-safe, it is required first to create multiple model instances by calling Clone() with
ParameterCloningMethod.Share, and then each thread uses a separate model instance for evaluation. The
EvaluateMultipleImagesInParallelAsync() demonstrates how to evaluate concurrent requests using CNTK C#/.NET Managed API.
Run evaluation asynchronously
CNTK C# API does not have an asynchronous method for Evaluate(), because the evaluation is a CPU-bound operation (Please refer to this article for detailed explanation). However, it is desired to run evaluation asynchronously in some use cases, e.g. offloading for responsiveness, we show in the example
EvaluationSingleImageAsync() how to achieve that by using the extension method
Evaluate intermediate layers
CNTK C# API is able to evaluate an intermediate layer in a model graph. To that end, the desired layer can be found by its name, and the intermediate layer can be evaluated. The example EvaluateIntermediateLayer() in C# demonstrates how to evaluate an intermediate layer in a model.
Evaluate outputs from multiple nodes
CNTK can evaluate multiple nodes in a model graph by combining their outputs into a single node. The example EvaluateCombinedOutputs() in C# demonstrates how to evaluate multiple outputs.
If you do not want to use NuGet Package, you can add
Cntk.Core.Managed-<VERSION>.dll as reference to your project. The
Cntk.Core.Managed DLL and all dependent DLLs can be found in the CNTK binary release package on the CNTK Releases page. Please make sure that the path to
Cntk.Core.Managed DLL and its dependencies (see list at the end of this page) are included in the search path of DLLs for your application.
The CNTK Library C++ API supports model evaluation in C++ applications. Both standard desktop applications and Universal Windows Platform applications are supported. Visual Studio 2017 is required and the target platform in the project property should be X64.
The CNTK NuGet package CNTK.CPUOnly and CNTK.GPU contain C++ CNTK Library for using CNTK in desktop applications. For using CNTK on Universal Windows Platform, please install CNTK.UWP.CPUOnly. By using NuGet packages, configuration about include and library directories is automatically added to your project property. Both Debug and Release build are supported. The NuGet Package page explains how to get and install CNTK NuGet Packages.
C++ Programming Guide
The following steps describe how to use the C++ CNTK Library for model evaluation.
- Link the
Cntk.Core-<VERSION>.libimport library into the application. This step can be omitted if you use the NuGet package. For UWP application, the library name is
- Include the evaluation header file "CNTKLibrary.h".
- Load the model by
CNTK::Function::Load(). The returned Function object represents the computation graph of the model.
- Prepare data for each input variable. You can use
CNTK::Value::CreateBatchOfSequences()to create a Value object from your input data representing a batch of samples, a sequence of samples, or a batch of sequences of samples respectively.
CNTK::Function::Evaluate()for evaluation. The
Evaluate()expects as input an unordered_map for input variables and input values, and an unordered_map for output variables and output values. The output value object could be
nullwhich means that CNTK Library allocates the actual storage for this output value. On return, the Value objects associated with the output variables contain the results of evaluation.
- Get output data from evaluation results. Use
CNTK::Value::CopyVariableValueTo()to copy the data stored in the Value object into the buffer as a list of sequences with variable length of samples.
For concurrent evaluation of multiple requests,
CNTK::Function::Clone() should be called before Evaluate(). The
ParallelEvaluationExample() demonstrates how to evaluate multiple requests in parallel using CNTK C++ Library API.
For details on C++ CNTK Library API for evaluation, please refer to the CNTK Library C++ Evaluation Interface page.
The C++ examples
CNTKLibraryCPPEvalGPUExamples illustrate the usage pattern above. The UWPImageRecognition contains an example using CNTK UWP library for model evaluation. The Eval Examples page provides detailed steps about building and running examples.
Alternatively, you can use C++ CNTK Library without the NuGet Package. You can either get
Cntk.Core_app-<Version>.lib) and all dependent DLLs from the CNTK Releases page or build them from CNTK source code. You also need to configure include and library path to point to the correct directory, and make sure that the build configuration (Debug or Release) of the CNTK Library is the same as that of your project setting (Otherwise you will get unexpected exceptions). The CNTK release package contains only the release build of binaries.
Shipping CNTK Library with your Windows application
CNTK Library requires the Visual C++ Redistributable Package for Visual Studio 2017 to be installed on the system where your application is going to run. And the target platform in the project property should be X64.
This page describes how CNTK binary files are named.
For desktop C++ applications, the following DLLs need to be distributed with your application.
If you use the C#/.NET Managed Library, beside the DLLs mentioned above, you need to additionally include the following DLLs:
For using GPU, you need in addition to include the following NVIDIA CUDA related DLLs:
Changed in CNTK version 2.1.
CNTK NuGet packages are shipping with cuDNN 6 (
All these DLLs can be found in the CNTK binary release version, see the CNTK Releases page.
For UWP applications, the following DLLs need to be distributed with your application. You can find these DLLs in the Nuget package
You can use Python to evaluate a pre-trained model. Examples can be found here.
CNTK also provides APIs for evaluating model in Java application. Please note that the CNTK Java API is still experimental and subject to change.
The Java example shows how to evaluate a CNN model using the Java API.
For using CNTK Java Library, please add the
cntk.jar file to the
classpath of your Java project. If you are working with an IDE you should add this as an unmanaged jar. The cntk.jar file can be found in the CNTK binary release package (in the folder cntk\cntk\java). You can also build cntk.jar from CNTK source. Please also set
java.library.path to the directory containing
Cntk.Core.JavaBinding-<Version>.dll, and make sure that the directory containing CNTK binary DLLs is in the DLL search path of your system, e.g. by adding the directory to the PATH environment variable. Please note that you also need to install Visual C++ Redistributable Package for Visual Studio 2017. If you get
UnsatisfiedLinkErrors in Java, it is typically because that the directory is not in the DLL search path (or in the wrong order).
The Java library is currently built and tested with 64-bit Oracle JDK 8.