DataOperationsCatalog.LoadFromEnumerable Method

Definition

Namespace:

Microsoft.ML

Assembly:

Microsoft.ML.Data.dll

Package:

Microsoft.ML v3.0.1

Overloads

LoadFromEnumerable<TRow>(IEnumerable<TRow>, SchemaDefinition)

Create a new IDataView over an enumerable of the items of user-defined type. The user maintains ownership of the data and the resulting data view will never alter the contents of the data. Since IDataView is assumed to be immutable, the user is expected to support multiple enumerations of the data that would return the same results, unless the user knows that the data will only be cursored once. One typical usage for streaming data view could be: create the data view that lazily loads data as needed, then apply pre-trained transformations to it and cursor through it for transformation results.

LoadFromEnumerable<TRow>(IEnumerable<TRow>, DataViewSchema)

Create a new IDataView over an enumerable of the items of user-defined type using the provided DataViewSchema, which might contain more information about the schema than the type can capture.

LoadFromEnumerable<TRow>(IEnumerable<TRow>, SchemaDefinition)

Create a new IDataView over an enumerable of the items of user-defined type. The user maintains ownership of the data and the resulting data view will never alter the contents of the data. Since IDataView is assumed to be immutable, the user is expected to support multiple enumerations of the data that would return the same results, unless the user knows that the data will only be cursored once.

One typical usage for streaming data view could be: create the data view that lazily loads data as needed, then apply pre-trained transformations to it and cursor through it for transformation results.

public Microsoft.ML.IDataView LoadFromEnumerable<TRow> (System.Collections.Generic.IEnumerable<TRow> data, Microsoft.ML.Data.SchemaDefinition schemaDefinition = default) where TRow : class;

member this.LoadFromEnumerable : seq<'Row (requires 'Row : null)> * Microsoft.ML.Data.SchemaDefinition -> Microsoft.ML.IDataView (requires 'Row : null)

Public Function LoadFromEnumerable(Of TRow As Class) (data As IEnumerable(Of TRow), Optional schemaDefinition As SchemaDefinition = Nothing) As IDataView

Type Parameters

TRow

The user-defined item type.

Parameters

data

IEnumerable<TRow>

The enumerable data containing type TRow to convert to a IDataView.

schemaDefinition

SchemaDefinition

The optional schema definition of the data view to create. If null, the schema definition is inferred from TRow.

Returns

IDataView

The constructed IDataView.

Examples

using System;
using System.Collections.Generic;
using Microsoft.ML;
using Microsoft.ML.Data;

namespace Samples.Dynamic
{
    public static class LoadFromEnumerable
    {
        // Creating IDataView from IEnumerable, and setting the size of the vector
        // at runtime. When the data model is defined through types, setting the
        // size of the vector is done through the VectorType annotation. When the
        // size of the data is not known at compile time, the Schema can be directly
        // modified at runtime and the size of the vector set there. This is
        // important, because most of the ML.NET trainers require the Features
        // vector to be of known size. 
        public static void Example()
        {
            // Create a new context for ML.NET operations. It can be used for
            // exception tracking and logging, as a catalog of available operations
            // and as the source of randomness.
            var mlContext = new MLContext();

            // Get a small dataset as an IEnumerable.
            IEnumerable<DataPointVector> enumerableKnownSize = new DataPointVector[]
            {
               new DataPointVector{ Features = new float[]{ 1.2f, 3.4f, 4.5f, 3.2f,
                   7,5f } },

               new DataPointVector{ Features = new float[]{ 4.2f, 3.4f, 14.65f,
                   3.2f, 3,5f } },

               new DataPointVector{ Features = new float[]{ 1.6f, 3.5f, 4.5f, 6.2f,
                   3,5f } },

            };

            // Load dataset into an IDataView. 
            IDataView data = mlContext.Data.LoadFromEnumerable(enumerableKnownSize);
            var featureColumn = data.Schema["Features"].Type as VectorDataViewType;
            // Inspecting the schema
            Console.WriteLine($"Is the size of the Features column known: " +
                $"{featureColumn.IsKnownSize}.\nSize: {featureColumn.Size}");

            // Preview
            //
            // Is the size of the Features column known? True.
            // Size: 5.

            // If the size of the vector is unknown at compile time, it can be set 
            // at runtime.
            IEnumerable<DataPoint> enumerableUnknownSize = new DataPoint[]
            {
               new DataPoint{ Features = new float[]{ 1.2f, 3.4f, 4.5f } },
               new DataPoint{ Features = new float[]{ 4.2f, 3.4f, 1.6f } },
               new DataPoint{ Features = new float[]{ 1.6f, 3.5f, 4.5f } },
            };

            // The feature dimension (typically this will be the Count of the array 
            // of the features vector known at runtime).
            int featureDimension = 3;
            var definedSchema = SchemaDefinition.Create(typeof(DataPoint));
            featureColumn = definedSchema["Features"]
                .ColumnType as VectorDataViewType;

            Console.WriteLine($"Is the size of the Features column known: " +
                $"{featureColumn.IsKnownSize}.\nSize: {featureColumn.Size}");

            // Preview
            //
            // Is the size of the Features column known? False.
            // Size: 0.

            // Set the column type to be a known-size vector.
            var vectorItemType = ((VectorDataViewType)definedSchema[0].ColumnType)
                .ItemType;
            definedSchema[0].ColumnType = new VectorDataViewType(vectorItemType,
                featureDimension);

            // Read the data into an IDataView with the modified schema supplied in
            IDataView data2 = mlContext.Data
                .LoadFromEnumerable(enumerableUnknownSize, definedSchema);

            featureColumn = data2.Schema["Features"].Type as VectorDataViewType;
            // Inspecting the schema
            Console.WriteLine($"Is the size of the Features column known: " +
                $"{featureColumn.IsKnownSize}.\nSize: {featureColumn.Size}");

            // Preview
            //
            // Is the size of the Features column known? True. 
            // Size: 3.
        }
    }

    public class DataPoint
    {
        public float[] Features { get; set; }
    }

    public class DataPointVector
    {
        [VectorType(5)]
        public float[] Features { get; set; }
    }
}

LoadFromEnumerable<TRow>(IEnumerable<TRow>, DataViewSchema)

Create a new IDataView over an enumerable of the items of user-defined type using the provided DataViewSchema, which might contain more information about the schema than the type can capture.

public Microsoft.ML.IDataView LoadFromEnumerable<TRow> (System.Collections.Generic.IEnumerable<TRow> data, Microsoft.ML.DataViewSchema schema) where TRow : class;

member this.LoadFromEnumerable : seq<'Row (requires 'Row : null)> * Microsoft.ML.DataViewSchema -> Microsoft.ML.IDataView (requires 'Row : null)

Public Function LoadFromEnumerable(Of TRow As Class) (data As IEnumerable(Of TRow), schema As DataViewSchema) As IDataView

Type Parameters

TRow

The user-defined item type.

Parameters

data

IEnumerable<TRow>

The enumerable data containing type TRow to convert to an IDataView.

schema

DataViewSchema

The schema of the returned IDataView.

Returns

IDataView

An IDataView with the given schema.

Remarks

The user maintains ownership of the data and the resulting data view will never alter the contents of the data. Since IDataView is assumed to be immutable, the user is expected to support multiple enumerations of the data that would return the same results, unless the user knows that the data will only be cursored once. One typical usage for streaming data view could be: create the data view that lazily loads data as needed, then apply pre-trained transformations to it and cursor through it for transformation results. One practical usage of this would be to supply the feature column names through the DataViewSchema.Annotations.