TokenizingByCharactersEstimator Class

  • Reference

Definition

Namespace:
Microsoft.ML.Transforms.Text
Assembly:
Microsoft.ML.Transforms.dll
Package:
Microsoft.ML v3.0.1
Package:
Microsoft.ML v1.0.0
Package:
Microsoft.ML v1.1.0
Package:
Microsoft.ML v1.2.0
Package:
Microsoft.ML v1.3.1
Package:
Microsoft.ML v1.4.0
Package:
Microsoft.ML v1.5.5
Package:
Microsoft.ML v1.6.0
Package:
Microsoft.ML v1.7.0
Package:
Microsoft.ML v2.0.0

IEstimator<TTransformer> for the TokenizingByCharactersTransformer.

public sealed class TokenizingByCharactersEstimator : Microsoft.ML.Data.TrivialEstimator<Microsoft.ML.Transforms.Text.TokenizingByCharactersTransformer>
type TokenizingByCharactersEstimator = class
    inherit TrivialEstimator<TokenizingByCharactersTransformer>
Public NotInheritable Class TokenizingByCharactersEstimator
Inherits TrivialEstimator(Of TokenizingByCharactersTransformer)
Inheritance
Object
TrivialEstimator<TokenizingByCharactersTransformer>
TokenizingByCharactersEstimator

Remarks

Estimator Characteristics
Does this estimator need to look at the data to train its parameters? Yes
Input column data type Scalar or Vector of Text
Output column data type Variable-sized vector of key type.
Exportable to ONNX Yes

The estimator tokenizes characters by splitting text into sequences of characters using a sliding window. During training, the estimator builds a key-value pair dictionary with the encountered sequences of characters.

The TokenizingByCharactersTransformer resulting from fitting the estimator creates a new column, named as specified in the output column name parameters, which contains the keys of the sequences of characters that were encountered in the input.

Check the See Also section for links to usage examples.

Methods

Fit(IDataView) (Inherited from TrivialEstimator<TTransformer>)
GetOutputSchema(SchemaShape)

Returns the SchemaShape of the schema which will be produced by the transformer. Used for schema propagation and verification in a pipeline.

Extension Methods

AppendCacheCheckpoint<TTrans>(IEstimator<TTrans>, IHostEnvironment)

Append a 'caching checkpoint' to the estimator chain. This will ensure that the downstream estimators will be trained against cached data. It is helpful to have a caching checkpoint before trainers that take multiple data passes.
WithOnFitDelegate<TTransformer>(IEstimator<TTransformer>, Action<TTransformer>)

Given an estimator, return a wrapping object that will call a delegate once Fit(IDataView) is called. It is often important for an estimator to return information about what was fit, which is why the Fit(IDataView) method returns a specifically typed object, rather than just a general ITransformer. However, at the same time, IEstimator<TTransformer> are often formed into pipelines with many objects, so we may need to build a chain of estimators via EstimatorChain<TLastTransformer> where the estimator for which we want to get the transformer is buried somewhere in this chain. For that scenario, we can through this method attach a delegate that will be called once fit is called.

Applies to

See also