LexicalTokenizer Class
Definition
Important
Some information relates to prerelease product that may be substantially modified before it’s released. Microsoft makes no warranties, express or implied, with respect to the information provided here.
Base type for tokenizers. Please note LexicalTokenizer is the base class. According to the scenario, a derived class of the base class might need to be assigned here, or this property needs to be casted to one of the possible derived classes. The available derived classes include ClassicTokenizer, EdgeNGramTokenizer, KeywordTokenizer, KeywordTokenizer, MicrosoftLanguageStemmingTokenizer, MicrosoftLanguageTokenizer, NGramTokenizer, PathHierarchyTokenizer, PatternTokenizer, LuceneStandardTokenizer, LuceneStandardTokenizer and UaxUrlEmailTokenizer.
public class LexicalTokenizertype LexicalTokenizer = classPublic Class LexicalTokenizer- Inheritance
-
LexicalTokenizer
- Derived
Properties
| Name |
The name of the tokenizer. It must only contain letters, digits, spaces, dashes or underscores, can only start and end with alphanumeric characters, and is limited to 128 characters. |
Applies to
Azure SDK for .NET
Feedback
Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see: https://aka.ms/ContentUserFeedback.
Submit and view feedback for