WordsSegmenter WordsSegmenter WordsSegmenter WordsSegmenter WordsSegmenter Class

Definition

A segmenter class that is able to segment provided text into words or word stems (depending on the particular language).

public : sealed class WordsSegmenter
struct winrt::Windows::Data::Text::WordsSegmenter
public sealed class WordsSegmenter
Public NotInheritable Class WordsSegmenter
var wordsSegmenter = new wordsSegmenter(language);
Attributes

Windows 10 requirements

Device family
Windows 10 (introduced v10.0.10240.0 - for Xbox, see UWP features that aren't yet supported on Xbox)
API contract
Windows.Foundation.UniversalApiContract (introduced v1)

Remarks

For languages that do not use spaces between words (such as Japanese, Chinese, Korean, and Thai), use of a segmenter is the only way to obtain individual words for textual processing scenarios such as keyword search.

The language supplied when this object is constructed is matched against the languages with word breakers on the system, and the best word segmentation rules available are used. The language need not be one of the app's supported languages. If there are no supported language rules available specifically for that language, the language-neutral rules are used (an implementation of Unicode Standard Annex #29 Unicode Text Segmentation), and the ResolvedLanguage property is set to "und" (undetermined language).

For keyword search scenarios, it is always recommended to request a segmenter in the language of the text contents.

For spellchecking scenarios, some language segmenters (such as German) may return multiple word stem segments for a single compound word. In contrast, the spellchecking APIs may expect the words to be kept together as a single word. For such languages, you may choose to force language-neutral segmenting rules by explicitly requesting the "und" (undetermined language) segmenter. However, doing so will greatly reduce the breaking quality of non-spaced languages. Therefore, it is recommended that you use the Language.Script API to determine if the content language uses one of the following non-spaced scripts:

Script Language
Bopo Bopomofo
Brah Brahmi
Egyp Egyptian Hieroglyphs
Goth Gothic
Hang Hangul
Hang Hiragana
Hang Old Hangul
Hani Han
Ital Old Italic
Java Javanese
Kana Katakana
Khar Kharoshthi
Khmr Khmer
Laoo Lao
Lisu Lisu
Mymr Myanmar
Talu New Tai Lue
Thai Thai
Tibt Tibetan
Xsux Cuneiform
Yiii Yi

If none of these scripts are found, then it should be safe to use "und" for spellchecking scenario segmentation.

Constructors

WordsSegmenter(String) WordsSegmenter(String) WordsSegmenter(String) WordsSegmenter(String) WordsSegmenter(String)

Creates a WordsSegmenter object. See the introduction in WordsSegmenter for a description of how the language supplied to this constructor is used.

Properties

ResolvedLanguage ResolvedLanguage ResolvedLanguage ResolvedLanguage ResolvedLanguage

Gets the language of the rules used by this WordsSegmenter object.

"und" (undetermined) is returned if we are using language-neutral rules.

Methods

GetTokenAt(String, UInt32) GetTokenAt(String, UInt32) GetTokenAt(String, UInt32) GetTokenAt(String, UInt32) GetTokenAt(String, UInt32)

Determines and returns the word or word stem which contains or follows a specified index into the provided text.

GetTokens(String) GetTokens(String) GetTokens(String) GetTokens(String) GetTokens(String)

Determines and returns all of the words or word stems in the provided text.

Tokenize(String, UInt32, WordSegmentsTokenizingHandler) Tokenize(String, UInt32, WordSegmentsTokenizingHandler) Tokenize(String, UInt32, WordSegmentsTokenizingHandler) Tokenize(String, UInt32, WordSegmentsTokenizingHandler) Tokenize(String, UInt32, WordSegmentsTokenizingHandler)

Calls the provided handler with two iterators that iterate through the words prior to and following a given index into the provided text.