Convert Word to Vector module
This article describes how to use the Convert Word to Vector module in Azure Machine Learning designer to do these tasks:
- Apply various Word2Vec models (Word2Vec, FastText, GloVe pretrained model) on the corpus of text that you specified as input.
- Generate a vocabulary with word embeddings.
This module uses the Gensim library. For more information about Gensim, see its official website, which includes tutorials and an explanation of algorithms.
More about converting words to vectors
Converting words to vectors, or word vectorization, is a natural language processing (NLP) process. The process uses language models to map words into vector space. A vector space represents each word by a vector of real numbers. It also allows words with similar meanings have similar representations.
Use word embeddings as initial input for NLP downstream tasks such as text classification and sentiment analysis.
Among various word embedding technologies, in this module, we implemented three widely used methods. Two, Word2Vec and FastText, are online-training models. The other is a pretrained model, glove-wiki-gigaword-100.
Online-training models are trained on your input data. Pretrained models are trained offline on a larger text corpus (for example, Wikipedia, Google News) that usually contains about 100 billion words. Word embedding then stays constant during word vectorization. Pretrained word models provide benefits such as reduced training time, better word vectors encoded, and improved overall performance.
Here's some information about the methods:
Word2Vec is one of the most popular techniques to learn word embeddings by using a shallow neural network. The theory is discussed in this paper, available as a PDF download: Efficient Estimation of Word Representations in Vector Space. The implementation in this module is based on the Gensim library for Word2Vec.
The FastText theory is explained in this paper, available as a PDF download: Enriching Word Vectors with Subword Information. The implementation in this module is based on the Gensim library for FastText.
The GloVe pretrained model is glove-wiki-gigaword-100. It's a collection of pretrained vectors based on a Wikipedia text corpus, which contains 5.6 billion tokens and 400,000 uncased vocabulary words. A PDF download is available: GloVe: Global Vectors for Word Representation.
How to configure Convert Word to Vector
This module requires a dataset that contains a column of text. Preprocessed text is better.
Add the Convert Word to Vector module to your pipeline.
As input for the module, provide a dataset that contains one or more text columns.
For Target column, choose only one column that contains text to process.
Because this module creates a vocabulary from text, the content of columns differs, which leads to different vocabulary contents. That's why the module accepts only one target column.
For Word2Vec strategy, choose from GloVe pretrained English Model, Gensim Word2Vec, and Gensim FastText.
If Word2Vec strategy is Gensim Word2Vec or Gensim FastText:
For Word2Vec Training Algorithm, choose from Skip_gram and CBOW. The difference is introduced in the original paper (PDF).
The default method is Skip_gram.
For Length of word embedding, specify the dimensionality of the word vectors. This setting corresponds to the
sizeparameter in Gensim.
The default embedding size is 100.
For Context window size, specify the maximum distance between the word being predicted and the current word. This setting corresponds to the
windowparameter in Gensim.
The default window size is 5.
For Number of epochs, specify the number of epochs (iterations) over the corpus. Corresponds to the
iterparameter in Gensim.
The default epoch number is 5.
For Maximum vocabulary size, specify the maximum number of the words in the generated vocabulary.
If there are more unique words than the max size, prune the infrequent ones.
The default vocabulary size is 10,000.
For Minimum word count, provide a minimum word count. The module will ignore all words that have a frequency lower than this value.
The default value is 5.
Submit the pipeline.
The module has one output:
- Vocabulary with embeddings: Contains the generated vocabulary, together with each word's embedding. One dimension occupies one column.
The following example shows how the Convert Word to Vector module works. It uses Convert Word to Vector with default settings to the preprocessed Wikipedia SP 500 Dataset.
The dataset contains a category column, along with the full text fetched from Wikipedia. The following table shows a few representative examples.
|nasdaq 100 component s p 500 component foundation founder location city apple campus 1 infinite loop street infinite loop cupertino california cupertino california location country united states...|
|br nasdaq 100 nasdaq 100 component br s p 500 s p 500 component industry computer software foundation br founder charles geschke br john warnock location adobe systems...|
|s p 500 s p 500 component industry automotive industry automotive predecessor general motors corporation 1908 2009 successor...|
|s p 500 s p 500 component industry conglomerate company conglomerate foundation founder location city fairfield connecticut fairfield connecticut location country usa area...|
|br s p 500 s p 500 component foundation 1903 founder william s harley br arthur davidson harley davidson founder arthur davidson br walter davidson br william a davidson location...|
Output vocabulary with embeddings
The following table contains the output of this module, taking the Wikipedia SP 500 dataset as input. The leftmost column indicates the vocabulary. Its embedding vector is represented by values of remaining columns in the same row.
|Vocabulary||Embedding dim 0||Embedding dim 1||Embedding dim 2||Embedding dim 3||Embedding dim 4||Embedding dim 5||...||Embedding dim 99|
In this example, we used the default Gensim Word2Vec for Word2Vec strategy, and Training Algorithm is Skip-gram. Length of word Embedding is 100, so we have 100 embedding columns.
This section contains tips and answers to frequently asked questions.
Difference between online-training and pretrained model:
In this Convert Word to Vector module, we provided three different strategies: two online-training models and one pretrained model. The online-training models use your input dataset as training data, and generate vocabulary and word vectors during training. The pretrained model is already trained by a much larger text corpus, such as Wikipedia or Twitter text. The pretrained model is actually a collection of word/embedding pairs.
The GloVe pre-trained model summarizes a vocabulary from the input dataset and generates an embedding vector for each word from the pretrained model. Without online training, the use of a pretrained model can save training time. It has better performance, especially when the input dataset size is relatively small.
In general, the length of word embedding is set to a few hundred. For example, 100, 200, 300. A small embedding size means a small vector space, which could cause word embedding collisions.
The length of word embeddings is fixed for pretrained models. In this example, the embedding size of glove-wiki-gigaword-100 is 100.
See the set of modules available to Azure Machine Learning.
For a list of errors specific to the designer modules, see Machine Learning error codes.