Choosing a natural language processing technology in Azure

• 02/12/2018
• 2 minutes to read
• Contributors
  • 
  • 
  • 
  • 
  • 

Free-form text processing is performed against documents containing paragraphs of text, typically for the purpose of supporting search, but is also used to perform other natural language processing (NLP) tasks such as sentiment analysis, topic detection, language detection, key phrase extraction, and document categorization. This article focuses on the technology choices that act in support of the NLP tasks.

What are your options when choosing an NLP service?

In Azure, the following services provide natural language processing (NLP) capabilities:

• Azure HDInsight with Spark and Spark MLlib
• Azure Databricks
• Microsoft Cognitive Services

Key selection criteria

To narrow the choices, start by answering these questions:

• Do you want to use prebuilt models? If yes, consider using the APIs offered by Microsoft Cognitive Services.

• Do you need to train custom models against a large corpus of text data? If yes, consider using Azure HDInsight with Spark MLlib and Spark NLP.

• Do you need low-level NLP capabilities like tokenization, stemming, lemmatization, and term frequency/inverse document frequency (TF/IDF)? If yes, consider using Azure HDInsight with Spark MLlib and Spark NLP.

• Do you need simple, high-level NLP capabilities like entity and intent identification, topic detection, spell check, or sentiment analysis? If yes, consider using the APIs offered by Microsoft Cognitive Services.

Capability matrix

The following tables summarize the key differences in capabilities.

General capabilities

Low-level natural language processing capabilities

High-level natural language processing capabilities

See also

Natural language processing