Detects the language of each line in the input file
Category: Text Analytics
Applies to: Machine Learning Studio
This content pertains only to Studio. Similar drag and drop modules have been added to the visual interface in Machine Learning service. Learn more in this article comparing the two versions.
This article describes how to use the Detect Languages module in Azure Machine Learning Studio to analyze text input and identify the language associated with each record in the input.
The language detection algorithm can identify many different languages. Just specify the string column to analyze, and the total number of languages to detect. The algorithm will analyze each row of text, and assign a probability score for each language. The language in the first result column is the language that got the highest score.
How to configure Detect Languages
Add the dataset containing the text you want to analyze to an experiment in Azure Machine Learning Studio. The column with the text to analyze must be the string data type.
The datset need not contain a label column; the language detection algorithm works purely on linguistic features of the supported languages.
If you are importing new data, make sure that your data is saved in the UTF-8 format. Other Unicode formats are not supported.
Add the Detect Languages module to your experiment, and connect the dataset with the text for language detection.
For Text column, choose the column you want to analyze.
For Upper bound on number of languages to detect, indicate the maximum number of languages to detect.
Setting an upper bound on the number of languages can improve performance.
Run the experiment.
The Detect Languages module outputs a language identifier and score for each row.
For example, the following table contains a sample analysis on test data.
The first two columns col1 and language label are columns passed through from the input dataset. In this example, because the input dataset was designed for testing the module, the expected language was already known, and is provided in the label column.
The remaining columns are generated by the Detect Languages module. If there are equi-probable language matches, several languages might be listed, with a score for each. In this case, the module predicts just one language for each row, together with the probability score for that language.
If the module fails to detect any language with a sufficiently high score, a result of (Unknown) with a score of 0 is output. However, the languages supported by the module can change over time as the API is updated.
|Col1||Language label||Col1 Language||Col1 Iso6391 Language||Col1 Iso6391 Language Score|
|It was a wonderful hotel with a friendly staff and good service||English||English||en||100|
|Es war ein wunderbares Hotel mit freundlichem Personal und guter service||German||German||de||100|
|C’est un magnifique hôtel avec un personnel sympathique et un service de qualité||French||French||fr||100|
|Det var et dejligt hotel med et venligt personale og god service||Danish||Danish||nl||100|
|Va ser un magnífic hotel amb un personal amable i bon servei||Catalan||Catalan||ca||92.30769348|
|qu mebpa'mey naQ friendly QaQ chavmoH je||Klingon||French||fr||77.5|
For examples of how the Detect Languages module is used in an experiment, see the Azure AI Gallery:
- Filter Movie Titles by Language: Detects the language used in movie names, and then uses the language identifier to split the dataset into English vs non-English movies.
For a general idea of the languages that potentially can be detected, refer to Bing Translator.
Many more languages can be detected than Azure Machine Learning currently supports for advanced text analytics. We recommend that you use the results of Detect Languages to filter the results that you send to other modules that require language-specific processing.
|Dataset||Data Table||The input|
|Upper bound on number of languages to detect||Integer||[1;184]||Required||1||Upper bound on number of languages to detect.|
|Text column||ColumnSelection||Required||Name or one-based index of text column.|
|Results dataset||Data Table||The result|
|Error 0003||Exception occurs if one or more of inputs are null or empty.|
|Error 0010||Exception occurs if input datasets have column names that should match but do not.|
|Error 0016||Exception occurs if input datasets passed to the module should have compatible column types but do not.|
|Error 0008||Exception occurs if parameter is not in range.|
For a list of errors specific to Studio modules, see Machine Learning Error codes
For a list of API exceptions, see Machine Learning REST API Error Codes.