# Language Detector

## Classifying

The OpenNLP Language Detector classifies a document in ISO-639-3 languages according to the model capabilities. A model can be trained with Maxent, Perceptron or Naive Bayes algorithms. By default normalizes a text and the context generator extracts n-grams of size 1, 2 and 3. The n-gram sizes, the normalization and the context generator can be customized by extending the LanguageDetectorFactory.

The default normalizers are:

Normalizers

Normalizer | Description
--- | ---
EmojiCharSequenceNormalizer | Replaces emojis by blank space
UrlCharSequenceNormalizer | Replaces URLs and E-Mails by a blank space.
TwitterCharSequenceNormalizer | Replaces hashtags and Twitter user names by blank spaces.
NumberCharSequenceNormalizer | Replaces number sequences by blank spaces
ShrinkCharSequenceNormalizer | Shrink characters that repeats three or more times to only two repetitions.

## Language Detector Tool

The easiest way to try out the language detector is the command line tool. The tool is only intended for demonstration and testing. The following command shows how to use the language detector tool.

    $ bin/opennlp LanguageDetector model

The input is read from standard input and output is written to standard output, unless they are redirected or piped.

## Language Detector API

To perform classification you will need a machine learning model - these are encapsulated in the LanguageDetectorModel class of OpenNLP tools.

First you need to grab the bytes from the serialized model on an InputStream - we'll leave it you to do that, since you were the one who serialized it to begin with. Now for the easy part:

    InputStream is = ...
    LanguageDetectorModel m = new LanguageDetectorModel(is);

With the LanguageDetectorModel in hand we are just about there:

    String inputText = ...
    LanguageDetector myCategorizer = new LanguageDetectorME(m);

    // Get the most probable language
    Language bestLanguage = myCategorizer.predictLanguage(inputText);
    System.out.println("Best language: " + bestLanguage.getLang());
    System.out.println("Best language confidence: " + bestLanguage.getConfidence());

    // Get an array with the most probable languages
    Language[] languages = myCategorizer.predictLanguages(null);

Note that the both the API or the CLI will consider the complete text to choose the most probable languages. To handle mixed language one can analyze smaller chunks of text to find language regions.

## Training

The Language Detector can be trained on annotated training material. The data can be in OpenNLP Language Detector training format. This is one document per line, containing the ISO-639-3 language code and text separated by a tab. Other formats can also be available. The following sample shows the sample from above in the required format.

    spa     A la fecha tres calles bonaerenses recuerdan su nombre (en Ituzaingó, Merlo y Campana). A la fecha, unas 50 \
    		naves y 20 aviones se han perdido en esa área particular del océano Atlántico.
    deu     Alle Jahre wieder: Millionen Spanier haben am Dienstag die Auslosung in der größten Lotterie der Welt verfolgt.\
     		Alle Jahre wieder: So gelingt der stressfreie Geschenke-Umtausch Artikel per E-Mail empfehlen So gelingt der \
     		stressfre ie Geschenke-Umtausch Nicht immer liegt am Ende das unter dem Weihnachtsbaum, was man sich gewünscht hat.
    srp     Већина становника боравила је кућама од блата или шаторима, како би радили на својим удаљеним пољима у долини \
    		Јордана и напасали своје стадо оваца и коза. Већина становника говори оба језика.
    lav     Egija Tri-Active procedūru īpaši iesaka izmantot siltākajos gadalaikos, jo ziemā aukstums var šķist arī \
    		nepatīkams. Valdība vienojās, ka izmaiņas nodokļu politikā tiek konceptuāli atbalstītas, tomēr deva \
    		nedēļu laika Ekonomikas ministrijai, Finanšu ministrijai un Labklājības ministrijai, lai ar vienotu \
    		pozīciju atgrieztos pie jautājuma izskatīšanas.

Note: The line breaks marked with a backslash are just inserted for formatting purposes and must not be included in the training data.

### Training Tool

The following command will train the language detector and write the model to langdetect.bin:

    $ bin/opennlp LanguageDetectorTrainer[.leipzig] -model modelFile [-params paramsFile] [-factory factoryName] -data sampleData [-encoding charsetName]

Note: To customize the language detector, extend the class opennlp.tools.langdetect.LanguageDetectorFactory add it to the classpath and pass it in the -factory argument.

### Training with Leipzig

The Leipzig Corpora collection presents corpora in different languages. The corpora is a collection of individual sentences collected from the web and newspapers. The Corpora is available as plain text and as MySQL database tables. The OpenNLP integration can only use the plain text version. The individual plain text packages can be downloaded here: http://corpora.uni-leipzig.de/download.html

This corpora is specially good to train Language Detector and a converter is provided. First, you need to download the files that compose the Leipzig Corpora collection to a folder. Apache OpenNLP Language Detector supports training, evaluation and cross validation using the Leipzig Corpora. For example, the following command shows how to train a model.

    $ bin/opennlp LanguageDetectorTrainer.leipzig -model modelFile [-params paramsFile] [-factory factoryName] \
    	-sentencesDir sentencesDir -sentencesPerSample sentencesPerSample -samplesPerLanguage samplesPerLanguage \
    	[-encoding charsetName]

The following sequence of commands shows how to convert the Leipzig Corpora collection at folder leipzig-train/ to the default Language Detector format, by creating groups of 5 sentences as documents and limiting to 10000 documents per language. Them, it shuffles the result and select the first 100000 lines as train corpus and the last 20000 as evaluation corpus:

    $ bin/opennlp LanguageDetectorConverter leipzig -sentencesDir leipzig-train/ -sentencesPerSample 5 -samplesPerLanguage 10000 > leipzig.txt
    $ perl -MList::Util=shuffle -e 'print shuffle(<STDIN>);' < leipzig.txt > leipzig_shuf.txt
    $ head -100000 < leipzig_shuf.txt > leipzig.train
    $ tail -20000 < leipzig_shuf.txt > leipzig.eval

### Training API

The following example shows how to train a model from API.

    InputStreamFactory inputStreamFactory = new MarkableFileInputStreamFactory(new File("corpus.txt"));

    ObjectStream<String> lineStream =
      new PlainTextByLineStream(inputStreamFactory, StandardCharsets.UTF_8);
    ObjectStream<LanguageSample> sampleStream = new LanguageDetectorSampleStream(lineStream);

    TrainingParameters params = ModelUtil.createDefaultTrainingParameters();
    params.put(TrainingParameters.ALGORITHM_PARAM,
      PerceptronTrainer.PERCEPTRON_VALUE);
    params.put(TrainingParameters.CUTOFF_PARAM, 0);

    LanguageDetectorFactory factory = new LanguageDetectorFactory();

    LanguageDetectorModel model = LanguageDetectorME.train(sampleStream, params, factory);
    model.serialize(new File("langdetect.bin"));
    }