# Morfologik Addon

Morfologik provides tools for finite state automata (FSA) construction and dictionary-based morphological dictionaries.

The Morfologik Addon implements OpenNLP interfaces and extensions to allow the use of FSA Morfologik dictionary tools.

## Morfologik Integration

To allow for an easy integration with OpenNLP, the following implementations are provided:

- The `MorfologikPOSTaggerFactory` extends `POSTaggerFactory`, which helps create a POSTagger model with an embedded FSA TagDictionary.

- The `MorfologikTagDictionary` implements an FSA based `TagDictionary`, allowing for much smaller files than the default XML based with improved memory consumption.

- The `MorfologikLemmatizer` implements an FSA based `Lemmatizer` dictionaries.

The first two implementations can be used directly from command line, as in the example bellow. Having an FSA Morfologik dictionary (see next section how to build one), you can train a POS Tagger model with an embedded FSA dictionary.

The example trains a POSTagger with a CONLL corpus named `portuguese_bosque_train.conll` and an FSA dictionary named `pt-morfologik.dict`. It will output a model named `pos-pt_fsadic.model`.

    $ bin/opennlp POSTaggerTrainer -type perceptron -lang pt -model pos-pt_fsadic.model \
    																				-data portuguese_bosque_train.conll -encoding UTF-8 \
    																				-factory opennlp.morfologik.tagdict.MorfologikPOSTaggerFactory \
    																				-dict pt-morfologik.dict

Another example follows. It shows how to use the `MorfologikLemmatizer`. You will need a lemma dictionary and info file, in this example, we will use a very small Portuguese dictionary. Its syntax is `lemma,lexeme,postag`.

File `lemmaDictionary.txt:`

    casa,casa,NOUN
    casar,casa,V
    casar,casar,V-INF
    Casa,Casa,PROP
    casa,casinha,NOUN
    casa,casona,NOUN
    menino,menina,NOUN
    menino,menino,NOUN
    menino,meninão,NOUN
    menino,menininho,NOUN
    carro,carro,NOUN

Mandatory metadata file, which must have the same name but .info extension `lemmaDictionary.info:`

    #
    # REQUIRED PROPERTIES
    #

    # Column (lemma, inflected, tag) separator. This must be a single byte in the target encoding.
    fsa.dict.separator=,

    # The charset in which the input is encoded. UTF-8 is strongly recommended.
    fsa.dict.encoding=UTF-8

    # The type of lemma-inflected form encoding compression that precedes automaton
    # construction. Allowed values: [suffix, infix, prefix, none].
    # Details are in Daciuk's paper and in the code.
    # Leave at 'prefix' if not sure.
    fsa.dict.encoder=prefix

The following code creates a binary FSA Morfologik dictionary, loads it in MorfologikLemmatizer and uses it to find the lemma the word "casa" noun and verb.

    // Part 1: compile an FSA lemma dictionary

    // we need the tabular dictionary. It is mandatory to have info
    //  file with same name, but .info extension
    Path textLemmaDictionary = Paths.get("dictionaryWithLemma.txt");

    // this will build a binary dictionary located in compiledLemmaDictionary
    Path compiledLemmaDictionary = new MorfologikDictionayBuilder()
        .build(textLemmaDictionary);

    // Part 2: load a MorfologikLemmatizer and use it
    MorfologikLemmatizer lemmatizer = new MorfologikLemmatizer(compiledLemmaDictionary);

    String[] toks = {"casa", "casa"};
    String[] tags = {"NOUN", "V"};

    String[] lemmas = lemmatizer.lemmatize(toks, tags);
    System.out.println(Arrays.toString(lemmas)); // outputs [casa, casar]

## Morfologik CLI Tools

The Morfologik addon provides a command line tool. `XMLDictionaryToTable` makes easy to convert from an OpenNLP XML based dictionary to a tabular format. `MorfologikDictionaryBuilder` can take a tabular dictionary and output a binary Morfologik FSA dictionary.

    $ sh bin/morfologik-addon
    OpenNLP Morfologik Addon. Usage: opennlp-morfologik-addon TOOL
    where TOOL is one of:
      MorfologikDictionaryBuilder    builds a binary POS Dictionary using Morfologik
      XMLDictionaryToTable           reads an OpenNLP XML tag dictionary and outputs it in a tabular file
    All tools print help when invoked with help parameter
    Example: opennlp-morfologik-addon POSDictionaryBuilder help