=============================================================
   Osalausestaja: Clause Segmenter for Estonian
=============================================================

  Clause Segmenter is a program that splits long and complex natural 
 language sentences into smaller segments (clauses). For example, the 
 sentence "Mees, keda seal kohtasime, oli tuttav ja teretas meid." 
 will be split into following clauses:

     "[Mees, [keda seal kohtasime,] oli tuttav ja] [teretas meid.]"
     (in the example, clauses are surrounded by brackets)

   The algorithm mainly relies on punctuation, conjunction words, and
  finite verb forms on identifying the clause boundaries.
  For linguistic details/motivations behind the algorithm, see (Kaalep, 
  Muischnek 2012).

=========================
   Requirements
=========================
 For building the program (JAR file):
  ** Java JDK (at least version 1.8.x is expected);
  ** Apache Ant (at least version 1.8.2);
  
 For using the program:
  ** A sentence segmentator;
  ** A word tokenizer;
  ** Estonian morphological analyzer, possible options:
     -- Filosoft Vabamorf: https://github.com/Filosoft/vabamorf
     -- PyVabamorf:        https://github.com/estnltk/pyvabamorf
     -- T3MESTA (a commercial morphological analyzer);
  ** Estonian morphological disambiguator;
     -- Vabamorf's disambiguator: 
        https://github.com/Filosoft/vabamorf
     NB! The clause segmenter also works on morphologically ambiguous 
        input, but the quality of the analysis is expected to be lower 
        than in the case of morphologically disambiguated text.

=========================
   Building the program
=========================
   The most straightforward way for compiling the program is by using
  Apache Ant and the build script ("build.xml" in root dir);
  
   Before building, correct path to JDK must be set in the file 
  "build.properties" (variable "java.home.location"). Then, building
  and deploying can be evoked with the command:

      ant deploy

  (in the same directory where "build.xml" is located);
   This compiles the Java source code, makes the JAR file (Osalau.jar), 
  and copies the JAR file along with required files into the folder 
  "test";
  
=========================
   Using the program
=========================

    Basic usage
   ---------------
   Before the clause segmenter can be applied on a text, a number of 
  text preprocessing steps must be made: text must be split into 
  sentences and tokens (words), and words must be morphologically 
  analysed (and disambiguated).
   Core of these functionalities is provided by EstNLTK toolkit, so the 
  easiest way to use the clause segmenter is within this toolkit ( see 
  https://github.com/estnltk/estnltk   for more details ).

   It is expected that the input of the clause segmenter is in the same 
  format as the output of Vabamorf's command 'etana analyze' - a JSON 
  structured text in UTF8 encoding. Note that the clause segmenter expects 
  that word root analyses are 'clean', without any phonetic markup symbols 
  (which can be optionally added in 'etana' with flag '-phonetic').

  An example of JSON input can be found in file "test/example_input.json";
  In the "test" folder, following command evokes clause segmenter on
  the input file "example_input.json" and outputs the results to standard 
  input:
  
     java -jar Osalau.jar -in file example_input.json -pretty_print

  (flag "-pretty_print" switches on the pretty printing mode, otherwise, 
  all of the output JSON is on single line);

  Alternatively, output can also be directed to a file by specifying:

     java -jar Osalau.jar -in file example_input.json -pretty_print -out file my_output.json

  Flag "-pyvabamorf" evokes the program in a special standard input/output 
  processing mode, where the program reads a line from the standard input,
  analyzes the line, and outputs the results (in a single line) to the standard 
  output.

     java -jar Osalau.jar -pyvabamorf

   More details about Vabamorf and its JSON format:   
        https://github.com/Filosoft/vabamorf 


    The "insensitive to missing commas" mode
   -------------------------------------------
    The clause segmenter can also be executed in the mode in which the program 
   attempts to be less sensitive to missing commas while detecting clause boundaries. 
   
    The flag "-ins_comma_mis" can be used to switch this mode on:
    
     java -jar Osalau.jar -in file example_input_missing_commas.json -pretty_print -ins_comma_mis
   
    (the file "example_input_missing_commas.json" can be found in the folder "test");
    
     Note that this mode is experimental and compared to the default mode, it can 
   introduce additional incorrect clause boundaries, although it also improves clause 
   boundary detection in texts with (a lot of) missing commas.
   
============================
   Interpreting the output
============================

  The clause segmenter marks clause boundaries: boundaries between regular 
 clauses, and start and end positions of embedded clauses. 
 
  In JSON input/output format, the clause boundary is indicated by adding object 
 'clauseAnnotation' to the token (at the same level as objects 'text' and 
 'analysis'). The 'clauseAnnotation' (which is a list of strings) can contain 
 three types of boundary markings:
    KINDEL_PIIR -- indicates that there is a clause boundary AFTER current 
                   token: one clause ends and another starts;
    KIILU_ALGUS -- marks a beginning of a new embedded clause BEFORE current 
                   token;
    KIILU_LOPP  -- marks ending of an embedded clause AFTER current token;

  Example:
    The sentence
       "Mees, keda seal kohtasime, oli tuttav ja teretas meid."

    will obtain following clause annotations:
        {'words': [ {'analysis': [ ... ],
                      'text': 'Mees,'},
                     {'analysis': [ ... ],
                      'clauseAnnotation': ['KIILU_ALGUS'],
                      'text': 'keda'},
                     {'analysis': [ ... ],
                      'text': 'seal'},
                     {'analysis': [ ... ],
                      'clauseAnnotation': ['KIILU_LOPP'],
                      'text': 'kohtasime,'},
                     {'analysis': [ ... ],
                      'text': 'oli'},
                     {'analysis': [ ... ],
                      'text': 'tuttav'},
                     {'analysis': [ ... ],
                      'clauseAnnotation': ['KINDEL_PIIR'],
                      'text': 'ja'},
                     {'analysis': [ ... ],
                      'text': 'teretas'},
                     {'analysis': [ ... ],
                      'text': 'meid.'} ]}

    which should be interpreted as:
          "keda" (KIILU_ALGUS) -- an embedded clause begins before "keda";
          "kohtasime," (KIILU_LOPP) -- the embedded clause ends after "kohtasime,";
          "ja" (KINDEL_PIIR)   -- one clause ends after "ja" and another begins;
          
    so, the corresponding clause structure should look like:
       "[Mees, [keda seal kohtasime,] oli tuttav ja] [teretas meid.]"
       (clauses are surrounded by brackets)
       
  Note that embedded clauses can contain other clauses and other embedded 
 clauses, and so the whole clause structure has a recursive nature.

=========================
   References
=========================

  *) Kaalep, Heiki-Jaan; Muischnek, Kadri (2012). Osalausete tuvastamine 
     eestikeelses tekstis kui iseseisev ülesanne. Helle Metslang, Margit 
     Langemets, Maria-Maren Sepper (Toim.). Eesti Rakenduslingvistika Ühingu 
     aastaraamat (55 - 68). Tallinn: Eesti Rakenduslingvistika Ühing;

  *) Kaalep, Heiki-Jaan; Muischnek, Kadri (2012). Robust clause boundary 
     identification for corpus annotation. Nicoletta Calzolari, Khalid Choukri, 
     Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mar (Toim.). 
     Proceedings of the Eight International Conference on Language Resources 
     and Evaluation (LREC'12) (1632 - 1636). Istanbul, Türgi: ELRA;