Convert FLEx data to CLDF-ready CSV.

Many descriptive linguists have annotated language data in a FLEx (SIL's Fieldworks Lexical Explorer) database, which provides perhaps the most popular and accessible assisted segmentation and annotation workflow. However, a reasonably complete data export is only available in XML, which is not human-friendly, and is not readily converted to other data. A data format growing in popularity is the CLDF standard, a table-based approach with human-readable datasets, designed to be used in CLLD apps and easily processable by any software that can read CSV files, including R, pandas or spreadsheet applications. The goal of cldflex is to convert lexicon and corpus data stored in FLEx to CSV tables, primarily for use in CLDF datasets.

cldflex is available on PyPI:

pip install cldflex

At the moment, there are two commands: cldflex flex2csv processes .flextext (corpora), and cldflex lift2csv processes .lift (lexica) files. Both commands create a number of CSV files. One can either use cldfbench to create one's own CLDF datasets from these files, or add the --cldf argument to create (simple) datasets. Project-specific configuration can be passed via --conf your/config.yaml

Basic usage:

cldflex flex2csv texts.flextext

Connect the corpus with the lexicon:

cldflex flex2csv texts.flextext --lexicon lexicon.lift

Create a CLDF dataset:

cldflex flex2csv texts.flextext --lexicon lexicon.lift --cldf

Extract morphemes, morphs, and entries from lexicon.lift:

cldflex lift2csv lexicon.lift

Create a CLDF dataset with a Dictionary module:

cldflex lift2csv lexicon.lift --cldf

The functions corresponding to the commands above are cldflex.flex2csv.convert() and cldflex.lift2csv.convert().

There is no default configuration. Rather, cldflex will guess values for most of the parameters below and tell you what it's doing. It is suggested to start out configuration-free until something goes wrong or you want to change something. Create a YAML file for CLI usage, pass a dict to the convert methods.

• obj_lg: the object language
• gloss_lg: the language used for glossing / translation
• msa_lg: the language used for storing POS information
• Language_ID: the value to be used in the created tables
• Glottocode: used to look up language metadata from glottolog
• csv_cell_separator: if there are multiple values in a cell (allomorphs, polysemy...), they are by default separated by "; "
• form_slices: set to false if you don't want form slices connecting morphs and word forms
• cldf:
  • lexicon: can be set to "wordlist" to create a Wordlist instead of a dictionary
• mappings: a dictionary specifying name changes of columns in the created CSV files