- Add the GloVe algorithm for
training word embeddings and a library class
word_embeddingsfor loading and querying trained embeddings. To facilitate returning word embeddings, a simpleutil::array_viewclass was added. - Add simple vector math library (and move
fastapproxinto themathnamespace).
- Fix
probe_map::extract()forinline_key_value_storagetype; old implementation forgot to delete all sentinel values before returning the vector. - Fix incorrect definition of
l1norm()insgd_model. - Fix
gmapcalculation where 0 average precision was ignored - Fix progress output in
multiway_merge.
- Improve performance of
printing::progress. Before,progress::operator()in tight loops could dramatically hurt performance, particularly due to frequent calls tostd::chrono::steady_clock::now(). Now,progress::operator()simply sets an atomic iteration counter and a background thread periodically wakes to update the progress output. - Allow full text storage in index as metadata field. If
store-full-text = true(default false) in the corpus config, the string metadata field "content" will be added. This is to simplify the creation of full text metadata: the user doesn't have to duplicate their dataset inmetadata.dat, andmetadata.datwill still be somewhat human-readable without large strings of full text added. - Allow
make_indexto take a user-supplied corpus object.
- ZLIB is now a required dependency.
- Switch to just using the standalone
./unit-testinstead ofctest. There aren't really many advantages for us to using CTest at this point with the new unit test framework, so just use our unit test executable.
- Fix issue where
metadata_parserwould not consume spaces in string metadata fields. Thanks to @hopsalot on the forum for the bug report! - Fix build issue on OS X with Xcode 6.4 and
clangrelated to their shipped version ofstring_viewlacking a constto_string()method
- The
./profileexecutable ensures that the file exists before operating on it. Thanks to @domarps for the PR! - Add a generic
util::multiway_mergealgorithm for performing the merge-step of an external memory merge sort. - Build with the following Xcode versions on Travis CI:
- Xcode 6.1 and OS X 10.9 (as before)
- Xcode 6.4 and OS X 10.10 (new)
- Xcode 7.1.1 and OS X 10.10 (new)
- Xcode 7.2 and OS X 10.11 (new)
- Index format rewrite: both inverted and forward indices now use the same compressed postings format, and intermediate chunks are now also compressed on-the-fly. There is now a built in tool to dump any forward index to libsvm format (as this is not the on-disk format for that type of index anymore).
- Metadata support: indices can now store arbitrary metadata associated with individual documents with string, integer, unsigned integer, and floating point values
- Corpus configuration is now stored within the corpus directory itself, allowing for corpora to be distributed with their proper configurations rather than having to bake this into the main configuration file
- RAM limits can be set for the indexing process via the configuration file. These are approximate and based on heuristics, so you should always set these to lower than available RAM.
- Forward indices can now be created directly instead of forcing the creation of an inverted index first
- ICU will be built and statically linked if the system provided library is too old on both OS X and Linux platforms. MeTA now will specify an exact version of ICU that should be used per release for consistency. That version is 56.1 as of this release.
- Analyzers have been modified to support both integral and floating point
values via the use of the
featurizerobject passed totokenize() - Documents no longer store any count information during the analysis process
- Postings lists can now be read in a streaming fashion rather than all at
once via
postings_stream - Ranking is now performed using a document-at-a-time scheme
- Ranking functions now use fast approximate math from fastapprox
- Rank correlation measures have been added to the evaluation library
- Rewrite of the language model library which can load models from the .arpa format
- SyntacticDiff implementation for comparative text mining, which may include grammatical error correction, summarization, or feature generation
- A feature selection library for selecting features for machine learning using chi square, information gain, correlation coefficient, and odds ratio has been added
- The API for the machine learning algorithms has been changed to use
datasetclasses; these are separate from the index classes and represent data that is memory-resident - Support for regression has been added (currently only via SGD)
- The SGD algorithm has been improved to use a normalized adaptive gradient method which should make it less sensitive to feature scaling
- The SGD algorithm now supports (approximate) L1 regularization via a cumulative penalty approach
- The libsvm modules are now also built using CMake
- Packed binary I/O functions allow for writing integers/floating point values in a compressed format that can be efficiently decoded. This should be used for most binary I/O that needs to be performed in the toolkit unless there is a specific reason not to.
- An interactive demo application has been added for the shift-reduce constituency parser
- A
string_viewclass is provided in themeta::utilnamespace to be used for non-owning references to strings. This will usestd::experimental::string_viewif available and our own implementation if not meta::util::optionalwill resolve tostd::experimental::optionalif it is available- Support for jemalloc has been added to the build system. We strongly recommend installing and linking against jemalloc for improved indexing performance.
- A tool has been added to print out the top k terms in a corpus
- A new library for hashing has been added in namespace
meta::hashing. This includes a generic framework for writing hash functions that are randomly keyed as well as (insertion only) probing-based hash sets/maps with configurable resizing and probing strategies - A utility class
fixed_heaphas been added for places where a fixed size set of maximal/minimal values should be maintained in constant space - The filesystem management routines have been converted to use STLsoft in
the event that the filesystem library in
std::experimental::filesystemis not available - Building MeTA on Windows is now officially supported via MSYS2 and MinGW-w64, and continuious integration now builds it on every commit in this environment
- A small support library for things related to random number generation
has been added in
meta::random - Sparse vectors now support
operator+andoperator- - An STL container compatible allocator
aligned_allocator<T, Alignment>has been added that can over-align data (useful for performance in some situations) - Bandit is now used for the unit tests, and these have been substantially improved upon
io::parserdeprecated and removed; most uses simply converted tostd::fstreambinary_file_{reader,writer}deprecated and removed;io::packedorio::{read,write}_binaryshould be used instead
- knn classifier now only requests the top k when performing classification
- An issue where uncompressed model files would not be found if using a zlib-enabled build (#101)
- Travis CI integration has been switched to their container infrastructure, and it now builds with OS X with Clang in addition to Linux with Clang and GCC
- Appveyor CI for Windows builds alongside Travis
- Indexing speeds are dramatically faster (thanks to many changes both in the in-memory posting chunks as well as optimizations in the tokenization process)
- If no build type is specified, MeTA will be built in Release mode
- The cpptoml dependency version has been bumped, allowing the use of
things like
value_orfor cleaner code - The identifiers library has been dramatically simplified
- Fix issue with
confusion_matrixwhere precision and recall values were swapped. Thanks to @husseinhazimeh for finding this!
- Better unit tests for
confusion_matrix - Add functions to
confusion_matrixto directly access precision, recall, and F1 score - Create a
predicted_labelopaque identifier to emphasizeclass_labelsthat are output from some model (and thus shouldn't be interchangeable)
- Fix inconsistent behavior of
utf::segmenter(and thusicu_tokenizer) for different locales. Thanks @CanoeFZH and @tng-konrad for helping debug this!
- Allow for specifying the language and country for locale generation in
setting up
utf::segmenter(and thusicu_tokenizer) - Allow for suppression of
<s>and</s>tags withinicu_tokenizer, mostly useful for information retrieval experiments with unigram words. Thanks @husseinhazimeh for the suggestion! - Add a
default-unigram-chainfilter chain preset which is suitable for information retrieval experiments using unigram words. Thanks @husseinhazimeh for the suggestion!
- Fix potential off-by-one when calculating the number of documents in a
line_corpuswhen its files do not end in a newline
- Change
score_datato support floating-point weights on query terms
- Fix missing support for sequence/parser analyzers in the classify tools
- Support building with biicode
- Add Vagrantfile for virtual machine configuration
- Add Dockerfile for Docker support
- Improve
ir_evalunit tests
- Fix
ir_eval::ndcgincorrect log base and addition instead of subtraction in IDCG calculation - Fix
ir_eval::avg_pincorrect early termination
- Fix issues with system-defined integer widths in binary model files (mainly impacted the greedy tagger and parser); please re-download any parser model files you may have had before
- Fix bug where parser model directory is not created if a non-standard prefix is used (anything other than "parser")
- Silence inconsistent missing overrides warning on clang >= 3.6
- fix potentially incorrect generation of vocabulary map files on 32-bit systems (this appears to have only impacted non-default block sizes)
- fix calculation of average precision in
ir_eval(the denominator was incorrect) - specify that labels are required for the
file_corpusdocument list; this allows spaces in the path to each document
- additions to the graph library:
- myopic search
- BFS
- preferential attachment graph generation model (supports node attractiveness from different distributions)
- betweenness centrality
- eigenvector centrality
- added a new natural language parsing library:
- parse tree library (visitor-based)
- shift-reduce constituency parser for generating phrase structure trees
- reimplementation of evalb metrics for evaluating parsers
- new filter for Penn Treebank-style normalization
- added a greedy averaged Perceptron-based tagger
- demo application for various basic text processing (profile)
- basic iostreams that support gzip compression (if compiled with ZLib support)
- added iteration method for
stats::multinomialseen events - added expected value and entropy functions to
statsnamespace - added
linear_model: a generic multiclass classifier storage class - added
gz_corpus: a compressed version ofline_corpus - added macros for generating type safe identifiers with user defined literal suffixes
- added a persistent stack data structure to
meta::util
- added operator== for
util::optional<T> - better CMake support for building the libsvm modules
- better CMake support for downloading unit-test data
- improved setup guide in README (for OS X, Ubuntu, Arch, and EWS/ENGRIT)
- tree analyzers refactored to use the new parser library (removes dependency on outside toolkits for generating tree files)
- analyzers that are not part of the "core" have been moved into their
respective folders (so
ngram_pos_analyzeris insrc/sequence,tree_analyzeris insrc/parser) make_indexnow checks if the files exist before loading an index, and if they are missing creates a new one (as opposed to just throwing an exception on a nonexistent file)- cpptoml upgraded to support TOML v0.4.0
- enable extra warnings (-Wextra) for clang++ and g++
- fix
sequence_analyzer::analyze() constwhen applied to untagged sequences (was throwing when it shouldn't) - ensure that the inverted index object is destroyed first before
uninverting occurs in the creation of a
forward_idnex - fix bug where
icu_tokenizerwould output spaces as tokens - fix bugs where index objects were not destroyed before trying to delete their files in the unit tests
- fix bug in
sparse_vector::find()where it would return a non-end iterator when asked to find an element that does not exist
- demo application for CRF-based POS tagging
nearest_centroidclassifier- basic statistics library for representing relevant probability distributions
sparse_vectorutility class
ngram_pos_analyzernow uses the CRf internally (see issue #46)knnclassifier new supports weighted knnfilesystem::copy_file()no longer hangs without progress reporting with large files- CMake build system now includes
INTERFACEtargets (better inclusion as a subproject in external projects) - MeTA can now (optionally) be built with C++14 support
language_model_rankerscoring function corrected (see issue #50)naive_bayesclassifier scoring corrected- several incorrect instances of
numeric_limits<double>::min()replaced with the intendednumeric_limits<double>::lowest() - fix compilation with versions of ICU < 4.4
- sequence analyzer and CRF implementation
- basic language model
- basic directed and undirected graphs
- restructure CMakeLists
- Initial release.