Skip to content

Split parameter (and other) files into parts to facilitate embedding large files into shared library #85

@ellio167

Description

@ellio167

Currently parameter (metadata, etc.) files are processed through xxd to create C source code and then compiled into a binary .o file for linking into an item's shared library.

This fails for large files due to excessive compiler memory usage.
--- See below link ---
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1040r6.html
https://thephd.dev/_vendor/future_cxx/papers/d1967.html
https://github.com/ThePhD/embed
--- xxxx ---
The above proposed standard is evolving and being accepted, but it is nowhere near ready for availability and wide use in the wild.

See this message on openkim-dev:
https://groups.google.com/d/msg/openkim-dev/aTjMuUxRNJY/VrqiPfFcCAAJ
where we discuss dropping xxd in favor of a native cmake implementation. In the end we implemented both. xxd is (much) faster and used if it exists. If not, a cmake implementation is used. -- This is independent of the time needed to compile the output source code.

--- some not-so-portable options for embedding files into binaries ---
https://unix.stackexchange.com/questions/494844/embedding-binary-data-into-an-executable-using-llvm-tools
--- xxx ---

To continue with this approach the kim-item shared-library format will need to be updated to understand that files are embedded in a series of chunks and the cmake processing of kim-items will need to be updated to split the files and handle all the data pointers appropriately.

This will be a significant effort, and we will end up with rather large shared library files (which might become an issue sooner than later--I don't know). My preference was to abandon embedded files as described in #59. However, #59 is a rather bigger (more involved) change than split.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions