-
Notifications
You must be signed in to change notification settings - Fork 19
Description
Currently parameter (metadata, etc.) files are processed through xxd to create C source code and then compiled into a binary .o file for linking into an item's shared library.
This fails for large files due to excessive compiler memory usage.
--- See below link ---
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1040r6.html
https://thephd.dev/_vendor/future_cxx/papers/d1967.html
https://github.com/ThePhD/embed
--- xxxx ---
The above proposed standard is evolving and being accepted, but it is nowhere near ready for availability and wide use in the wild.
See this message on openkim-dev:
https://groups.google.com/d/msg/openkim-dev/aTjMuUxRNJY/VrqiPfFcCAAJ
where we discuss dropping xxd in favor of a native cmake implementation. In the end we implemented both. xxd is (much) faster and used if it exists. If not, a cmake implementation is used. -- This is independent of the time needed to compile the output source code.
--- some not-so-portable options for embedding files into binaries ---
https://unix.stackexchange.com/questions/494844/embedding-binary-data-into-an-executable-using-llvm-tools
--- xxx ---
To continue with this approach the kim-item shared-library format will need to be updated to understand that files are embedded in a series of chunks and the cmake processing of kim-items will need to be updated to split the files and handle all the data pointers appropriately.
This will be a significant effort, and we will end up with rather large shared library files (which might become an issue sooner than later--I don't know). My preference was to abandon embedded files as described in #59. However, #59 is a rather bigger (more involved) change than split.