Skip to content

Conversation

nicola-cab
Copy link
Member

@nicola-cab nicola-cab commented Apr 8, 2024

What, How & Why?

Compression is only supported for Integer arrays (and Strings in the future).
In order to decide whether to turn on compression or not, we need to:

  1. Mark the array as "good to compress", navigating the cluster tree and mark the array only if compression is enabled for the particular type (integers, mixed of integers, collections of integers and any other integer alike properties, NOT timestamps)
  2. Add the needed machinery at storage level in order to change the array layout.

This PR addresses 1. Next PRs will be focused on 2.

Most importantly, the commit machinery has changed, because instead of calling Array::write recursively, we, now inspect the properties of the array before to mark it "ready for compression."

@cla-bot cla-bot bot added the cla: yes label Apr 8, 2024
@nicola-cab nicola-cab changed the base branch from nc/rcore-2050 to nc/merge_all_together April 8, 2024 15:32
@nicola-cab nicola-cab changed the base branch from nc/merge_all_together to nc/rcore-2050 April 8, 2024 15:39
@nicola-cab nicola-cab self-assigned this Apr 9, 2024
Base automatically changed from nc/rcore-2050 to next-major April 11, 2024 15:13
Copy link
Contributor

@jedelbo jedelbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@finnschiermer promised that this typed_print would be deleted again. If not, then all these functions could be put together in one file. Then they would not be included in the binary, if not called.

virtual ref_type typed_write(ref_type ref, _impl::ArrayWriterBase& out, const Table& table, bool deep,
bool only_modified, bool compress) const = 0;

virtual void typed_print(std::string prefix, const Table& table) const
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why have table as parameter? It is available through get_owning_table()

auto ref = top.get_as_ref(Group::s_evacuation_point_ndx);
REALM_ASSERT(ref);
Array::destroy(ref, m_alloc);
Array destroy_array(m_alloc);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks odd, we're only constructing the array for destroying it…

@nicola-cab nicola-cab merged commit 549c707 into next-major Apr 23, 2024
@nicola-cab nicola-cab deleted the nc/rcore-2055 branch April 23, 2024 15:47
@nicola-cab nicola-cab restored the nc/rcore-2055 branch May 2, 2024 16:36
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 1, 2024
@nicola-cab nicola-cab changed the title RCORE-2055 Array Classification for enabling compression RCORE-1624 RCORE-2055 Array Classification for enabling compression Jun 6, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants