Nullable Attribute Support - #1895
Merged
Merged
Conversation
joe-maley
force-pushed
the
jpm/null-support
branch
from
November 5, 2020 23:13
abafe5a to
2b84af7
Compare
Member
There was a problem hiding this comment.
Excellent work @joe-maley! I have just a few comments.
Also we need to edit HISTORY.md and the c-api.rst file in the docs for the added C API functions.
joe-maley
force-pushed
the
jpm/null-support
branch
from
November 9, 2020 16:18
2b84af7 to
12890c1
Compare
Contributor
Author
|
@stavrospapadopoulos Done -- I've updated the |
Attributes can be defined as nullable. Nullable attributes require a "validity
vector" buffer for both read and write queries, similar to how var-sized
attributes require an additional "offsets" buffer. Both fixed and var-sized
attributes may be nullable.
Using the C API, attributes must be set nullable before adding them to the
schema, e.g.:
```
tiledb_attribute_t* attr;
tiledb_attribute_alloc(ctx, "my_attr", TILEDB_INT32, &attr);
tiledb_attribute_set_nullable(ctx, attr, 1 /* nullable */);
tiledb_array_schema_t* array_schema;
tiledb_array_schema_alloc(ctx_, TILEDB_DENSE, &array_schema);
tiledb_array_schema_add_attribute(ctx_, array_schema, attr);
```
Write queries require a validity vector (bytemap) for nullable attributes. In
the below example, values "200" and "300" are null. These values may or may not
be written to the disk. TileDB may treat them as garbage.
```
int32_t buffer = {100, 200, 300, 400};
uint64_t buffer_size = sizeof(buffer);
uint8_t buffer_validity = {1, 0, 0, 1};
uint64_t buffer_validity_size = sizeof(buffer_validity);
tiledb_query_set_buffer_nullable(
ctx,
query,
"my_attr",
buffer,
buffer_size,
buffer_validity,
buffer_validity_size);
```
Overview:
- Format version bumped from 6 to 7.
- Validity vector buffers are written to their own tile, similar to how offset
buffers are written to their own tile, separate from the value tile.
- Currently, the "validity vector" is a bytemap in all usage (APIs, in-memory,
and on-disk). In the future, we could like to store the validity vector as
a bitmap in-memory and on-disk, but allowing the user to use an API that
uses either a bitmap or bytemap.
- A new, internal `ValidityVector` class has been introduced to store the
validity vector in-memory. This may seem extraneous because it wraps a simple
buffer, but this will change in the future when we support bitmaps.
- Similar to the existing "sm.memory_budget" and "sm.memory_budget_var" config
parameters, there is now a "sm.memory_budget_validity" for budgeting the
validity vector buffers.
- Similar to offset tiles, validity tiles have their own compressor that is
independent of the user-defined attribute filter. I have tentatively chosen
RLE compression.
- C/C++ APIs has been added.
- The `QueryBuffer` class has been moved from `misc/query_buffer.h` to
`query/query_buffer.h` because it now depends on `query/validity_vector`,
which is outside of the `misc` directory.
- Many of the internal classes are now nullable-aware (`Reader`, `Writer`,
`Query`, `FilterPipeline`, `Subarray`, `SubarrayPartitioner`).
joe-maley
force-pushed
the
jpm/null-support
branch
from
November 10, 2020 14:35
12890c1 to
33a376d
Compare
stavrospapadopoulos
approved these changes
Nov 10, 2020
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Attributes can be defined as nullable. Nullable attributes require a "validity
vector" buffer for both read and write queries, similar to how var-sized
attributes require an additional "offsets" buffer. Both fixed and var-sized
attributes may be nullable.
Using the C API, attributes must be set nullable before adding them to the
schema, e.g.:
Write queries require a validity vector (bytemap) for nullable attributes. In
the below example, values "200" and "300" are null. These values may or may not
be written to the disk. TileDB may treat them as garbage.
Overview:
Format version bumped from 6 to 7.
Validity vector buffers are written to their own tile, similar to how offset
buffers are written to their own tile, separate from the value tile.
Currently, the "validity vector" is a bytemap in all usage (APIs, in-memory,
and on-disk). In the future, we could like to store the validity vector as
a bitmap in-memory and on-disk, but allowing the user to use an API that
uses either a bitmap or bytemap.
A new, internal
ValidityVectorclass has been introduced to store thevalidity vector in-memory. This may seem extraneous because it wraps a simple
buffer, but this will change in the future when we support bitmaps.
Similar to the existing "sm.memory_budget" and "sm.memory_budget_var" config
parameters, there is now a "sm.memory_budget_validity" for budgeting the
validity vector buffers.
Similar to offset tiles, validity tiles have their own compressor that is
independent of the user-defined attribute filter. I have tentatively chosen
RLE compression.
C/C++ APIs has been added.
The
QueryBufferclass has been moved frommisc/query_buffer.htoquery/query_buffer.hbecause it now depends onquery/validity_vector,which is outside of the
miscdirectory.Many of the internal classes are now nullable-aware (
Reader,Writer,Query,FilterPipeline,Subarray,SubarrayPartitioner).