Skip to content

Nullable Attribute Support - #1895

Merged
joe-maley merged 2 commits into
devfrom
jpm/null-support
Nov 10, 2020
Merged

joe-maley merged 2 commits into
devfrom
jpm/null-support

Conversation

@joe-maley

Copy link
Copy Markdown
Contributor

Attributes can be defined as nullable. Nullable attributes require a "validity
vector" buffer for both read and write queries, similar to how var-sized
attributes require an additional "offsets" buffer. Both fixed and var-sized
attributes may be nullable.

Using the C API, attributes must be set nullable before adding them to the
schema, e.g.:

tiledb_attribute_t* attr;
tiledb_attribute_alloc(ctx, "my_attr", TILEDB_INT32, &attr);
tiledb_attribute_set_nullable(ctx, attr, 1 /* nullable */);

tiledb_array_schema_t* array_schema;
tiledb_array_schema_alloc(ctx_, TILEDB_DENSE, &array_schema);
tiledb_array_schema_add_attribute(ctx_, array_schema, attr);

Write queries require a validity vector (bytemap) for nullable attributes. In
the below example, values "200" and "300" are null. These values may or may not
be written to the disk. TileDB may treat them as garbage.

int32_t buffer = {100, 200, 300, 400};
uint64_t buffer_size = sizeof(buffer);
uint8_t buffer_validity = {1, 0, 0, 1};
uint64_t buffer_validity_size = sizeof(buffer_validity);
tiledb_query_set_buffer_nullable(
  ctx,
  query,
  "my_attr",
  buffer,
  buffer_size,
  buffer_validity,
  buffer_validity_size);

Overview:

  • Format version bumped from 6 to 7.

  • Validity vector buffers are written to their own tile, similar to how offset
    buffers are written to their own tile, separate from the value tile.

  • Currently, the "validity vector" is a bytemap in all usage (APIs, in-memory,
    and on-disk). In the future, we could like to store the validity vector as
    a bitmap in-memory and on-disk, but allowing the user to use an API that
    uses either a bitmap or bytemap.

  • A new, internal ValidityVector class has been introduced to store the
    validity vector in-memory. This may seem extraneous because it wraps a simple
    buffer, but this will change in the future when we support bitmaps.

  • Similar to the existing "sm.memory_budget" and "sm.memory_budget_var" config
    parameters, there is now a "sm.memory_budget_validity" for budgeting the
    validity vector buffers.

  • Similar to offset tiles, validity tiles have their own compressor that is
    independent of the user-defined attribute filter. I have tentatively chosen
    RLE compression.

  • C/C++ APIs has been added.

  • The QueryBuffer class has been moved from misc/query_buffer.h to
    query/query_buffer.h because it now depends on query/validity_vector,
    which is outside of the misc directory.

  • Many of the internal classes are now nullable-aware (Reader, Writer,
    Query, FilterPipeline, Subarray, SubarrayPartitioner).

@stavrospapadopoulos stavrospapadopoulos left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent work @joe-maley! I have just a few comments.

Also we need to edit HISTORY.md and the c-api.rst file in the docs for the added C API functions.

Comment thread format_spec/FORMAT_SPEC.md
Comment thread tiledb/sm/config/config.cc Outdated
Comment thread tiledb/sm/cpp_api/query.h Outdated
@joe-maley

Copy link
Copy Markdown
Contributor Author

@stavrospapadopoulos Done -- I've updated the HISTORY.md and c-api.rst.

Comment thread tiledb/sm/config/config.cc Outdated
Attributes can be defined as nullable.  Nullable attributes require a "validity
vector" buffer for both read and write queries, similar to how var-sized
attributes require an additional "offsets" buffer. Both fixed and var-sized
attributes may be nullable.

Using the C API, attributes must be set nullable before adding them to the
schema, e.g.:
```
tiledb_attribute_t* attr;
tiledb_attribute_alloc(ctx, "my_attr", TILEDB_INT32, &attr);
tiledb_attribute_set_nullable(ctx, attr, 1 /* nullable */);

tiledb_array_schema_t* array_schema;
tiledb_array_schema_alloc(ctx_, TILEDB_DENSE, &array_schema);
tiledb_array_schema_add_attribute(ctx_, array_schema, attr);
```

Write queries require a validity vector (bytemap) for nullable attributes. In
the below example, values "200" and "300" are null. These values may or may not
be written to the disk. TileDB may treat them as garbage.
```
int32_t buffer = {100, 200, 300, 400};
uint64_t buffer_size = sizeof(buffer);
uint8_t buffer_validity = {1, 0, 0, 1};
uint64_t buffer_validity_size = sizeof(buffer_validity);
tiledb_query_set_buffer_nullable(
  ctx,
  query,
  "my_attr",
  buffer,
  buffer_size,
  buffer_validity,
  buffer_validity_size);
```

Overview:
- Format version bumped from 6 to 7.

- Validity vector buffers are written to their own tile, similar to how offset
  buffers are written to their own tile, separate from the value tile.

- Currently, the "validity vector" is a bytemap in all usage (APIs, in-memory,
  and on-disk). In the future, we could like to store the validity vector as
  a bitmap in-memory and on-disk, but allowing the user to use an API that
  uses either a bitmap or bytemap.

- A new, internal `ValidityVector` class has been introduced to store the
  validity vector in-memory. This may seem extraneous because it wraps a simple
  buffer, but this will change in the future when we support bitmaps.

- Similar to the existing "sm.memory_budget" and "sm.memory_budget_var" config
  parameters, there is now a "sm.memory_budget_validity" for budgeting the
  validity vector buffers.

- Similar to offset tiles, validity tiles have their own compressor that is
  independent of the user-defined attribute filter. I have tentatively chosen
  RLE compression.

- C/C++ APIs has been added.

- The `QueryBuffer` class has been moved from `misc/query_buffer.h` to
  `query/query_buffer.h` because it now depends on `query/validity_vector`,
  which is outside of the `misc` directory.

- Many of the internal classes are now nullable-aware (`Reader`, `Writer`,
  `Query`, `FilterPipeline`, `Subarray`, `SubarrayPartitioner`).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants