Index zero and False values in the tabular pipeline - #1207

Problem

Tabular.concat() builds the text that gets indexed for each row. It tested every column value for truthiness:

column = self.column(row[column])
if column:
    parts.append(str(column))

column() normalizes NaN to None and returns everything else unchanged, so None is the marker for a missing value. The truthy test also drops 0, 0.0 and False, which are real data.

A row of {"id": 1, "quantity": 0, "text": "Widget"} with Tabular("id", ["quantity", "text"]) indexes as "Widget". The quantity is gone, and nothing reports it.

Measured on current master:

Change

concat() now compares against the marker column() actually produces:

if column is not None and column != "":

Empty strings stay excluded on purpose. Including them adds a ". " separator and no content, so {"quantity": "", "text": "Widget"} would index as ". Widget".

After the change:

Tests

Two tests added to test/python/testpipeline/testdata/testtabular.py:

• testFalsyValues asserts 0 and False reach the indexed text
• testNullValues asserts NaN and empty strings are still skipped

Reverting the one-line change makes testFalsyValues fail while testNullValues still passes, so the new test covers the behaviour that changed and not the behaviour that did not.

testtabular.py reports 8 passed. testCSV fails in my environment because it wants /tmp/txtai/tabular.csv, and it fails the same way on an unmodified checkout.

black --check reports both files unchanged.