Summary

• flush logs should flush every ~10k log rows (ideal per the clickhouse docs)
• previously, it was using the Kafka batch size, but each Kafka message has multiple log rows, so there would be unread messages until the next flush
• now, loop and flush all messages until the message queue is empty

How did you test this change?

• tested locally to make sure log ingestion was working, will monitor in prod

Are there any deployment considerations?

• no