Skip to content

jabr/pgfeed

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

20 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

pgfeed

This utility reads a standard PostgreSQL pgoutput logical replication slot (in text or binary mode) and sends INSERT records to the configured upstream sink.

Supported sinks

  • stdout
  • AMQP (todo)

LSN save point

The replicated LSN position is periodically written to file (e.g. replicated.lsn) and used as the starting point sent to Postgres on subsequent starts.

Architecture

  • PostgreSQL replication runs in an isolated thread, using the libpq library with blocking I/O.
  • Writes to upstream sink use an asynchronous I/O runtime in its own, dedicated thread.
  • Logging output and periodic replicated LSN position checkpoint writes occur in a third (parent) thread.

Crystal

This currently requires the -Dpreview_mt -Dexecution_context options to enable execution contexts in Crystal.

Replication setup in Postgres

-- Enable logical replication in postgresql.conf:
-- wal_level = logical
-- max_replication_slots = 10
-- max_wal_senders = 10

-- Create a replication role
CREATE ROLE repuser WITH REPLICATION LOGIN PASSWORD 'reppass';

-- Create a publication
CREATE PUBLICATION my_pub FOR ALL TABLES;

-- Create a logical replication slot
SELECT * FROM pg_create_logical_replication_slot('my_slot', 'pgoutput');

References

License

This project is licensed under the terms of the MIT license.

About

Consume PostgreSQL logical replication INSERT stream and send to sink

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published