Releases: wal-g/wal-g
v3.0.8
Major features of this stable release are:
-
Infrastructure & CI
- Switched tests and workflows to ubuntu-latest due to 20.04 deprecation.
- Updated Go version to 1.24 and refreshed linters (golangci-lint v2.0+).
- Multiple dependency upgrades (Azure SDK, OAuth2, crypto, JWT, gRPC, pgx, compress, etc.).
-
PostgreSQL / Greenplum / Cloudberry
- Added DIRECT-IO Reader for PostgreSQL.
- New commands for continuous recovery in GPDB/CBDB.
- Fixes for WAL page size, block size, checksums, and Cloudberry test stability.
- Support for xid64 patch page layout.
-
Redis
- Enhanced support for sharded Redis backups/restores: slots info, hostname/config parsing, validation fixes.
-
MongoDB
- Introduced partial restore and partial oplog replay.
- Improved metadata handling and collection in separate goroutines.
- Dropped tests for deprecated Mongo versions.
-
MySQL
- Added binlog-list command and reconnection to binlog server.
- Expanded unit tests for metadata fetch/set.
-
Storage Backends
- New implementation for Alibaba Cloud OSS with multipart upload support.
- Fixes for S3 client headers, batch deletion, SSE-C bug, and versioning behavior.
- Added S3_DELETE_BATCH_SIZE and none compression method.
-
Exporter & Monitoring
- Created Prometheus exporter with improved metrics (backup size, reduced binary size, better scrape intervals).
- Performance optimizations in WAL-G exporter.
-
Other Fixes
- Improved error handling (e.g., corrupted WAL metadata, connection refused).
- Added retries for restore point creation.
- Various test workflow improvements and cleanup.
What's Changed
- Refactor: Move CheckArchiveCommand from "connect.go" to queryRunner by @boosterKRD in #1882
- CI: switch unit tests to ubuntu-latest by @chipitsine in #1965
- Linters update to v2.0 by @debebantur in #1955
- Add config option WALG_FORCE_WAL_DELTA by @fanfuxiaoran in #1940
- Switch to container due to Ubuntu 20.04 deprecation on GitHub (fix the worflow release.yml) by @boosterKRD in #1970
- switch docker tests to ubuntu-latest by @chipitsine in #1973
- Disable WALG_FORCE_WAL_DELTA in tests by @fanfuxiaoran in #1975
- update azure sdk by @serprex in #1982
- Disable HTTP keepalive in endpoint requests by @EinKrebs in #1976
- golangci-lint 2.1 by @serprex in #1981
- add slots info for sharded Redis restore by @kronwerk in #1989
- some fixes for slot collection by @kronwerk in #1992
- fix Redis slots data paths by @kronwerk in #1994
- Added retries for restore point creation by @debebantur in #2000
- Added PGAPPNAME to allowed settings by @debebantur in #2002
- wal-verify flags with specified lsn and timeline to check by @instahipsta14 in #2001
- Fix failed restore gp cbdb by @Mulily0513 in #1974
- DIRECT-IO Reader for PostgreSQL by @vadv in #2003
- add slots info for sharded rdb backups also by @kronwerk in #2007
- Fix WalPageSize and BlockSize for gp and cloudberry by @fanfuxiaoran in #1968
- drop 4.2 and 5.0 mongo_features tests due to deprecated versions by @NotAmigo in #2009
- add partially restore to mongodb by @NotAmigo in #1995
- add sharded redis hostname parsing options by @kronwerk in #2014
- add redis sharded config parsing options by @kronwerk in #2015
- fix slots validation for redis sharded backups by @kronwerk in #2016
- update golang.org/x/oauth2 to v0.27.0 by @chipitsine in #2017
- Add
HandleDetailedBackupListtable output unit-tests by @BasedDepartment1 in #2013 - etcd: Delete target backup by @diPhantxm in #2018
- Update go version to 1.24 by @ostinru in #2025
- feat: add alicloud OSS storage implementation by @imrenagi in #2022
- Move copy composer file upload logic to its own method to create a different scope and avoid race condition by @dkhorasanizadeh in #2020
- Fix the issue where a corrupted WAL metadata file caused failure in pushing WALs by @dkhorasanizadeh in #2021
- fix: remove warning when alicloud oss valid configurations are present by @imrenagi in #2028
- Fix greenplum_path.sh usage for Cloudberry by @ostinru in #2029
- Fix WALG_FAILOVER_STORAGES_CHECK unknown setting warning by @de1ayer in #2019
- Update partial restore for mongodb by @NotAmigo in #2026
- tests: rebuild docker image for pg_storage_ssh_* test as well by @chipitsine in #2030
- Fix GPDB/CBDB/etcd build by @ostinru in #2033
- Feat: Add new commands to support gpdb/cbdb continuous recovery by @Mulily0513 in #1998
- remove debug prints by @NotAmigo in #2032
- Check for pageHeader.isNew before pageHeader.isValid by @bonnefoa in #2037
- build & release: pick only one tag if there are multiple by @chipitsine in #2006
- tests: mute ubuntu 20.04 EOL on mariadb setup by @chipitsine in #2039
- fix: HTTP-headers sent by S3 client by @ostinru in #2036
- Support the xid64 patch's page layout. by @e-voro in #2040
- feat: use multipart upload for alicloud oss by @imrenagi in #2043
- fix cross building of sql server by @chipitsine in #2046
- Fix cloudberry test by pinning Cloudberry 2.0 as stable version we support by @ostinru in #2045
- Bump actions/checkout from 4 to 5 by @dependabot[bot] in #2048
- CI: add sql server cross build by @chipitsine in #2047
- Add binlog-list command for mysql by @zkramorev in #2051
- download MongoDB backup metadata only if at least one of white or black lists are specified by @NotAmigo in #2053
- Added gcflags to enable debugging for build by @debebantur in #2052
- Bump gopkg.in/yaml.v3 from 3.0.0 to 3.0.1 in /internal/tools by @dependabot[bot] in #2054
- Bump actions/checkout from 4 to 5 by @dependabot[bot] in #2055
- add flag to count journal info by @kronwerk in #2049
- Bump actions/setup-go from 5 to 6 by @dependabot[bot] in #2059
- do not delete key from tarfileset if file has been already found by @NotAmigo in #2057
- Use cast to convert config values by @NikkoFox in #2034
- Add 'connection refused' to ShouldRetry by @ostinru in #2061
- Separate metadata MongoDB operations to goroutine && add mechanism of metadata collectiong by @NotAmigo in #2060
- fix: defer to delete temporary directory by @imrenagi in #2066
- Greenplum follow-primary: use absolute path to pg_ctl by @ostinru in #2063
- Create prometheus exporter by @dementii-priadko in #2064
- Greenplum follow-primary: print missing WAL file names by @ostinru in #2070
- tests: do not use MONGO_MAJOR anymore by @chipitsine in #2058
- tests: migrate from "bitnami" to "bitnamilegacy" by @chipitsine in #2076
- cleanup: do not install "pinentry-qt" to pg test container by @chipitsine in #2078
- do not use deprecated "-d" flag in "go get" by @chipitsine in #2079
- PostgreSQL & Greenplum: Reduce memory usage by @ostinru in #2056
- sequential test workflow: allow manual execution in forks by @chipitsine in #2077
- prepare test scripts for Postgresql > 10 by @chipitsine in #2080
- added GODEBUG to known settings by @debebantur in #2082
- Partial oplog replay by @NotAmigo in #2069
- preparation for Postgres > 10 by @chipitsine in #2084
- tests: run workflows on any branch by @chipitsine in #2083
- tests: drop /tmp/scripts/wait_while_pg_not_ready.sh wherever possible by @chipitsine in #2085
- Reduce walg-exporter binary siz...
v3.0.7
Major features of this stable release are:
-
From Greenplum's perspective, the following updates are beneficial: Commit 92e3995 introduces a --force-delete flag to the delete command. This enhancement enables users to effectively remove backups, even in cases where earlier deletion attempts were unsuccessful.
-
Commit 094bcb4 for Cloudberry: Added support for recognizing 'Apache Cloudberry' in the version() command, facilitating users in working with Cloudberry from the trunk branch.
-
Commit 44e4386 enhances the supply chain, resolving issues flagged by the Trivy security scanner.
What's Changed
- tests: remove few "sleep" by @chipitsine in #1883
- add unit test to /wal-g/internal/databases/greenplum/backup_push_hand… by @hairy-snake in #1890
- tests: follow redirects when building MariaDB image by @chipitsine in #1889
- Bump golangci/golangci-lint-action from 6.1.1 to 6.2.0 by @dependabot in #1891
- Add JournalInfo deletion logic by @noname0443 in #1874
- Refactor tests for postgres cleaner using gomock by @kozlov-ma in #1892
- Fixing issue by @M1ha-Shvn in #1842
- storages (s3): Add AWS_DUAL_STACK configuration by @sundbry in #1691
- Update pgx to v5 by @debebantur in #1893
- docker: use minio from docker hub (every time) by @chipitsine in #1895
- add user to Redis connection by @kronwerk in #1894
- replace redis aof backup pin method by @kronwerk in #1899
- remove folder cleanup by @kronwerk in #1901
- add user data into etcd sentinel by @diPhantxm in #1902
- Greenplum: add --force-delete flag to delete command by @ostinru in #1903
- Added working with S3 versions 2 by @debebantur in #1900
- pg_tests: try to prevent race condition on background start by @chipitsine in #1896
- Add unit tests for internal/databases/redis.HandleDetailedBackupList by @kozlov-ma in #1876
- Bump golangci/golangci-lint-action from 6.2.0 to 6.3.1 by @dependabot in #1907
- Reuse multistorage for daemon mode by @debebantur in #1908
- MongoDB oplog replay: skip startIndexBuild and abortIndexBuild ops + some refactoring by @VGoshev in #1911
- Better gp multistorage by @munakoiso in #1898
- Add @kronwerk to MongoDB codeowners by @VGoshev in #1914
- MongoDB Oplog replay: fix skipping some ops + fix issue in indexSpecFromCommitIndexBuilds by @VGoshev in #1913
- remove one more sleep from tests by @chipitsine in #1904
- MySQL: apply xtrabackup deltas by @ostinru in #1831
- fix assume-role by @dbarrosop in #1906
- Fix connection break by @debebantur in #1917
- Cloudberry: accept 'Apache Cloudbery' in version() by @ostinru in #1919
- Wal-g write an error to logs when there are no backups in a S3 storage by @noname0443 in #1918
- Remove many recursive queries in metadata collection by @debebantur in #1920
- add max db number to walg sentinel by @kronwerk in #1921
- differ 0 and missing data by @kronwerk in #1922
- Bump uraimo/run-on-arch-action from 2 to 3 by @dependabot in #1925
- Fix index out of range in
wal-g xb extractby @ostinru in #1923 - Add
wal-g st get-streamcommand to fetch backup as a single stream by @ostinru in #1924 - fix typo by @munakoiso in #1926
- MySQL: fix how --inplace handles delta files by @ostinru in #1928
- Do not run tests twice when PR submitted by @ostinru in #1931
- Remove redundant debug by @munakoiso in #1935
- Fix Cloudberry Tests - follow upstream changes by @ostinru in #1933
- Fix postgresql tests by @debebantur in #1934
- migrate to new issue templates by @boosterKRD in #1943
- Allow restore between different patches of the same redis minor version by @zykovd in #1942
- Added deduplication of pgdata in path by @debebantur in #1946
- Reduce logging for partial database restore [pg/gp] by @ostinru in #1947
- move pgconn and pgproto used into pgx by @fepape-ionos in #1941
- golangci-lint 1.64 by @serprex in #1910
- refactoring oplog replay by @Apelsin234 in #1950
- releases: add ubuntu22 for aarch64 by @chipitsine in #1956
- Fix catchup issuing premature stop by @x4m in #1957
- Prefetched wal is expected in default directory by @debebantur in #1954
New Contributors
- @hairy-snake made their first contribution in #1890
- @kozlov-ma made their first contribution in #1892
- @M1ha-Shvn made their first contribution in #1842
- @sundbry made their first contribution in #1691
- @diPhantxm made their first contribution in #1902
- @dbarrosop made their first contribution in #1906
Full Changelog: v3.0.5...v3.0.7
v3.0.6
Prefetched wal is expected in default directory (#1954) * get prefetch location from config * added some logging * added test
v3.0.5
First stable release in 2025. Major features of this stable release are:
- Cloudberry database support
- MongoDB 8.0 support + restore for sharded cluster with PiTR
- PostgreSQL - fix
wal-g copy
Additionally this release contains a bunch of bugfixes and documentation updates. For MySQL we fixed regression (all versions since v3.0.0 affected) that leads to broken backups to be marked as good ones. We strongly recommend MySQL users to upgrade.
What's Changed
- Add option to disable 100 continue in s3 by @fepape-ionos in #1829
- pg_checksum_verification_fix by @boosterKRD in #1828
- fix: don't ignore libsodiumKeyPath by @fepape-ionos in #1832
- CI: drop ubuntu-18.04 from release pipeline by @chipitsine in #1825
- fix data race on accessing bool var from different goroutines by @kronwerk in #1835
- MongoDB: Support restore of sharded cluster with PiTR by @Apelsin234 in #1797
- MongoDB: don't skip error if file was deleted while make backup by @Apelsin234 in #1838
- CI: cleanup build process a bit by @chipitsine in #1834
- add ubuntu-22, ubuntu-24 to release matrix by @chipitsine in #1841
- fix IsDirectoryEmpty error by @NotAmigo in #1836
- feat(file): add sync mutex to TarFileSet by @SoulKyu in #1843
- bump golang version in docker tests by @ostinru in #1840
- update brotli submodule by @chipitsine in #1791
- fix: correct LSN format string in delta backup logging by @robertmu in #1848
- Unit-tests to Upload with mocks by @Amyton0 in #1837
- Changed error handling while collecting tables data by @debebantur in #1849
- MySQL: Fix regression - broken backup considered as good one by @ostinru in #1852
- postgres: refactor backup to skip non-physical storage objects (#1850) by @robertmu in #1851
- release: unify amd64 <--> arm64 names by @chipitsine in #1854
- Fix wal-g copy for pg by @munakoiso in #1856
- MySQL: don't add
--apply-log-onlyto the latest stage of backup preparation. by @ostinru in #1857 - MongoDB: Add 8.0 support + bump other versions minors by @VGoshev in #1845
- Fix a grammar error in the PR template by @dligr in #1865
- wal-g copy for greenplum by @munakoiso in #1862
- Add logging to file by @EinKrebs in #1866
- fix(postgres): resolve race condition in pg_wal_restore_test by @robertmu in #1868
- fix readthedocs build definition by @chipitsine in #1847
- bump libsodium to 1.0.20 by @chipitsine in #1870
- do not run scheduled tests in forks by @chipitsine in #1858
- make mongo docker compose tests more stable by @chipitsine in #1789
- fix(PostgreSQL): exclude transaction state files from page file detec… by @robertmu in #1864
- docs: update typo in MySQL.md by @mr-ssd in #1877
- tests: remove few "sleep" by @chipitsine in #1878
- tests: use $(nproc) instead of hardcoded "8" by @chipitsine in #1881
- Fix SIGSEGV errors of wal-g delete by @ostinru in #1879
- WALG-1873. MySQL > use datadir as incremental dir in tests by @ostinru in #1880
- Greenplum multistorage support by @munakoiso in #1875
- Fixed backward compatibility with files metadata by @debebantur in #1886
- Fix Postgres 12+/Cloudberry concurrent backups. by @ostinru in #1813
New Contributors
- @fepape-ionos made their first contribution in #1829
- @SoulKyu made their first contribution in #1843
- @robertmu made their first contribution in #1848
- @Amyton0 made their first contribution in #1837
- @dligr made their first contribution in #1865
- @mr-ssd made their first contribution in #1877
Full Changelog: v3.0.4...v3.0.5
v3.0.4
This is a pre-release to test new AWS SDK version #1799
What's Changed
- sqlserver: special url for proxy health-checks by @mialinx in #1709
- try to switch to docker compose v2 by @chipitsine in #1765
- add universal backup tool by @danilkaz in #1689
- aof datadir clean fix by @kronwerk in #1767
- MySQL: Add xbstream unarchiver to wal-g by @ostinru in #1744
- Update README.md by @reshke in #1772
- Fixed working without metadata by @debebantur in #1773
- clean only specified files and folder from data dir by @kronwerk in #1776
- fix checkout on Windows by @chipitsine in #1777
- delete single redis backup by @Believemenot in #1786
- Test postgres/generic_meta_interactor.go/SetIsPermanent by @pevel-dev in #1788
- add redis used_memory and used_memory_rss to sentinel on backup creation by @kronwerk in #1790
- greenplum: separate restore points fetch & find operations - so we can reuse fetched metadata. by @ostinru in #1793
- Correction for checksum verification PostgreSQL by @boosterKRD in #1779
- Bumped aws sdk version by @ItielOlenick in #1799
- sort detailed backup list too by @kronwerk in #1803
- Bump golangci/golangci-lint-action from 6.0.1 to 6.1.1 by @dependabot in #1801
- fix: fix slice init length by @cuishuang in #1800
- Cloudberry DB initial support by @ostinru in #1796
- Add makezero and asciicheck linters by @ostinru in #1812
- Continue Greenplum garbage delete when no segment data found by @ostinru in #1814
- Remove unexpected '\r\n' sequence at the end of multiline commands in the config by @noname0443 in #1817
- Greenplum 7 support by @ostinru in #1818
- clean data on rdb restore also by @kronwerk in #1819
- Add JournalSize to backup sentinels by @noname0443 in #1806
- Fixes for partial restore by @debebantur in #1816
- Update go version to resolve CVE-2024-34156 by @annielzy in #1820
- add whitelist for mongodb datadir deletion protection by @NotAmigo in #1808
New Contributors
- @chipitsine made their first contribution in #1765
- @Believemenot made their first contribution in #1786
- @pevel-dev made their first contribution in #1788
- @boosterKRD made their first contribution in #1779
- @ItielOlenick made their first contribution in #1799
- @cuishuang made their first contribution in #1800
- @annielzy made their first contribution in #1820
Full Changelog: v3.0.3...v3.0.4
v3.0.3
Major feature of this release is full support for OrioleDB.
WAL-G supported block-level incremental backups since v0.1.3, but it previously treated OrioleDB data as a collection of unknown files. Now WAL-G understands if OrioleDB is installed into cluster and makes efficient backup copies of OrioleDB data. Thanks to @homper and other Supabase engineers.
Additionally, this release includes two new Postgres commands catchup-send and catchup-receive. These commands are useful when you need to bring a lagging replica up to date without pushing a new backup to the storage. In essence, they work like pg_rewind but in reverse. Perhaps we should rename them to pg_wind. Thanks to @vbp1 for drawing my attention to the problem of current catchup implementation and architecting new design.
We've also added aof-based backups for redis (@kronwerk).
This release also mitigates several CVEs in dependencies (#1675, #1715) and fixes assorted bugs. Thanks to all the WAL-G community :)
What's Changed
- Allow users to expose debug & pprof endpoints for wal-g-gp by @ostinru in #1748
- Delete greenplum backups with higher concurrency by @ostinru in #1750
- Upgrade golang.org/x/net to resolve CVE-2023-45288 by @jorsol in #1737
- redis aof backup and fetch, basic integration tests by @kronwerk in #1749
- Added missing test to parallel by @debebantur in #1730
- Add extendBackupCursor MongoDB flag to make extension optional by @NotAmigo in #1752
- Update CODEOWNERS by @VGoshev in #1753
- up golang version to 1.21 by @kronwerk in #1755
- [orioledb] Using latest orioledb in tests and new pageHeader field order by @homper in #1758
- Reduce log clutter from storages by @x4m in #1760
- behave tests for redis aof backup by @kronwerk in #1759
- clean all redis data before aof backup, not only aof folder by @kronwerk in #1763
- fix LoadCustomCABundleError when setting WALG_S3_CA_CERT_FILE by @berler in #1764
New Contributors
Full Changelog: v3.0.2...v3.0.3
v3.0.2
This pre-release is intended to preview OrioleDB support.
Also it fixes some bugs, notably #1736: reading calls to S3 could be really slow due to unintended bucket listings. The bug seems to be introduced in 3.0.1 pre-release.
What's Changed
- Fixes for working with s3 by @debebantur in #1718
- add test cases for tryFindHighestTimelineID by @annetannet in #1720
- Migrate to docker compose v2 by @debebantur in #1727
- сhange redis installation in tests from pkg to repo by @kronwerk in #1732
- Create MockObject for storage.Object and add backup tests with gomock by @alexnik10 in #1731
- wal-g hangs when s3 access_id or access_secret_key contains line break by @mialinx in #1733
- Add tests for FindTargetByName by @RedSquid2212 in #1729
- Refactor mongodb and redis func tests docker images by @zykovd in #1735
- Fix link_libsodium.sh by @garry-t in #1739
- Fix "backup-push Connection cancelled by remote host" error on Windows by @rail-rain in #1745
- Small error cleanup in catchkup-receive by @x4m in #1701
- Orioledb basic support by @homper in #1698
- Disable storage validation by default by @x4m in #1747
- [mongodb] add opportunity set timeout for recovery oplog by @Apelsin234 in #1746
New Contributors
- @annetannet made their first contribution in #1720
- @kronwerk made their first contribution in #1732
- @alexnik10 made their first contribution in #1731
- @RedSquid2212 made their first contribution in #1729
- @zykovd made their first contribution in #1735
- @garry-t made their first contribution in #1739
- @rail-rain made their first contribution in #1745
Full Changelog: v3.0.1...v3.0.2
v3.0.1
This is a technical pre-release aimed at mitigating several CVEs in dependencies (#1675, #1715). These vulenrabilities have no known ways to be exploited against WAL-G. But these days we should keep CVE scanners calm.
Also this pre-release gives access to several features that didn't make it into 3.0.0 because were not tested enough (#1662, #1646, #1652).
As usual, handful of bugs fixed.
Thanks to everyone who contributed!
What's Changed
- Support DigitalOcean Spaces (NoSuchKey during ListFolder) by @turt2live in #1646
- Skip archive_command check on standby by @x4m in #1662
- fixes for ao check config by @debebantur in #1664
- Fix for GP incremental backups by @debebantur in #1666
- Add unit tests for internal.databases.greenplum.buildFetchCommand by @denisSurkov in #1661
- Catchup-send and catchup-receive commands by @x4m in #1652
- Collecting statistics about read/written bytes by @debebantur in #1670
- Add Docs for undocumented debug env variables by @sebastiangrill in #1640
- sanikadze:doc_gp_gp_pitr by @Sanikadze in #1672
- Implement lockFile for windows by @passcod in #1673
- upgrade golang.org/x/net to resolve CVE-2023-39325 by @devinturner in #1675
- Fix WALG_STATSD_EXTRA_TAGS by @nick-kentik in #1676
- Fix gp tests by @debebantur in #1677
- add path to recovery.conf for greenplum by @avkashin in #1679
- Clarified dev installing instruction on Ubuntu for go 1.16+ by @homper in #1681
- added IsTablespaceMapExists test by @NiripsaKakVsegda in #1680
- split mongodb restore on stages under flag by @Apelsin234 in #1685
- add support to redis acl user by @NotAmigo in #1682
- Add glob pattern support in storagetools by @ReshetnikovPavel in #1684
- add PGSSLCERT, PGSSLKEY, PGSSLROOTCERT support by @NiripsaKakVsegda in #1687
- WALG-1627. MySQL fix file double Close() on file by @ostinru in #1628
- add create upload input unit tests by @danilkaz in #1690
- add tests for SetUserData in postgres/generic_meta_interactor by @SixBlueBlack in #1697
- Fix lll lint violation, update golangci version by @serprex in #1686
- Exclude standby.signal by @munakoiso in #1702
- WALG-1665. User proper DSNs in MySQL documentation & validate DSN format it earlier by @ostinru in #1692
- unit test for internal/backup.go/FetchMetadata by @ValeriaIM in #1695
- Bump golangci/golangci-lint-action from 4.0.0 to 5.3.0 by @dependabot in #1703
- Save isIncremented attribute by @debebantur in #1704
- Add charset converter to convert non
utf-8char by @anisurrahman75 in #1705 - Bump golangci/golangci-lint-action from 5.3.0 to 6.0.1 by @dependabot in #1706
- WALG-1683. Save CPU arch / OS to mysql sentinel. by @ostinru in #1693
- added without metadata to gp by @debebantur in #1708
- Tests for
isPagedFileby @Themplarer in #1710 - Added validation for S3 credentials by @Themplarer in #1711
- fixed hex parsing in timeline by @debebantur in #1712
- Added retention period for s3 by @debebantur in #1713
- Add few unit test on azure ConfigureStorage by @Loraan in #1714
- Add Uploader interface mock, test on UploadSentinel and make mock command by @Loraan in #1716
- Update modules in order to resolve CVEs. by @montaguethomas in #1715
New Contributors
- @turt2live made their first contribution in #1646
- @sebastiangrill made their first contribution in #1640
- @Sanikadze made their first contribution in #1672
- @passcod made their first contribution in #1673
- @devinturner made their first contribution in #1675
- @avkashin made their first contribution in #1679
- @homper made their first contribution in #1681
- @NiripsaKakVsegda made their first contribution in #1680
- @NotAmigo made their first contribution in #1682
- @danilkaz made their first contribution in #1690
- @SixBlueBlack made their first contribution in #1697
- @ValeriaIM made their first contribution in #1695
- @anisurrahman75 made their first contribution in #1705
- @Themplarer made their first contribution in #1710
- @Loraan made their first contribution in #1714
- @montaguethomas made their first contribution in #1715
Full Changelog: v3.0.0...v3.0.1
v3.0.0
Hello everyone!
This is a much-anticipated release. In the future, we will strive to avoid accumulating such a large number of changes. We will code frequently, ship frequently, and release frequently!
Noteworthy Changes
- Every database engine now benefits from failover storage. This technique can save the day if one of the storage systems goes down. However, do not forget to monitor them anyway.
- For Postgres, we have a WAL archiving daemon available. Its primary purpose is to improve the efficiency of parallel archiving. Additionally, WAL-G supports the partial restoration of a cluster. If you have large, unnecessary tables, you can exclude them from the restoration process, the Postgres will skip over the WAL records for these tables.
- Greenplum's PITR (Point-in-Time Recovery) is undergoing its maturation process. Partial restore is supported in a similar way to Postgres. We now have tools to verify the integrity of AO\AOCS files in storage without actually restoring backups. This technique is necessary to ensure developers can sleep well, as Greenplum backups are relatively new, and extra precautions need to be taken to ensure they are safe and reliable.
- MongoDB backups can now benefit from binary backups, which significantly reduce the impact of backups on a running workload.
- MySQL now supports incremental backups, also known as delta backups. However, this feature is still in beta mode and the API may change in the future.
- Also, we now have beta support for etcd. Changes necessary for real PITR have not yet been merged into etcd, and can be found at etcd-io/etcd#17233.
Detailed changes log
- Correct the --without-files-metadata docs by @usernamedt in #1327
- expose zstd compression by @heinerstilz in #1326
- [WIP] Multithread downloading directory by @hariton27sy in #1297
- Add WALG_GP_AOSEG_SIZE_THRESHOLD setting by @usernamedt in #1329
- Respect WALG_NETWORK_RATE_LIMIT globally by @usernamedt in #1330
- Restrict rate limiting only to backup-push/backup-fetch commands by @usernamedt in #1331
- Initial mvp mongodb binary backup by @KhurtinDN in #1285
- Add delta backups for Greenplum by @usernamedt in #1333
- Fix sorting in detailed mongodb backup list by @KhurtinDN in #1334
- Add @VGoshev and @KhurtinDN as codeowners of MongoDB part of Wal-G by @VGoshev in #1335
- Add tests, docs and minor fixes for Greenplum delta backups by @usernamedt in #1337
- Fix mongodb binary backup during mongod do checkpoint by @KhurtinDN in #1342
- Add rate limiter for mongodb by @KhurtinDN in #1343
- Add data catalog size for PostgreSQL/Greenplum backups by @usernamedt in #1344
- Add minor improvements for Greenplum by @usernamedt in #1345
- postgresq.md add cmake installation by @rogaliiik in #1348
- Support ARM64 builds on MacOS by @usernamedt in #1349
- Implement delete target backup for Greenplum by @usernamedt in #1350
- Add delete garbage command for Greenplum by @usernamedt in #1351
- Add limiters to Greenplum segment backup-push/backup-fetch by @usernamedt in #1352
- Fix failing docs builds by @usernamedt in #1354
- Fix distro name for ubuntu-20.04 in arm release by @williancolognesitrimble in #1355
- Bump golangci/golangci-lint-action from 3.2.0 to 3.3.0 by @dependabot in #1361
- Add minimal mongod config path option by @KhurtinDN in #1360
- Fix closing mongodb backup cursor by @KhurtinDN in #1359
- Replace mongodb binary backup to tar archive format by @KhurtinDN in #1357
- Fix binary backup of mongodb 4.2 by @KhurtinDN in #1364
- Add daemon mode for wal-push command by @rogaliiik in #1353
- Change error to warning when creating symlink, that already exists by @IncubusRK in #1320
- [Greenplum] do not fail backup on increment reading error by @usernamedt in #1365
- Only dump relevant env variables with LOG_LEVEL=DEVEL by @LeGEC in #1367
- Do not print secrets in debug logs by @LeGEC in #1368
- Bump golangci/golangci-lint-action from 3.3.0 to 3.3.1 by @dependabot in #1380
- enable using aws iam roles by @Qwiz in #1377
- notify_d watchdog for daemon service by @rogaliiik in #1371
- Use pure go implementation for zstd by @LeGEC in #1376
- Restore debug logging improvements by @usernamedt in #1389
- postgresql: fix data path mismatch on local backups by @LeGEC in #1383
- Fix Greenplum deleted AO segment file error handling by @usernamedt in #1391
- Implement max file size for mysql by @bularond in #1374
- Fix non-string config settings handling by @usernamedt in #1393
- Makefile: change compilation and linking of libbrotli to optional (and off by default) by @LeGEC in #1379
- add unit tests for the WriteBackupListDetails function by @tewboat in #1395
- Add recovery_target_timeline setting to Greenplum restore config by @usernamedt in #1398
- Add detailed json backup-list for Greenplum by @usernamedt in #1399
- Add restore-point-list command for gp by @rogaliiik in #1397
- Update golangci-lint version to 1.50.1 by @serprex in #1381
- Add restore-point-ts flag for Greenplum backup-fetch by @usernamedt in #1403
- Greenplum: populate pg_hba.conf on master host during restore by @usernamedt in #1404
- enable linters for cmd/redis internal/databases/redis and pkg/storages by @teewar2 in #1407
- Fix wal-g-mongo mongod restore procedure by @KhurtinDN in #1414
- Add mongo-rs-members parameter to binary mongo restore by @KhurtinDN in #1418
- [MongoDB] add opportunity restore mongodb backup with + 1 version by @Apelsin234 in #1417
- Add mongo-rs-name and mongo-rs-members parameters to binary mongo restore by @KhurtinDN in #1419
- add unit test for greenplum/backup_fetch_handler by @teewar2 in #1406
- fix bug with mixed up arguments by @Apelsin234 in #1421
- Postgres requires data dir to be 0700 or 0750. by @Aevin1387 in #1420
- Add support mongodb 6.0 by @KhurtinDN in #1416
- Fix connect to mongodb during restore procedure by @KhurtinDN in #1428
- Bump golangci/golangci-lint-action from 3.3.1 to 3.4.0 by @dependabot in #1426
- MDB-21194: [MongoDB] fix bug with purge garbage and change logic for retains by @Apelsin234 in #1429
- Implementation of the new binlog-server command by @Fizic in #1408
- Bump github.com/prometheus/client_golang from 1.7.1 to 1.11.1 in /internal/tools by @dependabot in #1433
- Bump golang.org/x/text from 0.3.6 to 0.3.8 in /internal/tools by @dependabot in #1437
- Implement retry download backup by @bularond in #1394
- adding push metrics as a defer in the upload method to make sure metrics are pushed even on failure by @francoispqt in #1423
- Revert #1423 by @usernamedt in #1442
- Implement database composer and partial fetch for Postgres by @Catmoonlight in #1434
- [MongoDB] fix retryable error from code to name by @Apelsin234 in #1444
- Bump actions/setup-go from 3 to 4 by @dependabot in #1446
- Adaptive timeout for mongodb restore by @Apelsin234 in #1449
- [Greenplum] Store metadata about backup restore points by @usernamedt in #1448
- Add Postgres partial restore by database names by @Catmoonlight in #1447
- Add tests postgres sentinel dto is incremental by @AlexUran in #1455
- Add tests for NewDtoSerializer by @denisSurkov in #1460
- feat: removing binlogs after using them by @Fizic in #1457
- Update golangci-lint in .github/workflows to 1.52.2 by @serprex in ...
2.0.2-rc1
This is a pre-release to test the aws iam roles config setting.