Evaluation guide for the dedupable storage backend #2062

pstorz · 2024-12-13T13:32:18Z

pstorz
Dec 13, 2024
Maintainer

Evaluation Guide for the Bareos Dedupable Backend

Prerequisites

This guide assumes that you know how to set up Bareos, know how to configure and use multiple
storage devices and have successfully backed up to and restored from different storages.

This example works on the default configuration that is automatically installed when setting up Bareos.

To take advantage of the deduplicability, you will need a deduplicating storage like ZFS, VDO or other.
This guide will use ZFS.

Create a deduping ZFS mountpoint and allow the user "bareos" to write to it

zfs create tank/deduping_zfs
zfs set dedup=on tank/deduping_zfs
chown bareos:bareos /tank/deduping_zfs

Installing the dedupable backend

Install the package bareos-storage-dedupable

Configure the dedupable backend

Configure the Device in the Storage Daemon

# copy the example device configuation
$ cp -a /etc/bareos/bareos-sd.d/device/Dedupable.conf.example  /etc/bareos/bareos-sd.d/device/Dedupable.conf

Adapt the configuration to your environment and alter the settings for

Device Options and
Archive Device

In case of ZFS, the default block size is 128k.
Change the Archive Device to match your mountpoint of the deduplicating storage.

Device {
  Name = DedupableDevice
  Media Type = dedupable

  #
  # Device Options:
  #    block size=  - Dedup block size of the filesystem/actual device
  #

  Device Options = "block size=128k"

  Archive Device = /tank/deduping_zfs/

  Device Type = dedupable
  Label Media = yes                    # lets Bareos label unlabeled media
  Random Access = yes
  Automatic Mount = yes                # when device opened, read it
  Removable Media = no
  Always Open = no
  Description = "dedupable device"
  Maximum Concurrent Jobs = 1
}

Caution

It is very important that the block size of the dedup device matches the block size of the underlying storage to achieve optimal deduplication rates.

Configure the Device in the Storage Daemon

# copy the example storage definition
$ cp -a /etc/bareos/bareos-dir.d/storage/Dedupable.conf.example  /etc/bareos/bareos-dir.d/storage/Dedupable.conf

Edit the configuration to your environment. Set Address and Password as in File.conf

Storage {
  Name = Dedupable
  Address =   <get address from "File.conf">
  Password =  <get password from "File.conf">
  Device = DedupableDevice
  Media Type = dedupable
}

Restart director and storage daemons

systemctl restart bareos-sd bareos-dir

Verify that the dedupable device is available

*status storage=Dedupable 
Connecting to Storage daemon Dedupable at r9-dedup:9103
 Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3

bareos-sd Version: 24.0.0 (11 December 2024) Red Hat Enterprise Linux release 9.5 (Plow) 
Daemon started 12-Dec-24 11:49. Jobs: run=0, running=0, Bareos pre-release binary
 Sizes: boffset_t=8 size_t=8 int32_t=4 int64_t=8 bwlimit=0kB/s

There are WARNINGS for this storagedaemon's configuration!
See output of 'bareos-sd -t' for details.

Job inventory:

No Jobs running.
====

Jobs waiting to reserve a drive:
====

Terminated Jobs:
====

Device status:

Device "DedupableDevice" (/tank/deduping_zfs/) is not open.
==
====

Used Volume status:
====

====

Make a first full test backup

*run job=backup-bareos-fd storage=Dedupable level=Full

messages
.....

  FD termination status:  OK
  SD termination status:  OK
  Bareos binary info:     Bareos pre-release (UNSUPPORTED): Get professional support from https://www.bareos.com
  Job triggered by:       User
  Termination:            Backup OK

Check the contents of the dedupable storage

find /tank/deduping_zfs/
/tank/deduping_zfs/
/tank/deduping_zfs/Full-0001
/tank/deduping_zfs/Full-0001/config
/tank/deduping_zfs/Full-0001/blocks
/tank/deduping_zfs/Full-0001/parts
/tank/deduping_zfs/Full-0001/aligned.data
/tank/deduping_zfs/Full-0001/unaligned.data

Check the dedup status in ZFS

$ zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tank  99.5G  47.6M  99.5G        -         -     0%     0%  1.00x    ONLINE  -

Make a second full test backup

*run job=backup-bareos-fd storage=Dedupable level=Full

messages
.....

  FD termination status:  OK
  SD termination status:  OK
  Bareos binary info:     Bareos pre-release (UNSUPPORTED): Get professional support from https://www.bareos.com
  Job triggered by:       User
  Termination:            Backup OK

Check the dedup status in ZFS

$ zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tank  99.5G  72.3M  99.4G        -         -     0%     0%  1.61x    ONLINE  -

Results of the test backups

The dedup factor is 1.61x for the fileset being backed up twice.
The second backup only occupies 24.7M instead of 47.6M, which is roughly 52% of the original size.
Backups in the real-world probably have a higher deduplication potential than the small files of the SelfTest fileset.

Estimate the deduplication potential of your existing backups.

bdedupestimate is a tool which reads in your existing volumes and estimates their deduplication potential for different blocksizes.

To show how bdedupestimate works, we backup the test fileset twice to the conventional FileStorage device and then analyze it:

# first run
* run job=backup-bareos-fd storage=File level=Full
# second run
* run job=backup-bareos-fd storage=File level=Full

Run bdedupestimate

bdedupestimate --volumes Full-0002 --devicename FileStorage  --blocksize=128k
Using block size 128 k
bdedupestimate: stored/butil.cc:305-0 Using device: "FileStorage" for reading.
Records consumed: 3204
  Total record size: 98.25 MB
  Size after dedup: 60.63 MB
  Possible reduction: 37.61 MB
  Dedup factor: 1.62x

As you can see, the estimated Dedup Factor matches the ZFS reality quite well.

Smaller block sizes usually result in better deduplication ratio. VDO for example can work with 4K blocks:

$ bdedupestimate --volumes Full-0002 --devicename FileStorage  --blocksize=4k
Using block size 4 k
bdedupestimate: stored/butil.cc:305-0 Using device: "FileStorage" for reading.
Records consumed: 3204
  Total record size: 98.25 MB
  Size after dedup: 48.03 MB
  Possible reduction: 50.22 MB
  Dedup factor: 2.05x

Summary

The bareos dedupable storage backend is easy to configure and allows to write backup data in a dedupable-friendly format.
To achieve optimal results, the blocksize can be configured to match the block size of the deduplicating filesystem.
Using the bededupestimate tool, existing backups can be analyzed to estimate the achievable deduplication rate even before using the dedupable backend.

MrMEEE · 2025-01-28T13:55:55Z

MrMEEE
Jan 28, 2025

@pstorz I have seen some references saying that the "Maximum Network Buffer Size" option is important.. Is this true?

4 replies

arogge Feb 4, 2025
Maintainer

as you can read here, the maximum network buffer size should be divisible by your storage's deduplication block size. i.e. 128k dedup blocksize on ZFS divides the default network buffer size 256k, so it is fine.
What other references did you find?

troloff Jun 4, 2025

On a linux fd, I had to set it explicitly to have aligned.data and thus deduplication work. Maybe, the documentation is a little bit unclear about this:

If not specified the operating system will choose an acceptable size.

vs.

The default value is 256 KiB.

Without setting it:

# ls -lh /storage/bareos/File-RKP-Dedup-0010
insgesamt 100G
-rw-r----- 1 bareos bareos 1,0K  2. Jun 15:12 aligned.data
-rw-r----- 1 bareos bareos 5,5M  3. Jun 15:48 blocks
-rw-r----- 1 bareos bareos  189  3. Jun 15:48 config
-rw-r----- 1 bareos bareos  80M  3. Jun 15:48 parts
-rw-r----- 1 bareos bareos 100G  3. Jun 15:48 unaligned.data

With Maximum Network Buffer Size = 262144 in fd -> /etc/bareos/bareos-fd.d/client/myself.conf

# ls -lh /storage/bareos/File-RKP-Dedup-0014
insgesamt 100G
-rw-r----- 1 bareos bareos 100G  4. Jun 10:50 aligned.data
-rw-r----- 1 bareos bareos 5,5M  4. Jun 10:50 blocks
-rw-r----- 1 bareos bareos  189  4. Jun 10:50 config
-rw-r----- 1 bareos bareos  16M  4. Jun 10:50 parts
-rw-r----- 1 bareos bareos  11M  4. Jun 10:50 unaligned.data

arogge Jun 4, 2025
Maintainer

That's weird. As the documentation states the default is 256k. This has been changed in Bareos 23, so if you are still running a Bareos 22 or earlier FD you have to set this, but the latest FD should work out of the box.

troloff Jun 4, 2025

You are absolutely right, I picked a test fd with an outdated bareos version... after update, I removed maximum network buffer size setting and it still works as expected.

troloff · 2025-06-03T06:11:33Z

troloff
Jun 3, 2025

One more question: Are there any benefits from having compression activated in filesets, if compression is on on zfs volumes? Or maybe drawbacks?

2 replies

arogge Jun 3, 2025
Maintainer

Compression (and of course also encryption) on the SD will make deduplication impossible. The problem with compression is that the data records the SD will see are not aligned anymore, because their size was changed by compression.
As compression in ZFS happens after deduplication, the proper setup is to have compression disabled in Bareos and enabled in ZFS. If you have to have compression in the FD for some reason you can look into autoxflate to decompress the data before storing it.

troloff Jun 4, 2025

Thanks for the detailed explanation. And with dir->storage option like

Storage {
[...]
	Allow Compression = no
}

it even works without modifying each fileset.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Evaluation guide for the dedupable storage backend #2062

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments 6 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Evaluation guide for the dedupable storage backend #2062

Uh oh!

Uh oh!

pstorz Dec 13, 2024 Maintainer

Evaluation Guide for the Bareos Dedupable Backend

Prerequisites

Create a deduping ZFS mountpoint and allow the user "bareos" to write to it

Installing the dedupable backend

Configure the dedupable backend

Configure the Device in the Storage Daemon

Configure the Device in the Storage Daemon

Restart director and storage daemons

Verify that the dedupable device is available

Make a first full test backup

Check the contents of the dedupable storage

Check the dedup status in ZFS

Make a second full test backup

Check the dedup status in ZFS

Results of the test backups

Estimate the deduplication potential of your existing backups.

Run bdedupestimate

Summary

Replies: 2 comments · 6 replies

Uh oh!

MrMEEE Jan 28, 2025

Uh oh!

arogge Feb 4, 2025 Maintainer

Uh oh!

troloff Jun 4, 2025

Uh oh!

arogge Jun 4, 2025 Maintainer

Uh oh!

troloff Jun 4, 2025

Uh oh!

troloff Jun 3, 2025

Uh oh!

arogge Jun 3, 2025 Maintainer

Uh oh!

troloff Jun 4, 2025

pstorz
Dec 13, 2024
Maintainer

Replies: 2 comments 6 replies

MrMEEE
Jan 28, 2025

arogge Feb 4, 2025
Maintainer

arogge Jun 4, 2025
Maintainer

troloff
Jun 3, 2025

arogge Jun 3, 2025
Maintainer