Screenshots & sketches RSS feed
This is the first test of program's brand new ability to read and
use NTFS change journals when delta-copying files.
As you may know, delta copying speeds up file updates by not writing out file parts that didn't change since the last update.
By using file system change journals the program can now also be selective with what it reads. Less reading - faster updating.
The screenshot above shows an update of a 2 GB file with 4 modified blocks. These are detected using the journal and only these are read and processed, speeding up the update by a mere factor of 10.
Coming soon to a R83 preview release near you.
Happy Holidays, everyone !
As you may know, delta copying speeds up file updates by not writing out file parts that didn't change since the last update.
By using file system change journals the program can now also be selective with what it reads. Less reading - faster updating.
The screenshot above shows an update of a 2 GB file with 4 modified blocks. These are detected using the journal and only these are read and processed, speeding up the update by a mere factor of 10.
Coming soon to a R83 preview release near you.
Happy Holidays, everyone !
The next release will change the way the program
displays its change log when a newer version is
found by the in-app update check.
The
original approach
dates back to 2014 and it works by merging all changes
between the new version and the one
currently installed into a single list,
grouping them by type - features, bug fixes, etc. -
and displaying that :
In other words, this creates a categorized summary
of differences between what you have and what's available.
This format also allows trimming off irrelevant changes. That's when one of the in-between releases fixed a problem caused by an earlier release, with both being newer than the installed one.
This was the theory.
In reality this approach wasn't necessarily any clearer than the conventional chronological format, and it also complicated the management of the change by quite a bit.
The practice has once again diverged from the theory. Long story short, Release 82 will swtich to showing complete log of all changes from all preceding releases, but it will also dim notes for the currently installed version and earlier releases.
The second change is that release notes now support clickable links. This will come handy when a change has more context than what could fit into the change log.
In other words, this creates a categorized summary
of differences between what you have and what's available.
This format also allows trimming off irrelevant changes. That's when one of the in-between releases fixed a problem caused by an earlier release, with both being newer than the installed one.
This was the theory.
In reality this approach wasn't necessarily any clearer than the conventional chronological format, and it also complicated the management of the change by quite a bit.
The practice has once again diverged from the theory. Long story short, Release 82 will swtich to showing complete log of all changes from all preceding releases, but it will also dim notes for the currently installed version and earlier releases.
The second change is that release notes now support clickable links. This will come handy when a change has more context than what could fit into the change log.
Support for copying speed throttling is in.
Starting with the Release 82 it will be possible to limit reading and writing speeds of backup jobs, either individually or as a group. This feature is also included in R82 preview releases starting with the build 44.
This is supported because with delta copying the amount of data written during a file copy may be significantly less than what's read, so imposing a single "copy speed" limit would've not been very flexible.
The available bandwidth is split between active jobs based on their priorities, which are specified when the cap is configured.
This is helpful for managing the IO load on a storage device used by several jobs that run in parallel with each other.
This assumes full-load conditions, i.e. with all jobs in the group
being active and hungry for bandwidth.
In particular, while processing a backup, the program would typically have several files copied in parallel with each other. Meaning that in addition to distributing bandwidth across multiple jobs the IO shaper also needs to further distribute one job's allowance between all its active copy operations. This is one change from the stock FWQ.
Jobs may also be paused and resumed at will and, again, this is not something supported by FWQ out of the box.
All in all, this is a fairly interesting feature in that it's well-rooted in fundamental comp.sci. algorithmic base. The problem may seem difficult at first, but, turns out, it has a very elegant and efficient solution. All in all, this is an interesting feature because it comes from the fundamental comp. sci. algorithmic base. The problem may seem difficult at first, but it ends up having a very elegant solution.
Starting with the Release 82 it will be possible to limit reading and writing speeds of backup jobs, either individually or as a group. This feature is also included in R82 preview releases starting with the build 44.
Independent read and write caps
Reading and writing speeds can be capped independently.This is supported because with delta copying the amount of data written during a file copy may be significantly less than what's read, so imposing a single "copy speed" limit would've not been very flexible.
Shared caps
The program supports imposing a single shared speed limit on multiple jobs at once, treating them as a group of consumers competing for the bandwidth.The available bandwidth is split between active jobs based on their priorities, which are specified when the cap is configured.
This is helpful for managing the IO load on a storage device used by several jobs that run in parallel with each other.
Allocation breakdown
To help with visualizing the bandwidth split within the group, the program can display exact per-job limits, recacluated dynamically in response to the input.
This assumes full-load conditions, i.e. with all jobs in the group
being active and hungry for bandwidth.
The guts
Internally, the IO shaper is based on fair weighted queueing, but adapted to the specifics of Bvckup 2.In particular, while processing a backup, the program would typically have several files copied in parallel with each other. Meaning that in addition to distributing bandwidth across multiple jobs the IO shaper also needs to further distribute one job's allowance between all its active copy operations. This is one change from the stock FWQ.
Jobs may also be paused and resumed at will and, again, this is not something supported by FWQ out of the box.
All in all, this is a fairly interesting feature in that it's well-rooted in fundamental comp.sci. algorithmic base. The problem may seem difficult at first, but, turns out, it has a very elegant and efficient solution. All in all, this is an interesting feature because it comes from the fundamental comp. sci. algorithmic base. The problem may seem difficult at first, but it ends up having a very elegant solution.
Work in progress on adding UI support for configuring
advanced job settings that are currently accessible
only through manual
overrides.
Not every setting will be covered as there is well over a hundred of them in each job, but all crowd favourites will be on the list here.
Coming shortly to a preview build near you.
Not every setting will be covered as there is well over a hundred of them in each job, but all crowd favourites will be on the list here.
Coming shortly to a preview build near you.
Preview build 27 of R82 improves detection and reporting of
files and folders with really badly malformed names.
In theory,
Microsoft specs
state that file and folder names cannot include certain
reserved symbols (such as ?) and "non-printable" ASCII
characters in 0 to 31 range.
In practice, it is very much possible to run into names that include these forbidden fruits. This happens, for example, when working with non-Windows file storage that generally imposes fewer restrictions on file names. Previously, Bvckup 2 checked only for zero-length names. This is something that happens in the wild and appears to be caused by rootkit-ish software trying to hide files from the public view.
The new preview build adds two more checks.
1. The program now verifies that names don't contain a NUL character. Windows API works with zero-terminated strings, so having a NUL in a name will break things in peculiar ways and lead to bizarre errors.
2. Bvckup 2 now checks names for reserved symbols. Namely:
< >
" : | \
/ ? *
If these are ever present in the name, they will make respective file or folder all but inaccessible through Windows API, so there's no point in even trying to work with them.
These two cases are now caught early, reported and offending items are ignored. The non-printable characters - 1 through 31 ASCII - are allowed and not treated differently, because they create no operational difficulties. At least not at the moment. Finally, it's worth mentioning that it is perfectly possible to create files and folders with forbidden names directly on Windows itself. This requires using Native API to work around various parameter restrictions implemented at the Windows API level.
In practice, it is very much possible to run into names that include these forbidden fruits. This happens, for example, when working with non-Windows file storage that generally imposes fewer restrictions on file names. Previously, Bvckup 2 checked only for zero-length names. This is something that happens in the wild and appears to be caused by rootkit-ish software trying to hide files from the public view.
The new preview build adds two more checks.
1. The program now verifies that names don't contain a NUL character. Windows API works with zero-terminated strings, so having a NUL in a name will break things in peculiar ways and lead to bizarre errors.
2. Bvckup 2 now checks names for reserved symbols. Namely:
If these are ever present in the name, they will make respective file or folder all but inaccessible through Windows API, so there's no point in even trying to work with them.
These two cases are now caught early, reported and offending items are ignored. The non-printable characters - 1 through 31 ASCII - are allowed and not treated differently, because they create no operational difficulties. At least not at the moment. Finally, it's worth mentioning that it is perfectly possible to create files and folders with forbidden names directly on Windows itself. This requires using Native API to work around various parameter restrictions implemented at the Windows API level.
It's that time of season again - the progress bar
redesign time. It's been over 3 years since the
last one, so it's clearly justified by that fact alone.
For the previous installment of this saga see this post from 2018 with an archive of all past designs or read on for details of the new one.
For the previous installment of this saga see this post from 2018 with an archive of all past designs or read on for details of the new one.
When the program is
delta copying
a file, it renders progress bar using four levels
of gray to reflect how many blocks were copied
in different parts of the file.
Light gray stands for no changes, followed by few changes gray, lots of changes gray and every block changed black.
While this split may sound reasonable, it tends to produce uniformly-looking bars like in the screenshot above (R81). This happens because delta copying operates with 64KB blocks, emitting a binary "modified" flag for each block. For a 1GB file this yields 16384 flags. The UI then needs to digest and render them into 220 pixels of the progress bar.
That's a ratio of 70 blocks / pixel and it's for just 1 GB.
In practice, delta copying is used for files that are significantly larger and with much higher blocks/pixel ratios. So the change pattern - whatever it happens to be for a specific file - ends up being averaged into just a couple of gray levels. Sometimes, rarely - into three and almost never - into four.
Hence the boring looking bar. Quantization into 4 levels of gray was not a good choice. Let's try something different.
Opening screenshot is for a 256 GB file, giving us 16384 blocks per pixel. The file is not heavily modified, with per-pixel changes varying between 0 and 2684 blocks.
This means we can't just take per-pixel change count and divide it by 16384. This maximum value we get from this will be 0.16 and the bar will end up looking uniform gray.
Instead we can use the max observed value as a
base, i.e. 2568, and it will give us something like this:
Next, we'll make the no-changes pixels stand out
a bit more, because they are, after all, interesting.
To do this we use the lightest shade for 0.0 and map non-zero values to, say, 0.2 - 1.0 range (instead of 0.0 - 1.0). We'll get this:
Next, we are going to amplify differences between non-zero
shades a bit by applying
gamma correction:
This is not bad, but in this particular case only 1% of
the file was modified and copied over. The bar however
implies that it was higher than that. So we will tone
things down a bit by bumping up our observed max
of 2568 a bit closer to the true max of 16384:
For slices that represent smaller values, we are going to contract them vertically around their midpoint. So in addition to looking lighter, they will also look smaller.
Contracting proportionally to the value will yield a bar that looks like this:
That's hardly an improvement. But we can gamma-correct
the contraction here to reduce its effect on larger
values. This will give us this:
Finally, we can also apply contraction only if the
value is under 50% of the max, i.e. if it is indeed
small-ish (and already lighter):
To see the exact effect of slice contraction we
can take color gamma to 0 and it will render
like so:
And that's it. We introduced a bit of color and
geometrical variety to the bar, but without making
it too jarring or noisy-looking.
So here is a webpage that implements just that:
Make your own progress bar
It's an older version of the playground, so its parameter set is a bit different from what's described above.
In particular, the flutter parameter was a dead-end experiment that added random vertical offset to each slice. The idea was to break the symmetry a bit and to add some vertical movement.
If it's smaller, it causes the bar to slowly "roll" up and down:
If it's larger, it just randomizes the heck out of
the whole thing:
Combined with extreme gamma values it can also produce
bars like this:
These versions look as interesting as they are confusing.
The up and down movement doesn't really mean anything,
but it looks it does. So - the dead-end.
Defaults for the parameters are different from the playground defaults, skewed towards producing a softer looking bar, because it looks better against the glaring full-white iof the UI. They are all tweakable however via UI's ini, so there's that too.
That's it... and till the next redesign!
For the previous installment see here.
Light gray stands for no changes, followed by few changes gray, lots of changes gray and every block changed black.
While this split may sound reasonable, it tends to produce uniformly-looking bars like in the screenshot above (R81). This happens because delta copying operates with 64KB blocks, emitting a binary "modified" flag for each block. For a 1GB file this yields 16384 flags. The UI then needs to digest and render them into 220 pixels of the progress bar.
That's a ratio of 70 blocks / pixel and it's for just 1 GB.
In practice, delta copying is used for files that are significantly larger and with much higher blocks/pixel ratios. So the change pattern - whatever it happens to be for a specific file - ends up being averaged into just a couple of gray levels. Sometimes, rarely - into three and almost never - into four.
Hence the boring looking bar. Quantization into 4 levels of gray was not a good choice. Let's try something different.
More of the same
The first thing we are going to change is to use a full range of grays to render the progress.Opening screenshot is for a 256 GB file, giving us 16384 blocks per pixel. The file is not heavily modified, with per-pixel changes varying between 0 and 2684 blocks.
This means we can't just take per-pixel change count and divide it by 16384. This maximum value we get from this will be 0.16 and the bar will end up looking uniform gray.
Instead we can use the max observed value as a
base, i.e. 2568, and it will give us something like this:
Next, we'll make the no-changes pixels stand out
a bit more, because they are, after all, interesting.
To do this we use the lightest shade for 0.0 and map non-zero values to, say, 0.2 - 1.0 range (instead of 0.0 - 1.0). We'll get this:
Next, we are going to amplify differences between non-zero
shades a bit by applying
gamma correction:
This is not bad, but in this particular case only 1% of
the file was modified and copied over. The bar however
implies that it was higher than that. So we will tone
things down a bit by bumping up our observed max
of 2568 a bit closer to the true max of 16384:
Less of the same
The second change is that we are going to play with the vertical size of 1px slices.For slices that represent smaller values, we are going to contract them vertically around their midpoint. So in addition to looking lighter, they will also look smaller.
Contracting proportionally to the value will yield a bar that looks like this:
That's hardly an improvement. But we can gamma-correct
the contraction here to reduce its effect on larger
values. This will give us this:
Finally, we can also apply contraction only if the
value is under 50% of the max, i.e. if it is indeed
small-ish (and already lighter):
To see the exact effect of slice contraction we
can take color gamma to 0 and it will render
like so:
And that's it. We introduced a bit of color and
geometrical variety to the bar, but without making
it too jarring or noisy-looking.
Do it yourself
Since our goal here was to make things look good the fastest and simplest way to do it was to take the parameters (gammas, thresholds, etc.) and to write a little interactive playground for playing with them.So here is a webpage that implements just that:
Make your own progress bar
It's an older version of the playground, so its parameter set is a bit different from what's described above.
In particular, the flutter parameter was a dead-end experiment that added random vertical offset to each slice. The idea was to break the symmetry a bit and to add some vertical movement.
If it's smaller, it causes the bar to slowly "roll" up and down:
If it's larger, it just randomizes the heck out of
the whole thing:
Combined with extreme gamma values it can also produce
bars like this:
These versions look as interesting as they are confusing.
The up and down movement doesn't really mean anything,
but it looks it does. So - the dead-end.
Out on the [preview] channel
New bar styling is a part of Preview 26 of Release 82, available now.Defaults for the parameters are different from the playground defaults, skewed towards producing a softer looking bar, because it looks better against the glaring full-white iof the UI. They are all tweakable however via UI's ini, so there's that too.
That's it... and till the next redesign!
For the previous installment see here.
Maintenance release 81.20 adds an option of requesting a
license for an offline installation using QR codes.
As exciting as it sounds, this option is meant for a specific and very niche type of setup that allows no data to be copied from a machine, while still allowing things to be copied to it.
Prior to this release the only option was to copy activation URL by hand, character by character, which is obviously a major hassle.
With 81.20 it's now possible to open the URL by scanning its QR code. The system will then generate and display the license and may optionally send a copy of it by email.
The second "leg" of offline activation remains the same - the license still needs to be brought to the offline machine and registered with the program.
As exciting as it sounds, this option is meant for a specific and very niche type of setup that allows no data to be copied from a machine, while still allowing things to be copied to it.
Prior to this release the only option was to copy activation URL by hand, character by character, which is obviously a major hassle.
With 81.20 it's now possible to open the URL by scanning its QR code. The system will then generate and display the license and may optionally send a copy of it by email.
The second "leg" of offline activation remains the same - the license still needs to be brought to the offline machine and registered with the program.
Preview 16
ships a smaller UI enhancement that makes life easier
when working with very large backups.
The program now displays the progress of the backup planning activity as it moves through all three planning stages.
The program now displays the progress of the backup planning activity as it moves through all three planning stages.
Backup planning involves looking at the source and backup
trees, finding the differences and then generating a list
of simple steps that, when executed in correct order, bring
the backup in sync with the source.
As you may or may not know, Bvckup 2 creates a backup plan in three stages. The change in this release is that the program now reports how far along each stage it is at any given moment.
This makes little difference for smaller and medium-sized backups (up to 100K files, give or take), because planning is generally very fast. But for larger backups with millions of items this change makes the process a bit more predictable.
During the comparison stage the planner ingests source and backup file trees and spits out a list of create/delete steps needed to bring the latter in sync with the former.
This basic plan is then passed through the file/folder move detection. It also has various policies applied to it, e.g. the file archival, forced file updates, etc. This is the compilation phase and it produces the backup plan.
Finally, the plan is trimmed down and stripped of all temporary data that was used during the planning. This comprises the 3rd stage, the packaging. GIF screencap at the top of the post shows the program go through all three stages.
Keep in mind that the program was artificially slowed down for demo purposes. Regular production version will make a plan of this size in a fraction of a second.
As you may or may not know, Bvckup 2 creates a backup plan in three stages. The change in this release is that the program now reports how far along each stage it is at any given moment.
This makes little difference for smaller and medium-sized backups (up to 100K files, give or take), because planning is generally very fast. But for larger backups with millions of items this change makes the process a bit more predictable.
Planning stages
The three planning stages, as used by Bvckup 2, are the tree comparision, compilation and packaging.During the comparison stage the planner ingests source and backup file trees and spits out a list of create/delete steps needed to bring the latter in sync with the former.
This basic plan is then passed through the file/folder move detection. It also has various policies applied to it, e.g. the file archival, forced file updates, etc. This is the compilation phase and it produces the backup plan.
Finally, the plan is trimmed down and stripped of all temporary data that was used during the planning. This comprises the 3rd stage, the packaging. GIF screencap at the top of the post shows the program go through all three stages.
Keep in mind that the program was artificially slowed down for demo purposes. Regular production version will make a plan of this size in a fraction of a second.
Preview build 13
strikes one the oldest items off the ToDo list - the UI
support for archiving file backups.
Internally, the engine supported archiving of modified files
for quite some time now. It was first added
back in 2014,
but as it was deemed "temporary" in its design, the feature
could only be enabled with a manual configuration override.
Archiving was originally meant as a safety net for people who click their way through the warnings only to stare in a petrified disbelief at years worth of their precious family jewels vanishing from their one and only removable drive.
The archiving worked well for this purpose. It saved the jewels.
However it worked by appending a timestamp to the file or folder name, so retrieving items from the archive was tedious. It was not a good general-purpose archival approach. Fast forward 7 years to last week and preview build 12 rolls out support for a much simpler archive structure.
Details are at the link, but the keyword here is "human-friendly". The second part to the recipe was the new UI mechanism for editing details of a backup job.
Again, the details are at the link, but the gist of it is that the backup configuration now allows for more settings per section, while still remaining compact in size and light in function.
So the "Deleting" section became "Archiving", gained some weight and finally added UI support for the file versioning.
It took a while, but it's finally here.
It's in the fact that file versioning prevents delta copying from actually ** delta ** copying.
This copying method gets its speed benefits from being selective with updating an existing copy of a file. But the archiving ensures that there's no existing copy => the file gets copied in full, at full IO cost.
Hence this brand new warning:
To explain it differently:
When the archiving is enabled, there will be one extra full copy of a file after every backup run.
But delta copying is all about doing less than a full copy, so it's not possible in principle to mix one with another.
QED.
Archiving was originally meant as a safety net for people who click their way through the warnings only to stare in a petrified disbelief at years worth of their precious family jewels vanishing from their one and only removable drive.
The archiving worked well for this purpose. It saved the jewels.
However it worked by appending a timestamp to the file or folder name, so retrieving items from the archive was tedious. It was not a good general-purpose archival approach. Fast forward 7 years to last week and preview build 12 rolls out support for a much simpler archive structure.
Details are at the link, but the keyword here is "human-friendly". The second part to the recipe was the new UI mechanism for editing details of a backup job.
Again, the details are at the link, but the gist of it is that the backup configuration now allows for more settings per section, while still remaining compact in size and light in function.
So the "Deleting" section became "Archiving", gained some weight and finally added UI support for the file versioning.
It took a while, but it's finally here.
The caveat
There is always a caveat.It's in the fact that file versioning prevents delta copying from actually ** delta ** copying.
This copying method gets its speed benefits from being selective with updating an existing copy of a file. But the archiving ensures that there's no existing copy => the file gets copied in full, at full IO cost.
Hence this brand new warning:
To explain it differently:
When the archiving is enabled, there will be one extra full copy of a file after every backup run.
But delta copying is all about doing less than a full copy, so it's not possible in principle to mix one with another.
QED.
The next preview build of Release 82
will ship with a new mechanic for accessing configuration details
in the Backup Settings window.
Instead of expanding configuration sections in place by pushing bottom of the window down, the program will pull details to the center of the window and then de-emphasize the rest of the settings.
The change is not dramatic, but it looks nice (always a plus) and it also allows for larger detail sections... which will come into play shortly.
Stay tuned.
Instead of expanding configuration sections in place by pushing bottom of the window down, the program will pull details to the center of the window and then de-emphasize the rest of the settings.
The change is not dramatic, but it looks nice (always a plus) and it also allows for larger detail sections... which will come into play shortly.
Stay tuned.
Release 82 is an upcoming major Bvckup 2 update and it is now
starting to enter the preview update
channel.
First up is a new and important feature - the write verification.
First up is a new and important feature - the write verification.
Write verification does exactly what its name implies - for
every data block that the program writes out, it reads the
block back and checks that it remained intact.
This process helps to safeguard against two types of problems.
The relative risk of such corruption is exceedingly small, but the backups tend to push a lot of data around, so the absolute risk may grow with time to the non-negligible levels.
If the media is OK, the read-back will just complete nominally.
If the media is dying, i.e. if the data can still be read, but only after retrying or via the sector ECC recovery, then we will get our data back and trigger the sector relocation, avoiding potential data loss if the media were to degrade further.
If the media is dead, then we'll won't have our data back, but we'll learn about the problem now rather than later.
The program arranges for this by opening the target file for the second time and requesting a so-called unbuffered read access to it.
This doesn't prevent writes from being cached, but ensures that reads are served from the disk. Going around the cache automatically means that the copying becomes slower.
The degradation is in part mitigated by the asynchronous nature of the ultra copier and it's further reduced by the delta copying if it's in effect, but there is a slow down. No miracle there.
It will also be mentioned in the completion statement
for all non-empty files:
Finally, if a write verification fails, the log will
show the following:
Verification failures are non-retryable, the
program will not retrying the operation in an attempt
to correct the problem.
This is by design to keep the issue on the surface and to force investigating it. In prerelease builds the write verification can be enabled for a job via the right-click menu as shown in the opening screenshot.
In production builds it's likely to be the same, except the Preview submenu will have a different name and it will be visible only if the Ctrl held down. You can get a prerelease build by switching your installation to the "preview" update channel as described here.
This process helps to safeguard against two types of problems.
1. Data corruption on the way to the device
This may happen due to the software and firmware bugs, memory bit flips (both in the machine's own RAM and in the controller/device memory) and transport errors if going over the network.The relative risk of such corruption is exceedingly small, but the backups tend to push a lot of data around, so the absolute risk may grow with time to the non-negligible levels.
2. Latent storage media problems
Reading data back effectively acts as a form of scrubbing and helps flushing any pending issues with the storage media.If the media is OK, the read-back will just complete nominally.
If the media is dying, i.e. if the data can still be read, but only after retrying or via the sector ECC recovery, then we will get our data back and trigger the sector relocation, avoiding potential data loss if the media were to degrade further.
If the media is dead, then we'll won't have our data back, but we'll learn about the problem now rather than later.
How it works
The somewhat less obvious aspect of write verification is that it needs to bypass the file system cache and to try and read the data from actual the device.The program arranges for this by opening the target file for the second time and requesting a so-called unbuffered read access to it.
This doesn't prevent writes from being cached, but ensures that reads are served from the disk. Going around the cache automatically means that the copying becomes slower.
The degradation is in part mitigated by the asynchronous nature of the ultra copier and it's further reduced by the delta copying if it's in effect, but there is a slow down. No miracle there.
How it looks
Once enabled, the write verification setting will show up in the Configuration section:
It will also be mentioned in the completion statement
for all non-empty files:
Finally, if a write verification fails, the log will
show the following:
Verification failures are non-retryable, the
program will not retrying the operation in an attempt
to correct the problem.
This is by design to keep the issue on the surface and to force investigating it. In prerelease builds the write verification can be enabled for a job via the right-click menu as shown in the opening screenshot.
In production builds it's likely to be the same, except the Preview submenu will have a different name and it will be visible only if the Ctrl held down. You can get a prerelease build by switching your installation to the "preview" update channel as described here.
Maintenance release 81.7 adds a relatively simple
new feature - an option for automatically discarding files'
delta copying state
based on its age or the number of uses.
When enabled, this will force the re-copying of large files in full as per set thresholds. This may be desirable in some setups, e.g. as a data scrubbing mechanism for flushing out pending storage media issues.
When enabled, this will force the re-copying of large files in full as per set thresholds. This may be desirable in some setups, e.g. as a data scrubbing mechanism for flushing out pending storage media issues.
It's been a while since the last sketch,
so here is one.
This is for a rework of the web dashboard that aims to make it more compact and better suited for managing a large number of licenses. No doubt, a nice problem to have for everyone involved.
* Still very much a work in progress.
This is for a rework of the web dashboard that aims to make it more compact and better suited for managing a large number of licenses. No doubt, a nice problem to have for everyone involved.
* Still very much a work in progress.
Maintenance release 81.3 includes a rework of
how the program tracks, handles and reports errors
encountered during a backup run.
Starting with this release Bvckup 2 will report not only the error count and the number of retries, but also the number of errors actually corrected through retrying.
Corrected error count is needed, because a single backup step may sometimes fail with more than one error, e.g. -
When this happens, earlier Bvckup 2 versions would say
that the run completed with 2 errors and 1 retry...
which was not terribly helpful, even if it was technically correct.
With the new change, the program now tracks the exact number of errors associated with each failure and report how many of them were in fact corrected through retrying. Another change in 81.3 has to do with the retrying logic itself.
In the presence of multiple errors the program will now look at all errors in order to decide whether the operation should be retried. This is different from earlier versions that looked at the last error only.
It turns out that yanking a network cable from a NAS sometimes results in "An unexpected network error occurred" first, followed by "The file is in use" second. The "in-use" error is non-retryable, so looking at the last error only ended up suppressing a retry.
Starting with this release Bvckup 2 will report not only the error count and the number of retries, but also the number of errors actually corrected through retrying.
Corrected error count is needed, because a single backup step may sometimes fail with more than one error, e.g. -
When this happens, earlier Bvckup 2 versions would say
that the run completed with 2 errors and 1 retry...
which was not terribly helpful, even if it was technically correct.
With the new change, the program now tracks the exact number of errors associated with each failure and report how many of them were in fact corrected through retrying. Another change in 81.3 has to do with the retrying logic itself.
In the presence of multiple errors the program will now look at all errors in order to decide whether the operation should be retried. This is different from earlier versions that looked at the last error only.
It turns out that yanking a network cable from a NAS sometimes results in "An unexpected network error occurred" first, followed by "The file is in use" second. The "in-use" error is non-retryable, so looking at the last error only ended up suppressing a retry.
Another new feature in Release 81 is an option
for waking up the computer to run scheduled (periodic)
backups.
When enabled, this option will cause the program to ask Windows to wake up the computer around the time of the next scheduled backup run.
If no user interaction is registered while the machine is awake and the backup is running, Windows will put it back to sleep once the backup is done.
Any errors of this sort are reported through the log:
The second caveat is that wake-up timers are imprecise.
Asking for a wake-up at 12:00:00 may translate into computer just starting waking up some 10 to 30 seconds later.
To work around this Bvckup 2 schedules all wake-ups for a bit earlier than the actual backup time and it also tolerates the machine (still) waking up later than needed.
The defaults are 30 and 60 seconds respectively, but, as per usual, they are fully adjustable.
When enabled, this option will cause the program to ask Windows to wake up the computer around the time of the next scheduled backup run.
If no user interaction is registered while the machine is awake and the backup is running, Windows will put it back to sleep once the backup is done.
Caveats
One caveat is that the machine should support software-driven wake-ups in general, e.g. they should not disabled at the BIOS level or at the OS level.Any errors of this sort are reported through the log:
The second caveat is that wake-up timers are imprecise.
Asking for a wake-up at 12:00:00 may translate into computer just starting waking up some 10 to 30 seconds later.
To work around this Bvckup 2 schedules all wake-ups for a bit earlier than the actual backup time and it also tolerates the machine (still) waking up later than needed.
The defaults are 30 and 60 seconds respectively, but, as per usual, they are fully adjustable.
Upcoming Release 81 will introduce an option
for locking down UI access when the program is running in
service mode.
The engine runs autonomously in the background as
a system service, under a service user account.
The user interface runs as a regular
desktop program, under a desktop user account.
This split allows the UI to be started and stopped at will, with the engine running continuously even when no user is logged in. When the UI is started, it connects to the engine, at which point the engine starts feeding the UI with updates and the UI sends back control and configuration commands as per user's input.
The change in R81 allows to set the engine to ask for a password when the UI first connects to it.
Once enabled, it will cause the program to pop up a
password prompt when the UI is opened. No password = no UI.
This option is primarily meant for preventing non-admin
local users from opening the UI and messing with the setup.
2. Salted hash of the password is stored in the engine's config file. The password can be cleared by stopping the service, wiping password hash from the config file and restarting the service. This also requires Admin rights.
3. When the program is being updated, the UI knows how to restart and reconnect back to the service without requesting a password.
Due to how updates work, this is one of more complicated (and interesting!) parts, based on single-use authentication tokens that are issued by the engine to the UI specifically for this case. However it's completely invisible in use.
4. The UI does away with dual password entry fields (enter once, enter again to confirm). Instead, just as many modern UIs do, it allows revealing the password if needed.
5. Don't forget your password.
Details
In service mode, the program splits the backup engine and its user interface into two separate processes.
The engine runs autonomously in the background as
a system service, under a service user account.
The user interface runs as a regular
desktop program, under a desktop user account.
This split allows the UI to be started and stopped at will, with the engine running continuously even when no user is logged in. When the UI is started, it connects to the engine, at which point the engine starts feeding the UI with updates and the UI sends back control and configuration commands as per user's input.
The change in R81 allows to set the engine to ask for a password when the UI first connects to it.
Once enabled, it will cause the program to pop up a
password prompt when the UI is opened. No password = no UI.
This option is primarily meant for preventing non-admin
local users from opening the UI and messing with the setup.
Fine print
1. The password is verified by the engine, meaning that the check is isolated from the desktop user context and can't be worked around without having Admin rights.2. Salted hash of the password is stored in the engine's config file. The password can be cleared by stopping the service, wiping password hash from the config file and restarting the service. This also requires Admin rights.
3. When the program is being updated, the UI knows how to restart and reconnect back to the service without requesting a password.
Due to how updates work, this is one of more complicated (and interesting!) parts, based on single-use authentication tokens that are issued by the engine to the UI specifically for this case. However it's completely invisible in use.
4. The UI does away with dual password entry fields (enter once, enter again to confirm). Instead, just as many modern UIs do, it allows revealing the password if needed.
5. Don't forget your password.
When Pro licenses were introduced last year,
with that we also introduced a new support and software update model referred to
as Pro Support.
All details on this element of Pro licenses - coverage, eligibility, renewals and costs - are now available here:
Pro Support
* In particular, do note that renewal of the Pro Support while it is still active is offered at a
discounted rate.
All details on this element of Pro licenses - coverage, eligibility, renewals and costs - are now available here:
Preview 12 of Release 81 is out and
it adds UI-side support for configuring multi-threaded (parallel) processing.
The contents of Backup Settings window got squished together a bit and a new Processing section was added.
Surprisingly enough, the window actually got smaller as a result, but it does look a bit more crowded now.
There's also a couple of important options behind Edit details:
Two options in the Large files section define how the program handles the copying of "large" files.
The definition of "large" is based on a file size threshold and the question was if this threshold was important enough to have a place in the UI.
Initially, the idea was to both display and to allow editing it.
This gives too much prominence to an otherwise minor option and
it also invites to mess with it. We can do better.
The option is shown as a text, but can be edited by clicking on it.
Furthermore the idea is that it can be edited inline. That is, with the edit box just appearing in place and going away once the edit is done.
This required implementing an auto-sizing edit control, built into a radio button widget. Doable, but hard, and all of this is still for a secondary option.
We take this option out and put it into a separate text. Also
stick it at the very end of the text. This allows stretching
out the edit box to the max and don't bother with dynamically
sizing it:
We can also add a bit of transition to compensate for edit box
not aligning with the option value when it is static.
This is not bad. Let's try with a borderless edit box:
Not bad either and the slide is just a couple of pixels
now. How about we don't slide it at all?
Looks and feels quite nice, however...
When using it for the first time, it's very easy to discover that typing something and hitting Enter stands for "Accept the changes" and hitting Esc works as "Discard the changes".
But the user clicking outside of the edit box can be ambiguous. Some people will expect this to work as OK, while others will expect it to discard changes. Moreover, one may make more sense than the other depending on the context.
While it's possible to anticipate and to accommodate this sort of nuance, it gets us a bit too far into deep customization territory.
This could've been justified if inline editing were added to the UI arsenal and used a first class element across the entire program. But the way things stand, it will be a random widget used in one obscure corner of the UI.
As such, this option was shelved.
Instead the window uses an in-place hint to show the threshold value and related details.
The contents of Backup Settings window got squished together a bit and a new Processing section was added.
Surprisingly enough, the window actually got smaller as a result, but it does look a bit more crowded now.
There's also a couple of important options behind Edit details:
Breadcrumbs
While we are at it, here is a quick rundown of the design process leading up to the final version.Two options in the Large files section define how the program handles the copying of "large" files.
The definition of "large" is based on a file size threshold and the question was if this threshold was important enough to have a place in the UI.
Initially, the idea was to both display and to allow editing it.
Take 1
This gives too much prominence to an otherwise minor option and
it also invites to mess with it. We can do better.
Take 2
The option is shown as a text, but can be edited by clicking on it.
Furthermore the idea is that it can be edited inline. That is, with the edit box just appearing in place and going away once the edit is done.
This required implementing an auto-sizing edit control, built into a radio button widget. Doable, but hard, and all of this is still for a secondary option.
Take 3
We take this option out and put it into a separate text. Also
stick it at the very end of the text. This allows stretching
out the edit box to the max and don't bother with dynamically
sizing it:
We can also add a bit of transition to compensate for edit box
not aligning with the option value when it is static.
This is not bad. Let's try with a borderless edit box:
Not bad either and the slide is just a couple of pixels
now. How about we don't slide it at all?
Looks and feels quite nice, however...
The "details"
... the issue is with how this contraption behaves in edge cases.When using it for the first time, it's very easy to discover that typing something and hitting Enter stands for "Accept the changes" and hitting Esc works as "Discard the changes".
But the user clicking outside of the edit box can be ambiguous. Some people will expect this to work as OK, while others will expect it to discard changes. Moreover, one may make more sense than the other depending on the context.
While it's possible to anticipate and to accommodate this sort of nuance, it gets us a bit too far into deep customization territory.
This could've been justified if inline editing were added to the UI arsenal and used a first class element across the entire program. But the way things stand, it will be a random widget used in one obscure corner of the UI.
As such, this option was shelved.
Instead the window uses an in-place hint to show the threshold value and related details.
Introducing a brand new addition to the ever growing
family of Bvckup 2 spinoffs and companion products...
Please meet - byenow.
Multi-threaded folder removal utility, it speeds up the deletion by processing multiple subfolders in parallel - something that works especially well for deleting folders off network shares.
Check it out here - https://iobureau.com/byenow
Please meet - byenow.
Multi-threaded folder removal utility, it speeds up the deletion by processing multiple subfolders in parallel - something that works especially well for deleting folders off network shares.
Check it out here - https://iobureau.com/byenow
Next maintenance release - 80.9 - will improve how the program
estimates and displays completion time for longer file copies.
A brand new version of the copying speed-time estimator is in. It
is far more sensitive to the changes in copying conditions while
also producing estimates that are far less jittery.
New version also has a visually distinct mode when the program is correcting its estimate due to changes in the copying speed.
Significant copying speed-ups are not uncommon, but the slow-downs are even more routine. They happen few seconds into the copying, when the write cache fills in and the writing speed drops to the device's native value:
When this happens, the completion time gets pushed back and the
UI reacts to this by highlighting the estimate in red.
It also increases its rate of refresh to make changes more obvious.
The GIF at the top captures this in action with a backup that goes from an NVMe drive to an HDD.
The copying speed first spikes to 1600MB/s as the write cache is getting filled, but quickly drops to 100 MB/s, which is the native write speed of the HDD. When a change in speed like this happens, the estimator enters a correcting state and starts to gradually adjust the ETA.
Once the copying speed stabilizes, the estimator switches to the stable state and sticks to its most recent prediction. That is, it fixes the ETA and keeps it constant until further notice.
As the copying speed fluctuates, the effective ETA may go a bit up or down, but the estimator will ignore this jitter in favour of keeping the countdown steady.
This ensures that the countdown behaves like a clock, without sudden stalls and accelerations like this:
The estimator also keeps monitoring effective ETA to make sure
it doesn't drift too far away from the fixed estimate. If it
does, then the estimator initiates go back into correcting
state and the process repeats itself.
So there you have it - our new and shiny stabilized estimator.
Coming up in Release 80.9.
*
A broader context here is that even the fastest storage devices
will complete the IO unevenly. As a result the raw measurements
that go into an estimator will always be noisy and feeding
them into an estimator, as-is, will produce noisy estimates.
For example, here is (an otherwise truly excellent) Far Manager running a bulk copy and using an overly simplistic estimator:
This is more distracting than it is helpful.
A better, human-friendly estimator needs to de-noise the data and to smooth out the estimates. It also needs to make use of heuristics to further align the result with what a human would expect, e.g. to not re-adjust the ETA every chance it gets.
Long story short, calculating ETA is one of those things that look trivial on the surface, but in practice has quite a bit of depth and complexity. Just something to keep in mind.
New version also has a visually distinct mode when the program is correcting its estimate due to changes in the copying speed.
Significant copying speed-ups are not uncommon, but the slow-downs are even more routine. They happen few seconds into the copying, when the write cache fills in and the writing speed drops to the device's native value:
When this happens, the completion time gets pushed back and the
UI reacts to this by highlighting the estimate in red.
It also increases its rate of refresh to make changes more obvious.
The GIF at the top captures this in action with a backup that goes from an NVMe drive to an HDD.
The copying speed first spikes to 1600MB/s as the write cache is getting filled, but quickly drops to 100 MB/s, which is the native write speed of the HDD. When a change in speed like this happens, the estimator enters a correcting state and starts to gradually adjust the ETA.
Once the copying speed stabilizes, the estimator switches to the stable state and sticks to its most recent prediction. That is, it fixes the ETA and keeps it constant until further notice.
As the copying speed fluctuates, the effective ETA may go a bit up or down, but the estimator will ignore this jitter in favour of keeping the countdown steady.
This ensures that the countdown behaves like a clock, without sudden stalls and accelerations like this:
The estimator also keeps monitoring effective ETA to make sure
it doesn't drift too far away from the fixed estimate. If it
does, then the estimator initiates go back into correcting
state and the process repeats itself.
So there you have it - our new and shiny stabilized estimator.
Coming up in Release 80.9.
*
A broader context here is that even the fastest storage devices
will complete the IO unevenly. As a result the raw measurements
that go into an estimator will always be noisy and feeding
them into an estimator, as-is, will produce noisy estimates.
For example, here is (an otherwise truly excellent) Far Manager running a bulk copy and using an overly simplistic estimator:
This is more distracting than it is helpful.
A better, human-friendly estimator needs to de-noise the data and to smooth out the estimates. It also needs to make use of heuristics to further align the result with what a human would expect, e.g. to not re-adjust the ETA every chance it gets.
Long story short, calculating ETA is one of those things that look trivial on the surface, but in practice has quite a bit of depth and complexity. Just something to keep in mind.
Preview 7 of R81 adds support for working with live files.
This is a bit of a rabbit hole, so some background is in order.
This is a bit of a rabbit hole, so some background is in order.
If we are to open an existing file, append some data to it
and keep it open, then the dir command will
keep returning file's original size until we close the file:
The same will also happen to the last-modified timestamp. Even
though we "touch" the file, everyone else will be seeing its old
timestamp until we release the file reference.
This looks odd, doesn't it? The explanation to this behavior can be found here and it has to do with the fact that modern file systems such as NTFS and ReFS store file meta data alongside the file itself rather than with its folder entry.
This allows attaching arbitrary meta data to the files (i.e. not just attributes and timestamps), but it also means that a simple directory listing requires combing through each and every file to collect respective bits of meta.
Obviously, this doesn't scale well. This issue is resolved by caching parts of file's meta in its folder's entry. This speeds up directory listings, but it also comes at a cost of meta being replicated lazily.
For example, if we are to add data to a file, then its new file size will not get propagated to its folder record immediately. Instead, it will done at some point later on.
Depending on the Windows version, this may happen when we flush file buffers, query file size via a handle or close its handle. All this means that there's no reliable way of detecting actively modified files based on their directory listing alone.
So if we want to identify and back up such live files, we need a workaround.
One workaround is to just forcibly update them on every run, regardless of whether they have actually changed or not. It's crude and wasteful, but in a pinch it works. A better option, supported by the new preview release, is to double-check directory listing data by re-querying file size and timestamps directly from the files themselves.
The queries aren't cheap, so they are not issued for all files.
Instead, this option works similarly to the forced updates - we specify one or more files of interest and the program rechecks their meta if it runs into them during a scan. None of this is too complicated, but it takes time to understand why things work they way they do and how to work around that.
* Kudos to James Kindon for raising the issue and providing original traces to identify it.
The same will also happen to the last-modified timestamp. Even
though we "touch" the file, everyone else will be seeing its old
timestamp until we release the file reference.
This looks odd, doesn't it? The explanation to this behavior can be found here and it has to do with the fact that modern file systems such as NTFS and ReFS store file meta data alongside the file itself rather than with its folder entry.
This allows attaching arbitrary meta data to the files (i.e. not just attributes and timestamps), but it also means that a simple directory listing requires combing through each and every file to collect respective bits of meta.
Obviously, this doesn't scale well. This issue is resolved by caching parts of file's meta in its folder's entry. This speeds up directory listings, but it also comes at a cost of meta being replicated lazily.
For example, if we are to add data to a file, then its new file size will not get propagated to its folder record immediately. Instead, it will done at some point later on.
Depending on the Windows version, this may happen when we flush file buffers, query file size via a handle or close its handle. All this means that there's no reliable way of detecting actively modified files based on their directory listing alone.
So if we want to identify and back up such live files, we need a workaround.
One workaround is to just forcibly update them on every run, regardless of whether they have actually changed or not. It's crude and wasteful, but in a pinch it works. A better option, supported by the new preview release, is to double-check directory listing data by re-querying file size and timestamps directly from the files themselves.
The queries aren't cheap, so they are not issued for all files.
Instead, this option works similarly to the forced updates - we specify one or more files of interest and the program rechecks their meta if it runs into them during a scan. None of this is too complicated, but it takes time to understand why things work they way they do and how to work around that.
* Kudos to James Kindon for raising the issue and providing original traces to identify it.
Our pre-release program has been formally
opened to everyone.
In the weeks leading up to a release of the next major
version we will now be posting internal development builds
to the preview update channel.
These builds are meant to let anyone interested to kick the tires of what's coming and to allow us to see how the new version behaves outside our test environment.
All pre-releases are of production quality, with no known issues, but they may have some wrinkles and they may also not have all planned features implemented yet. More information on the program, including instructions on how to obtain a pre-release head over here -
Pre-release builds
Ready when you are.
These builds are meant to let anyone interested to kick the tires of what's coming and to allow us to see how the new version behaves outside our test environment.
All pre-releases are of production quality, with no known issues, but they may have some wrinkles and they may also not have all planned features implemented yet. More information on the program, including instructions on how to obtain a pre-release head over here -
Ready when you are.
Next maintenance update - 80.6 - will be adding
UI support for configuring program's update channel.
Update channels were first introduced back in 2016 and they got formally documented earlier this year.
However the selection of channels remained limited to newest, stable and, the everyone's favourite, symantec. There was no need for making channels easily configurable. Not until now. A brand new update channel called preview will go live shortly after the 80.6 release and it will host development builds of upcoming major releases, starting with R81.
A chance for you to see what's coming and a chance for us to crash on machines other than ours. Win-win. * Update channel configuration is an advanced option, so it will not be visible unless Help menu is opened with Ctrl being held.
Update channels were first introduced back in 2016 and they got formally documented earlier this year.
However the selection of channels remained limited to newest, stable and, the everyone's favourite, symantec. There was no need for making channels easily configurable. Not until now. A brand new update channel called preview will go live shortly after the 80.6 release and it will host development builds of upcoming major releases, starting with R81.
A chance for you to see what's coming and a chance for us to crash on machines other than ours. Win-win. * Update channel configuration is an advanced option, so it will not be visible unless Help menu is opened with Ctrl being held.
Release 81 will include one of the most interesting and
important improvements of recent times - the full-scale
support for parallel backup processing.
So here's a quick screencap showing it in action.
The test setup is C:\Windows\System32 with about 15K files and 1500 folders being copied from an NVMe to an SSD drive.
More on this to follow.
So here's a quick screencap showing it in action.
The test setup is C:\Windows\System32 with about 15K files and 1500 folders being copied from an NVMe to an SSD drive.
| R80 (before) | 17 sec |
| R81 (after) | 5.7 sec |
More on this to follow.
The second improvement to the handling of cloud-stored files
in 80.5 has to do with gracefully tolerating these files being offline.
Call it an opportunistic or best-effort backup if you will.
We will try and back these files up if we can, but we won't make much fuss if we can't.
Call it an opportunistic or best-effort backup if you will.
We will try and back these files up if we can, but we won't make much fuss if we can't.
When run, every backup job goes through 3 main phases:
Scanning creates file indexes of the source and backup locations.
Planning compares resulting two indexes, finds the differences and compiles a list of basic steps - create folder, copy file, move this, delete that, update third - that, when executed in order, bring the backup in sync with the source.
Executing merely goes through the plan and diligently executes each step. Having a formal planning phase comes with one very important benefit - it allows extending core backup logic with a minimum amount of changes, making it a very simple, fast and predictable process.
In case of offline files, the opportunistic backup is implemented by simply disabling scheduled file update if the source file has a Recall-On attribute set.
Simple, which is why the change can be rolled out as a part of a smaller update rather than needing to wait for the next major release. * The same mechanism is already used by the option that disables file updates when the backup copy is newer than the source.
It works the same way - an update is scheduled using regular file comparison logic, and then an additional check is made to compare file times and to disable the planned action if needed.
- Scanning
- Planning
- Executing
Scanning creates file indexes of the source and backup locations.
Planning compares resulting two indexes, finds the differences and compiles a list of basic steps - create folder, copy file, move this, delete that, update third - that, when executed in order, bring the backup in sync with the source.
Executing merely goes through the plan and diligently executes each step. Having a formal planning phase comes with one very important benefit - it allows extending core backup logic with a minimum amount of changes, making it a very simple, fast and predictable process.
In case of offline files, the opportunistic backup is implemented by simply disabling scheduled file update if the source file has a Recall-On attribute set.
Simple, which is why the change can be rolled out as a part of a smaller update rather than needing to wait for the next major release. * The same mechanism is already used by the option that disables file updates when the backup copy is newer than the source.
It works the same way - an update is scheduled using regular file comparison logic, and then an additional check is made to compare file times and to disable the planned action if needed.
Next update (80.5) will introduce several
improvements to the handling of
OneDrive
files.
One of these improvements is a new option for excluding and including cloud-stored files/folders based on their local state.
One of these improvements is a new option for excluding and including cloud-stored files/folders based on their local state.
As you probably know, files stored in the OneDrive folder may or
may not
actually have their data cached locally.
When there's no local copy available, the file is said to be offline and its contents will be retrieved from the cloud storage when the file is accessed.
An offline file like this will have either
Recall-On-Open or Recall-On-Data-Access
attribute set, allowing it to be identified as such.
The Pinned attribute will be set on files that
were marked for permanent local storage using
"Always keep on this device" -
These files will show up with the solid green icon
in the Explorer.
Conversely, the Unpinned attribute will be set
on files that were explicitly purged from the computer
using "Free up space" -
Finally, there's also an Offline attribute that
is set on offline files by older versions of OneDrive.
With release 80.5 it becomes possible to define rules
based on these four attributes.
For example, we can exclude all unpinned files from the backup. Once in place, a rule like this will cause Bvckup 2 to remove file's backup when its OneDrive copy is discarded with Free up space. While not dramatically useful by itself, this change paves the way for another improvement - an option for suppressing file updates if the source file is offline.
See Part 2 for details.
When there's no local copy available, the file is said to be offline and its contents will be retrieved from the cloud storage when the file is accessed.
An offline file like this will have either
Recall-On-Open or Recall-On-Data-Access
attribute set, allowing it to be identified as such.
The Pinned attribute will be set on files that
were marked for permanent local storage using
"Always keep on this device" -
These files will show up with the solid green icon
in the Explorer.
Conversely, the Unpinned attribute will be set
on files that were explicitly purged from the computer
using "Free up space" -
Finally, there's also an Offline attribute that
is set on offline files by older versions of OneDrive.
With release 80.5 it becomes possible to define rules
based on these four attributes.
For example, we can exclude all unpinned files from the backup. Once in place, a rule like this will cause Bvckup 2 to remove file's backup when its OneDrive copy is discarded with Free up space. While not dramatically useful by itself, this change paves the way for another improvement - an option for suppressing file updates if the source file is offline.
See Part 2 for details.
The company behind the best file replicator of the modern
age is now known as IO Bureau.
This change took an excruciatingly long time, so here's a bit of background along with some field notes and takeaways.
The plan back then was to make a distributed network profiling and monitoring system, tentatively called Notwerk. As quirky as it was, the name quite accurately predicted what happened next - the project did not werk out.
So not few months later we ended up switching to another idea - a proper, production rewrite of the original Bvckup. It was a file backup program that I wrote over a weekend few years prior and that was getting an unexpected amount of traction despite all my efforts to ignore it.
The company name remained though and according to it we were still knee-deep in the pipe measurement business. Even had a logo to match -
That's an undersea network cable cross-section... or at least
an artist's impression of the same.
In any case, all this was around 7 years ago.
And there we were, making an offline software for handling one's private data, and yet we were called ...metrics. What was it that we were merticsing exactly?
Neither of these concerns required an immediate name change, but they lingered. And so about 3 years in we started looking for a better name.
Naming things is not hard. It is an utterly frustrating and painful, painful process that drains all fun from life and creative juices even from those who didn't have them to begin with.
Sometimes you can stumble upon a great name in a matter of seconds, but more often you don't. Or you just google it up and realize that you are the N-th heir to the throne, ten years too late. Say, you want to name something that deals with data. Dat should not be hard. Ding - "dat" is taken, and you haven't even considered it yet.
DataWorks?
DataScape?
ProData?
Pfft, lol. Are you kidding? Taken.
How about Datto?
Quirky, weird. No way anyone else would think it up.
You'd think. Taken.
In fact, taken so long ago that people now complain how much better it was in the good old days, whenever that was.
Datrium?
You know - uranium, unobtanium, pinacoladium. No-pe, taken.
Days go by and all that you can come up with are just OK-ayish bland names, each flawed in more ways than one, and most of them even without a matching dot-com domain.
Inevitably, you start looking for a way to make the process more formal, structured, predictable. To keep track of the "progress" and not to cover the same ground twice.
The good news is that it is doable. The bad news is that genuinely great names are still a matter of luck and random insipiration.
The shorter the better. There are exceptions, of course, but the rule of thumb is that it should be 3-4 syllables max.
It needs to be pronounceable and it should not have any negative connotations. Yes, I know, there's an irony in here somewhere.
In fact, looking back this is probably the most important property of a good name - it needs to roll off the tongue and leave a good impression behind.
That impression should be strong and memorable, even more so than the name itself. While people tend to forget specific names, the concepts and feelings stick around for much longer.
To that end good names often have an unusual, but an otherwise obvious association with whatever they are attached to. An excellent example of this is a distributed database called...
Ready?...
Voldemort!
Distributed storage indeed. How ridiculously good. Naturally, there are great names that don't have any explicit associations.
For example, Akamai does have a very good explanation behind it, but it's not obvious. However the name is simple and it sounds nice, so it is easy to remember just because of that.
Same goes for Skype, originally standing for Sky Peer-to-Peer.
Kodak is yet another example, except that it's a completely made up word. George Eastman, the founder of the company, liked the impression the sound of the letter K made, so he went on to find a name that was "short, easy to pronounce, and not resemble any other name."
The year was 1888. Finding a good association is a matter of certain luck, but the chances can be improved by formalizing the process.
By brute-forcing it basically.
Start by defining what it is that the company (product, gadget, whatever) does and what it is about. Compile a list of relevant keywords. Then for each keyword add several more - synonyms, related concepts, ideas, references.
Repeat several times.
The end result will be a list comprising both very specific and very abstract things that can be used as latches for an association.
For us, being in a data preservation domain, it went something like this:
The list was over 50 entries long, stretching as far
as "bastion", "marmalade" and "formaldehyde".
The problem, of course, is that a keyword alone usually doesn't make for an unique name.
One such pattern is adding a trailer qualifier:
Similar, but with more "edge":
Alternatively, the keyword itself can be equipped with a suffix:
This tends to result in a rather pompous and boring enterprise-y names, which sometimes may actually be desirable:
Infocom, Unisys, Commware, Microsoft, Symantec, Datamark, Netcraft, etc.
Prepending prefix to the word is yet another option:
InterPlay (of the Descent game fame), ParaData, HyperLoop, etc. One other option I feel I must mention is that mixing Maxwell into any technical name tends to make it dramatically better.
The association here is that with Maxwell's equations that carry an incredible mix of mathematical elegance and complexity.
"Maxwell" also sounds nice, and it is just 2 syllables long.
As a result, we had Maxwell Storent on the short list. In the end it lost to other options, but I still think it was a great name, full of technical competence and credibility at no extra charge :)
If we were back in the .com era, prepending an e to everything would've sounded like an excellent idea. Still have fond memories working in a place called eTunnels.
Dropping a vowel was an equally acceptable option with a great following - Flickr, Tumblr, etc.
Duplicating letters also worked, e.g. Dribbble or FFFFound.
In fact, there was also Forrst, which boldly went where nobody dared, both dropping and duplicating like there's no tomorrow. Replacing c's with k's and dropping trailing vowels tends to make words sound vaguely Scandinavian.
Objective becomes Objektiv, which can pass for a noun.
Accurate - Akkurat, etc. Technik can help adding some German flavor to the name. We would've gone this route except that we are in a French-speaking canton and it didn't seem like the right thing to do. Replacing trailing -e with an -a tends to create Latin-sounding words, e.g. Apertura, Machina, Textura, etc.
Similarly, tweaking the word to end with -ium or -eum results in a Latin-scientific feel.
It's reasonably short.
It rhymes when pronounced.
It's generic enough to cover more than just backup software.
The iobureau.com and @iobureau were available.
It also can be made to look both like IO and 10 when rasterized down to a small size. Not terribly important, but having a simple symbol to go with the name is always a plus.
So, ladies and gentlemen, until we decide to rename ourselves
again we shall be known as the IO Bureau.
1. Reverse dictionary
To search for words with specific ending.
2. List of last names
To make your own Bang & Olufsen.
3. Lists of gods
Quite a few solid leads there.
4. List of Intel project codenames
Skylake, Ice Lake, etc. Mostly geographical landmarks.
5. List of Microsoft project codenames
Not as consistent as Intel's, but still worth a read.
6. List of Solar System objects
Including a handy list of moons.
7. List of planets in science fiction
Examples of names made up by other people.
8. List of fantasy worlds
Same here.
An NYT article that follows the naming of a very high-profile project. Perhaps with no directly actionable insight, but really interesting nonetheless.
Brand Names
This is a really really deep dive into scientific aspects behind naming and name perception. Fantastic resource... though a bit like trying to learn how to fry a steak by studying metallurgy and proteins :) Be warned that you may end up being even more confused than before.
Feeds
-
RSS
/
Twitter
/
Reddit
This change took an excruciatingly long time, so here's a bit of background along with some field notes and takeaways.
7 years ago
The original company name - Pipemetrics - was picked in 2012, when the company was formed.The plan back then was to make a distributed network profiling and monitoring system, tentatively called Notwerk. As quirky as it was, the name quite accurately predicted what happened next - the project did not werk out.
So not few months later we ended up switching to another idea - a proper, production rewrite of the original Bvckup. It was a file backup program that I wrote over a weekend few years prior and that was getting an unexpected amount of traction despite all my efforts to ignore it.
The company name remained though and according to it we were still knee-deep in the pipe measurement business. Even had a logo to match -
That's an undersea network cable cross-section... or at least
an artist's impression of the same.
In any case, all this was around 7 years ago.
4 years ago
The discrepancy between the product and the company name was one concern. Another concern surfaced a bit later on, when Windows 10 with its pervasive telemetry was released and more people started paying closer attention to the tracking and privacy matters.And there we were, making an offline software for handling one's private data, and yet we were called ...metrics. What was it that we were merticsing exactly?
Neither of these concerns required an immediate name change, but they lingered. And so about 3 years in we started looking for a better name.
Naming
There are only two hard things in computer science:
cache invalidation and naming things.
Now, that is just factually wrong.
cache invalidation and naming things.
Naming things is not hard. It is an utterly frustrating and painful, painful process that drains all fun from life and creative juices even from those who didn't have them to begin with.
Sometimes you can stumble upon a great name in a matter of seconds, but more often you don't. Or you just google it up and realize that you are the N-th heir to the throne, ten years too late. Say, you want to name something that deals with data. Dat should not be hard. Ding - "dat" is taken, and you haven't even considered it yet.
DataWorks?
DataScape?
ProData?
Pfft, lol. Are you kidding? Taken.
How about Datto?
Quirky, weird. No way anyone else would think it up.
You'd think. Taken.
In fact, taken so long ago that people now complain how much better it was in the good old days, whenever that was.
Datrium?
You know - uranium, unobtanium, pinacoladium. No-pe, taken.
Days go by and all that you can come up with are just OK-ayish bland names, each flawed in more ways than one, and most of them even without a matching dot-com domain.
Inevitably, you start looking for a way to make the process more formal, structured, predictable. To keep track of the "progress" and not to cover the same ground twice.
The good news is that it is doable. The bad news is that genuinely great names are still a matter of luck and random insipiration.
Part 1 - Keywords and associations
The name needs to be unique and memorable, obviously.The shorter the better. There are exceptions, of course, but the rule of thumb is that it should be 3-4 syllables max.
It needs to be pronounceable and it should not have any negative connotations. Yes, I know, there's an irony in here somewhere.
In fact, looking back this is probably the most important property of a good name - it needs to roll off the tongue and leave a good impression behind.
That impression should be strong and memorable, even more so than the name itself. While people tend to forget specific names, the concepts and feelings stick around for much longer.
To that end good names often have an unusual, but an otherwise obvious association with whatever they are attached to. An excellent example of this is a distributed database called...
Ready?...
Voldemort!
Distributed storage indeed. How ridiculously good. Naturally, there are great names that don't have any explicit associations.
For example, Akamai does have a very good explanation behind it, but it's not obvious. However the name is simple and it sounds nice, so it is easy to remember just because of that.
Same goes for Skype, originally standing for Sky Peer-to-Peer.
Kodak is yet another example, except that it's a completely made up word. George Eastman, the founder of the company, liked the impression the sound of the letter K made, so he went on to find a name that was "short, easy to pronounce, and not resemble any other name."
The year was 1888. Finding a good association is a matter of certain luck, but the chances can be improved by formalizing the process.
By brute-forcing it basically.
Start by defining what it is that the company (product, gadget, whatever) does and what it is about. Compile a list of relevant keywords. Then for each keyword add several more - synonyms, related concepts, ideas, references.
Repeat several times.
The end result will be a list comprising both very specific and very abstract things that can be used as latches for an association.
For us, being in a data preservation domain, it went something like this:
The list was over 50 entries long, stretching as far
as "bastion", "marmalade" and "formaldehyde".
The problem, of course, is that a keyword alone usually doesn't make for an unique name.
Part 2 - Prefixes and suffixes
Lots and lots of names follow certain patterns. These patterns can be identified, distilled and formalized.One such pattern is adding a trailer qualifier:
- ... technologies
- ... software
- ... systems
- ... project
- ... group
- ... labs
- ... associates
Similar, but with more "edge":
- ... initiative
- ... innovations
- ... engineering
- ... committee
- ... society
- ... masters
- ... theory
- ... method
- ... logic
- ... works
- ... matters
Alternatively, the keyword itself can be equipped with a suffix:
- -com
- -sys
- -ware
- -soft
- -tec
- -tech
- -tronic
- -mark
- -craft
This tends to result in a rather pompous and boring enterprise-y names, which sometimes may actually be desirable:
Infocom, Unisys, Commware, Microsoft, Symantec, Datamark, Netcraft, etc.
Prepending prefix to the word is yet another option:
- Para-...
- Meta-...
- Vari-...
- Poly-...
- Inter-...
- Super-...
- Hyper-...
InterPlay (of the Descent game fame), ParaData, HyperLoop, etc. One other option I feel I must mention is that mixing Maxwell into any technical name tends to make it dramatically better.
The association here is that with Maxwell's equations that carry an incredible mix of mathematical elegance and complexity.
"Maxwell" also sounds nice, and it is just 2 syllables long.
As a result, we had Maxwell Storent on the short list. In the end it lost to other options, but I still think it was a great name, full of technical competence and credibility at no extra charge :)
Part 3 - Word mangling
Mangling a keyword is yet another way of turning it into an unique name.If we were back in the .com era, prepending an e to everything would've sounded like an excellent idea. Still have fond memories working in a place called eTunnels.
Dropping a vowel was an equally acceptable option with a great following - Flickr, Tumblr, etc.
Duplicating letters also worked, e.g. Dribbble or FFFFound.
In fact, there was also Forrst, which boldly went where nobody dared, both dropping and duplicating like there's no tomorrow. Replacing c's with k's and dropping trailing vowels tends to make words sound vaguely Scandinavian.
Objective becomes Objektiv, which can pass for a noun.
Accurate - Akkurat, etc. Technik can help adding some German flavor to the name. We would've gone this route except that we are in a French-speaking canton and it didn't seem like the right thing to do. Replacing trailing -e with an -a tends to create Latin-sounding words, e.g. Apertura, Machina, Textura, etc.
Similarly, tweaking the word to end with -ium or -eum results in a Latin-scientific feel.
The yield
Fast forward 4 years of episodic search spurts and we had a list of names with quite a few candidates on it. Here it is, with just few entries trimmed:- Data Mechanics - really liked this one, actually, a good pun
- Data Matters - similar, also double-meaning
- Data Associates - only if in hopes of being acquired by CA
- Data Committee - implied that we don't get any real work done
- Swiss Data Works
- Swiss Data Reserve - implied that we are a storage provider
- Datori
- Datamata
- Datamation
- DataTechnik
- SuperData
- Redata
- Storewell
- Storent
- Prestor
- ByteMasters
- MetaByters
- Keepwell
- Keep Systems
- Parity Works
- Zeroloss Systems
- Data & Loss - the barristers
- Stray neutrino
* As per the theory that the bitrot is caused by neutrinos
- Ashwell
- Waltron
- Corelion - as in Coreli-on, not Core-lion
- Hyper - with "Hyper Abc" as product names
- Variance
- Wellmade
- Struktura - also liked this one a lot, but the .com is squatted
And the winner is...
It's reasonably short.
It rhymes when pronounced.
It's generic enough to cover more than just backup software.
The iobureau.com and @iobureau were available.
It also can be made to look both like IO and 10 when rasterized down to a small size. Not terribly important, but having a simple symbol to go with the name is always a plus.
So, ladies and gentlemen, until we decide to rename ourselves
again we shall be known as the IO Bureau.
Bonus material
In case you want to try your hand at finding a good sounding, but otherwise random name, here are some pointers:1. Reverse dictionary
To search for words with specific ending.
2. List of last names
To make your own Bang & Olufsen.
3. Lists of gods
Quite a few solid leads there.
4. List of Intel project codenames
Skylake, Ice Lake, etc. Mostly geographical landmarks.
5. List of Microsoft project codenames
Not as consistent as Intel's, but still worth a read.
6. List of Solar System objects
Including a handy list of moons.
7. List of planets in science fiction
Examples of names made up by other people.
8. List of fantasy worlds
Same here.
Samples
-
Brand new
Long-running series on re/branding by UnderConsideration,
including longer reviews, quick mentions and smaller bits.
-
Origins of well-known company names
3M = Minnesota Mining and Manufacturing Company, etc.
-
YC list
A list of Y-Combinator funded startups, if only to see how
hard it really is to come up with a truly memorable name.
Longer reads
The Weird Science of Naming New ProductsAn NYT article that follows the naming of a very high-profile project. Perhaps with no directly actionable insight, but really interesting nonetheless.
Brand Names
This is a really really deep dive into scientific aspects behind naming and name perception. Fantastic resource... though a bit like trying to learn how to fry a steak by studying metallurgy and proteins :) Be warned that you may end up being even more confused than before.
Release 80 has been out for over a week now and with it
came the change to the
licensing model that replaced Professional and Personal options with new Pro and Basic licenses.
To smooth out the transition the change includes a "grace" transition period, during which legacy licenses are still being available for purchase over at /buy/legacy page.
Once the grace period is over, legacy licenses will be removed from sale and only new Pro and Basic licenses will be available.
The grace period is set to end this Wednesday, June 26th. To emphasize - All existing licenses will continue working just as they do now and new program versions will recognize and support both new and old license types. However it will no longer be possible to buy the old licenses.
To smooth out the transition the change includes a "grace" transition period, during which legacy licenses are still being available for purchase over at /buy/legacy page.
Once the grace period is over, legacy licenses will be removed from sale and only new Pro and Basic licenses will be available.
The grace period is set to end this Wednesday, June 26th. To emphasize - All existing licenses will continue working just as they do now and new program versions will recognize and support both new and old license types. However it will no longer be possible to buy the old licenses.
Release 80 will be introducing a reworked mechanism for
in-program checks for updates.
In particular, the UI will now make a clear distinction between
upgrade releases and maintenance patches.
For maintenance releases it will also put more stress on the importance of an update if it happens to resolve a high-impact issue. See "Critical patch" screenshot above. Additionally, the release notes are now displayed off the main view, giving them a bit more space if and when it's needed:
For maintenance releases it will also put more stress on the importance of an update if it happens to resolve a high-impact issue. See "Critical patch" screenshot above. Additionally, the release notes are now displayed off the main view, giving them a bit more space if and when it's needed:
Release 80 will be making a small but important change
to the default behavior of the exclusion/inclusion rules.
Starting with R80 all explicit folder exclusions will be final.
In plain English it means that excluding a folder by clicking on it in the middle pane of the Backup What window will now prevent any further rules from being applied to its contents:
For example, if there were an include *.dll rule defined
further below, it will now no longer pick up basebrd.dll
from Basebrd subfolder.
It is, of course, still possible to re-include any items from an excluded folder just like so:
Prior to this change excluding a folder wasn't
"final",
meaning that it merely set the default action for all sub-items
to exclude, and this action could still be overridden
by other rules.
New behavior will be enabled for all backup jobs created with R80 and newer releases. All existing jobs will continue to use their original filtering logic and work exactly as they did before. There is also a way to tell if a job uses the new or the old logic. Here's an older job:
And here's one with the new logic:
In plain English it means that excluding a folder by clicking on it in the middle pane of the Backup What window will now prevent any further rules from being applied to its contents:
For example, if there were an include *.dll rule defined
further below, it will now no longer pick up basebrd.dll
from Basebrd subfolder.
It is, of course, still possible to re-include any items from an excluded folder just like so:
Prior to this change excluding a folder wasn't
"final",
meaning that it merely set the default action for all sub-items
to exclude, and this action could still be overridden
by other rules.
New behavior will be enabled for all backup jobs created with R80 and newer releases. All existing jobs will continue to use their original filtering logic and work exactly as they did before. There is also a way to tell if a job uses the new or the old logic. Here's an older job:
And here's one with the new logic:
Release 80 is nearly here and, coincidentally, so is the Bvckup's 5th anniversary of its first production release back in 2014.
So it's only fitting that the program's logo is getting a refresh.
So it's only fitting that the program's logo is getting a refresh.
Alte Haas Grotesk
has been replaced by a considerably more expressive
Brutal Type,
slightly customized and
re-kerned.
The symbol has also been redrawn from a bastardized V to a proper and better balanced geometrical construction. The symbol now comes in three versions. One is shaded suitable for larger sizes, where the gradients do not get lost:
Second is the monochrome, which is a reduction of the
first version to just black and white:
The last one is simple, which is just the shape without
any embellishments as used for program icons and such.
While we are at it, it's probably worth sharing several other
options that didn't make the cut.
An alternative weave:
An alternative shading style:
Something with color:
Truth be told, the one of the left was my absolute
favorite, but it ended up looking like a royal mess
at smaller sizes. And smaller sizes are
important, because, you know... Windows.
The symbol has also been redrawn from a bastardized V to a proper and better balanced geometrical construction. The symbol now comes in three versions. One is shaded suitable for larger sizes, where the gradients do not get lost:
Second is the monochrome, which is a reduction of the
first version to just black and white:
The last one is simple, which is just the shape without
any embellishments as used for program icons and such.
While we are at it, it's probably worth sharing several other
options that didn't make the cut.
An alternative weave:
An alternative shading style:
Something with color:
Truth be told, the one of the left was my absolute
favorite, but it ended up looking like a royal mess
at smaller sizes. And smaller sizes are
important, because, you know... Windows.
It's easy to do things when they go as planned. It's when they go sideways is what makes everything much more exciting.
In other words, the next maintenance release includes changes to better handle several new edge cases.
In other words, the next maintenance release includes changes to better handle several new edge cases.
When reading a file, the program does that by requesting blocks of data, back to back:
Then, as nears the end of a file two things can happen. If the
file size is not a multiple of block size, the program will
receive back an incomplete block:
And if the file is perfectly aligned to the block size, the
program will see either a zero-sized block or an explicit
"end of file" notification or EOF for short.
Upon seeing either of these lovely events, the program will
know that it reached the end of the file and there's nothing
else left to copy.
There is however a nuance. In theory, the program may receive a partial block back even if it's not at the end of a file.
It is declared, that if this happens, it indicates a busy device or some other intermittent issue. The program is then welcome to retry the request or to read the remainder with a separate query.
In practice, this means a faulty storage device or a severely stressed system.
Further to that, very few program implement retrying on partial reads and OS vendors fully realize that.
So the overwhelmingly common practice is for the OS to just fail such requests, automatically meaning that a partial read = EOF.
As it turns out there do exist storage devices that will return partial blocks mid-file if ... the file is corrupted:
Technically, it makes some sense - this allows salvaging
at least some data from a corrupted block. Practically however
this is completely useless unless you are in a data recovery
business.
More specifically, the program will now check that a partial block is the exact tail end of a file and it will raise an alarm if it's not.
With the new release, if it detects an EOF of the source file, it will then check any read requests for blocks beyond this EOF location and ensure that they too show the same EOF.
If the source file is being actively modified and it is growing very quickly, the program may see an EOF followed by a actual data block, because the latter gets added exactly between the EOF and the second requests.
This is now also detected and reported accordingly.
When this happens, Bvckup 2 will see an EOF first. It will realize that EOF is for a smaller file size than it saw when the copying started. So it will re-check the size and the current file size will now be even less than the recorded EOF.
Clearly, this is a mess and it is too now detected and logged. Will be in release 79.26.
Then, as nears the end of a file two things can happen. If the
file size is not a multiple of block size, the program will
receive back an incomplete block:
And if the file is perfectly aligned to the block size, the
program will see either a zero-sized block or an explicit
"end of file" notification or EOF for short.
Upon seeing either of these lovely events, the program will
know that it reached the end of the file and there's nothing
else left to copy.
There is however a nuance. In theory, the program may receive a partial block back even if it's not at the end of a file.
It is declared, that if this happens, it indicates a busy device or some other intermittent issue. The program is then welcome to retry the request or to read the remainder with a separate query.
In practice, this means a faulty storage device or a severely stressed system.
Further to that, very few program implement retrying on partial reads and OS vendors fully realize that.
So the overwhelmingly common practice is for the OS to just fail such requests, automatically meaning that a partial read = EOF.
Think different
However "overwhelmingly common" is not 100%.As it turns out there do exist storage devices that will return partial blocks mid-file if ... the file is corrupted:
Technically, it makes some sense - this allows salvaging
at least some data from a corrupted block. Practically however
this is completely useless unless you are in a data recovery
business.
Partial mid-file reads
The first edge case that is now being explicitly handled by Bvckup 2 is that of reading a partial block mid-file.More specifically, the program will now check that a partial block is the exact tail end of a file and it will raise an alarm if it's not.
Past-EOF reads
As you probably know Bvckup 2 uses async IO and it normally has several read/write requests "in flight".With the new release, if it detects an EOF of the source file, it will then check any read requests for blocks beyond this EOF location and ensure that they too show the same EOF.
Rapidly growing files
There are cases when being able past EOF marker is actually OK.If the source file is being actively modified and it is growing very quickly, the program may see an EOF followed by a actual data block, because the latter gets added exactly between the EOF and the second requests.
This is now also detected and reported accordingly.
Rapidly shrinking files
There's also an inverse case - the source file being very quickly shrunk down.When this happens, Bvckup 2 will see an EOF first. It will realize that EOF is for a smaller file size than it saw when the copying started. So it will re-check the size and the current file size will now be even less than the recorded EOF.
Clearly, this is a mess and it is too now detected and logged. Will be in release 79.26.
Next maintenance release adds support for indicating if
a backup may be in an indeterminate state due to Bvckup
2 not being shut down cleanly on its previous run.
In particular, this is meant to highlight backups that were active when the host computer ran into a blue screen or experienced a sudden power loss. Prior to this change the engine was already recording whether a backup job was running, used that to detect incomplete runs and to run a bit of house maintenance.
With this release, the engine also records at which exact phase the job was and then uses this information to purge any cached information that may no longer be accurate.
Most importantly this includes the file index snapshot, a.k.a. the "destination snapshot". If the job was already in the active phase, making changes to the backup location, the engine will discard the snapshot and rebuild it from scratch on the next run.
Will be in 79.25.
In particular, this is meant to highlight backups that were active when the host computer ran into a blue screen or experienced a sudden power loss. Prior to this change the engine was already recording whether a backup job was running, used that to detect incomplete runs and to run a bit of house maintenance.
With this release, the engine also records at which exact phase the job was and then uses this information to purge any cached information that may no longer be accurate.
Most importantly this includes the file index snapshot, a.k.a. the "destination snapshot". If the job was already in the active phase, making changes to the backup location, the engine will discard the snapshot and rebuild it from scratch on the next run.
Will be in 79.25.
As you may know, Bvckup 2 can run in a so-called service mode
which keeps the backups running even if the program itself is not
started or when no one is logged in at the computer.
The program can be switched between the service and desktop modes on the fly and this functionality has been improved for both speed and robustness in the upcoming 79.23 release.
This is different from before where any such recovery, while still always possible, needed to be done manually.
2. Switching back to the desktop mode now displays a % done for the slow part of the process ( see below ).
3. The slow part itself was reworked for speed and it was also fully parallelized. The more CPU cores you have, the faster it goes. It can still be slow, mind you, but it is less so now.
4. Finally, it's now possible to cancel the switch if desired.
One is the backup engine, which is a module that handles scheduling and execution of backup jobs, tracking of storage devices, dispatching of email alerts, etc.
Another is the user interface module - it talks to the engine, retrieves and displays a list of jobs and allows controlling and configuring them by sending requests to the engine.
Both modules operate independently from each other and communicate through a formal messaging protocol. Similarly their configuration data is also not mixed, with each module using their own directory to store all its data. When the program is switched to the service mode, the engine module is simply shut down and re-launched as a system service.
Switching back to the desktop mode reverses that - it stops the service and starts the engine inside the UI process. Easy.
But there is one nuance. There always is... When running as a service, the engine runs under a dedicated user account. This account is a member of the Local Admins group, so it has no problem accessing engine's config files even though they were created under a desktop user account.
However, any files it creates are owned by the Local Admins and they are not fully accessible by the original (desktop) process.
So, going back to the desktop mode requires another step -
we need to reset the owner on all engine config files.
This issue is further compounded by the fact that the number of
files in the engine config can be really quite large.
This is because the engine config is where settings for all backup jobs are kept, and this includes the blockhash data of files copied using the delta copier.
The more files are delta-copied, the larger the engine config will be and the longer it will take for the owner reset to complete.
So that is the slow part.
The program can be switched between the service and desktop modes on the fly and this functionality has been improved for both speed and robustness in the upcoming 79.23 release.
Changes
1. The program can now automatically recover from any issues stemming from being foricibly closed/killed midway through the mode switch process.This is different from before where any such recovery, while still always possible, needed to be done manually.
2. Switching back to the desktop mode now displays a % done for the slow part of the process ( see below ).
3. The slow part itself was reworked for speed and it was also fully parallelized. The more CPU cores you have, the faster it goes. It can still be slow, mind you, but it is less so now.
4. Finally, it's now possible to cancel the switch if desired.
The slow part
Under the hood Bvckup 2 is essentially a pair of programs sharing a single executable.One is the backup engine, which is a module that handles scheduling and execution of backup jobs, tracking of storage devices, dispatching of email alerts, etc.
Another is the user interface module - it talks to the engine, retrieves and displays a list of jobs and allows controlling and configuring them by sending requests to the engine.
Both modules operate independently from each other and communicate through a formal messaging protocol. Similarly their configuration data is also not mixed, with each module using their own directory to store all its data. When the program is switched to the service mode, the engine module is simply shut down and re-launched as a system service.
Switching back to the desktop mode reverses that - it stops the service and starts the engine inside the UI process. Easy.
But there is one nuance. There always is... When running as a service, the engine runs under a dedicated user account. This account is a member of the Local Admins group, so it has no problem accessing engine's config files even though they were created under a desktop user account.
However, any files it creates are owned by the Local Admins and they are not fully accessible by the original (desktop) process.
So, going back to the desktop mode requires another step -
we need to reset the owner on all engine config files.
This issue is further compounded by the fact that the number of
files in the engine config can be really quite large.
This is because the engine config is where settings for all backup jobs are kept, and this includes the blockhash data of files copied using the delta copier.
The more files are delta-copied, the larger the engine config will be and the longer it will take for the owner reset to complete.
So that is the slow part.
Work-in-progress for the new About window
for Release 80.
Incidentally, this is what this blog was originally
about - posting bits of work, UI sketches, coding
details - all the daily grind that goes into making
a seemingly simple piece of software and that stays
largely hidden once the result gets shipped.
When you see a product that is simple and logical, it's very rarely that it came together that way on its own. More often than not it's a result of a long iterative process of trying this, that and third, going back, tweaking, revising and retrying, and eventually converging to a stable point.
So here's something to acknowledge all other 20-plus versions of the About window that were also really not bad at all, but didn't quite make the cut! :)
When you see a product that is simple and logical, it's very rarely that it came together that way on its own. More often than not it's a result of a long iterative process of trying this, that and third, going back, tweaking, revising and retrying, and eventually converging to a stable point.
So here's something to acknowledge all other 20-plus versions of the About window that were also really not bad at all, but didn't quite make the cut! :)
All manually initiated checks for updates will now see
the latest available version.
This applies to the case when a recently released update is going through a gradual roll-out phase and it's not immediately visible to all installations.
This applies to the case when a recently released update is going through a gradual roll-out phase and it's not immediately visible to all installations.
Previously, both manual and scheduled (automated) checks for
new versions were treated the same, so it wasn't possible for
an installation to retrieve a freshly released update until
its IP address was included in the newest channel
coverage.
With this change, gradual roll-out provisions now apply to automated checks only. Manual checks will now always be served through the newest channel. They will also override Symantec-specific processing if one applies. For detailed description of gradual roll-outs, update channels, Symantec-specific handling and available configuration options see this post.
With this change, gradual roll-out provisions now apply to automated checks only. Manual checks will now always be served through the newest channel. They will also override Symantec-specific processing if one applies. For detailed description of gradual roll-outs, update channels, Symantec-specific handling and available configuration options see this post.
Preview of the Pro Support option for new Pro licenses.
Pro licenses will come with a year of Pro Support - an option that enables access to the program's upgrade releases (R80, R81, etc.) and priority tech support.
Basic licenses will not have this option, meaning that they can only be used with a program release that was current at the moment the license was issued. Additionally, Basic licenses will be supported through public support forum only.
Maintenance updates ("patches") will be provided for all license types, unconditionally. In other words, the bugfixes are on us.
After a year, Pro Support can be extended for another year. This extension is completely optional and it's not automated in any way. Extension fee is 20% of the original license cost.
See the original announcement for a bit more context.
Pro licenses will come with a year of Pro Support - an option that enables access to the program's upgrade releases (R80, R81, etc.) and priority tech support.
Basic licenses will not have this option, meaning that they can only be used with a program release that was current at the moment the license was issued. Additionally, Basic licenses will be supported through public support forum only.
Maintenance updates ("patches") will be provided for all license types, unconditionally. In other words, the bugfixes are on us.
After a year, Pro Support can be extended for another year. This extension is completely optional and it's not automated in any way. Extension fee is 20% of the original license cost.
See the original announcement for a bit more context.
Release 79.18 ships with a brand new storage tracking module.
Storage tracking module is responsible for maintaining detailed list of storage devices, file system volumes and drives.
The backup engine uses this information for tracking removable drives, running backups in response to drive arrivals, adapting bulk IO profile to the device specifics and for several other tasks.
Storage tracking module is responsible for maintaining detailed list of storage devices, file system volumes and drives.
The backup engine uses this information for tracking removable drives, running backups in response to drive arrivals, adapting bulk IO profile to the device specifics and for several other tasks.
In earlier releases volume information was queried in an ad-hoc
fashion, when it was needed or when it became stale.
This release gathers all this functionality in a single place, adds support for tracking device information and improves detection of changes to network drives.
It also reworks processing of Windows device notifications, which are the "pings" sent by the OS when devices arrive or depart, volumes get mounted, locked, formatted, renamed, etc.
The program is now capable of sourcing these notifications from more than one place, which gives it a fallback option should the default notification source stop working ... at least until you get to reboot that WS2012 instance.
For example, a system service will never receive any alerts for TC volumes being mounted or unmounted.
These quirks are transparently accommodated. It all just works as expected.
In fact, the plan has always been to use Diskovery as a testbed for all things storage-related and then fold them into Bvckup 2.
And so we are now at exactly that point - all hard-earned Diskovery smarts are now being merged into Bvckup 2.
Next, we'll be enabling the monitoring of device SMART status and tracking of SMART attributes. The code that supports this is ready to go, it's now only a matter of pacing the merge through its stages.
We are also on track for adding previously announced UI support for the storage monitor to the next major release, R80.
Stay tuned for updates.
This release gathers all this functionality in a single place, adds support for tracking device information and improves detection of changes to network drives.
It also reworks processing of Windows device notifications, which are the "pings" sent by the OS when devices arrive or depart, volumes get mounted, locked, formatted, renamed, etc.
The program is now capable of sourcing these notifications from more than one place, which gives it a fallback option should the default notification source stop working ... at least until you get to reboot that WS2012 instance.
TrueCrypt
Storage tracking module includes explicit support for TrueCrypt volumes. This is notable because TrueCrypt does not implement all required volume notifications, making it impossible to track volume activity via standard means.For example, a system service will never receive any alerts for TC volumes being mounted or unmounted.
These quirks are transparently accommodated. It all just works as expected.
Under the hood
New code comes from Diskovery, a storage inspection tool that we first released back in 2016 and have been steadily improving ever since.In fact, the plan has always been to use Diskovery as a testbed for all things storage-related and then fold them into Bvckup 2.
And so we are now at exactly that point - all hard-earned Diskovery smarts are now being merged into Bvckup 2.
Staged merge
First, with this release the program gets access to raw device information, which is vastly more detailed and accurate than what's available through conventional Windows APIs.Next, we'll be enabling the monitoring of device SMART status and tracking of SMART attributes. The code that supports this is ready to go, it's now only a matter of pacing the merge through its stages.
We are also on track for adding previously announced UI support for the storage monitor to the next major release, R80.
Stay tuned for updates.
With the next maintenance release it will be possible to
quickly reload a backup job after adding manual overrides
for program's many hidden options.
* As with other advanced options, the Reload item is shown only if the context menu is opened with Ctrl held down. Will be in 79.15.
* As with other advanced options, the Reload item is shown only if the context menu is opened with Ctrl held down. Will be in 79.15.
Support for new Basic licenses and their respective
restrictions, coming up in the next feature release (R80).
Looked a various options for enforcing restrictions
of simpler licenses and settled on the same
approach currently used
for expired trials.
That is, all and any features in any backup job can be used under any license, but if a feature is not covered by the license, then the backup job will be disabled after every run.
It's still perfectly possible to enable it back, but the only way to do this is to manually select the job and click on the Go button.
That is, all and any features in any backup job can be used under any license, but if a feature is not covered by the license, then the backup job will be disabled after every run.
It's still perfectly possible to enable it back, but the only way to do this is to manually select the job and click on the Go button.
It is now possible to pre-compute delta copying state
for large files when taking over existing backups with Bvckup 2.
The way the delta copying
works is that it relies
on block hashes saved from the last file update to
detect which parts of the file changed since then
and copy over only these modified blocks.
This means that the actual delta part of the copying kicks in only on the second update, because when we are copying file for the first time, we don't have the block hashes yet. Once the file is copied, we have the hashes and all set to do the delta updating. This inability to detect block changes on the first run is not a big deal. Most of the times we are adding new files to a backup and these need to be copied in full anyway, so we do just that and initialize their delta state at the same time.
However, when taking over existing backups, it presents a slight problem. Consider the case when you spent a day copying a very large file over a slow link to an off-site storage. Then, you created a backup job to handle all further updates.
If you are to touch the file now and run the backup, there'll be no block hashes yet, so the file will be re-copied in full. If only we could somehow initialize the delta copying state without copying a file...
That is, if you put an existing backup under Bvckup's control, you can now flip a switch, do a quick run and then have all files ready for delta copying on their first update.
† Quick here assumes that your source is local, reads are cheap and writes are expensive, which is how the setup should be for making full use of delta copying benefits. The screencap above shows taking over a Debian VM backup that is initially created by just copy-pasting VM folder in Windows Explorer.
Running a backup executes delta pre-comp for .vmdk files and patches up "created" times on backup items - all without writing a single byte to destination.
This means that the actual delta part of the copying kicks in only on the second update, because when we are copying file for the first time, we don't have the block hashes yet. Once the file is copied, we have the hashes and all set to do the delta updating. This inability to detect block changes on the first run is not a big deal. Most of the times we are adding new files to a backup and these need to be copied in full anyway, so we do just that and initialize their delta state at the same time.
However, when taking over existing backups, it presents a slight problem. Consider the case when you spent a day copying a very large file over a slow link to an off-site storage. Then, you created a backup job to handle all further updates.
If you are to touch the file now and run the backup, there'll be no block hashes yet, so the file will be re-copied in full. If only we could somehow initialize the delta copying state without copying a file...
Enter delta state pre-comp
Release 79.13 adds support for pre-computing block hashes for all files in the backup that qualify for delta copying but not yet have the delta state.That is, if you put an existing backup under Bvckup's control, you can now flip a switch, do a quick run and then have all files ready for delta copying on their first update.
† Quick here assumes that your source is local, reads are cheap and writes are expensive, which is how the setup should be for making full use of delta copying benefits. The screencap above shows taking over a Debian VM backup that is initially created by just copy-pasting VM folder in Windows Explorer.
Running a backup executes delta pre-comp for .vmdk files and patches up "created" times on backup items - all without writing a single byte to destination.
Next release is adding support for displaying
absolute date/time of the next backup run.
When the next run falls on today, the UI will show just the time. For runs falling on tomorrow it will say "tomorrow" plus the time. Otherwise it will show both the date and time. Another change is a reshuffle of the About window:
There's now Check for updates option
next to the version info, which is automatically
displayed as Newer version is available
notice if there was one seen at some point earlier
on:
The window also got a separate Validity field,
which is used for displaying remaining
trial time and also for confirming that
all production licenses are in fact perpetual.
When the next run falls on today, the UI will show just the time. For runs falling on tomorrow it will say "tomorrow" plus the time. Otherwise it will show both the date and time. Another change is a reshuffle of the About window:
There's now Check for updates option
next to the version info, which is automatically
displayed as Newer version is available
notice if there was one seen at some point earlier
on:
The window also got a separate Validity field,
which is used for displaying remaining
trial time and also for confirming that
all production licenses are in fact perpetual.
Next release is adding support for dynamically changing source
and backup paths using run-time variables.
In particular, it's now possible to direct backups into a folder that is named after current date, hour, day of the week, etc.
It's been possible to do this using a bit of pre-backup scripting and it will, of course, remain the path of a true Jedi. For the rest of us though there's now a (much) easier way.
In particular, it's now possible to direct backups into a folder that is named after current date, hour, day of the week, etc.
It's been possible to do this using a bit of pre-backup scripting and it will, of course, remain the path of a true Jedi. For the rest of us though there's now a (much) easier way.
If you want to have a day worth of hourly backups, you'd
just point them at ...\%HOUR%, and just as if by
magic backups will go into ...\00, ...\01,
etc. depending on the current hour.
For a year worth of weekly snapshots just use %WEEK%.
%DAY_OF_YEAR% will give you a year of dailies.
%DAY_OF_WEEK% - a week of dailies, and so forth and so on. One caveat is that when the backup destination switches to a new folder, it will invalidate destination snapshot and require a full destination rescan.
Similarly, it will also invalidate delta copying state, so actively modified files will end up being recopied in full on their next update. Finally, since no good feature is truly complete without a way to shoot oneself in a foot, be advised that the same set of variables can be used in ther source path as well.
For a year worth of weekly snapshots just use %WEEK%.
%DAY_OF_YEAR% will give you a year of dailies.
%DAY_OF_WEEK% - a week of dailies, and so forth and so on. One caveat is that when the backup destination switches to a new folder, it will invalidate destination snapshot and require a full destination rescan.
Similarly, it will also invalidate delta copying state, so actively modified files will end up being recopied in full on their next update. Finally, since no good feature is truly complete without a way to shoot oneself in a foot, be advised that the same set of variables can be used in ther source path as well.
An option for running Bvckup 2 at a low system priority has been around
for a long time.
This option is primarily meant for machines that are already running close to their capacity, often due to using newer Windows versions on older hardware.
Enabling it caused the program to set its process priority to so-called Below Normal level and that throttled down both its CPU and disk run-time allowances.
At least, that's the theory.
With an introduction of the ultra copier, this option has become less effective, because the program is now capable of pushing disks to their limits even on a low system priority.
For this reason the next maintenance release (79.6) will extend this option to explicitly throttle down program's disk activity.
This option is primarily meant for machines that are already running close to their capacity, often due to using newer Windows versions on older hardware.
Enabling it caused the program to set its process priority to so-called Below Normal level and that throttled down both its CPU and disk run-time allowances.
At least, that's the theory.
With an introduction of the ultra copier, this option has become less effective, because the program is now capable of pushing disks to their limits even on a low system priority.
For this reason the next maintenance release (79.6) will extend this option to explicitly throttle down program's disk activity.
Internally, this is done very simply.
When enabled, this option will cause the backup engine to use a much smaller count of memory buffers to move data from A to B.
* It is also perfectly possible to throttle disk load
dynamically
by looking at the actual disk performance and by yielding IO
capacity to other programs when required. This is planned and
it will be coming with later releases.
When enabled, this option will cause the backup engine to use a much smaller count of memory buffers to move data from A to B.
* It is also perfectly possible to throttle disk load
dynamically
by looking at the actual disk performance and by yielding IO
capacity to other programs when required. This is planned and
it will be coming with later releases.
As you might've heard (and more than once), the next
release will coincide with a change to our licensing model.
Existing Personal and Professional licenses will be phased out in favor of Basic, Pro and Server tiers.
The principal change is that licensing options will now differ in what the program can do as opposed to what it can be used for. Licenses will remain perpetual and require no paid renewal. The secondary change is that we'll also start making an explicit difference between feature releases and maintenance updates.
Feature releases are major versions that introduce
new features and significant improvements - R79, R80, etc.
Maintenance updates are minor releases that patch issues and fix bugs. In the current versioning notation these are indicated by sub-versions - 79.1, 79.2, etc.
Existing Personal and Professional licenses will be phased out in favor of Basic, Pro and Server tiers.
The principal change is that licensing options will now differ in what the program can do as opposed to what it can be used for. Licenses will remain perpetual and require no paid renewal. The secondary change is that we'll also start making an explicit difference between feature releases and maintenance updates.
Feature releases are major versions that introduce
new features and significant improvements - R79, R80, etc.
Maintenance updates are minor releases that patch issues and fix bugs. In the current versioning notation these are indicated by sub-versions - 79.1, 79.2, etc.
The change itself is that we will continue providing
maintenance updates for free, while the access to
new features will be free for one year
and an optional paid extension after that.
The one year starts from the moment of licensing an
installation. All major releases within one year
window are included, each of which coming with a
lifetime of maintenance updates.
The paid-for option will extend the access to feature releases
for another year and will be priced as a smaller percentage of
the original license cost.
Basically, the bugfixes are on us. The new features, should you need them, are at small cost after the first year. If you don't find any new features useful, you will be able to keep on using your existing release that will remain fully functional and supported. Finally, all legacy licenses (of Personal and Professional types) will continue receiving all feature releases free of charge, just as they do now. Questions or comments? Post them here or send them privately.
The one year starts from the moment of licensing an
installation. All major releases within one year
window are included, each of which coming with a
lifetime of maintenance updates.
The paid-for option will extend the access to feature releases
for another year and will be priced as a smaller percentage of
the original license cost.
Basically, the bugfixes are on us. The new features, should you need them, are at small cost after the first year. If you don't find any new features useful, you will be able to keep on using your existing release that will remain fully functional and supported. Finally, all legacy licenses (of Personal and Professional types) will continue receiving all feature releases free of charge, just as they do now. Questions or comments? Post them here or send them privately.
Rolled out a web dashboard option
for reseller partners to allow assigning purchases to
their customers.
This option passes control over purchased activation codes and licenses to a customer, granting them access to the dashboard, but without exposing any transaction details. This also conceals code and license details from the reseller, leaving only purchase details visible. To each their own. * We are always looking for new distributor partners. If you are a VAR, an IT service provider or a system integrator, drop us a line and let's talk.
This option passes control over purchased activation codes and licenses to a customer, granting them access to the dashboard, but without exposing any transaction details. This also conceals code and license details from the reseller, leaving only purchase details visible. To each their own. * We are always looking for new distributor partners. If you are a VAR, an IT service provider or a system integrator, drop us a line and let's talk.
Here's a little fun data visualization.
This is a heatmap of a day worth of changes to a large TrueCrypt container file that was backed up with delta copying and all block changes diligently saved as a bitmap. Block changed - its bit is 1, unchanged - the bit is 0.
Container file is 256 GB, the block size is 64 KB, which gives us 4M of blocks to the total of 512 KB when packed into a bitmap. With the bitmap at hand, we head over to binvis.io to generate a 2D entropy map using a Hilbert curve fill, overlay with absolute changes, then pass it through some cosmetic filters and, voilà - a simple way to see where the delta copying gets its speed from.
The brighter parts are the 10 Gigs worth of changes, the dark background is 246 GB of unchanged space... meaning that we ended up copying under 4% of the file and spent a fraction of time doing that.
* For a detailed tour of delta copying see here.
This is a heatmap of a day worth of changes to a large TrueCrypt container file that was backed up with delta copying and all block changes diligently saved as a bitmap. Block changed - its bit is 1, unchanged - the bit is 0.
Container file is 256 GB, the block size is 64 KB, which gives us 4M of blocks to the total of 512 KB when packed into a bitmap. With the bitmap at hand, we head over to binvis.io to generate a 2D entropy map using a Hilbert curve fill, overlay with absolute changes, then pass it through some cosmetic filters and, voilà - a simple way to see where the delta copying gets its speed from.
The brighter parts are the 10 Gigs worth of changes, the dark background is 246 GB of unchanged space... meaning that we ended up copying under 4% of the file and spent a fraction of time doing that.
* For a detailed tour of delta copying see here.
By popular request - an option to suppress
tray icon blinking
when a periodic backup job misses its scheduled run.
This covers all causes - the computer being off, sleeping or storage devices missing or being inaccessible.
Will be in Release 79.
This covers all causes - the computer being off, sleeping or storage devices missing or being inaccessible.
Will be in Release 79.
This is the final part of the ultra copier tour
and it covers a much improved support for
resuming copying after cancellations and
IO errors.
It works just as you'd expect - if a copying of a file is interrupted part-way through, then on the next run the copying will resume from around where it was before. The most interesting bit here is that this works even if the copy is aborted by disconnecting a drive, in the middle of a write. More often than not this will cause data corruption, but the program is now capable of detecting the extent of such corruption and correcting it without recopying the file in full.
Second, the source file must remain unchanged between the attempts (as witnessed by its size and timestamps). Because if the source changes, we don't know where the change was, so we must start from the beginning.
Third, the destination file is, obviously, expected to also stay the same, but that's a more general requirement for the delta copying as a whole. Touching target file between the runs will automatically invalidate its delta state and trigger a re-copying.
* For people lacking certain excitement in their lives it is possible to suppress the last check. Inquire within for details. In addition to recording how far it went in the source file, the ultra copier also remembers the reason for aborting a copy. Three main reasons are the read errors, write errors and user cancellations.
In case of read errors and user cancellations the copier gets a chance to shut down the copying process in an orderly fashion. This ensures a consistent state of the backup copy, so in these cases the copying is always resumable.
In case of write errors it depends.
So when we are resuming after a write failure, we need a way to ensure that the state of the destination file is consistent with our (or rather delta copier's) view of it.
Ultra does that by keeping track of file ranges that it successfully updated on the last run in a form of a write log.
However since we are resuming after a write
failure, some of these writes may actually
have never hit the storage media even though
they were reported as "completed". This is due to all
the caching, lazy-writing and outright lying that
modern drives do to improve their performance.
For this reason when resuming after a write
failure the ultra will go through the write
log, read respective parts of the backup copy
and check their hashes against
those stored in the delta state. If there's
a match, we are in clear. Otherwise, it's a
block we need to re-copy.
The earliest non-matching block from the write log gives us an adjusted resume point. Simple... with a hint of elegance.
2. Faster delta copying #
3. Resuming support and error recovery
All courtesy of the new ultra copier. Coming to an update server near you in a few days...
It works just as you'd expect - if a copying of a file is interrupted part-way through, then on the next run the copying will resume from around where it was before. The most interesting bit here is that this works even if the copy is aborted by disconnecting a drive, in the middle of a write. More often than not this will cause data corruption, but the program is now capable of detecting the extent of such corruption and correcting it without recopying the file in full.
† Some conditions apply
First, this is enabled only for files that are delta-copied, because all resume-related data is saved with the file's delta state.Second, the source file must remain unchanged between the attempts (as witnessed by its size and timestamps). Because if the source changes, we don't know where the change was, so we must start from the beginning.
Third, the destination file is, obviously, expected to also stay the same, but that's a more general requirement for the delta copying as a whole. Touching target file between the runs will automatically invalidate its delta state and trigger a re-copying.
* For people lacking certain excitement in their lives it is possible to suppress the last check. Inquire within for details. In addition to recording how far it went in the source file, the ultra copier also remembers the reason for aborting a copy. Three main reasons are the read errors, write errors and user cancellations.
In case of read errors and user cancellations the copier gets a chance to shut down the copying process in an orderly fashion. This ensures a consistent state of the backup copy, so in these cases the copying is always resumable.
In case of write errors it depends.
Recovering from write errors
If we are to yank out an USB drive while it is writing data, the state of the backup copy will be somewhat uncertain. Ditto for the network copies when the router decides it had enough for the day and dies.So when we are resuming after a write failure, we need a way to ensure that the state of the destination file is consistent with our (or rather delta copier's) view of it.
Ultra does that by keeping track of file ranges that it successfully updated on the last run in a form of a write log.
However since we are resuming after a write
failure, some of these writes may actually
have never hit the storage media even though
they were reported as "completed". This is due to all
the caching, lazy-writing and outright lying that
modern drives do to improve their performance.
For this reason when resuming after a write
failure the ultra will go through the write
log, read respective parts of the backup copy
and check their hashes against
those stored in the delta state. If there's
a match, we are in clear. Otherwise, it's a
block we need to re-copy.
The earliest non-matching block from the write log gives us an adjusted resume point. Simple... with a hint of elegance.
So there you have it...
1. Faster bulk copying #2. Faster delta copying #
3. Resuming support and error recovery
All courtesy of the new ultra copier. Coming to an update server near you in a few days...
This is a part 2 of a tour of the Bvckup's brand new
copying engine dubbed the ultra copier.
Part 1 covered improvements to the engine's bulk copying and in this post we will look at changes to its incremental file updater, a.k.a. the delta copier.
When files change, more often than not they change only
here and there, in an isolated manner. This makes
delta copying a godsend for backing up all sorts of large,
slowly changing files from large RAW
photos, to database files, to VM images and
encrypted file containers.
If you are familiar with rsync, it is based on the same idea to derive the same speed benefits. However where Bvckup 2 differs from rsync is how it does change detection.
We still end up reading a file in full, but we save time on being selective with writing, which is usually an expensive thing to do. Bvckup 2 works by saving block hashes between the runs and then using these as a reference point on the next update.
This requires only basic IO access to source/backup copies and some local space to store the block hashes. That is, the delta copying works with any storage devices that are accessible for reading/writing.
* In comparison, rsync requires a copy of itself to be running on receiving end (that's when copying over the network). It doesn't require any interim storage, but it ends up reading both source and target files in full. This makes rsync poorly suited for local copying and it also explains the r in its name.
With delta copying if we ever run into a hash collision, we will end up skipping a modified block ... which is something we most certainly don't want to be doing.
Avoiding false negatives comes down to reducing the chances of running into them on a real-world data. This is done by using cryptographic hashing for its avalanche effect - a property that causes a hash to change drastically in response to small changes to the input.
Using longer hashes is another option.
And using multiple hashes computed with different algorithms is yet another one.
This yields 20 bytes of hash per block, meaning that the delta state of a file is about 0.06% of its size.
The purpose of this hash was to catch cases when no modified blocks were detected but the full-file hash changed.
That was *not* a very good idea. It added very little in terms of the false negative protection and it created a bottleneck.
See, SHA-1 can't be parallelized. In fact, very few cryptographic hash functions can be and they all expect to be fed blocks in sequence, one by one.
This means that performance of the delta copier was effectively bounded by the speed of SHA-1 on a single CPU core.
For mechanical drives it's usually not an issue, because they are quite slow. But SSDs and NVMe drives can be read faster than we can hash what we get from them:
* These are read rates for the drives and bulk rates for SHA-1 using a well-optimized implementation.
So the full-file hash is a bottleneck and it should be eliminated, which is exactly what the ultra copier in delta mode does.
Per-block hash size is now 32 bytes - 16 from Blake2b, 8 from xxHash and 8 from Spooky, but the blocks are twice as large, so the ratio of delta-state / file-size is actually lower, ~ 0.05%.
Non-crypto hashes, xxHash and Spooky, are running at 10.8 GB/s and 11.7 GB/s respectively. That's gigabytes per second, so these hashes effectively come free of charge.
But more importantly, we are no longer constricted by the full-file hash, so we can push these 762 MB/s per core.
* Numbers are obviously specific to our test machine, but they give a good idea of relative performance of the algorithms. The net effect of these changes is that the delta copier now can fully saturate an IO of a very fast drive while using just a small handful of hashing threads.
This in turn means that delta copying now comes with virtually no performance impact even when it ends up copying files in full. It basically goes as fast as a regular bulk copier.
Copying file afresh now runs at the write speed of the target drive and updating it afterwards (with little to no changes) runs at the read speed of the source drive.
Needless to say, that's a huge improvement. Next up is the part 3 of the ultra copier series. It covers resuming of interrupted copies and fast error recovery.
Part 1 covered improvements to the engine's bulk copying and in this post we will look at changes to its incremental file updater, a.k.a. the delta copier.
Quick recap
Delta copying is a way of updating files whereby only blocks that changed since the last update are copied over.
When files change, more often than not they change only
here and there, in an isolated manner. This makes
delta copying a godsend for backing up all sorts of large,
slowly changing files from large RAW
photos, to database files, to VM images and
encrypted file containers.
If you are familiar with rsync, it is based on the same idea to derive the same speed benefits. However where Bvckup 2 differs from rsync is how it does change detection.
Detecting changes
The recipe is simple - split the file in large-ish blocks, compute block hashes and then see if the hash changed from the last time we looked at the file. If it did, we got a modified block, so we copy it over.We still end up reading a file in full, but we save time on being selective with writing, which is usually an expensive thing to do. Bvckup 2 works by saving block hashes between the runs and then using these as a reference point on the next update.
read block
hash block
load hash from the last time
if (new-hash ≠ old-hash) then
write block
save new-hash
hash block
load hash from the last time
if (new-hash ≠ old-hash) then
write block
save new-hash
This requires only basic IO access to source/backup copies and some local space to store the block hashes. That is, the delta copying works with any storage devices that are accessible for reading/writing.
* In comparison, rsync requires a copy of itself to be running on receiving end (that's when copying over the network). It doesn't require any interim storage, but it ends up reading both source and target files in full. This makes rsync poorly suited for local copying and it also explains the r in its name.
False negatives
It is possible that two different blocks of data will have the same hash. This is a so-called hash collision.With delta copying if we ever run into a hash collision, we will end up skipping a modified block ... which is something we most certainly don't want to be doing.
Avoiding false negatives comes down to reducing the chances of running into them on a real-world data. This is done by using cryptographic hashing for its avalanche effect - a property that causes a hash to change drastically in response to small changes to the input.
Using longer hashes is another option.
And using multiple hashes computed with different algorithms is yet another one.
Old delta setup
From its very first public release and up to Release 79 the program used the following delta copying setup:| Block size | 32KB |
| Block hashes | MD5 + a variant of CRC32 |
| Full-file hash | SHA-1 |
This yields 20 bytes of hash per block, meaning that the delta state of a file is about 0.06% of its size.
Full-file hash
In addition to hashing individual blocks, delta copier was also computing a SHA-1 hash of the whole file and storing it with the delta state.The purpose of this hash was to catch cases when no modified blocks were detected but the full-file hash changed.
That was *not* a very good idea. It added very little in terms of the false negative protection and it created a bottleneck.
See, SHA-1 can't be parallelized. In fact, very few cryptographic hash functions can be and they all expect to be fed blocks in sequence, one by one.
This means that performance of the delta copier was effectively bounded by the speed of SHA-1 on a single CPU core.
For mechanical drives it's usually not an issue, because they are quite slow. But SSDs and NVMe drives can be read faster than we can hash what we get from them:
| HDD | Seagate ST9500325AS | 70.5 MB/s |
| HDD | WDC WD5000LPVX | 118.8 MB/s |
| SHA-1 | Intel i3 @ 2.53GHz | 332.4 MB/s |
| SHA-1 | Xeon E5 v4 @ 2.10GHz | 380.1 MB/s |
| SSD | Samsung T3, USB 3.0 | 429.7 MB/s |
| SHA-1 | Intel i5 @ 2.70GHz | 520.9 MB/s |
| SSD | Samsung 860 Pro | 534.8 MB/s |
| NVMe | Samsung MZVPV512HDGL | 2157.9 MB/s |
* These are read rates for the drives and bulk rates for SHA-1 using a well-optimized implementation.
So the full-file hash is a bottleneck and it should be eliminated, which is exactly what the ultra copier in delta mode does.
New delta setup
Starting with Release 79 the delta copying uses a larger block size, replaces MD5 with Blake2b and compliments it with two weaker, but very fast hashes called xxHash and spooky.| Block size | 64KB |
| Block hashes | Blake2b + xxHash + Spooky |
| Full-file hash | - |
Per-block hash size is now 32 bytes - 16 from Blake2b, 8 from xxHash and 8 from Spooky, but the blocks are twice as large, so the ratio of delta-state / file-size is actually lower, ~ 0.05%.
Impact on performance
Block hashing is now noticeably faster:| MD5 | 595.4 MB/s |
| SHA-1 | 561.9 MB/s |
| SHA-256 | 247.4 MB/s |
| SHA-512 | 391.4 MB/s |
| Blake2b | 762.3 MB/s |
Non-crypto hashes, xxHash and Spooky, are running at 10.8 GB/s and 11.7 GB/s respectively. That's gigabytes per second, so these hashes effectively come free of charge.
But more importantly, we are no longer constricted by the full-file hash, so we can push these 762 MB/s per core.
* Numbers are obviously specific to our test machine, but they give a good idea of relative performance of the algorithms. The net effect of these changes is that the delta copier now can fully saturate an IO of a very fast drive while using just a small handful of hashing threads.
This in turn means that delta copying now comes with virtually no performance impact even when it ends up copying files in full. It basically goes as fast as a regular bulk copier.
An example
Copying a 16 GB file from an NVMe to an SSD drive using the old delta copier vs the new one:| Release 78 | Release 79 | |
| Creating | 262.8 MB/s | 547.2 MB/s |
| Updating | 272.5 MB/s | 2026.6 MB/s |
Copying file afresh now runs at the write speed of the target drive and updating it afterwards (with little to no changes) runs at the read speed of the source drive.
Needless to say, that's a huge improvement. Next up is the part 3 of the ultra copier series. It covers resuming of interrupted copies and fast error recovery.
The backup engine now understands OneDrive failures
related to files being unpinned and offline.
The next release (R79) will add support for
automatically retrying file system operations
that fail for network-related reasons.
More specifically, the backup engine will retry an operation for up to 20 times in 15 sec intervals when running into Unexpected network error, Location is unreachable and several other failures.
This should come especially handy when running larger backups over long connections, e.g. inter-branch VPNs or similar.
More specifically, the backup engine will retry an operation for up to 20 times in 15 sec intervals when running into Unexpected network error, Location is unreachable and several other failures.
This should come especially handy when running larger backups over long connections, e.g. inter-branch VPNs or similar.
Prior to this release the engine already maintained
a list of so-called fatal errors that causes
a backup to be aborted if any of them is encountered.
These are errors like Disk corrupted or IO device error and they signal an unrecoverable condition that will cause all remaining backup steps to fail the same way.
Network-related failures were also on the list, but now they've been moved on a separate list of retryable errors, for which the engine is, basically, "trying harder". The UI will still show a total error count in the job's summary, but it will render it in black if all errors were recovered from by retrying. See the In action screengrab for details.
The retry count and the pause interval are configurable, but the 20 x 15 sec default should work well in most cases.
It's also possible to add/remove error codes to both fatal and retryable lists, so if you ever need to retry on, say, Disk full, you can certainly do that too.
These are errors like Disk corrupted or IO device error and they signal an unrecoverable condition that will cause all remaining backup steps to fail the same way.
Network-related failures were also on the list, but now they've been moved on a separate list of retryable errors, for which the engine is, basically, "trying harder". The UI will still show a total error count in the job's summary, but it will render it in black if all errors were recovered from by retrying. See the In action screengrab for details.
The retry count and the pause interval are configurable, but the 20 x 15 sec default should work well in most cases.
It's also possible to add/remove error codes to both fatal and retryable lists, so if you ever need to retry on, say, Disk full, you can certainly do that too.
Ultra copier is an internal name of a brand
new bulk and delta copying module, shipping with R79.
A complete rewrite, it builds on what we learned in the past 4 years and implements significant improvements in several key areas.
The per-file overhead of prep/post work is often comparable to the time needed for the actual copying. This fixed cost is split between program's own overhead and the time spent actually opening and closing files, copying meta info, etc.
With older drives this was not an area worthy of optimization, because the cost of merely opening/creating a file dwarfed that of any prep work that the app itself was doing.
However with newer drives and faster machines it's no longer the case. All that trivial activity like allocating buffers, writing to the log, pre-configuring the IO - it all suddenly adds up and starts to matter.
For this reason the ultra copier now aggressively pre-allocates, caches, recycles and otherwise streamlines the prep/post phases to keep its per-file overhead to an absolute minimum.
The effect of this obviously varies,
but it can be as eye-popping
as a threefold speed-up,
for example, when cloning C:\Windows on an
NVMe
drive.
The core of the technique is that the program doesn't wait for read/write requests to complete, but it rather just queues them with Windows and later checks if they are done.
That last bit - check if it's done - is where the new code does things differently. Ultra's IO pipeline is built around IO completion ports (IOCP) which it uses to track, well, the completion of IO requests.
The program issues read/write requests as before, but it also asks Windows to queue a "done" notification once a request is completed.
This queue is called an "IO completion... port", which tends to muddy the waters somewhat, but it is one of more elegant and useful mechanisms of the Windows kernel.
The key point of IOCP is in the last line:
With IOCP we no longer need to drag the full list
of pending requests across the userspace boundary
just to learn which of them might've been completed.
This makes IOCP really quite fast. But wait, there's more.
IOCP can also accommodate async non-IO operations.
The ultra copier makes a full use of this when delta-copying a file. It feeds a stream of block hashing requests to a pool of worker threads and then receives their completions events via the same port that it uses for IO requests.
This allows for uniform handling of all async
operations in the IO code. Reading, hashing
and writing now become equal parts of the IO pipeline,
leading to a simpler code.
We care about synchronous completion, because it makes the pipeline stutter, so it's not good for performance.
When this happens, Windows still queues an IOCP notification, so the simplest thing to do is to ignore how request completes and just wait for an IOCP ping.
This however adds a small delay to the IO flow, because we end up completing a request later than we could've.
You probably see where this is headed and you are correct - the ultra copier suppresses IOCP pings for sync reads/writes and processes them immediately.
If you ever wondered what SetFileCompletionNotificationModes is for - here you go, now you know :)
There also happens to be a way to further improve performance by locking IO buffers with SetFileIoOverlappedRange.
* Some conditions apply
In particular, this requires holding a rather exotic privilege and it just may lead to memory starvation for the rest of the system. Ask me how I know.
But when used with care it does appear to improve bulk IO rate on faster drives. Next up, the delta copying improvements.
A complete rewrite, it builds on what we learned in the past 4 years and implements significant improvements in several key areas.
- Copying of smaller files
- Working with very fast drives
- Faster scalable delta copying
- First-class support for resuming of copying
Smaller files
Copying lots of small files quickly is a challenge.The per-file overhead of prep/post work is often comparable to the time needed for the actual copying. This fixed cost is split between program's own overhead and the time spent actually opening and closing files, copying meta info, etc.
With older drives this was not an area worthy of optimization, because the cost of merely opening/creating a file dwarfed that of any prep work that the app itself was doing.
However with newer drives and faster machines it's no longer the case. All that trivial activity like allocating buffers, writing to the log, pre-configuring the IO - it all suddenly adds up and starts to matter.
For this reason the ultra copier now aggressively pre-allocates, caches, recycles and otherwise streamlines the prep/post phases to keep its per-file overhead to an absolute minimum.
The effect of this obviously varies,
but it can be as eye-popping
as a threefold speed-up,
for example, when cloning C:\Windows on an
NVMe
drive.
Faster drives
Bvckup 2 has been using multi-buffer async IO from its very first release.The core of the technique is that the program doesn't wait for read/write requests to complete, but it rather just queues them with Windows and later checks if they are done.
That last bit - check if it's done - is where the new code does things differently. Ultra's IO pipeline is built around IO completion ports (IOCP) which it uses to track, well, the completion of IO requests.
The program issues read/write requests as before, but it also asks Windows to queue a "done" notification once a request is completed.
This queue is called an "IO completion... port", which tends to muddy the waters somewhat, but it is one of more elegant and useful mechanisms of the Windows kernel.
The key point of IOCP is in the last line:
With IOCP we no longer need to drag the full list
of pending requests across the userspace boundary
just to learn which of them might've been completed.
This makes IOCP really quite fast. But wait, there's more.
IOCP can also accommodate async non-IO operations.
The ultra copier makes a full use of this when delta-copying a file. It feeds a stream of block hashing requests to a pool of worker threads and then receives their completions events via the same port that it uses for IO requests.
This allows for uniform handling of all async
operations in the IO code. Reading, hashing
and writing now become equal parts of the IO pipeline,
leading to a simpler code.
Synchronous IO
There are documented cases when async IO requests may complete synchronously. In fact, Microsoft also says that your code should be prepared for this to happen at all times.We care about synchronous completion, because it makes the pipeline stutter, so it's not good for performance.
When this happens, Windows still queues an IOCP notification, so the simplest thing to do is to ignore how request completes and just wait for an IOCP ping.
This however adds a small delay to the IO flow, because we end up completing a request later than we could've.
You probably see where this is headed and you are correct - the ultra copier suppresses IOCP pings for sync reads/writes and processes them immediately.
If you ever wondered what SetFileCompletionNotificationModes is for - here you go, now you know :)
Locked IO buffers
Among smaller performance tweaks, the ultra copier defaults to using unbuffered IO when reading larger files. Larger files aren't likely to be cached in full, so bypassing the cache has a small, but noticeable effect on the reading speed.There also happens to be a way to further improve performance by locking IO buffers with SetFileIoOverlappedRange.
* Some conditions apply
In particular, this requires holding a rather exotic privilege and it just may lead to memory starvation for the rest of the system. Ask me how I know.
But when used with care it does appear to improve bulk IO rate on faster drives. Next up, the delta copying improvements.
The UI will now show read/write byte counts for longer
copies, and it will also display the percentage of data
that was written out during delta updates. The lower it is,
the better.
* Also, the progress bar got a facelift... which should
be of little surprise, because it's hardly the first
time it ever happened.
May 2013:
Jun 2013:
Sep 2013:
Nov 2013:
Nov 2013:
Feb 2014:
Aug 2015:
Dec 2017:
The original plan was to switch to the last style,
but after testing it on live data it proved to animate
very poorly. When IO buffers are large (which is
common) that nice right-to-left sliding animation
turns into a jumpy-jerky movement which is plain
confusing.
In any case, here's a version that made it into R79...
...though that's certainly not the final word in
the progress bar saga. Far from it.
Jun 2013:
Sep 2013:
Nov 2013:
Nov 2013:
Feb 2014:
Aug 2015:
Dec 2017:
The original plan was to switch to the last style,
but after testing it on live data it proved to animate
very poorly. When IO buffers are large (which is
common) that nice right-to-left sliding animation
turns into a jumpy-jerky movement which is plain
confusing.
In any case, here's a version that made it into R79...
...though that's certainly not the final word in
the progress bar saga. Far from it.
Just a quick pair of screenshots to introduce a
little something called the ultra copier.
It's a faster, lighter and altogether more beautiful rewrite of the core copying module which handles both bulk and delta copying. The numbers are for copying ~100K files from C:\Windows from one location on a fast drive to another, with the OS file cache purged beforehand. The speed up is indeed almost threefold...
More to follow shortly...
It's a faster, lighter and altogether more beautiful rewrite of the core copying module which handles both bulk and delta copying. The numbers are for copying ~100K files from C:\Windows from one location on a fast drive to another, with the OS file cache purged beforehand. The speed up is indeed almost threefold...
More to follow shortly...
This is Disk IO Monitor - a fairly simple
utility that samples and displays, in real-time,
read/write rates of a physical drive.
Read activity is on the top, write activity is at the bottom. What's interesting about this graph is that it can display samples as large as gigabytes and still clearly show byte-level differences between them.
Read activity is on the top, write activity is at the bottom. What's interesting about this graph is that it can display samples as large as gigabytes and still clearly show byte-level differences between them.
To that end, it spreads each sample across
several horizontal bands, each displaying
a count of full GBs, MBs and KBs,
plus "leftover" bytes in each sample.
If we want to display, say, 12345678, then it's 11 MB + 792 KB + 445 bytes, so the band closest to the zero line shows a bar filled to 445/1024, the next band - 792/1024 and the 3rd - 11/1024. This is basically a mod-1024 factorization of a sample value.
It is also a discrete (rather than continuous) variation of a log scale, which is very commonly used for "squeezing" large values into more manageable ranges. In practical terms this allows us to quickly see how different any two samples are, even if they are very large.
For example -
These two samples are both in the MB range,
very close to each other, but looking at
the KB band it's immediately obvious which
one is larger and approximately by how much.
This sort of analysis is often needed when trying
to spot jitter that is order of magnitude
smaller than absolute sample values.
In our case, this need surfaces when analyzing the effects of OS file caching on bulk IO performance.
Here's a 16 GB file being copied using Windows Explorer:
Here's the same file being copied with a custom,
somewhat better optimized code:
* Both tests are run with the file cache purged.
Windows Explorer makes uses synchronous IO that goes through the Windows File Cache. This delays writes and it also adds jitter to read operations.
In comparison, the second graph is for the code that uses asynchronous IOCP-based copying engine that bypasses the OS cache. The resulting data flow is far more stable and predictable. It's also substantially faster.
io-monitor.exe
~ 72KB, no dependencies, signed. Just save somewhere and run.
It defaults to monitoring of the C: drive, but another drive can be selected by clicking on "Disk C:". The sampling rate is 500 ms and it is fixed for now.
The Purge cache option will clear the so-called Windows Standby List, which effectively empties the OS file cache. Use with care, obviously.
Happy graphing :)
If we want to display, say, 12345678, then it's 11 MB + 792 KB + 445 bytes, so the band closest to the zero line shows a bar filled to 445/1024, the next band - 792/1024 and the 3rd - 11/1024. This is basically a mod-1024 factorization of a sample value.
It is also a discrete (rather than continuous) variation of a log scale, which is very commonly used for "squeezing" large values into more manageable ranges. In practical terms this allows us to quickly see how different any two samples are, even if they are very large.
For example -
These two samples are both in the MB range,
very close to each other, but looking at
the KB band it's immediately obvious which
one is larger and approximately by how much.
This sort of analysis is often needed when trying
to spot jitter that is order of magnitude
smaller than absolute sample values.
In our case, this need surfaces when analyzing the effects of OS file caching on bulk IO performance.
Here's a 16 GB file being copied using Windows Explorer:
Here's the same file being copied with a custom,
somewhat better optimized code:
* Both tests are run with the file cache purged.
Windows Explorer makes uses synchronous IO that goes through the Windows File Cache. This delays writes and it also adds jitter to read operations.
In comparison, the second graph is for the code that uses asynchronous IOCP-based copying engine that bypasses the OS cache. The resulting data flow is far more stable and predictable. It's also substantially faster.
Download
Here's a dev build if you feel like seeing this thing in action:~ 72KB, no dependencies, signed. Just save somewhere and run.
It defaults to monitoring of the C: drive, but another drive can be selected by clicking on "Disk C:". The sampling rate is 500 ms and it is fixed for now.
The Purge cache option will clear the so-called Windows Standby List, which effectively empties the OS file cache. Use with care, obviously.
Happy graphing :)
Next version of Bvckup 2 will have a proper
understanding of machine's entire storage stack -
lettered drives, simple, dynamic and virtual volumes,
physical disks, their types, models, connection
details, health data, etc.
This will allow the backup engine to automatically adapt its IO strategy to specific devices used in any given backup job. It will also let the program to keep an eye on the devices' health, which is not a bad thing to have either.
This will allow the backup engine to automatically adapt its IO strategy to specific devices used in any given backup job. It will also let the program to keep an eye on the devices' health, which is not a bad thing to have either.
The storage stack scanner is adapted from
another project of ours called
Diskovery
and the IO profiling logic is based on the
data gathered through
CCSIO Benchmark.
While the scanner part is anything but trivial,
the most time-consuming part, as per usual,
was the UI.
There's a lot of information to be displayed - both the properties of volumes and devices, and how they relate to each other.
As you likely know, a device may host multiple volumes and a volume may span multiple devices. So in a most general case it's a full-mesh, which is the reason why we can't display volumes and devices as a simple hierarchy.
So, after going...
... through ...
... some ...
... options ...
... it became obvious that having two lines per entry
was a way to go and that volumes and devices should be on
separate lists, both visible simultaneously.
After a bit more sketching the lists arranged themselves
into two columns and the scroll bar of the left list went
to the left of it:
* I must say that the idea of placing that damn scroll bar
on the left took waaay too long to converge to. For
whatever reason it's just a strangely un-obvious thing
to do...
In any case, the result is a layout where we can display relations between volumes and devices with literal, drawn links. After that just needed to take care of smaller details like scrolling slits...
out-of-view arrows...
and connector points...
and, voilà, we now have something that is both
usable and reasonably good-looking.
Still to be added is a separate pane for displaying
further details - device hardware info, SMART data,
etc. This pane will sit at the bottom of the window
and will switch between showing volume and device
details depending on what's selected above.
Stay tuned for the updates :)
There's a lot of information to be displayed - both the properties of volumes and devices, and how they relate to each other.
As you likely know, a device may host multiple volumes and a volume may span multiple devices. So in a most general case it's a full-mesh, which is the reason why we can't display volumes and devices as a simple hierarchy.
So, after going...
... through ...
... some ...
... options ...
... it became obvious that having two lines per entry
was a way to go and that volumes and devices should be on
separate lists, both visible simultaneously.
After a bit more sketching the lists arranged themselves
into two columns and the scroll bar of the left list went
to the left of it:
* I must say that the idea of placing that damn scroll bar
on the left took waaay too long to converge to. For
whatever reason it's just a strangely un-obvious thing
to do...
In any case, the result is a layout where we can display relations between volumes and devices with literal, drawn links. After that just needed to take care of smaller details like scrolling slits...
out-of-view arrows...
and connector points...
and, voilà, we now have something that is both
usable and reasonably good-looking.
Still to be added is a separate pane for displaying
further details - device hardware info, SMART data,
etc. This pane will sit at the bottom of the window
and will switch between showing volume and device
details depending on what's selected above.
Stay tuned for the updates :)
You know how some people, when tired of their
established ways, tend to move furniture around
the house to make the same thing look like
something new?
Here's the same idea applied to the world of progress bars.
Here's the same idea applied to the world of progress bars.
This is a new and improved progress bar that
is making its way into R79. The idea is to
use the vacant space on the right-hand side
(the "todo" part) to show the sub-progress.
In comparision, here's a lively version we have now:
† This is a GIF, trust me.
In Bvckup's case a progress bar also needs to
accommodate the delta copying. In particular,
the bar should be able to display processed file
blocks that were copied in full, copied
in part or not copied at all.
So here's a short list of options to achieve just that:
Will probably go either with the 2nd from
the top or the 5th from the bottom. Maybe
both.
In comparision, here's a lively version we have now:
† This is a GIF, trust me.
In Bvckup's case a progress bar also needs to
accommodate the delta copying. In particular,
the bar should be able to display processed file
blocks that were copied in full, copied
in part or not copied at all.
So here's a short list of options to achieve just that:
Will probably go either with the 2nd from
the top or the 5th from the bottom. Maybe
both.
There are several ways in which a program can query security
attributes (Owner, Group and ACLs) of one file and copy
them to another file.
In a true Microsoft's way, some are well-documented, but they don't work as advertised, and some work as needed, but they are not documented. First hit on Google is Get/SetFileSecurity. The name checks out, it accepts a HANDLE, so it should be quick and simple way to do what we want.
Nope, it's not.
It's an older API that doesn't correctly handle inheritable ACLs and its use is not recommended. Not by the API documentation, mind you, but by a random blog post from one of many Microsoft development teams, which you still need to find first. What they recommend instead is Get/SetSecurityInfo.
And you'd still naively think it's the one. Nope, it's not.
The problem is that SetSecurityInfo will fail with "access denied" if you are trying to set DACL and the file is not opened with READ_CONTROL.
You may wonder why the heck it needs read control... excellent question. That's because it wants to look up the name of the file and then *feed that name into SetNamedSecurityInfo function*. The latter will then re-open the file and start spraying kernel with requests, which is the last thing you need when you are already quite busy with copying a million of files from A to B.
Looking at the trace of SetSecurityInfo with Procmon shows that it generates over a dozen of kernel calls... whereas only one is really needed. The name of that one call that we want is NtSetSecurityObject, but it's a part of so-called Native API and as such it is officially off-limits for any application use.
However, if we are to comb several pagefuls of relevant APIs, we may spot something called SetKernelObjectSecurity, which looks suspiciously like NtSetSecurityObject. Long story short, you want to copy security attributes - use Get / SetKernelObjectSecurity. So now you know. Congratulations :)
In a true Microsoft's way, some are well-documented, but they don't work as advertised, and some work as needed, but they are not documented. First hit on Google is Get/SetFileSecurity. The name checks out, it accepts a HANDLE, so it should be quick and simple way to do what we want.
Nope, it's not.
It's an older API that doesn't correctly handle inheritable ACLs and its use is not recommended. Not by the API documentation, mind you, but by a random blog post from one of many Microsoft development teams, which you still need to find first. What they recommend instead is Get/SetSecurityInfo.
And you'd still naively think it's the one. Nope, it's not.
The problem is that SetSecurityInfo will fail with "access denied" if you are trying to set DACL and the file is not opened with READ_CONTROL.
You may wonder why the heck it needs read control... excellent question. That's because it wants to look up the name of the file and then *feed that name into SetNamedSecurityInfo function*. The latter will then re-open the file and start spraying kernel with requests, which is the last thing you need when you are already quite busy with copying a million of files from A to B.
Looking at the trace of SetSecurityInfo with Procmon shows that it generates over a dozen of kernel calls... whereas only one is really needed. The name of that one call that we want is NtSetSecurityObject, but it's a part of so-called Native API and as such it is officially off-limits for any application use.
However, if we are to comb several pagefuls of relevant APIs, we may spot something called SetKernelObjectSecurity, which looks suspiciously like NtSetSecurityObject. Long story short, you want to copy security attributes - use Get / SetKernelObjectSecurity. So now you know. Congratulations :)
R78.6 adds support for
alternate data streams.
This is a rather obscure, but interesting feature of NTFS that allows storing secondary data blobs under the same file name.
You'd give such blob a name - say, ABC - and then you'd be able to read, write and delete it by referring to it as filename.ext:ABC.
No doubts, some of you may be thinking why though?
This is a rather obscure, but interesting feature of NTFS that allows storing secondary data blobs under the same file name.
You'd give such blob a name - say, ABC - and then you'd be able to read, write and delete it by referring to it as filename.ext:ABC.
No doubts, some of you may be thinking why though?
The answer is that alt. streams is a
generic and more flexible replacement for an older "extended attributes"
mechanism. It is meant for the cases when you may want to pin an extra
bit of info to a file (or a folder), the info that is application-specific,
but that doesn't belong in a file itself.
An example here is a small tag that browsers add to executable files downloaded from the Internet. This tag is used by Windows Explorer to show its "Ugh-oh, something from the Internets" warning when you are trying to run such files. The tag is stored as an alternate stream called Zone.Identifier. You delete this stream - you won't get the warning.
Long story short, Bvckup 2 now can replicate these secondary data streams. This option is in Backup Settings > More Options > Copy Also section and it is Off by default.
An example here is a small tag that browsers add to executable files downloaded from the Internet. This tag is used by Windows Explorer to show its "Ugh-oh, something from the Internets" warning when you are trying to run such files. The tag is stored as an alternate stream called Zone.Identifier. You delete this stream - you won't get the warning.
Long story short, Bvckup 2 now can replicate these secondary data streams. This option is in Backup Settings > More Options > Copy Also section and it is Off by default.
New version of
Backup Settings > More Options > Copy Also
is coming up in Release 78.6.
• File attributes are out.
• Owner and Group are now a single multi-state option.
• DACL is now called Security info.
• SACL is now in, as Auditing info.
• Alternate NTFS streams are in as well.
The rationale behind File attributes option removal is that it was the only one from the list that was familiar to non-professional users. That instilled a false sense of confidence when looking at other options, enticing to tick them all on.
From now on the list comprises exclusively... erm... alien-looking stuff, so hopefully this will deter people from randomly enabling options they should not really be enabling. This also lets those in need of a complete replication configure it now directly in the UI.
• File attributes are out.
• Owner and Group are now a single multi-state option.
• DACL is now called Security info.
• SACL is now in, as Auditing info.
• Alternate NTFS streams are in as well.
The rationale behind File attributes option removal is that it was the only one from the list that was familiar to non-professional users. That instilled a false sense of confidence when looking at other options, enticing to tick them all on.
From now on the list comprises exclusively... erm... alien-looking stuff, so hopefully this will deter people from randomly enabling options they should not really be enabling. This also lets those in need of a complete replication configure it now directly in the UI.
Here's a way to present two closely-related options
with a single checkbox with 4 distinct states.
It is very rarely (if ever) that anyone would need to copy Owner information without also copying file's Group info. However, these are separate bits, so bunching them together doesn't seem like a right thing to do. But giving each of them their own piece of UI real-estate doesn't feel right either.
So how can we pack them up?
It is very rarely (if ever) that anyone would need to copy Owner information without also copying file's Group info. However, these are separate bits, so bunching them together doesn't seem like a right thing to do. But giving each of them their own piece of UI real-estate doesn't feel right either.
So how can we pack them up?
First, let's look at how this can be done conventionally.
We'll just be verbose about it and list all the options as
is:
This is OK in terms of usability, but when taken as a part
of the whole it makes window look cluttered:
And *unnecessary* cluttered at that, because in a vast
majority of cases Owner and Group will be either both checked
or unchecked. Ditto for Security and Audit options.
So let's try something else. Windows natively supports something called "3-state" checkboxes that can be in checked, unchecked or indeterminate states.
"Indeterminate" is a rather strange term as if
the program is not sure in which state the option is.
However we can think of it as an in-between or
partial state and do something like this:
This is not bad, however we lose the overall meaning
of the option when it is in a partial state:
So let's put the context back:
Better, but now we are back to things being too
verbose and looking cluttered. Let's simplify
it for two most common cases:
Better, but now it's off visually:
How about with chop the specifics and move them to
the side:
Almost there, just need to patch up visual hierarchy
a bit to help telling principal parts from the
secondary ones:
And that's it. The result is an UI that looks clean
and simple for two most common cases, fully supports
marginal ones and packs 4 options in a space of two.
This is OK in terms of usability, but when taken as a part
of the whole it makes window look cluttered:
And *unnecessary* cluttered at that, because in a vast
majority of cases Owner and Group will be either both checked
or unchecked. Ditto for Security and Audit options.
So let's try something else. Windows natively supports something called "3-state" checkboxes that can be in checked, unchecked or indeterminate states.
"Indeterminate" is a rather strange term as if
the program is not sure in which state the option is.
However we can think of it as an in-between or
partial state and do something like this:
This is not bad, however we lose the overall meaning
of the option when it is in a partial state:
So let's put the context back:
Better, but now we are back to things being too
verbose and looking cluttered. Let's simplify
it for two most common cases:
Better, but now it's off visually:
How about with chop the specifics and move them to
the side:
Almost there, just need to patch up visual hierarchy
a bit to help telling principal parts from the
secondary ones:
And that's it. The result is an UI that looks clean
and simple for two most common cases, fully supports
marginal ones and packs 4 options in a space of two.
Earlier today we rolled out a new revision of
purchase discounts.
Discounts are now dead simple - they are based on the number of licenses that you already have plus the size of your order.
As simple as that.
Also, the purchase form was reworked to display effective pricing if you are logged into your dashboard account (and we basically know who you are).
Discounts are now dead simple - they are based on the number of licenses that you already have plus the size of your order.
As simple as that.
Also, the purchase form was reworked to display effective pricing if you are logged into your dashboard account (and we basically know who you are).
Licensing server now recognizes a handful of cheat
codes. These codes are still refused, but at least
in a more respectful manner :)
Also rolled out today is the first revision of
Dashboard API.
The API is primarily meant for integrating with external inventory management systems by exporting dashboard data in json/csv formats.
A bit later we'll be expanding it to allow changing things as well, so there's that.
The API is in pre-beta, so it's available on opt-in basis only - if interested, drop us a line and we'll enable it for your account.
The API is primarily meant for integrating with external inventory management systems by exporting dashboard data in json/csv formats.
A bit later we'll be expanding it to allow changing things as well, so there's that.
The API is in pre-beta, so it's available on opt-in basis only - if interested, drop us a line and we'll enable it for your account.
Rolled out a couple of improvements to the
web dashboard.
One is an option to hide revoked licenses from the list and another is a support for automatically adjusting all displayed dates/times to browser's native time zone, with an option to override if needed. Previously all times were displayed in CET time zone (because that's where our backend servers are).
One is an option to hide revoked licenses from the list and another is a support for automatically adjusting all displayed dates/times to browser's native time zone, with an option to override if needed. Previously all times were displayed in CET time zone (because that's where our backend servers are).
Release 78 will introduce changes to file/folder filtering
system to make it a bit more flexible and logical, while
also keeping it as point-and-clicky as possible.
In pre-78 releases if you excluded a folder, you weren't
able to then go and include some of its subfolders or files.
Similarly, it wasn't possible to exclude some of the items
in an explicitly included folder.
In fact, it wasn't possible to exclude anything at all if using the Start with an empty list mode or include any item when using the Include everything option. While this approach mapped on a simpler code and sort of made sense when it was first coded, in retrospect it wasn't that good. So, R78 will come with a fundamental overhaul of the filtering module... and whatever new and exciting bugs that may come with it. What the middle pane now does is it allows quickly setting the action (include or exclude) for folder and files by simply clicking on them. For files the action is final - if you exclude it, then it is excluded. End of story.
For folders however it also sets a default action for all its files and subfolders. This action may be overriden for any item by clicking on it, which simply reverses the default.
So when bvckup2 looks at what to do with any given file
or folder, it will first pass it through this set of
filters, yielding a preliminary include or
exclude action.
Next, the item will be passed through generic filters, from the bottom pane.
First hit on any of these
determines the outcome. If there are no hits, then
the default action it is.
There are also new, finer controls over the filtering
flow. In short - it's now possible to mark any generic
filter as final, which will suppress generic
filter matching for all folder contents.
is actually no longer needed!
All it does is merely sets the default action for the top directory - and we may just allow doing that by simply clicking on it in the middle pane.
However I think having it a separate option helps making the UI a bit more approachable and easier to use, so it will stay.
In fact, it wasn't possible to exclude anything at all if using the Start with an empty list mode or include any item when using the Include everything option. While this approach mapped on a simpler code and sort of made sense when it was first coded, in retrospect it wasn't that good. So, R78 will come with a fundamental overhaul of the filtering module... and whatever new and exciting bugs that may come with it. What the middle pane now does is it allows quickly setting the action (include or exclude) for folder and files by simply clicking on them. For files the action is final - if you exclude it, then it is excluded. End of story.
For folders however it also sets a default action for all its files and subfolders. This action may be overriden for any item by clicking on it, which simply reverses the default.
So when bvckup2 looks at what to do with any given file
or folder, it will first pass it through this set of
filters, yielding a preliminary include or
exclude action.
Next, the item will be passed through generic filters, from the bottom pane.
First hit on any of these
determines the outcome. If there are no hits, then
the default action it is.
There are also new, finer controls over the filtering
flow. In short - it's now possible to mark any generic
filter as final, which will suppress generic
filter matching for all folder contents.
Bonus material
If you are to think about all this for as looong as I have, you might realize that the Include everything / Start with an empty list selector at the top, this one -
is actually no longer needed!
All it does is merely sets the default action for the top directory - and we may just allow doing that by simply clicking on it in the middle pane.
However I think having it a separate option helps making the UI a bit more approachable and easier to use, so it will stay.
Release 77 will introduce the Errors only view for the logs.
As you might've heard the log viewer happens to be one of the most complex parts of Bvckup 2. It looks simple, but on the back it is anything but.
As you might've heard the log viewer happens to be one of the most complex parts of Bvckup 2. It looks simple, but on the back it is anything but.
1. Log files are created by the backup engine,
but displayed by the UI module. In desktop mode this
is easy enough to handle, but in service mode the engine
runs in a process of its own and under a different user
account, meaning the logs are simply inaccessible to the
UI.
So what ends up happening is that the engine opens logs on behalf of the UI and passes open file handles back to it.
PS. Ever wondered what all those "process" handles are doing on the DuplicateHandle() argument list? Exactly to allow one process to do something privileged on behalf another process. 2. Log files are rotated, so what you see in the UI is actually a seamlessly glued together list of all log entries across all copies of a backup log.
This would've not been a big deal if it weren't for... Finally, the UI needs to expand/collapse log parts and do it nearly instantaneously. This is a hard problem, especially considering that logs can grow very big.
As the engine writes out the logs, the UI scans them and creates an auxiliary index, which it then uses to quickly understand what should and should not be shown in any given state.
These ndecies cannot be kept in memory, not in full. Instead the UI stores them in a specialized on-disk database, which too needs to be very quick. However, all this complexity had a nice little side-effect. It made adding the Error view very easy.
All the UI had to do is to build another index and include just the error entries and their immediate context. This way switching to/from the Error view is a simple matter of swapping one index file for another. That's it.
This might not be the most exciting thing ever, but still has some elegance to it, doesn't it?
So what ends up happening is that the engine opens logs on behalf of the UI and passes open file handles back to it.
PS. Ever wondered what all those "process" handles are doing on the DuplicateHandle() argument list? Exactly to allow one process to do something privileged on behalf another process. 2. Log files are rotated, so what you see in the UI is actually a seamlessly glued together list of all log entries across all copies of a backup log.
This would've not been a big deal if it weren't for... Finally, the UI needs to expand/collapse log parts and do it nearly instantaneously. This is a hard problem, especially considering that logs can grow very big.
As the engine writes out the logs, the UI scans them and creates an auxiliary index, which it then uses to quickly understand what should and should not be shown in any given state.
These ndecies cannot be kept in memory, not in full. Instead the UI stores them in a specialized on-disk database, which too needs to be very quick. However, all this complexity had a nice little side-effect. It made adding the Error view very easy.
All the UI had to do is to build another index and include just the error entries and their immediate context. This way switching to/from the Error view is a simple matter of swapping one index file for another. That's it.
This might not be the most exciting thing ever, but still has some elegance to it, doesn't it?
One of several tweaks coming up in Release 77 - the
on-hover extension of long log lines.
Windows comes with native support for this sort of functionality and it's called "in-place tooltips". For example, they are used in Windows Explorer when displaying file and folder names that don't quite fit the allotted space.
But in a true Windows fashion, while it works, it also... flickers!
Blinkety-blink. Here's a slo-mo in all its glory:
So after killing several hours on trying to make it
behave, it became clear that it's faster and simpler
to just redo it from scratch.
Not the first time it happened and, sure as hell, not the last :-|
Windows comes with native support for this sort of functionality and it's called "in-place tooltips". For example, they are used in Windows Explorer when displaying file and folder names that don't quite fit the allotted space.
But in a true Windows fashion, while it works, it also... flickers!
Blinkety-blink. Here's a slo-mo in all its glory:
So after killing several hours on trying to make it
behave, it became clear that it's faster and simpler
to just redo it from scratch.
Not the first time it happened and, sure as hell, not the last :-|
Today we rolled out a new addition to the
/purchase page.
Internally known as Perks for Tweets it allows receiving a bit of a discount in exchange for a Bvckup 2 mention on Twitter.
The main reason we are doing this is to try and reach out to new people, so the fine print is as follows. Your Twitter account should be at least a month old and you should have some number of followers and updates (tweets) on record. A tweet should include a link to bvckup2.com and/or an @bvckup mention (but without being directed at @bvckup).
Keep in mind that this is one of them wild marketing experiments, so things are subject to change in the next few weeks.
Internally known as Perks for Tweets it allows receiving a bit of a discount in exchange for a Bvckup 2 mention on Twitter.
The main reason we are doing this is to try and reach out to new people, so the fine print is as follows. Your Twitter account should be at least a month old and you should have some number of followers and updates (tweets) on record. A tweet should include a link to bvckup2.com and/or an @bvckup mention (but without being directed at @bvckup).
Keep in mind that this is one of them wild marketing experiments, so things are subject to change in the next few weeks.
Starting with 76.9 when the app is switched to run as a
service, the engine will now run under a dedicated
user account.
The account is called "Bvckup 2 Service". It is automatically added when the app is switched to the service mode and removed when the app is switched back to the desktop mode or uninstalled.
The account is made a member of Administrators and Backup Operators groups, which gives it the same set of privileges as provided by the LocalSystem account.
Ideally, it should've only needed Backup Operators membership, but in some cases this group lacks network access privileges, so the default is to add it to the Administrators group as well. It is still possible to install the service under the LocalSystem account - click the More button at the bottom left of the Switch Mode dialog and flip the Run service under option. This change has been long coming as it's an altogether better and tidier setup. It helps isolating Bvckup 2 from the rest of the services, eliminating effects of any custom LocalSystem policies that might be in place and it also allows for a fine-grained control over its permission set. So if another Windows update comes out and (again) breaks things, it should be much easier to hammer it all back in place.
The account is called "Bvckup 2 Service". It is automatically added when the app is switched to the service mode and removed when the app is switched back to the desktop mode or uninstalled.
The account is made a member of Administrators and Backup Operators groups, which gives it the same set of privileges as provided by the LocalSystem account.
Ideally, it should've only needed Backup Operators membership, but in some cases this group lacks network access privileges, so the default is to add it to the Administrators group as well. It is still possible to install the service under the LocalSystem account - click the More button at the bottom left of the Switch Mode dialog and flip the Run service under option. This change has been long coming as it's an altogether better and tidier setup. It helps isolating Bvckup 2 from the rest of the services, eliminating effects of any custom LocalSystem policies that might be in place and it also allows for a fine-grained control over its permission set. So if another Windows update comes out and (again) breaks things, it should be much easier to hammer it all back in place.
With the next maintenance release (76.8) Bvckup 2 will log
device status when a backup job is created, loaded or enabled
(after being in a disabled state).
The app was already logging device arrival/deparature events and OK'ays from the share monitoring module, but the messages were lacking consistency and no initial device state was logged.
All tidied up now.
The app was already logging device arrival/deparature events and OK'ays from the share monitoring module, but the messages were lacking consistency and no initial device state was logged.
All tidied up now.
options:
Reworking the welcome popup that is shown on
the first launch. it will now briefly go over
the differences between file and
system backups, explain how the trial
works and what licensing options are available.
This set of sketches shows various styling options for the index on the left hand side of the window.
In comparison, here's the existing version in all its one-page glory:
This set of sketches shows various styling options for the index on the left hand side of the window.
In comparison, here's the existing version in all its one-page glory:
Rolled out a couple of
Customer dashboard
improvements.
First is a cosmetic change to how an inline editing is done - click on a Pen icon, start typing, press Enter to save, Escape to cancel (or click on respective button if that's your preference). Second change is a bit more interesting - it adds support for deactivating licenses via the dashboard.
Please note that this feature is currently
reserved for site license and larger resellers
only, and it is enabled on a case-by-case basis.
First is a cosmetic change to how an inline editing is done - click on a Pen icon, start typing, press Enter to save, Escape to cancel (or click on respective button if that's your preference). Second change is a bit more interesting - it adds support for deactivating licenses via the dashboard.
Please note that this feature is currently
reserved for site license and larger resellers
only, and it is enabled on a case-by-case basis.
Another change in upcoming 76.5 is a new
warning that is shown when copying of
owner, group or DACL attributes is enabled
for a backup job that
goes across the
computer boundary
.
As you might guess the warning is due to what we are seeing on the support end, when some people enable some backup options merely because they look reasonable, but not because they actually need them.
As you might guess the warning is due to what we are seeing on the support end, when some people enable some backup options merely because they look reasonable, but not because they actually need them.
Added support for pre-seeding backups and
adjusting source or backup locations when
the on-disk data is being moved around.
Previously, if either source or destination location was changed in backup settings, the app would purge all delta copying data and the snapshot of destination if present. This caused destination to be re-scanned on the next run and all files to be re-copied in full next time they needed an update.
With the next relase (76.5) the app will now ask if the backup state should be purged or not. This way if you are setting up a remote backup job, you may first pre-seed the backup by attaching the drive locally and making a backup on it. They you'd take the device to the remote machine, change Destination path in the backup job, say No to the state reset and - voila - you are back in business as if the drive has always been remote.
Previously, if either source or destination location was changed in backup settings, the app would purge all delta copying data and the snapshot of destination if present. This caused destination to be re-scanned on the next run and all files to be re-copied in full next time they needed an update.
With the next relase (76.5) the app will now ask if the backup state should be purged or not. This way if you are setting up a remote backup job, you may first pre-seed the backup by attaching the drive locally and making a backup on it. They you'd take the device to the remote machine, change Destination path in the backup job, say No to the state reset and - voila - you are back in business as if the drive has always been remote.
Bvckup 2 is getting media and device ejection support.
Media ejection refers to, well, ejecting storage media from an otherwise fixed device such as a tape drive.
Device ejection has to do with "safe removal" of USB drives, the same thing you can do by clicking on a standard Windows "safe remove" icon in the system tray before yanking drive's USB cable out.
However, in a true Windows fashion ejecting a media sometimes also ejects a device and sometimes you can also eject media from an external USB HDD (and, yes, this makes absolutely no sense).
Media ejection refers to, well, ejecting storage media from an otherwise fixed device such as a tape drive.
Device ejection has to do with "safe removal" of USB drives, the same thing you can do by clicking on a standard Windows "safe remove" icon in the system tray before yanking drive's USB cable out.
However, in a true Windows fashion ejecting a media sometimes also ejects a device and sometimes you can also eject media from an external USB HDD (and, yes, this makes absolutely no sense).
In any case, the next maintenance release (76.5) will
get ejection support, but without the UI part. It will
be possible to set it up with a good old INI file edit
only.
Then, next full release (R77) will get the UI support and above sketches are for that bit. The tricky part is that ejection settings are ultimately per-location preference, so they are needed for both source and destination drives and this takes 2x of screen estate to accommodate.
Secondly, ejection is available only for local drives and not network shares, so the Option 2 series aren't really good. They all allows enabling ejection for non-local drives, meaning that we have to do extra "sanity checks" and error messaging when validating backup settings. That's just... messy.
In comparison, Option 1 simply doesn't show ejection option for non-local drives, just like it doesn't show device tracking (the pushpin icon) in that case. So it's tidy and logical.
Then, next full release (R77) will get the UI support and above sketches are for that bit. The tricky part is that ejection settings are ultimately per-location preference, so they are needed for both source and destination drives and this takes 2x of screen estate to accommodate.
Secondly, ejection is available only for local drives and not network shares, so the Option 2 series aren't really good. They all allows enabling ejection for non-local drives, meaning that we have to do extra "sanity checks" and error messaging when validating backup settings. That's just... messy.
In comparison, Option 1 simply doesn't show ejection option for non-local drives, just like it doesn't show device tracking (the pushpin icon) in that case. So it's tidy and logical.
The 9th revision of the website is now live.
New tray icon state for the case when all configured
jobs are disabled. Will be in the next R76 maintenance
update.
This is a development test of new parallel copying
feature that may be making its way into the next release.
If you are familiar with robocopy, it's an equivalent of its /mt option, which causes several files to be copied, well, in parallel.
As it's probably obvious from the above screenshot it works really well, especially for smaller files when an overhead of opening and closing file is comparable to the bulk I/O time. Here, for example, C:\Windows\Fonts folder with about 1500+ items is getting copied to a Synology NAS over 1Gbps link.
This feature is going to be available under a Pro license only. More on this to follow.
If you are familiar with robocopy, it's an equivalent of its /mt option, which causes several files to be copied, well, in parallel.
As it's probably obvious from the above screenshot it works really well, especially for smaller files when an overhead of opening and closing file is comparable to the bulk I/O time. Here, for example, C:\Windows\Fonts folder with about 1500+ items is getting copied to a Synology NAS over 1Gbps link.
This feature is going to be available under a Pro license only. More on this to follow.
First stab at the Drive Health Monitor UI
for the next Bvckup 2 release.
There's lots of information to be displayed, so the idea is to recycle columns for hosting multiple releated bits of data.
For example, you can toggle between showing drive vendor and its model in the 2nd column, and between disk capacity and its current utilization in the 5th column.
Naturally, none of this is final, but I think for the first pass this is not bad.
There's lots of information to be displayed, so the idea is to recycle columns for hosting multiple releated bits of data.
For example, you can toggle between showing drive vendor and its model in the 2nd column, and between disk capacity and its current utilization in the 5th column.
Naturally, none of this is final, but I think for the first pass this is not bad.
A list of storage device vendors
as observed by Diskovery to date.
I must admit that I haven't even heard of at least 3 of these.
Here's few more notable findings from the aggregated analysis
of 200+ logs submitted so far:
- Top three vendors - Seagate (19%), WD (18%) and Samsung (12%)
- 219 HDDs and 58 SSDs - 79% and 21% respectively.
- The only SMART attribute present in all SMART reading was #12 - the power cycle count.
-
All but one device also reported power on hours and
reallocated
sector count
.
This is good news, because the "reallocated" attribute (#5) is the principal predictor of an imminent drive failure. Once you see it starting to climb up, it's time to change the drive. - The temperature attribute (#194) was reported by 71% of HDDs and 51% of SSDs. However some devices report temperature readings in different attributes, so these percentages may be off.
- Attributes in the 17 to 159 are unused (with an exception of one SSD that reports something #64).
- Lots and lots undocumented attributes. Some vendors have incomplete specs and others (e.g. Seagate) just don't release SMART specs at all. This means that we can't even label some of the attributes.
-
Lots of inaccuracies in the specs too, meaning that it's hard to reliably
interpret the attribute's raw values.
Raw value is a 6-byte sequence of vendor-specific data, so you *must* have a spec to understand them. And you do want to understand them, because that's where the actually interesting data is.
Regrettably some specs would say that it's just a 48-bit counter, whereby in reality it'll be a 16-bit something followed by a 32-bit counter. Furthermore, certain vendors (e.g. Kingston) tend to change the format not just between their product lines, but between sibling products that are just a model away from each other. It's really quite nuts.
Two years worth of changes from the last version of the
beta cycle (April 2014) to the upcoming production release
(April 2016).
Just thought I'd share.
Just thought I'd share.
Finalized the last change that will go into R75 -
full UI support for advanced filtering rules that
have been supported internally
since last August.
More specifically, the UI can now be used to set up rules for matching on the file size and created/last-modified times. Attribute matching has also been extended to support more exotic values such as Offline and Don't-Index.
More specifically, the UI can now be used to set up rules for matching on the file size and created/last-modified times. Attribute matching has also been extended to support more exotic values such as Offline and Don't-Index.
Added an option to copy parts of the log to the
clipboard - either just a single line or the line
and all its children (the block).
If the block is large (over 64K), then the app will also display its size on the right-hand side on the menu. The neat thing is that this information is readily available in the log index, so this requires no log scanning of any kind or no interim memory allocations. Basically, it's as O(1) as it gets.
Will be in R75.
If the block is large (over 64K), then the app will also display its size on the right-hand side on the menu. The neat thing is that this information is readily available in the log index, so this requires no log scanning of any kind or no interim memory allocations. Basically, it's as O(1) as it gets.
Will be in R75.
Network Logon configuration has now been reworked
to allow configuring a couple of advanced aspects
of the share access.
Will be in R75.
Will be in R75.
Back to Bvckup 2 now and its upcoming Release 75.
With the next release the UI will start adopting a new "progressive refinement" approach for managing advanced and less-frequently used features.
After trying this and that, the design ended up converging to a simple More button at the bottom left corner of the dialog that, when pressed, injects additional settings into the dialog.
This is certainly not something new and revolutionary and it must've been done before, but it does feel a bit more natural compared to a more conventional approach of popping up a separate window with all the advanced extras.
Also, the initial iterations had the More button toggle between "More..." and "Less..." when clicked, but it looked a tad too busy, hence the current revision of a simple "sticky" background highlight when in the expanded state.
With the next release the UI will start adopting a new "progressive refinement" approach for managing advanced and less-frequently used features.
After trying this and that, the design ended up converging to a simple More button at the bottom left corner of the dialog that, when pressed, injects additional settings into the dialog.
This is certainly not something new and revolutionary and it must've been done before, but it does feel a bit more natural compared to a more conventional approach of popping up a separate window with all the advanced extras.
Also, the initial iterations had the More button toggle between "More..." and "Less..." when clicked, but it looked a tad too busy, hence the current revision of a simple "sticky" background highlight when in the expanded state.
Added support for displaying both effective and
all supported SATA signaling rates for physical
drives. These are also interchangeably knowns
as "transfer speeds" and "link rates".
Detecting supported signaling rates is trivial. After all, it is a part of the drive's ATA IDENTIFY block, word 76.
In comparison, drive's effective signaling rate is a bit trickier to determine. In theory, it is also a part of IDENTIFY block, word 77. In practice, this is only available from drives that support ACS-3, which is a more recent revision of the ATA command set.
So for drives that don't indicate their effective signaling rate Diskovery looks at "phy link rate" as reported by CSMI-compliant drive controllers.
But, wait!
What if both rate readings are available and they are different? Not possible, you say? "Ha-ha" says an outdated Intel RST driver that ships with Windows 8.1. In this case Diskovery will default to the drive's version of the value and also issue a warning of the discrepancy.
Detecting supported signaling rates is trivial. After all, it is a part of the drive's ATA IDENTIFY block, word 76.
In comparison, drive's effective signaling rate is a bit trickier to determine. In theory, it is also a part of IDENTIFY block, word 77. In practice, this is only available from drives that support ACS-3, which is a more recent revision of the ATA command set.
So for drives that don't indicate their effective signaling rate Diskovery looks at "phy link rate" as reported by CSMI-compliant drive controllers.
But, wait!
What if both rate readings are available and they are different? Not possible, you say? "Ha-ha" says an outdated Intel RST driver that ships with Windows 8.1. In this case Diskovery will default to the drive's version of the value and also issue a warning of the discrepancy.
Diskovery now recognizes commonly used emulated
RAM-based storage, like SoftPerfect RAM Disk
and Dataram RAMDisk.
The empty space at the bottom of the Diskovery
window is now used to show quick status messages.
A little opening animation is included.
Diskovery can now also talk to the
Virtual Disk Service
and extract all sort of interesting information from it,
including exact volume type (spanned, striped, mirrored,
etc.),
its online/offline status and its health.
Diskovery now detects the USB drive type and its effective
connection speed. No more USB3 drives sitting on USB2 hubs.
An opening screen for Diskovery, must be read
in Jeeves'es voice for full effect.
Not long now, almost done tying up all loose ends...
With Diskovery coming together,
I wanted to take a moment and show what a UI sketching
process for an app looks like.
In a vast majority of cases native UI controls (widgets)
work just fine. There's really no point in re-skinning,
say, a button unless, of course, your goal is to make it
stick out like a sore thumb.
There are however cases when the app needs to display information in a way that is not natively supported by the OS. In other cases, while it might be possible to render the data with a native control, it won't do it well or the result won't look good.
In these cases you have no option but to go with a custom UI, but, luckily, if you start with a need (rather than a quest for bells and whistles) the process is fairly predictable and logical. Diskovery deals with displaying the bits and pieces of the computer storage stack - from physical drives, to parititions, to volumes, to logical drives and mount points.
These fall naturally into a hierarchy, with each level comprising a list of key-value(s) pairs.
So taking the first stab at the UI, we would quickly arrive at something like this:
Primary list of drives and volumes on the left,
with details of a selected item on the right, organized
into a tree of "key:value" entries.
This is functional, but the right pane is not exactly easy to follow. So the first change will be to tabulate the key-value pairs:
Next, note that items in bold act like section
headers, so they don't really need to be collapsible.
So we do away with the buttons and connecting lines
between top-level items:
This gets us a bit of extra horizontal space and trims
a useless, but still actionable UI element.
Next, a couple of tweaks to how we render selected items.
There's no need for any items on the right to be selectable at all, so we won't render a selection highlight in the right pane.
The left pane can use a less jarring selection style, consistent with how Windows itself does it.
Next, we try and improve readability of the data on the
right by helping to guide the eye from a key to its
value.
The simplest option is the grid lines:
At this point, we take a sip of our cold coffee, squint
an eye and assess if we like how it looks.
It's not bad, but we can try something else:
Now, that's not bad either and it has a nice
visual consistency with the selection bar in the
left pane.
However a minor nit here is that we now have bold items (section headers) paired randomly with row highlights. See how Storage space is on white and Partition table is on light gray?
Moreover, adding or expanding items will cause this pairing to flip-flop, e.g. we add an extra row about Partition table and it now sits on the white background:
That's not a disaster, but we can do better.
The idea is to recognize section headers as such and style them independently:
With this change in place we can just restart the row
highlight sequence at the top of each section and
all's good.
Next, we apply the same styling to the left pane as it happens to also comprise of sections:
Another sip of coffee, re-squint the eye and
the whole thing now looks a bit cluttered and
busy.
Easy enough to rectify by adding a bit of vertical padding:
Next, we remove vertical connecting lines leading
from section headers to the top-level items.
However, we remove them on the right, but keep them on the left. This is because on the left we have a hierarchy of devices, so connection lines make sense. It also seems to look better this way.
Next, we move the expand/collapse buttons to
the right.
In this particular app expandable items contain secondary details that 99% of users will not ever need. These items will also be collapsed by default (even though they are expanded here, but that's just because we are sketching through the details).
This change also helps decreasing the leading pad of top-level items and providing for a more compact horizontal packing of the data.
Alright, so that's about it as far as functional
tweaks go. So now it's where it gets subjective. We are
going to spruce it up a bit.
It's all about colors, padding and little nuances. If we were designing for the web, it would also involve texturing, but for a desktop app texturing is a sure way to ruin native look and feel.
In any case, moving on - invert the headers from dark-on-light to light-on-dark:
Add a bit of hue to the row highlights, tint their
edges a bit and also flatten the look of
expand/collapse buttons:
Splice in a scroll bar and make sure our precious design
still looks reasonable:
Re-style the left pane to look more like an actual connected
hierarchy of disks and volumes. The rationale here is to make
it more obvious that items on the left are clickable.
Almost there. Take a final look and adjust vertical padding
and spacing to de-clump items a bit more:
And here it is. Compare to the
starting point and you'd
probably agree that there's at least some improvement.
Needless to say,
one man's extra vertical padding is another man's complete
waste of space, but any customization is always a combination
of rational and subjective choices. The former is what gives
an app the ease of use and the latter is what give it the
personality. Can't have one without the other.
There are however cases when the app needs to display information in a way that is not natively supported by the OS. In other cases, while it might be possible to render the data with a native control, it won't do it well or the result won't look good.
In these cases you have no option but to go with a custom UI, but, luckily, if you start with a need (rather than a quest for bells and whistles) the process is fairly predictable and logical. Diskovery deals with displaying the bits and pieces of the computer storage stack - from physical drives, to parititions, to volumes, to logical drives and mount points.
These fall naturally into a hierarchy, with each level comprising a list of key-value(s) pairs.
So taking the first stab at the UI, we would quickly arrive at something like this:
Primary list of drives and volumes on the left,
with details of a selected item on the right, organized
into a tree of "key:value" entries.
This is functional, but the right pane is not exactly easy to follow. So the first change will be to tabulate the key-value pairs:
Next, note that items in bold act like section
headers, so they don't really need to be collapsible.
So we do away with the buttons and connecting lines
between top-level items:
This gets us a bit of extra horizontal space and trims
a useless, but still actionable UI element.
Next, a couple of tweaks to how we render selected items.
There's no need for any items on the right to be selectable at all, so we won't render a selection highlight in the right pane.
The left pane can use a less jarring selection style, consistent with how Windows itself does it.
Next, we try and improve readability of the data on the
right by helping to guide the eye from a key to its
value.
The simplest option is the grid lines:
At this point, we take a sip of our cold coffee, squint
an eye and assess if we like how it looks.
It's not bad, but we can try something else:
Now, that's not bad either and it has a nice
visual consistency with the selection bar in the
left pane.
However a minor nit here is that we now have bold items (section headers) paired randomly with row highlights. See how Storage space is on white and Partition table is on light gray?
Moreover, adding or expanding items will cause this pairing to flip-flop, e.g. we add an extra row about Partition table and it now sits on the white background:
That's not a disaster, but we can do better.
The idea is to recognize section headers as such and style them independently:
With this change in place we can just restart the row
highlight sequence at the top of each section and
all's good.
Next, we apply the same styling to the left pane as it happens to also comprise of sections:
Another sip of coffee, re-squint the eye and
the whole thing now looks a bit cluttered and
busy.
Easy enough to rectify by adding a bit of vertical padding:
Next, we remove vertical connecting lines leading
from section headers to the top-level items.
However, we remove them on the right, but keep them on the left. This is because on the left we have a hierarchy of devices, so connection lines make sense. It also seems to look better this way.
Next, we move the expand/collapse buttons to
the right.
In this particular app expandable items contain secondary details that 99% of users will not ever need. These items will also be collapsed by default (even though they are expanded here, but that's just because we are sketching through the details).
This change also helps decreasing the leading pad of top-level items and providing for a more compact horizontal packing of the data.
Alright, so that's about it as far as functional
tweaks go. So now it's where it gets subjective. We are
going to spruce it up a bit.
It's all about colors, padding and little nuances. If we were designing for the web, it would also involve texturing, but for a desktop app texturing is a sure way to ruin native look and feel.
In any case, moving on - invert the headers from dark-on-light to light-on-dark:
Add a bit of hue to the row highlights, tint their
edges a bit and also flatten the look of
expand/collapse buttons:
Splice in a scroll bar and make sure our precious design
still looks reasonable:
Re-style the left pane to look more like an actual connected
hierarchy of disks and volumes. The rationale here is to make
it more obvious that items on the left are clickable.
Almost there. Take a final look and adjust vertical padding
and spacing to de-clump items a bit more:
And here it is. Compare to the
starting point and you'd
probably agree that there's at least some improvement.
Needless to say,
one man's extra vertical padding is another man's complete
waste of space, but any customization is always a combination
of rational and subjective choices. The former is what gives
an app the ease of use and the latter is what give it the
personality. Can't have one without the other.
Diskovery is a little companion app that throws
a simple UI on top of the new S.M.A.R.T. diagnostics
module of Bvckup 2.
The app will go live shortly and its main purpose is to let us estimate how well the diagnostics module fares against the real world hardware. Because there are specs and there's life.
In particular, certain USB drives and older RAID controllers don't always cooperate fully (or at all), and it would help to know what- and how common they are.
Once released, Diskovery will remain publicly available and we'll update and support it. It's a simple tool for quickly inspecting one's storage hardware and I can think of a number of cases where it can come handy.
The app will go live shortly and its main purpose is to let us estimate how well the diagnostics module fares against the real world hardware. Because there are specs and there's life.
In particular, certain USB drives and older RAID controllers don't always cooperate fully (or at all), and it would help to know what- and how common they are.
Once released, Diskovery will remain publicly available and we'll update and support it. It's a simple tool for quickly inspecting one's storage hardware and I can think of a number of cases where it can come handy.
Having a working backup is great, but knowing
if any drives are nearing their lifespan is
just as essential.
As you might've heard the next release of Bvckup 2 will be adding support for the drive health monitoring via S.M.A.R.T. diagnostics .
However, first things first, and today I'd like to do a quick review of what's what in the Windows data storage stack.
Read in full here => Drives, Partitions, Volumes
As you might've heard the next release of Bvckup 2 will be adding support for the drive health monitoring via S.M.A.R.T. diagnostics .
However, first things first, and today I'd like to do a quick review of what's what in the Windows data storage stack.
Read in full here => Drives, Partitions, Volumes
Final remark on general email config improvements in R75.
When connecting to a mail server over SSL/TLS, it's not uncommon for the server to present a certificate that doesn't match its hostname.
In particular, this happens when a custom domain is used with a managed (shared) mail service. You'd grab a domain, set its MX to mx.domain, add CNAME for mx.domain to point at mx.mail-service and - voilà! - you are served with mx.mail-service cert when connecting to mx.domain.
Ultimately, this is a mail server mis-configuration issue - your MX should really be pointing at mx.mail-service. In theory this shouldn't be happening, but in practice it does. To address this, it's now possible to specify the exact domain the app should be looking for in an SSL certificate:
In the above example, you'd simply set this field to mx.mail-service and that's it. To make things a bit easier that app now also helps you to pre-populate this field with the actual domain from the server's certificate.
It does that when you send out a test email and the field is left blank. As you can see from an opening screenshot, the app will recognize the certificate mismatch and re-run the test, accepting any and all certificates. It will then make note of the domain in the cert and copy it into the "SSL domain" field for you. Finally, setting "SSL domain" to * will effectively disable all certificate checks and force the app to accept any certificate that the server sends, even if it's an expired or a self-signed one.
When connecting to a mail server over SSL/TLS, it's not uncommon for the server to present a certificate that doesn't match its hostname.
In particular, this happens when a custom domain is used with a managed (shared) mail service. You'd grab a domain, set its MX to mx.domain, add CNAME for mx.domain to point at mx.mail-service and - voilà! - you are served with mx.mail-service cert when connecting to mx.domain.
Ultimately, this is a mail server mis-configuration issue - your MX should really be pointing at mx.mail-service. In theory this shouldn't be happening, but in practice it does. To address this, it's now possible to specify the exact domain the app should be looking for in an SSL certificate:
In the above example, you'd simply set this field to mx.mail-service and that's it. To make things a bit easier that app now also helps you to pre-populate this field with the actual domain from the server's certificate.
It does that when you send out a test email and the field is left blank. As you can see from an opening screenshot, the app will recognize the certificate mismatch and re-run the test, accepting any and all certificates. It will then make note of the domain in the cert and copy it into the "SSL domain" field for you. Finally, setting "SSL domain" to * will effectively disable all certificate checks and force the app to accept any certificate that the server sends, even if it's an expired or a self-signed one.
New alert configuration flow - coming up in R75.
† GIF is a bit choppy, sorry about that.
† GIF is a bit choppy, sorry about that.
Earlier versions had general and alert-specific settings
mixed up. For example, "From" doesn't
really belong to the alert configuration,
while "Include" from
General Settings does.
This has been sorted out.
A larger picture here is that the app will be getting additional alert types - e.g. for S.M.A.R.T. diagnostics and backup verification - and this requires a proper structure to support these cleanly. Also, while at it, the UI got a new framework for handling window transitions between different states.
For example, in the above screencap there's are 5 primary states and 2 intermediate ones. Prior to R75 moving window between these states required quite a bit of hand coding. This is no more, so we can now animate the hell out of any window. Consider yourselves warned :)
This has been sorted out.
A larger picture here is that the app will be getting additional alert types - e.g. for S.M.A.R.T. diagnostics and backup verification - and this requires a proper structure to support these cleanly. Also, while at it, the UI got a new framework for handling window transitions between different states.
For example, in the above screencap there's are 5 primary states and 2 intermediate ones. Prior to R75 moving window between these states required quite a bit of hand coding. This is no more, so we can now animate the hell out of any window. Consider yourselves warned :)
How about a rant? :)
The year is 2015.
The age of cars trying to drive themselves and rockets trying to land on sea barges. Graphics cards are effortlessly churning photo-realistic images in real-time, mining crypto money and cracking passwords.
And here we have Windows still defiantly refusing to use any of that double-buffering nonsense when re-painting one of its basic common controls. Because who wouldn't want a bit of flickering?
Argh.
The age of cars trying to drive themselves and rockets trying to land on sea barges. Graphics cards are effortlessly churning photo-realistic images in real-time, mining crypto money and cracking passwords.
And here we have Windows still defiantly refusing to use any of that double-buffering nonsense when re-painting one of its basic common controls. Because who wouldn't want a bit of flickering?
Argh.
Tab control for multi-page configuration windows in general,
and for the email notification configuration in particular.
Prior to R75 there was one primary email alert, which was sent on backup completion. Now the app is getting other types of alerts, so the UI needs to be adapted to support that.
Prior to R75 there was one primary email alert, which was sent on backup completion. Now the app is getting other types of alerts, so the UI needs to be adapted to support that.
The idea is that the config window for specific alert type
will give access to both the alert details and general
email settings that are recycled between all alert types.
So here are several sketches for organizing page index of a multi-page configuration window. Nothing terribly novel here, it's more just a matter of getting styling details right - make sure it's obvious that it's an index and that its items are selectable. Option Z (right-most) looks OK, but it's not obvious that index items are actionable as they look like static section headers.
Option Y is a bit better, but the arrow actually points at "From" field and you can bet some people will end up in a stupor because of that.
Option X uses standard Windows white-on-blue for selected item, so it's better in the actionability department. But it doesn't look nice.
Option B attempts to fix that.
Option A attempts the same but using app's own styling of selected items, which is already used in several places in the UI.
Thoughts or comments? There is a thread for that!
So here are several sketches for organizing page index of a multi-page configuration window. Nothing terribly novel here, it's more just a matter of getting styling details right - make sure it's obvious that it's an index and that its items are selectable. Option Z (right-most) looks OK, but it's not obvious that index items are actionable as they look like static section headers.
Option Y is a bit better, but the arrow actually points at "From" field and you can bet some people will end up in a stupor because of that.
Option X uses standard Windows white-on-blue for selected item, so it's better in the actionability department. But it doesn't look nice.
Option B attempts to fix that.
Option A attempts the same but using app's own styling of selected items, which is already used in several places in the UI.
Thoughts or comments? There is a thread for that!
Minor tweak to the look of the delta copying progress bar -
a bit more contrast, both for completed and outstanding
parts.
Will be in R74.15
Will be in R74.15
Third pass.
First stab at backup grouping UI for the next release (R75).
Thoughts or comments? There is a thread for that!
Thoughts or comments? There is a thread for that!
There's a good guiding principle coming from Paul Graham that says
make something people want.
It's very simple and it's very effective.
It also makes real-world user opinions a critical part of any
good product design.
In other words - please take a moment and talk to us! :)
In other words - please take a moment and talk to us! :)
Unicode,
as you may know, is a standard that assigns pretty pictograms to
numerical codes from the 0 to 0x0010FFFF range.
In simpler cases pictograms represent letters, numbers and symbols. In other cases they are decorative glyph elements, like umlauts, that are meant to be combined with the letters.
In simpler cases pictograms represent letters, numbers and symbols. In other cases they are decorative glyph elements, like umlauts, that are meant to be combined with the letters.
There are also Unicode encodings - UTF-8, UTF-16, etc. -
these deal with how specific Unicode numerical IDs are translated
into sequences of *bytes* for storage, transmission, etc.
The simplest encoding is UTF-32 - it simply stores Unicode ID as a 32-bit number. This is however rather wasteful as at most 21 bit is needed to represent any Unicode ID.
Prior to Windows 2000, Microsoft used a simple 16-bit encoding that mapped every Unicode ID from the 0 to 0x00FFFF range to the exact same 16-bit number. This is called UCS-2 encoding. The Unicode range was smaller back then, so this approach worked.
Then, Unicode range grew and IDs no longer fit into 16 bits. So to address this, UCS-2 was modified to encode symbols from the 0x010000 - 0x10FFFF range using two 16-bit numbers, from the 0xD800 - 0xDFFF range. Exact details of this encoding are not important, but what it meant was that some 16-bit numbers, e.g. 0xD812, were no longer valid Unicode symbols just on their own.
This is called UTF-16 encoding, and this is what Windows uses at the moment.
But wait !
As Apple likes to say after a screw-up - "it turns out" that despite routinely talking about UTF-16 in their API documentation, Windows file system still uses plain 16-bit encoding for the file names!
For example, "\xD812Haha" is a valid file name even though it is *not* a valid UTF-16 string. This in turn means that if the app is converting file names into other encoding, it should go easy on full UTF-16 compliance checks and be ready to digest ill-formed UTF-16 string.
Bvckup 2 keeps all text in UTF-8 format internally, so it does quite a bit of conversion from/to UTF-16 and this leads to some interesting things when the app runs into an malformed UTF-16 name. Like those created by *ahem* Adobe *ahem* After Effects *ahem*, for example.
The solution is simple - try and convert as if we have the UTF-16 encoding, but fallback to UCS-2 when running into invalid UTF-16 sequences. Also, apply the same approach when converting back to UTF-16 and all will be good.
Will be in Release 74.8.
* Kudos to Andrea for reporting this.
The simplest encoding is UTF-32 - it simply stores Unicode ID as a 32-bit number. This is however rather wasteful as at most 21 bit is needed to represent any Unicode ID.
Prior to Windows 2000, Microsoft used a simple 16-bit encoding that mapped every Unicode ID from the 0 to 0x00FFFF range to the exact same 16-bit number. This is called UCS-2 encoding. The Unicode range was smaller back then, so this approach worked.
Then, Unicode range grew and IDs no longer fit into 16 bits. So to address this, UCS-2 was modified to encode symbols from the 0x010000 - 0x10FFFF range using two 16-bit numbers, from the 0xD800 - 0xDFFF range. Exact details of this encoding are not important, but what it meant was that some 16-bit numbers, e.g. 0xD812, were no longer valid Unicode symbols just on their own.
This is called UTF-16 encoding, and this is what Windows uses at the moment.
As Apple likes to say after a screw-up - "it turns out" that despite routinely talking about UTF-16 in their API documentation, Windows file system still uses plain 16-bit encoding for the file names!
For example, "\xD812Haha" is a valid file name even though it is *not* a valid UTF-16 string. This in turn means that if the app is converting file names into other encoding, it should go easy on full UTF-16 compliance checks and be ready to digest ill-formed UTF-16 string.
Bvckup 2 keeps all text in UTF-8 format internally, so it does quite a bit of conversion from/to UTF-16 and this leads to some interesting things when the app runs into an malformed UTF-16 name. Like those created by *ahem* Adobe *ahem* After Effects *ahem*, for example.
The solution is simple - try and convert as if we have the UTF-16 encoding, but fallback to UCS-2 when running into invalid UTF-16 sequences. Also, apply the same approach when converting back to UTF-16 and all will be good.
Will be in Release 74.8.
* Kudos to Andrea for reporting this.
Release 74 also implements new delta copying defaults
and the results are ... interesting :)
Delta copying, when enabled, has always been used conditionally
and the criteria was the size of the file. For smaller
files the overhead of setting up the delta copier is
larger than any potential savings, so these files
are simply copied in full.
The file size threshold was initially set to 128KB and the value was inherited from the original Bvckup version, where it made sense because the I/O was generally slower back then.
Fast forward few years and 128KB is no longer a good default, so as of Release 71 it was increased to 2MB. With Release 74 the criteria changes again.
First, the threshold goes up to 32MB.
Second, the delta copying will also be used for files between 2MB and 32MB, but only IF they were modified in the last 30 days OR if they were previously delta-copied.
The rationale is pretty simple - we try and not bother with delta copying unless files are large or they change frequently. This change has two main effects.
1. Smaller backup state.
Delta copying stores per-block hashes in a backup's configuration directory. This is done for every file that is delta-copied, so the fewer files are, the less data ends up being in the \deltas folder.
The above screenshot shows the \deltas folder stats after backing up \Windows\System32 directory. And here are the same stats after backing up with an older version -
That is, twice the disk usage and nearly 10x more files and folders in the \deltas folder.
2. Faster overall backups.
For smaller files the overhead of setting up the delta state is comparable to the time needed for the actual copying. We skip the delta setup -> we complete the copying faster. Magic.
Before:
After:
That's nearly a 10% speed up. This is obviously specific to the contents of \Windows\System32, but it's not hard to extrapolate from here.
The file size threshold was initially set to 128KB and the value was inherited from the original Bvckup version, where it made sense because the I/O was generally slower back then.
Fast forward few years and 128KB is no longer a good default, so as of Release 71 it was increased to 2MB. With Release 74 the criteria changes again.
First, the threshold goes up to 32MB.
Second, the delta copying will also be used for files between 2MB and 32MB, but only IF they were modified in the last 30 days OR if they were previously delta-copied.
The rationale is pretty simple - we try and not bother with delta copying unless files are large or they change frequently. This change has two main effects.
1. Smaller backup state.
Delta copying stores per-block hashes in a backup's configuration directory. This is done for every file that is delta-copied, so the fewer files are, the less data ends up being in the \deltas folder.
The above screenshot shows the \deltas folder stats after backing up \Windows\System32 directory. And here are the same stats after backing up with an older version -
That is, twice the disk usage and nearly 10x more files and folders in the \deltas folder.
2. Faster overall backups.
For smaller files the overhead of setting up the delta state is comparable to the time needed for the actual copying. We skip the delta setup -> we complete the copying faster. Magic.
Before:
After:
That's nearly a 10% speed up. This is obviously specific to the contents of \Windows\System32, but it's not hard to extrapolate from here.
Release 74 will also be adding an option
for exporting app's complete configuration
as a single .zip file.
Restoring app's config from this backup is a simple matter of shutting down the app and extracting contents of the .zip into %LocalAppData%\Bvckup2\ folder.
But since we can hardly expect anyone to know this location by heart, a short text file called How to restore.txt is also included with each backup zip.
Restoring app's config from this backup is a simple matter of shutting down the app and extracting contents of the .zip into %LocalAppData%\Bvckup2\ folder.
But since we can hardly expect anyone to know this location by heart, a short text file called How to restore.txt is also included with each backup zip.
Run-time memory usage is now capped.
Prior to Release 74 app's memory usage was proportional to the number of items in the backup. This was because both the source and destination file trees were kept in memory in their entirety, as was the backup plan. While not a big deal for smaller backups, it does translate into quite a bit of RAM when the file count starts to climb into a million range.
Prior to Release 74 app's memory usage was proportional to the number of items in the backup. This was because both the source and destination file trees were kept in memory in their entirety, as was the backup plan. While not a big deal for smaller backups, it does translate into quite a bit of RAM when the file count starts to climb into a million range.
Release 74 implements the off-RAM storage for
large data sets. In previous release the run-time
arrangement was like this -
With the new release the objects are kept in the app's memory only if they are in active use. The rest is trickled down into a swap file for storage, through a couple of intermediate caches.
The object cache holds fully assembled instances of application data objects and its size is kept at or below preset item count.
When the object cache grows too big, least recently used objects are written down to the swap file. The writes are passed through a typical page cache to coalesce the I/O into larger chunks. This lunch is obviously not free.
There's a cost to even successful cache lookups and there's most certainly an expense to doing the disk I/O.
The app mitigates this by using aggressive cache optimization (buckets, MRU lookups, etc), raw IO caching and also by allowing keeping swap files in a custom location, and presumably on a different physical drive. But, again, all this starts to really matter only when backups grow into a million item range.
While reasonable defaults are provided, the cache sizes, the trimming policy and other details are fully configurable through the .ini files. The app may also be set up to log cache/swap performance if required. This change is big, it took nearly two months to implement and it means that Bvckup 2 can now handle backups of unlimited size. To say that it's a very big deal would be an understatement.
With the new release the objects are kept in the app's memory only if they are in active use. The rest is trickled down into a swap file for storage, through a couple of intermediate caches.
The object cache holds fully assembled instances of application data objects and its size is kept at or below preset item count.
When the object cache grows too big, least recently used objects are written down to the swap file. The writes are passed through a typical page cache to coalesce the I/O into larger chunks. This lunch is obviously not free.
There's a cost to even successful cache lookups and there's most certainly an expense to doing the disk I/O.
The app mitigates this by using aggressive cache optimization (buckets, MRU lookups, etc), raw IO caching and also by allowing keeping swap files in a custom location, and presumably on a different physical drive. But, again, all this starts to really matter only when backups grow into a million item range.
While reasonable defaults are provided, the cache sizes, the trimming policy and other details are fully configurable through the .ini files. The app may also be set up to log cache/swap performance if required. This change is big, it took nearly two months to implement and it means that Bvckup 2 can now handle backups of unlimited size. To say that it's a very big deal would be an understatement.
Still working on major engine changes to cap its
run-time use (and I will be posting data from the
test runs shortly), but meanwhile here's something
simple that I'm considering adding to the next
release.
It's an option for the app to report your backup statistics to one of our servers and then being able to display these stats as a simple PNG served over HTTP from the same stats server.
It's really meant for trying to motivate other people to back up their things as well as to indirectly promote the app as well. And, yes, I lived through the 90s, that's where I got the idea from :)
It's an option for the app to report your backup statistics to one of our servers and then being able to display these stats as a simple PNG served over HTTP from the same stats server.
It's really meant for trying to motivate other people to back up their things as well as to indirectly promote the app as well. And, yes, I lived through the 90s, that's where I got the idea from :)
Currently working on the improvements to the
scanning and planning modules of the app. No pretty
UI screenshots to show, just some performance numbers.
The ultimate goal of this rework is to put a cap on the
run-time memory usage, which in turn is needed to better
accommodate very large backups (tens of millions files
and up).
The app comes with a formal backup planner, which is
a module that accepts two full directory trees (one
for the source location and another for the backup one),
digests them and produces a list of simple steps that,
when executed in order, bring one tree in sync with another.
The app is then diligently goes through the list
and creates, updates, renames and deletes files
and folders as directed.
An alternative to formal planning is to to traverse the trees and make on-disk changes just as you spot the differences. There are several drawback to this approach, including not being able to estimate the amount of work needed, to understand disk space requirements and to detect file/folder moves. It's just an altogether messier and less predictable way to go about the whole thing.
That's how Bvckup 1 worked and that's how robocopy works, for example. Long story short, having a backup planner is the right thing to do. However, it means that the app needs to have two full file trees readily available for the planner to digest. If these trees are kept entirely in memory, it may get expensive with large backups. At 100 to 300 bytes per item, a million file backup means 200 to 600 megs of RAM just for the trees.
That's quite a bit, but more importantly this memory usage grows linearly with the size of the backup. This is no good. The solution obviously enough is to not keep full trees in memory. This is a fairly complicated endeavour and it maps onto several large changes to the app inner structure.
One such change is nearly done. Both the scanning planning modules no longer require for the trees to be resident in memory at all times. Instead, they now use them in a piecemeal fashion.
Behind the scenes, trees still reside in memory for the time being, but the underlying rework has some nice side effects. Trees now take less memory, they are faster to populate, faster to save and load to/from a file, they take substantally less space on disk and they compress better with NTFS compression.
This will ship as a part of R74.
After that, R75 will replace in-memory trees with version that can swap out tree parts to the disk, significantly reducint the run-time memory footprint of the scanning/planning process.
An alternative to formal planning is to to traverse the trees and make on-disk changes just as you spot the differences. There are several drawback to this approach, including not being able to estimate the amount of work needed, to understand disk space requirements and to detect file/folder moves. It's just an altogether messier and less predictable way to go about the whole thing.
That's how Bvckup 1 worked and that's how robocopy works, for example. Long story short, having a backup planner is the right thing to do. However, it means that the app needs to have two full file trees readily available for the planner to digest. If these trees are kept entirely in memory, it may get expensive with large backups. At 100 to 300 bytes per item, a million file backup means 200 to 600 megs of RAM just for the trees.
That's quite a bit, but more importantly this memory usage grows linearly with the size of the backup. This is no good. The solution obviously enough is to not keep full trees in memory. This is a fairly complicated endeavour and it maps onto several large changes to the app inner structure.
One such change is nearly done. Both the scanning planning modules no longer require for the trees to be resident in memory at all times. Instead, they now use them in a piecemeal fashion.
Behind the scenes, trees still reside in memory for the time being, but the underlying rework has some nice side effects. Trees now take less memory, they are faster to populate, faster to save and load to/from a file, they take substantally less space on disk and they compress better with NTFS compression.
This will ship as a part of R74.
After that, R75 will replace in-memory trees with version that can swap out tree parts to the disk, significantly reducint the run-time memory footprint of the scanning/planning process.
Release 73 adds copying speed and completion
time estimation for longer file copies. To
toggle between the views just click on the
text next to the progress bar.
The estimation is calculated in three steps.
First, the app looks at effective copying speed and uses sliding window to average it across last 25 samples taken 650 ms apart.
Second, it uses this average speed and the remaining byte count to approximate the completion time.
Finally, the completion time is averaged with its own sliding window, but using only 4 past samples.
This double-averaging helps mitigating the effects of sudden short-lived I/O bursts and hiccups on the estimation.
The parameters - 25, 650 and 4 - are configurable through the engine .ini file, so the whole thing is open for tuning if needs be.
First, the app looks at effective copying speed and uses sliding window to average it across last 25 samples taken 650 ms apart.
Second, it uses this average speed and the remaining byte count to approximate the completion time.
Finally, the completion time is averaged with its own sliding window, but using only 4 past samples.
This double-averaging helps mitigating the effects of sudden short-lived I/O bursts and hiccups on the estimation.
The parameters - 25, 650 and 4 - are configurable through the engine .ini file, so the whole thing is open for tuning if needs be.
The initial revision of the auto-update opton
is in.
It's a simple extension of existing "check for updates" option that skips showing the "new version found" prompt and just goes straight to the updating part. There are two caveats, both applicable only when the app is running in service mode and both stemming from the fact the updates are driven by the UI.
First is that closing the UI will effectively disable auto-updates. Second - if the UI is running without admin privileges, running the installer will trigger the UAC prompt.
Both will be resolved shortly by moving all update logic into the engine module. Meanwhile...
The auto-update option will be hidden by default unless the Preferences window is opened with the Control button held down .
It's a simple extension of existing "check for updates" option that skips showing the "new version found" prompt and just goes straight to the updating part. There are two caveats, both applicable only when the app is running in service mode and both stemming from the fact the updates are driven by the UI.
First is that closing the UI will effectively disable auto-updates. Second - if the UI is running without admin privileges, running the installer will trigger the UAC prompt.
Both will be resolved shortly by moving all update logic into the engine module. Meanwhile...
The auto-update option will be hidden by default unless the Preferences window is opened with the Control button held down .
Release 73 will also be adding support for monthly
backups. It will now be possible to set a backup
to run on the N-th of every M-th month.
You surely are thinking "but 31st!", "February!" or perhaps even "every 6 months staring on Jan 31st!"
All fair points. Fear not, the UI got this covered:
You surely are thinking "but 31st!", "February!" or perhaps even "every 6 months staring on Jan 31st!"
All fair points. Fear not, the UI got this covered:
A rework of the loggging module, coming up in Release 73.
As you may or may not know, Bvckup 2 uses a
dual-module architecture of the inside.
One module is the backup engine. It maintains
the list of backup jobs, schedules, executes
backups and generally does other useful
things that don't require direct user involvement.
The second module is the user interface. This is the part that displays the app's window and allows users to monitor and control whatever the engine module is doing.
The principal point is that the engine and the UI are isolated from each other. They share no run-time data and communicate with each other using a messaging protocol.
This isolation is what enables the app to run in service mode. In this mode the engine runs in its own process, as a system service, and the UI runs as a regular desktop app.
It was all well and beautiful, but there was, however, a caveat. The caveat was the backup logs.
Logs are inherently shared data - they are written by the engine and they are read and displayed by the UI. Logs tend to be bulky, so it is shared app data of a special kind and it requires special handling.
Prior to R73 the app included a third module (in its dual-module architecture, no less) - the log manager. It was a complicated contraption that abstracted log access for the engine and the UI, and the biggest issue with it was that it behaved completely differently depending on which mode the app was in.
In desktop mode, the manager was writing *and* indexing logs on behalf of the engine and the UI was merely displaying them. However, in service mode, the engine did no indexing, just the writing, while the UI was making copies of all logs for its own use and then it was indexing *them*.
This is exactly the kind of non-uniform conditional processing that the dual-module design was meant to avoid! Enter R73.
The separate logging module is no more. The engine no longer does any log indexing, it is now entirely the UI's reponsibility. Indexes can now be rebuilt on demand at any time, and finally, the UI no longer creates its own copies of the log files. Instead, it works with the engine to gain access to the engine's copy of the logs, using DuplicateHandle to move open file handles across the process boundary when needed.
The UI now can display logs while it is still catching up with them.
Think -- running in service mode and launching the UI only once a week. On each launch the UI needs to index a week's worth of log entries it hasn't seen before. Prior to R73, the UI would just sit there and show "synching ... XX% done". Now the sync is done in the background (as per the screenshot).
The UI now can also recover gracefully from indexes getting corrupted (e.g. due to a cold power cycle or blue screen).
Just as important, the underlying code is now much simpler, lighter and more uniform, the executable is 8KB smaller and the developer is generally happier. Good news all around.
The second module is the user interface. This is the part that displays the app's window and allows users to monitor and control whatever the engine module is doing.
The principal point is that the engine and the UI are isolated from each other. They share no run-time data and communicate with each other using a messaging protocol.
This isolation is what enables the app to run in service mode. In this mode the engine runs in its own process, as a system service, and the UI runs as a regular desktop app.
It was all well and beautiful, but there was, however, a caveat. The caveat was the backup logs.
Logs are inherently shared data - they are written by the engine and they are read and displayed by the UI. Logs tend to be bulky, so it is shared app data of a special kind and it requires special handling.
Prior to R73 the app included a third module (in its dual-module architecture, no less) - the log manager. It was a complicated contraption that abstracted log access for the engine and the UI, and the biggest issue with it was that it behaved completely differently depending on which mode the app was in.
In desktop mode, the manager was writing *and* indexing logs on behalf of the engine and the UI was merely displaying them. However, in service mode, the engine did no indexing, just the writing, while the UI was making copies of all logs for its own use and then it was indexing *them*.
This is exactly the kind of non-uniform conditional processing that the dual-module design was meant to avoid! Enter R73.
The separate logging module is no more. The engine no longer does any log indexing, it is now entirely the UI's reponsibility. Indexes can now be rebuilt on demand at any time, and finally, the UI no longer creates its own copies of the log files. Instead, it works with the engine to gain access to the engine's copy of the logs, using DuplicateHandle to move open file handles across the process boundary when needed.
The UI now can display logs while it is still catching up with them.
Think -- running in service mode and launching the UI only once a week. On each launch the UI needs to index a week's worth of log entries it hasn't seen before. Prior to R73, the UI would just sit there and show "synching ... XX% done". Now the sync is done in the background (as per the screenshot).
The UI now can also recover gracefully from indexes getting corrupted (e.g. due to a cold power cycle or blue screen).
Just as important, the underlying code is now much simpler, lighter and more uniform, the executable is 8KB smaller and the developer is generally happier. Good news all around.
Running an extended set of tests to fine-tune the
performance of the Bvckup 2 bulk copier.
It's an inherently simple matter of determining an optimal size and count of read/write buffers used to ferry data from source to destination. There are however ... erm ... details.
It's an inherently simple matter of determining an optimal size and count of read/write buffers used to ferry data from source to destination. There are however ... erm ... details.
A source can be local or it can be over-the-network.
There are slow local devices (e.g. a USB2 drive) and
there are fast local devices. There is a warm, there
is a cold cache and there are files that are cached
only partially.
The writes can go out in lots of smaller chunks, all in parallel, or with a smaller number of large buffers. Again, it can go to a local device or over the network. The app can also explicitly flush all buffers after copying or leave flushing to the system. The above graph is the I/O throughput of a 0.5 GB file being repeatedly copied from the local HDD to Synology Diskstation over a Gigabit wire. What changes between the runs is the buffering strategy and as you may notice there is a difference.
Still need to run several over-nighters to gather more statistics, but there is already a couple of things that can be changed in the copying code to make it run faster.
Scheduled to appear in the Release 72, coming up shortly.
The writes can go out in lots of smaller chunks, all in parallel, or with a smaller number of large buffers. Again, it can go to a local device or over the network. The app can also explicitly flush all buffers after copying or leave flushing to the system. The above graph is the I/O throughput of a 0.5 GB file being repeatedly copied from the local HDD to Synology Diskstation over a Gigabit wire. What changes between the runs is the buffering strategy and as you may notice there is a difference.
Still need to run several over-nighters to gather more statistics, but there is already a couple of things that can be changed in the copying code to make it run faster.
Scheduled to appear in the Release 72, coming up shortly.
"Send test email..." is now done.
R71 adds an option of automatically enabling
NTFS compression
for the backup logs.
Based on what I've seen so far, both .log and .tdb files compress to about 20-30% of their raw sizes.
Based on what I've seen so far, both .log and .tdb files compress to about 20-30% of their raw sizes.
Reworked part of the More Options page in
the backup configuration window.
* The animation here is a bit jerky, but it's just a GIF encoding artefact.
* The animation here is a bit jerky, but it's just a GIF encoding artefact.
Email template editor. "Show variables" opens
up another part that lists all variables that
can be used in the form fields.
This is getting a bit too complex for my taste, but it's the part of the UI with little foot traffic, so it's probably OK.
This is getting a bit too complex for my taste, but it's the part of the UI with little foot traffic, so it's probably OK.
Upon even further consideration it looks like
the From field is not of a prime interest either,
so it too can go into the Details.
This cuts down the number of fields and renders section headers unnecessary, so they too can go.
Basically the idea is to reduce the dialog down to fields that actually need user's input and stash everything else away... I don't know if it's the right thing to do though. Perhaps having a summary of the whole email setup (including the subject, From and the message) is not such a bad thing. Not sure. Need to sleep on it.
This cuts down the number of fields and renders section headers unnecessary, so they too can go.
Basically the idea is to reduce the dialog down to fields that actually need user's input and stash everything else away... I don't know if it's the right thing to do though. Perhaps having a summary of the whole email setup (including the subject, From and the message) is not such a bad thing. Not sure. Need to sleep on it.
Upon further consideration, decided to remove
both Subject and Message off this dialog to
keep it simple(r) and to the point.
First stab at the email configuration window.
Too many options, too little space. As usual.
EHLO there!
Release 71 is getting email notifications, so here's all you always wanted to know about teaching your app speak SMTP.
First of all, let's get MAPI non-sense out of the way. As per usual, it sounds good in theory, but an awful mess in practice. So, no, MAPI is not an option unless your goal is to burn through your support budget or to practice your communication skills with the help of less-than-happy users.
This leaves us with raw SMTP.
SMTP is an inherently simple protocol - send a text command, receive a text response. Repeat. Open a socket, connect, send(), recv(), done. A couple of hours of work at most. Ha. It starts to get complicated when you throw TLS in a mix to secure the communication with the server. In particular, this is required when you want to send emails in a way that requires logging into a server first.
There are two ways in which SMTP can be combined with TLS.
First is when you take an existing SMTP session and say STARTTLS. The server says OK and then you basically stop talking SMTP over your TCP connection and start talking TLS. Then, once the TLS handshake completes, you go back to SMTP, but now it all goes over the TLS session, nice and encrypted. This is a standard way, that's what every decent SMTP client should do. This is TCP/25 and TCP/587.
Second option is SMTPS. You first establish a TLS session and then start talking SMTP over it. Just like HTTPS. Apparently, this is non-standard and discouraged way, but still widely used. This is TCP/465. Then, there's a matter of how to add TLS support to the app.
The obvious choice is OpenSSL, but that will cost you 200-300KB in extra binary code, so unless you already have OpenSSL linked in, it's not really an option, not for a *backup* app.
Digging through MSDN you may come across something called SSPI or Security Support Provider Interface. It is an attempt to stuff several security protocols like SSL and Kerberos into a single framework in a faint hope for a miracle of reuse. One of SSPI providers is a mysterious entity called "Secure channel", which, upon closer inspection, turns out to be ... yes, TLS!
So, the next step is try and work through all the abstraction fluff of SSPI to make it actually do the TLS handshake. The idea is actually quite neat - you can create a stateful object (context) that takes in data you get from the peer and spits out some data for the peer. At some point it completes the handshake and becomes available for accepting app's own data, which it encrypts and hands you a blob to send to the peer. You would then also get blobs from the peer, pass it to this context/object and get back plaintext data for the app. Whether it's TLS or ROT13 on the back is completely transparent to the app.
That's the idea. In reality, the documentation is subpar and lacking and there are exactly two examples - one is incomplete and another one is buggy.
The good news is though is that once this bloody contraption gets off the ground, you get full TLS support at the cost of just several kilobytes. With TLS done, there's still a bunch of smaller things to be taken care of, e.g. lack of asynchronous DNS query API in pre-W8 windows. Not a big deal, nothing a spawning of a worker thread would solve, but still ... a bit here, a bit there and it all adds up.
In other words, it's all coming together, but as per usual it all takes longer than anticipated. Shocking, I know :)
Finalized the UX for auto-configuration of
the device tracking. It's essentially the
same as animation immediately below except
for a brief flash of blue when tracking
config is touched by the app.
Another subtle UI change in Release 71.
When configuring or reconfiguring a backup, the app would automatically enable/disable device tracking when a location is changed to use a different drive.
Since this change happens behind the scenes, the app will attract the attention to the fact with this little animation. When hovered over, the tooltip for the pushpin button will explain what's up and suggest reviewing the changes.
My only concern is that this still might be a bit too subtle, but it's better than what's in there now... which is no notification at all.
When configuring or reconfiguring a backup, the app would automatically enable/disable device tracking when a location is changed to use a different drive.
Since this change happens behind the scenes, the app will attract the attention to the fact with this little animation. When hovered over, the tooltip for the pushpin button will explain what's up and suggest reviewing the changes.
My only concern is that this still might be a bit too subtle, but it's better than what's in there now... which is no notification at all.
New UI for configuring device tracking in R71.
This is one of those changes that look almost trivial when done, but behind the scenes take a disproportionate amount of time to actually flesh out.
In this case, the main problem was that the feature needs a detailed explanation and the question, of course, was how one can fit a bunch of text into a reasonably small space so not to have this text take over the actual configuration section.
This is one of those changes that look almost trivial when done, but behind the scenes take a disproportionate amount of time to actually flesh out.
In this case, the main problem was that the feature needs a detailed explanation and the question, of course, was how one can fit a bunch of text into a reasonably small space so not to have this text take over the actual configuration section.
The first iteration was the good old message box. You
click on the "More on this..." and you get :
Nice, but looks like a hackjob, does't it? So, naturally, the next step was to spruce it up a bit - put text into a dedicated window, use RTF to add a bit of in-text formatting and a bit of animation to liven things up :
Nice again, looks better, but feels off. Too much going on. It looks like a conventional help window, but ... crippled. It's also not clear how to act when the user wants to resize the window - allow it, not allow it, restict minimum/maximum size, etc.
In short, what's lacking here is the cohesion. Luckily, the solution is right on the surface:
... whereby the text panel slides out on the "More info" click.
This is better, but now we have a window that sits lopsided on the desktop when the panel is open. It is also becoming wide, meaning that the app needs to mind the desktop edges, re-center the window, etc.
But if we just sleep on it, we might suddenly decide to make it an "either-or" window and then the UI magically becomes
(a) compact
(b) cohesive
(c) balanced
and
(d) with unlimited text space!
So there you have it - a new mini-help subsystem coming to Bvckup 2 in the Release 71, deployed on as-needed basis across the app.
Don't know about you, but I think it's neat, tidy and pretty damn clever :)
Nice, but looks like a hackjob, does't it? So, naturally, the next step was to spruce it up a bit - put text into a dedicated window, use RTF to add a bit of in-text formatting and a bit of animation to liven things up :
Nice again, looks better, but feels off. Too much going on. It looks like a conventional help window, but ... crippled. It's also not clear how to act when the user wants to resize the window - allow it, not allow it, restict minimum/maximum size, etc.
In short, what's lacking here is the cohesion. Luckily, the solution is right on the surface:
... whereby the text panel slides out on the "More info" click.
This is better, but now we have a window that sits lopsided on the desktop when the panel is open. It is also becoming wide, meaning that the app needs to mind the desktop edges, re-center the window, etc.
But if we just sleep on it, we might suddenly decide to make it an "either-or" window and then the UI magically becomes
(a) compact
(b) cohesive
(c) balanced
and
(d) with unlimited text space!
So there you have it - a new mini-help subsystem coming to Bvckup 2 in the Release 71, deployed on as-needed basis across the app.
Don't know about you, but I think it's neat, tidy and pretty damn clever :)
Added support for the reseller licensing
to the app. Any license can now be tagged as
being sold through a distributor, in which
case Help/About will show distributor's name
and link to its site.
Additionally, Tech Support and Homepage links from the Help menu may be configured to go to the distributor's pages.
It may not look like much, but it's the most important non-technical feature to be added to the app to date.
More on this to follow...
Additionally, Tech Support and Homepage links from the Help menu may be configured to go to the distributor's pages.
It may not look like much, but it's the most important non-technical feature to be added to the app to date.
More on this to follow...
Another option, as per
this comment.
It's a really nice concept, but it requires a squint
to realize what the icon is. Highly-detailed 16x16 px
icons are hard to pull off.
Release 71 is going to reshuffle the contents of
the backup configuration window a bit and location
fields are getting equipped with a "device tracking"
icon as a result.
In other words, for a network location the app will display the "key" icon (see /wip entry immediately below) and for local paths it will show "tracking" icon. Both fronting respective dialog for configuring the details.
The main question, of course, is what the icon should be ...
In other words, for a network location the app will display the "key" icon (see /wip entry immediately below) and for local paths it will show "tracking" icon. Both fronting respective dialog for configuring the details.
The main question, of course, is what the icon should be ...
In a process of adding a way to configure
network logon credentials for remote locations.
Will be in Release 70.
Release 69 also adds an option of simulated backups
or, as they are also known, "dry runs".
The option is accessible via the right-click context menu and, when invoked, it puts the backup through its preparation phase, including snapshot creation, scanning and planning steps, and then dumps compiled backup plan to the log without executing it.
Simulation also executes the archive pruning without making any actual changes to the destination.
The option is accessible via the right-click context menu and, when invoked, it puts the backup through its preparation phase, including snapshot creation, scanning and planning steps, and then dumps compiled backup plan to the log without executing it.
Simulation also executes the archive pruning without making any actual changes to the destination.
Release 69 adds support for optional VSS writers.
VSS writers are the components of the shadow copying
framework that are tasked with ensuring consistency
of specific files in VSS snapshots. For example,
when creating a snapshot, Windows would ask the MSSQL
writer to flush any pending database changes in a way
that would make on-disk copy of the database self-consistent.
Some of the writers are required and always participate in the snapshot creation process, but some are optional and they need to be explicitly enabled by the initiating app. Starting with R69 Bvckup will be enabling all optional writers and it will also provide a mechanism for excluding specific ones if needed.
Some of the writers are required and always participate in the snapshot creation process, but some are optional and they need to be explicitly enabled by the initiating app. Starting with R69 Bvckup will be enabling all optional writers and it will also provide a mechanism for excluding specific ones if needed.
It's shipped!
Here it is
Here it is
Nearly done with the checkout flow on the
website. Integrated with Stripe yesterday,
now finalizing the visuals.
This is a purchase receipt. It is both sent out and shown on the site when the payment goes through.
PS. All numbers and codes are made up. Don't get any ideas :)
This is a purchase receipt. It is both sent out and shown on the site when the payment goes through.
PS. All numbers and codes are made up. Don't get any ideas :)
Working on the production version of the website.
Cleaned up overall look and feel a bit, got rid of baby blue theme and anorexic body font, reworked the copy, expanded Features section and so on. Still need to slice and code it. Also asked a couple of people for the feedback, so waiting on that too.
Cleaned up overall look and feel a bit, got rid of baby blue theme and anorexic body font, reworked the copy, expanded Features section and so on. Still need to slice and code it. Also asked a couple of people for the feedback, so waiting on that too.
Working through the necessary evil - the app licensing :)
The plan is to have a 14 day trial that doesn't require any explicit licensing. Just install and run. It will pop up a notification half way through the trial period and it will also say something on the last day.
Once the trial period is over and until a production (paid) license is installed, the app will go into an unlicensed mode and it will activate two restrictions (see screenshot). It will remain largely functional, but presumably not convenient enough for regular or unattended use. You would then hit /purchase page on the website, go through the purchase process and get an activation code in return. The code will then need to be sticked into the activation form on the client (see 2nd screenshot) and it will fetch the license from the server.
If the licensing server is inaccessible for whatever reason, there's a couple of ways to retrieve the license manually (see last screenshot). For those with Beta licenses, they will be accepted at the checkout as a discount code.
For those looking at large deployments / enterprise setups, there will be an option for automating licensing process. Get in touch for details if you have questions.
Updated the About window a bit.
Current design has somehow always felt ... cramped?
Super-final version of the progress bar as seen in Beta 60.
Note how colors of some parts are inverted under selection,
took me disturbingly long to arrive at this solution :-/
Finalized changes to the progress bar - now with more
contrast, a bit of depth and beautifully centered
splatter for partially copied blocks.
And here is a bunch of runners-up if anyone's interested.
And here is a bunch of runners-up if anyone's interested.
Just waiting for the test build to compile...
so here's another take on the delta copier
progress bar (and here's
existing version for comparison).
Clicking and holding error navigation arrow for 750 ms
now activates auto-scrolling. It starts at 10 errors
per second and accelerates linearly to 100 err/sec,
which is pretty damn fast.
Additionally, holding Ctrl while scrolling with the mouse wheel makes it go in 10 line increments.
Additionally, holding Ctrl while scrolling with the mouse wheel makes it go in 10 line increments.
It is now possible to clear the log view and
then revert it back to the full log view.
A smaller UI change.
When hopping through errors in a backup log, log viewer will now briefly highlight next entry if it cannot be pulled directly under the cursor. This basically adds a visual terminator to the interaction sequence, which is a surprsingly useful thing to have.
Otherwise it works as before.
When hopping through errors in a backup log, log viewer will now briefly highlight next entry if it cannot be pulled directly under the cursor. This basically adds a visual terminator to the interaction sequence, which is a surprsingly useful thing to have.
Otherwise it works as before.
Having thought about it a bit more, I can even
remove that slight upward slope from the hump
of the blue graph. The usage still climbs just
a little bit during the run, but for practical
intents and purposes it's flat.
Wrapping up changes to the logging system. Here's
a quick graph to demonstrate the effects of the
rework.
For an initial full run of a 100,000 file backup
memory usage is down from about 200 to 50 Megs.
For a 1,000,000 file backup the effect is similar
- peak memory usage dropped
from 2.9GB to 0.58GB
.
Previously, the UI would keep logs of last few
backup runs in the memory and for larger backups
this would eat up a good chunk of memory. This
was especially true for the initial backups where
all files needed copying and thus for 100K files
it would generate close to half a million log entries.
For subsequent backups it wasn't a big deal as they would handle just the changes and generate only a handful of steps and a bit of log data So in the end, the memory usage was capped, but the cap was higher than it could've been.
Beta 60 addresses that.
For subsequent backups it wasn't a big deal as they would handle just the changes and generate only a handful of steps and a bit of log data So in the end, the memory usage was capped, but the cap was higher than it could've been.
Beta 60 addresses that.
Only a very small part of the
log is now kept in memory, just enough to fill the
log viewer panel in the UI. The rest is kept on the
disk and read from there on as-needed basis.
However this is easier said than done.
Bvckup logs are hierarchical, meaning that they are generic trees with nodes that may have an arbitrary large number of children. Running a 1 million file backup would yield a "Processing..." entry with 1 million children, each with up to a dozen of descendants. Displaying this tree would've not been a big deal if all log entries were always visible at all times. When the UI would've wanted to show entries 700,503 through 700,523 entries, it would've simply gone to a respective location in the log index file and plucked the data out. But...
The nodes can be collapsed and expanded. This hides and shows arbitrary chunks of the log tree and then the UI has this issue of finding 700,503-th *visible* item. And this is hard.
This implies quick look-ups in some sort of super-imposed binary search tree of visible items. A tree that changes every time a node is expanded or collapsed and that sits in a disk file, which restricts how you can walk and modify it without getting an I/O penalty.
This is fun. This is what cost me 3 weeks of shuffling through CS books and several complete rewrites of the tree indexing code. Believe it or not, but this is basically an original research and it seems that no one had to deal with this issue before. At least not publicly. The solution is to use slightly customized version of an AVL tree to store the list of node's children (that's using tree within a tree), use it for the visible item look-ups and then optimize the hell out of the index I/O through aggressive write caching.
When all i's are dotted and t's crossed, this yields a logging system that can push up to 1 mil entries per second on a resonably equipped box. This might not be knock-your-socks-off impressive, but it's plenty enough for the task at hand.
However this is easier said than done.
Bvckup logs are hierarchical, meaning that they are generic trees with nodes that may have an arbitrary large number of children. Running a 1 million file backup would yield a "Processing..." entry with 1 million children, each with up to a dozen of descendants. Displaying this tree would've not been a big deal if all log entries were always visible at all times. When the UI would've wanted to show entries 700,503 through 700,523 entries, it would've simply gone to a respective location in the log index file and plucked the data out. But...
The nodes can be collapsed and expanded. This hides and shows arbitrary chunks of the log tree and then the UI has this issue of finding 700,503-th *visible* item. And this is hard.
This implies quick look-ups in some sort of super-imposed binary search tree of visible items. A tree that changes every time a node is expanded or collapsed and that sits in a disk file, which restricts how you can walk and modify it without getting an I/O penalty.
This is fun. This is what cost me 3 weeks of shuffling through CS books and several complete rewrites of the tree indexing code. Believe it or not, but this is basically an original research and it seems that no one had to deal with this issue before. At least not publicly. The solution is to use slightly customized version of an AVL tree to store the list of node's children (that's using tree within a tree), use it for the visible item look-ups and then optimize the hell out of the index I/O through aggressive write caching.
When all i's are dotted and t's crossed, this yields a logging system that can push up to 1 mil entries per second on a resonably equipped box. This might not be knock-your-socks-off impressive, but it's plenty enough for the task at hand.
A quick update on what's happening.
Disk-based virtual data structures, that's what is :)
This is a part of an effort to further reduce and cap app's run-time memory usage. More specifically, it is meant to help the app cope gracefully with multi-million file backups that involve millions of steps and generate lots and lots of log data.
Disk-based virtual data structures, that's what is :)
This is a part of an effort to further reduce and cap app's run-time memory usage. More specifically, it is meant to help the app cope gracefully with multi-million file backups that involve millions of steps and generate lots and lots of log data.
At the moment Bvckup's UI keeps backup logs in
memory. It puts an effort to trim them to a
reasonable size, but this still eats quite a
bit of RAM, especially with large backups.
It costs about 120 bytes plus the size of an entry to
keep a single log line in memory, whereby these 120 bytes
help weave entries into a hierarchy (a tree) and track
their state.
If we want to reduce the memory usage, then the most obvious optimization would be to keep the actual entry text in a disk file and load it on demand, when it's actually visible in the log viewer.
This certainly helps, but as the log grows, the memory usage still climbs. A million file backup generates about 4 million log entries. At 120 bytes per entry that's 0.5 Gig in just supporting structures. This is unacceptable. So it means that the entire tree structure of the log needs to sit in a disk file and to be read from there as needed.
Disk-based tree structures are routinely used in databases and file systems to store indecies of data sets, typically in a form of a B-Tree and its variations. The kicker is that these structures are meant for storing sorted data and they optimize for searching. Trying to adapt them for storing unsorted data would be nothing short of fitting a square peg in a round hole. Long story short - there are no ready-made code for storing and manipulating generic data trees on a disk, leave alone fast code. This appears to be an esoteric problem that everyone solves on their own.
The screenshot above shows a part of a small "tree database" library that I ended up writing, together with its own caching module, predictive page loader and a pony.
Took almost two weeks. On a bright side though, this code is perfectly reusable for storing disk snapshots as well and this paves way for making the planner module work almost entirely off the memory.
Once done, Bvckup memory usage will NOT depend on the particulars of a backup at all - this a very big deal and an incredibly improtant feature to have for any robust backup software.
So, stay tuned...
If we want to reduce the memory usage, then the most obvious optimization would be to keep the actual entry text in a disk file and load it on demand, when it's actually visible in the log viewer.
This certainly helps, but as the log grows, the memory usage still climbs. A million file backup generates about 4 million log entries. At 120 bytes per entry that's 0.5 Gig in just supporting structures. This is unacceptable. So it means that the entire tree structure of the log needs to sit in a disk file and to be read from there as needed.
Disk-based tree structures are routinely used in databases and file systems to store indecies of data sets, typically in a form of a B-Tree and its variations. The kicker is that these structures are meant for storing sorted data and they optimize for searching. Trying to adapt them for storing unsorted data would be nothing short of fitting a square peg in a round hole. Long story short - there are no ready-made code for storing and manipulating generic data trees on a disk, leave alone fast code. This appears to be an esoteric problem that everyone solves on their own.
The screenshot above shows a part of a small "tree database" library that I ended up writing, together with its own caching module, predictive page loader and a pony.
Took almost two weeks. On a bright side though, this code is perfectly reusable for storing disk snapshots as well and this paves way for making the planner module work almost entirely off the memory.
Once done, Bvckup memory usage will NOT depend on the particulars of a backup at all - this a very big deal and an incredibly improtant feature to have for any robust backup software.
So, stay tuned...
Effects of delta copying as seen in Task Manager.
1.28 TB read, 0.37 TB written out - just the changes!
1.28 TB read, 0.37 TB written out - just the changes!
Switched delta copying progress bar from varying color
of the partially copied blocks to varying their height.
This should help with telling apart skipped, partial and
fully-copied blocks.
SAMBA !
This one - the *nix implementation of Windows Networking. If you have a NAS box the chances are that it's using Samba to support network access from Windows computers. It is a good software, simpler and much snappier than its native Windows counterpart, but in case of NAS devices it typically runs on Linux. This means that the underlying file system is anything but NTFS. This in turn translates into no support for "created" timestamps and no support for certain attributes, including System and Hidden.
Moreover, the attributes that cannot be set or cleared depend on whether it's a dot file or a regular file. Who would've thought? :-|
From what I understand it *is* possible to set things up so that Samba would record these timestamps and attributes and make things transparent to its Windows clients, but in reality this doesn't seem to be happening. Long story short - starting with Beta 49 Bvckup will now be explicitly testing destination file system to check whether c-times are supported and which attributes can be set.
Additionally, it will also explicitly test both source and destination file systems to determine the granularity of c-times and m-times. This is again thanks to some NAS devices that are customized to use 1 sec for m-times and 1 usec (micro) for c-times.
By popular request.
When ticked, Minimize button minimizes to the taskbar (as usual) and Close, the (x), minimizes to the systray.
When unticked, Minimize minimizes to the systray and Close exits the app.
Trivial in retrospect, but took me ages to realize that behavior of Close and Minimize can be controlled with just one setting.
When ticked, Minimize button minimizes to the taskbar (as usual) and Close, the (x), minimizes to the systray.
When unticked, Minimize minimizes to the systray and Close exits the app.
Trivial in retrospect, but took me ages to realize that behavior of Close and Minimize can be controlled with just one setting.
Second pass over the same sketch. Trimmed down
decorations a bit, reworked content to have a
summary, added footer, etc.
Full-length version is here.
What do you think? Let me know - @apankrat or over an email.
Full-length version is here.
What do you think? Let me know - @apankrat or over an email.
Started sketching production version of the website.
As much as I personally like the minimalist version
that's currently up there, it just
doesn't seem to work that well :-/
When periodic backup misses its last scheduled run,
it's now possible to tell the app how to handle it.
There are two common cases - (a) the app itself wasn't running and (b) the backup drive is a removable and it wasn't plugged in.
With certain setups it makes sense to run the backup as soon as the app can, to "catch up" with any changes and get them backed up quickly.
In other cases, a backup may be large and heavy and set up to run every midnight so not to disturb anyone. Running it ASAP will generally be not a very good idea.
Hence this option.
There are two common cases - (a) the app itself wasn't running and (b) the backup drive is a removable and it wasn't plugged in.
With certain setups it makes sense to run the backup as soon as the app can, to "catch up" with any changes and get them backed up quickly.
In other cases, a backup may be large and heavy and set up to run every midnight so not to disturb anyone. Running it ASAP will generally be not a very good idea.
Hence this option.
Context hints for each section in More options.
Here's the screen-by-screen flow of the
licensing process.
It starts with the above window and directs users to the website to pick and purchase a license code, which is given on spot and also sent in the receipt email.
Then copy-paste the code, click on Verify, get this -
If this works (the server is accessible, the license code is valid, etc), then this is it. Thank you very much, click Ok, you are done -
Otherwise, it explains the failure and offers either to retry the activation or to use an alternative -
There are two alternatives, both are simple -
For offline boxes and locked-down setups there's the Copy To Clipboard option. Take the URL to a networked machine, open it in a browser, get a licensing file back and then drop it into the app's config folder.
The URL is exactly the same as the one used by the app's own licensing code and it comprises of the license code and installation ID and the fingerprint .
Prepare the email option creates a new email with default email client and puts the Installation ID, Fingerprint and License Code in the message along with some pleasantries. The license file will be attached to the reply and it will simply need to be dropped into %LocalAppData%\Bvckup2\engine.
It starts with the above window and directs users to the website to pick and purchase a license code, which is given on spot and also sent in the receipt email.
Then copy-paste the code, click on Verify, get this -
If this works (the server is accessible, the license code is valid, etc), then this is it. Thank you very much, click Ok, you are done -
Otherwise, it explains the failure and offers either to retry the activation or to use an alternative -
There are two alternatives, both are simple -
For offline boxes and locked-down setups there's the Copy To Clipboard option. Take the URL to a networked machine, open it in a browser, get a licensing file back and then drop it into the app's config folder.
The URL is exactly the same as the one used by the app's own licensing code and it comprises of the license code and installation ID and the fingerprint .
Prepare the email option creates a new email with default email client and puts the Installation ID, Fingerprint and License Code in the message along with some pleasantries. The license file will be attached to the reply and it will simply need to be dropped into %LocalAppData%\Bvckup2\engine.
A bit of UI animation for the license
activation window. This is a animated
GIF, so the colors and the timing are
all off. It looks tidier and faster in
reality.
Licensing support. Yay!
This is one of those things that is rather simple conceptually, but translates into a lot of actual petty work.
This is one of those things that is rather simple conceptually, but translates into a lot of actual petty work.
When the app is started, it looks at several system
properties including the CPU model, Windows'
MachineGuid value and others, hashes them together
to produce an "Installation ID". For example -
The ID stays the same under normal computer use, but it changes with hardware upgrades or when the OS is reinstalled.
The app also generates an "Installation fingerprint" by hashing each of the properties separately and stringing them together. This yields a longer hash -
Like ID, the fingerprint changes with hardware upgrades and OS reinstalls, but, unlike ID, it changes only in parts. This makes it suitable for detecting incremental changes to the user's computer.
Both ID and Fingerprint identify user's installation, the difference is that the ID is used on the app side and the Fingerprint - on the licensing server side. Next, ID and fingerprint are submitted to the licensing server, either by the app itself or by a human through a website. When licensing for production use (as opposed to beta or evaluation), the request also includes a license code - a random token that is issued to the user in exchange for payment.
The server responds with a license, which is a short text file that binds together few pieces of information and seals them with an RSA signature of the licensing server.
For beta and evaluation licenses the "license code" is generated automatically by the licensing server. The large block of gibberish at the bottom of the license is the RSA signature. When the app starts, it loads the license and verifies it using public key of the licensing server that is embedded in the app.
If the license is valid, the app works as normal. If the license is missing or expired or doesn't match the installation ID, the app disables certain features and periodically nags to do the right thing and what not. Now, let say I upgraded my box. This screws up the Installation ID and invalidates the license. The app recomputes new ID and Fingerprint and sends them to the licensing server.
This is where Fingerprint comes into play. The server takes old and new fingerprints, chops them into individual hashes and does a "fuzzy" match. If enough hashes match, we can assume an incremental software or hardware change on user's end and re-issue the license for new Installation ID.
If fingerprints are wildly different, then we make fuss and require contacting human tech support. In conclusion, needless to say that the whole scheme relies on the validation code in the app being trustworthy. If someone pries the app open and replaces selected IFs with NOPs, then obviously no licensing will work.
The answer to that is that the app should implement self-consistency checks and deploy counter-measures when these checks fail. This is a separate and very large and interesting subject ... which I have no plans on covering here :-) A good starting point here would be fravia.org archives and then follow from there.
$c!f2ZoPxOlnpLWraJZ3F1nm7JcA
The ID stays the same under normal computer use, but it changes with hardware upgrades or when the OS is reinstalled.
The app also generates an "Installation fingerprint" by hashing each of the properties separately and stringing them together. This yields a longer hash -
YQdAXlZR0FLXZ0KpYxlOk8sJ9Z$xtq19!rp6i3C6P5LiUc0J
x2Ynqe29i41qb!1w5wEzot!q0EH8SsLWfY6gvkyoZPtfn3bW
uYnSwItsG8Wh1FkQFWBCaNqdndeiBiDgkrTde$wJUib9wR58
GJ5bQU1Zr1iewtgyRhLWigHyzKDO3oYztT4hMw6MnP6voupb
J6An5QE4!etkMJx!qzOaYmioIOQ0GiQyapcPQeyYLCi4CN28
CEev0!FZFZz3aa$0dCmNnMRBohZ9z6EZH5NxROhokF8pXWVM
9y$bzQAA
x2Ynqe29i41qb!1w5wEzot!q0EH8SsLWfY6gvkyoZPtfn3bW
uYnSwItsG8Wh1FkQFWBCaNqdndeiBiDgkrTde$wJUib9wR58
GJ5bQU1Zr1iewtgyRhLWigHyzKDO3oYztT4hMw6MnP6voupb
J6An5QE4!etkMJx!qzOaYmioIOQ0GiQyapcPQeyYLCi4CN28
CEev0!FZFZz3aa$0dCmNnMRBohZ9z6EZH5NxROhokF8pXWVM
9y$bzQAA
Like ID, the fingerprint changes with hardware upgrades and OS reinstalls, but, unlike ID, it changes only in parts. This makes it suitable for detecting incremental changes to the user's computer.
Both ID and Fingerprint identify user's installation, the difference is that the ID is used on the app side and the Fingerprint - on the licensing server side. Next, ID and fingerprint are submitted to the licensing server, either by the app itself or by a human through a website. When licensing for production use (as opposed to beta or evaluation), the request also includes a license code - a random token that is issued to the user in exchange for payment.
The server responds with a license, which is a short text file that binds together few pieces of information and seals them with an RSA signature of the licensing server.
License code: EVAL-8028-8316-0400
Installation: $c!f2ZoPxOlnpLWraJZ3F1nm7JcA
Expires on: 20/12/2013
J/8natmgEo1fFr0NSBtw3fsderaMsv6gwELBa7P4tsjtkPxO
Y/4z930IfpujPKwen6QB9Gb33q7lyXQNkAVcvMV5hlXwKNV+
eifzJUqyOE1GdmYd8Z4Io0OuFr2V4tD5vnDmYTFoOye4gd67
D40G/hoKgcvGtX82w6bjLTae+vzm7u6GwowTGrbEmqqJDPNI
4Qc6un+aGuOEWEY0X25DkPYnFy/VnPusWgLRma1eTCrmvOqD
T6WX27++Bwq2+YH1Aw+Br9xEQF6SZOBp9OR1Xp/K24vYSN52
MRlLjhrBIQXmJTTepLKAHEsnA3pvdinU6pzpYexygfjtciUm
VZlgvxfUICWI3pKBNRJtlOQKc9LIyr9jVXQ1jP4QVdY5JNZW
y+71b8cSEtWgRRkYVGZMyUXeJjU7AoW/PhDNWdKvDPBwBInQ
QuNqA1ncxjWMRdLYogR/VY41yKYs9cPaFTPFb4y1XLPiqVRK
vEOMMUvE1kArZS/IuC5yB7Jqd3mXlZaRMacwuhWVcd+fgbFj
wVbLJvRRePKEoP/Kk0CtDPIQNdYVxlEwdZi/S+cFMdfZLiWU
o8HFpT247WWEyElUaDahfCREgjPbs9Iw5pik82bm42phns4I
1936WVycSATBQZp3JC1KXX8iQeDBfOgwe3QQcnWIrpclI/oI
57Mknd6bxeI=
Installation: $c!f2ZoPxOlnpLWraJZ3F1nm7JcA
Expires on: 20/12/2013
J/8natmgEo1fFr0NSBtw3fsderaMsv6gwELBa7P4tsjtkPxO
Y/4z930IfpujPKwen6QB9Gb33q7lyXQNkAVcvMV5hlXwKNV+
eifzJUqyOE1GdmYd8Z4Io0OuFr2V4tD5vnDmYTFoOye4gd67
D40G/hoKgcvGtX82w6bjLTae+vzm7u6GwowTGrbEmqqJDPNI
4Qc6un+aGuOEWEY0X25DkPYnFy/VnPusWgLRma1eTCrmvOqD
T6WX27++Bwq2+YH1Aw+Br9xEQF6SZOBp9OR1Xp/K24vYSN52
MRlLjhrBIQXmJTTepLKAHEsnA3pvdinU6pzpYexygfjtciUm
VZlgvxfUICWI3pKBNRJtlOQKc9LIyr9jVXQ1jP4QVdY5JNZW
y+71b8cSEtWgRRkYVGZMyUXeJjU7AoW/PhDNWdKvDPBwBInQ
QuNqA1ncxjWMRdLYogR/VY41yKYs9cPaFTPFb4y1XLPiqVRK
vEOMMUvE1kArZS/IuC5yB7Jqd3mXlZaRMacwuhWVcd+fgbFj
wVbLJvRRePKEoP/Kk0CtDPIQNdYVxlEwdZi/S+cFMdfZLiWU
o8HFpT247WWEyElUaDahfCREgjPbs9Iw5pik82bm42phns4I
1936WVycSATBQZp3JC1KXX8iQeDBfOgwe3QQcnWIrpclI/oI
57Mknd6bxeI=
For beta and evaluation licenses the "license code" is generated automatically by the licensing server. The large block of gibberish at the bottom of the license is the RSA signature. When the app starts, it loads the license and verifies it using public key of the licensing server that is embedded in the app.
If the license is valid, the app works as normal. If the license is missing or expired or doesn't match the installation ID, the app disables certain features and periodically nags to do the right thing and what not. Now, let say I upgraded my box. This screws up the Installation ID and invalidates the license. The app recomputes new ID and Fingerprint and sends them to the licensing server.
This is where Fingerprint comes into play. The server takes old and new fingerprints, chops them into individual hashes and does a "fuzzy" match. If enough hashes match, we can assume an incremental software or hardware change on user's end and re-issue the license for new Installation ID.
If fingerprints are wildly different, then we make fuss and require contacting human tech support. In conclusion, needless to say that the whole scheme relies on the validation code in the app being trustworthy. If someone pries the app open and replaces selected IFs with NOPs, then obviously no licensing will work.
The answer to that is that the app should implement self-consistency checks and deploy counter-measures when these checks fail. This is a separate and very large and interesting subject ... which I have no plans on covering here :-) A good starting point here would be fravia.org archives and then follow from there.
Reworking About window a bit to include license
information and associated trinkets (not shown).
This will probably need another iteration or two...
Basically the idea is to show back/forth arrows only on hover and make them pull next error to the spot where the cursor is. So to hop through errors you would just go click-click-click without moving the mouse.
This compliments well existing feature of jumping to the first error of the last run by clicking on "Xxx errors" in backup's main window entry.
As per previous post, this leaves a question of how one would get to the *nearest* error if none is currently visible and the answer is that one would simply use conventional scroll for that.
All in all, having played with this for a bit I think it's a good solution. If it doesn't seem like one to you, give it a try and it will grow on you.
Tying up few loose ends in the UI.
The above screenshot is the first option for navigating between the errors in the log pane -- see two red arrows next to the "x" in the pane header. The second option is this -
This time the next/previous arrows are in the error entry itself. I like the second option better, because it's contextual and I think it's more obvious. However, first option is superior in functionality because it allows jumping to next or previous error when there's currently no error visible in the pane.
Still deciding...
The above screenshot is the first option for navigating between the errors in the log pane -- see two red arrows next to the "x" in the pane header. The second option is this -
This time the next/previous arrows are in the error entry itself. I like the second option better, because it's contextual and I think it's more obvious. However, first option is superior in functionality because it allows jumping to next or previous error when there's currently no error visible in the pane.
Still deciding...
Tweaked the progress indicator a bit.
For one, replaced the fixed-width byte countdown with a simple "% out of [size]". The countdown didn't work out, it was just... odd.
Also made the % adjust the number of shown decimal digits depending on how fast the copying is going.
For fast copies there are no decimals. For slower ones it shows one, for slower yet - two, and for the slowest (like the one on the screenshot) it goes down to 3 decimal places.
The idea, obviously enough, to always show some movement, but try and avoid being too excessive with it.
For one, replaced the fixed-width byte countdown with a simple "% out of [size]". The countdown didn't work out, it was just... odd.
Also made the % adjust the number of shown decimal digits depending on how fast the copying is going.
For fast copies there are no decimals. For slower ones it shows one, for slower yet - two, and for the slowest (like the one on the screenshot) it goes down to 3 decimal places.
The idea, obviously enough, to always show some movement, but try and avoid being too excessive with it.
Added code to replicate folder timestamps. It's ON
by default as it takes only several seconds to go
through a hundred thousand folders, but it can be
disabled if needed
Final version of the update window. De-emphasized the
"View changes" button and tweaked the colors a bit.
Reworked the software update mechanism a little.
First, when the update server is processing an update check from the program, it now takes into account program's current version.
It uses this version to compile a consolidated list of all changes between program's version and the latest version available. It then uses this list to understand if there are any critical or important outstanding changes. This yields an importance indicator of the update, when one is available.
Secondly, both the list and the indicator are now sent back to the program.
First, when the update server is processing an update check from the program, it now takes into account program's current version.
It uses this version to compile a consolidated list of all changes between program's version and the latest version available. It then uses this list to understand if there are any critical or important outstanding changes. This yields an importance indicator of the update, when one is available.
Secondly, both the list and the indicator are now sent back to the program.
On program's end, the indicator translates into the
"Update is recommended" line. It is one three options
with "optional" and "critical" being the other two.
"Recommended" is a routine update that adds this and fixes that. It would better be installed, but it's not the end of the world if it's not.
"Critical" is critical - "you'd be sorry if you don't install it" kind of update.
An example of "optional" update is a new version that updates program's French translation when the user is running an English version, i.e. an update that can be ignored altogether. There's now also the "View changes" button that expands the window and shows a neatly formatted (RTF) version of the change list.
The biggest deal, of course, is that it's a consolidated list of all changes between the installed and the latest version, rather than a laundry list of all changes since the dawn of time. It makes the list infinitely more useful compared to the raw version.
"Recommended" is a routine update that adds this and fixes that. It would better be installed, but it's not the end of the world if it's not.
"Critical" is critical - "you'd be sorry if you don't install it" kind of update.
An example of "optional" update is a new version that updates program's French translation when the user is running an English version, i.e. an update that can be ignored altogether. There's now also the "View changes" button that expands the window and shows a neatly formatted (RTF) version of the change list.
The biggest deal, of course, is that it's a consolidated list of all changes between the installed and the latest version, rather than a laundry list of all changes since the dawn of time. It makes the list infinitely more useful compared to the raw version.
Adding support for running external command
before and after each backup.
Both commands have a configurable timeout value. Additionally, the pre-backup command can be tagged as critical, in which case bvckup checks the exit code and aborts the backup if the code doesn't match specified value.
The screenshot shows touching a TrueCrypt volume before the backup. This is needed, because TrueCrypt goes out of its way to preserve timestamps on its container files and bvckup sees such files as unmodified.
Both commands have a configurable timeout value. Additionally, the pre-backup command can be tagged as critical, in which case bvckup checks the exit code and aborts the backup if the code doesn't match specified value.
The screenshot shows touching a TrueCrypt volume before the backup. This is needed, because TrueCrypt goes out of its way to preserve timestamps on its container files and bvckup sees such files as unmodified.
Reworked the More Options section.
It has already grown quite tall and it still needs to take on few more configuration options, so expanding it in place as it's done for all other sections doesn't work. The window gets too big and simply doesn't fit on a desktop.
Instead, it pushes everything up and out of the window and floods it with its own content. As I add more config options to the section, it will also expand the window down a bit if needed. All this should help keep all configuration within single window and also keep window's height in check.
Took me a while to converge to this solution, but it is a pretty decent way to structure it I think.
It has already grown quite tall and it still needs to take on few more configuration options, so expanding it in place as it's done for all other sections doesn't work. The window gets too big and simply doesn't fit on a desktop.
Instead, it pushes everything up and out of the window and floods it with its own content. As I add more config options to the section, it will also expand the window down a bit if needed. All this should help keep all configuration within single window and also keep window's height in check.
Took me a while to converge to this solution, but it is a pretty decent way to structure it I think.
Sketching up a dedicated device tracking
section. Not sure if it's worth it, still
have to try few things out...
Done with the inclusion/exclusion filters
The tree selector widget is a piece of art even though it doesn't look like one :) It is basically a front of an asynchronous parallel disk scanner that dynamically rearranges its scanning queue to first scan subdirectories that a user might be accessing next.
In other words, if you expand a directory, then the scanner will postpone scanning anything else and focus instead on all its subdirectories. So if you are making your way through a massively populated disk, it will try to pre-load what you are looking at and scan the rest on a lower priority.
Also, since the widget starts showing tree contents right away, it allows quickly navigating to and excluding a desired item without needing to wait for the entire scan to complete. Again, quite a bit of speed up with huge trees.
The tree selector widget is a piece of art even though it doesn't look like one :) It is basically a front of an asynchronous parallel disk scanner that dynamically rearranges its scanning queue to first scan subdirectories that a user might be accessing next.
In other words, if you expand a directory, then the scanner will postpone scanning anything else and focus instead on all its subdirectories. So if you are making your way through a massively populated disk, it will try to pre-load what you are looking at and scan the rest on a lower priority.
Also, since the widget starts showing tree contents right away, it allows quickly navigating to and excluding a desired item without needing to wait for the entire scan to complete. Again, quite a bit of speed up with huge trees.
Advanced filter configuration. The pattern
matching is all brand new and much improved
over the disaster that it was in Bvckup 1.
It also adds an option of matching on item
attributes, both set and cleared.
Finishing up the dialog for configuring
inclusion/exclusion filters.
In inclusive mode (pictured) the tree view tags excluded items with 'x'. In exclusive mode, when "Start with an empty list" selected, the tree view tags included items with checkmarks.
Currently leaning towards Option 1, and not yet decided if to strike-out excluded items or not.
In inclusive mode (pictured) the tree view tags excluded items with 'x'. In exclusive mode, when "Start with an empty list" selected, the tree view tags included items with checkmarks.
Currently leaning towards Option 1, and not yet decided if to strike-out excluded items or not.
Well into the beta, just dotting the i's and
crossing the t's.
One of the outstanding features is the archival of deleted items and this is the supporting UI for it.
Now deciding between a compact format of the config widget or a verbose one. Compact format has a single descriptive field that is re-used to show the explanation of a currently selected option. Verbose format has descriptions for each option visible at all times - it is likely easier on first time users, but it is also a waste of screen space on all subsequent uses.
One of the outstanding features is the archival of deleted items and this is the supporting UI for it.
Now deciding between a compact format of the config widget or a verbose one. Compact format has a single descriptive field that is re-used to show the explanation of a currently selected option. Verbose format has descriptions for each option visible at all times - it is likely easier on first time users, but it is also a waste of screen space on all subsequent uses.
For very large files (say, 256 Gigs), there was a clear lack of UI acknowledgement of the backup progress. So the completed percentage got a couple of point digits and a byte countdown for the file.
Added the first draft of the Miscellanea section.
Also renamed More Options button to
Rabbit Hole, because that's essentially
where it leads to :)
( the original version is here )
Significantly improved performance through
multi-core, parallel hashing, asynchronous I/O and
more aggressive caching of the state data.
Strengthened the change detection algorithm
through addition of the whole file hash. When all
individual block hashes come back unchanged,
the algorithm now uses the file hash to confirm
that the file has indeed remained unchanged.
Combined with the original use of two separate hashes per data block, this eliminates the need for a precautionary full-file sync every N copies. Improved handling of cancelled and aborted delta copies by introducing support for partial updates .
The algorithm now remembers how far along the file it managed to proceed and stashes this information for the next run.
When the copying is cancelled partway through, the file is marked as out-of-sync, but the delta state is updated to capture what has been already done. This comes very handy and speeds things up when updating very large files. Improved handling of large file counts.
Version 1 maintained an in-memory index of all delta copied files and, surprisingly, this didn't scale well. It also kept all delta state information in a single folder, which too led to some pain and suffering when the file count grew large.
New version eliminates both these problems. Lastly, all copier parameters - block sizes, hash algorithms, cache buffer sizes, threading parameters, etc. - are now configurable on a per-job basis. All in all - THIS IS IT.
This is how a simple and efficient delta copying should be done. Can't wait to release it to the real world.
Combined with the original use of two separate hashes per data block, this eliminates the need for a precautionary full-file sync every N copies. Improved handling of cancelled and aborted delta copies by introducing support for partial updates .
The algorithm now remembers how far along the file it managed to proceed and stashes this information for the next run.
When the copying is cancelled partway through, the file is marked as out-of-sync, but the delta state is updated to capture what has been already done. This comes very handy and speeds things up when updating very large files. Improved handling of large file counts.
Version 1 maintained an in-memory index of all delta copied files and, surprisingly, this didn't scale well. It also kept all delta state information in a single folder, which too led to some pain and suffering when the file count grew large.
New version eliminates both these problems. Lastly, all copier parameters - block sizes, hash algorithms, cache buffer sizes, threading parameters, etc. - are now configurable on a per-job basis. All in all - THIS IS IT.
This is how a simple and efficient delta copying should be done. Can't wait to release it to the real world.
Shadow copying is in,
for those pesky locked Outlook mailboxes and other "in use" files.
There are three big things left to weave into the program - file system monitoring, new delta copier and software updates - and the beta is good to go. A matter of few days now.
There are three big things left to weave into the program - file system monitoring, new delta copier and software updates - and the beta is good to go. A matter of few days now.
If your message box includes the "Don't ask again"
option, just make sure to disable all but one
button, when it checked.
Oddly enough, I don't know a single app that does that.
Oddly enough, I don't know a single app that does that.
Feeling creative? Make your own version, save as a PNG, drop it in a config directory and - voilà - your systray looks just like you want it to.
There are two sets of icons on the sprite sheet. The first three columns is the XP style, which is a bit puffy and matches well older Windows styling. The last three columns is a flat version that goes better with native W7 tray icon style.
Each icon comes in 3 sizes - 16x16, 20x20, 24x24. This is needed to accommodate 3 common DPI levels, see this earlier entry for details.
"Message pending" indication, whereby a message
can be either an error, a warning or a notice
and these are differentiated with the ! color.
I realize it's not color-blind friendly,
but the goal is to attract the attention and
make user bring the app window up to read the
message --(therefore)--> reflecting the message
type via an icon level is not exactly critical.
Revised Preferences window a bit.
Added basic control over backup scheduling and switched from "close to systray" to "minimize to systray" - the (x) at the top right corner will now always act as an Exit, just as it is supposed to.
Added basic control over backup scheduling and switched from "close to systray" to "minimize to systray" - the (x) at the top right corner will now always act as an Exit, just as it is supposed to.
More of the UI work.
Custom pop-out button that stays nice, flat and out of the way when idle, but acts like a regular button otherwise. Not exactly a UI innovation, but it was nice to be able to use it in a real project :)
Custom pop-out button that stays nice, flat and out of the way when idle, but acts like a regular button otherwise. Not exactly a UI innovation, but it was nice to be able to use it in a real project :)
Took out the engine mode switch from the Preferences
and into a dialog of its own. Still need to understand
if it's worth mentioning that there's an engine per se
or just stick to more amorphous "application mode"...
Looked at few options for splitter styling. First
one is final. Tentatively though... heh.
Done with the split-view mode for the main window.
It's one of those things that everyone takes for
granted and that, in reality, maps to quite a bit
of work if done right.
For example, resizing window by the top edge should resize the top pane and leave the bottom part alone.
Likewise, resizing by the bottom edge should
expand/compress the bottom pane, but not the
top one.
However, if we are compressing window by the top edge and the top pane is at its minimal height, then we should the bottom pane starting to compress.
All good so far, but, say, we change our mind and start
expanding the window back (without releasing the mouse
button).
We should see the bottom pane brought back to its original size first, followed by the expansion of the top part... even though we are, technically, dragging by the top edge.
This sort of UX detail is something that few people will notice if it's done right, because it feels natural and aligns well with expected behavior. But cut some corners - and you bet everyone will notice.
For example, resizing window by the top edge should resize the top pane and leave the bottom part alone.
Likewise, resizing by the bottom edge should
expand/compress the bottom pane, but not the
top one.
However, if we are compressing window by the top edge and the top pane is at its minimal height, then we should the bottom pane starting to compress.
All good so far, but, say, we change our mind and start
expanding the window back (without releasing the mouse
button).
We should see the bottom pane brought back to its original size first, followed by the expansion of the top part... even though we are, technically, dragging by the top edge.
This sort of UX detail is something that few people will notice if it's done right, because it feels natural and aligns well with expected behavior. But cut some corners - and you bet everyone will notice.
Going through styling options for the expand/collapse buttons.
Presently leaning towards C or D, because I'm considering making the text selectable with the mouse for easy copy- pasting. For that to work the buttons need to be as unobtrusive as possible.
Still WiP though...
Presently leaning towards C or D, because I'm considering making the text selectable with the mouse for easy copy- pasting. For that to work the buttons need to be as unobtrusive as possible.
Still WiP though...
Working on the backup log viewer.
Basically the idea for the log is to support
multiple detail levels, each capturing
progressively more context that its parent.
With all details collapsed, the log gives a quick overview of when the job was running and if it completed OK or with errors.
Expanding details allows for a drill-down - to look at the details such as the file and byte counts, the timing and performance data and the details of any errors that have occured.
I'm still going to shuffle things around the window, e.g. push the -/+ buttons to the right, tweak the styling too, but the general hierarchical format will stay. It is a very convenient way of organizing the log info.
With all details collapsed, the log gives a quick overview of when the job was running and if it completed OK or with errors.
Expanding details allows for a drill-down - to look at the details such as the file and byte counts, the timing and performance data and the details of any errors that have occured.
I'm still going to shuffle things around the window, e.g. push the -/+ buttons to the right, tweak the styling too, but the general hierarchical format will stay. It is a very convenient way of organizing the log info.
Finished a piece of code that determines
effective timestamp resolution of a file
system.
Every file system keeps track of when a file
was created and when it was last modified.
These timestamps are the natural properties
of a file and they are readily available for
inspection in the Windows Explorer or any
file manager of your choice.
The less obvious aspect of timestamping is that timestamps have resolution. For example, FAT tracks file modification with a measly 2 second precision. In comparison, NTFS uses the 100 ns (nano-second) precision. It is very typical for a backup software to rely on timestamps to determine if a file has been modified and requires copying. But if the source file sits on an NTFS volume and the backup goes onto a FAT disk, then comparing timestamp directly simply won't work, because FAT timestamp will be rounded-up to a 2-second mark.
In other words, the timestamp granularity needs to be taken into an account when comparing the timestamps. The question is how to determine what it is exactly. First of all, the granularity for creation and modification times can be different. FAT has them at 10 ms and 2 s respectively, but NTFS has them both at 100 ns.
To complicate matters, there appears to be NAS devices that report using NTFS, but in reality having the granularity that is not 100 ns.
There are also NAS devices that mangle timestamps so spectacularly that it can't be explained in any rational way. Like rounding them up to 1 sec and then subtracting 1 ms. So what Bvckup 2 does is it probes the file system and attempts to determine effective resolution of both timestamps. This involves dropping a temporary file and then trying to set its timestamps to this, that and 3rd and see what they end up at. From that it's possible to deduce the resolution.
In cases when such probing fails, the app falls back to guessing resolution by the file system name. It is also possible to override the resolution values via the config file, just in case.
Additionally, the program also keeps track of a maximum observed timestamp difference for the backup volume. That is, when replicating the timestamps, it will make note of the values it is asking to set and values that end up being actually recorded for a file or a folder.
This difference is used as a lower threshold for the probed and estimated resolutions and, over time, it converges to the actual volume resolution.
The less obvious aspect of timestamping is that timestamps have resolution. For example, FAT tracks file modification with a measly 2 second precision. In comparison, NTFS uses the 100 ns (nano-second) precision. It is very typical for a backup software to rely on timestamps to determine if a file has been modified and requires copying. But if the source file sits on an NTFS volume and the backup goes onto a FAT disk, then comparing timestamp directly simply won't work, because FAT timestamp will be rounded-up to a 2-second mark.
In other words, the timestamp granularity needs to be taken into an account when comparing the timestamps. The question is how to determine what it is exactly. First of all, the granularity for creation and modification times can be different. FAT has them at 10 ms and 2 s respectively, but NTFS has them both at 100 ns.
To complicate matters, there appears to be NAS devices that report using NTFS, but in reality having the granularity that is not 100 ns.
There are also NAS devices that mangle timestamps so spectacularly that it can't be explained in any rational way. Like rounding them up to 1 sec and then subtracting 1 ms. So what Bvckup 2 does is it probes the file system and attempts to determine effective resolution of both timestamps. This involves dropping a temporary file and then trying to set its timestamps to this, that and 3rd and see what they end up at. From that it's possible to deduce the resolution.
In cases when such probing fails, the app falls back to guessing resolution by the file system name. It is also possible to override the resolution values via the config file, just in case.
Additionally, the program also keeps track of a maximum observed timestamp difference for the backup volume. That is, when replicating the timestamps, it will make note of the values it is asking to set and values that end up being actually recorded for a file or a folder.
This difference is used as a lower threshold for the probed and estimated resolutions and, over time, it converges to the actual volume resolution.
Done with the backup scheduler. Comes with a
couple of neat improvements over v1.
The Go and Stop buttons work almost exactly
as in v1. If the backup is running, click
Stop once to Pause it, click again to Cancel,
click third time to disable the job.
If the job is disabled, click once to enable
it, click again to manually start it. And so
on, you get the idea.
First improvement over v1 are the Go!
and Stop! commands that are activated by
holding Ctrl down when clicking on the
toolbar buttons.
Go! moves the job straight into the running state.
It's the "Run it. NOW!" command.
Stop! stops and disables the job straight away.
It's the "Stop it. NOW!" command. Second improvement is that multiple jobs can now be run in parallel. Generally, this has a negative impact on the overall performance, but there are certain cases when this is desirable. One example would be running a smaller backup in parallel with a much larger and slower one.
There are three ways to do this:
Go! moves the job straight into the running state.
It's the "Run it. NOW!" command.
Stop! stops and disables the job straight away.
It's the "Stop it. NOW!" command. Second improvement is that multiple jobs can now be run in parallel. Generally, this has a negative impact on the overall performance, but there are certain cases when this is desirable. One example would be running a smaller backup in parallel with a much larger and slower one.
There are three ways to do this:
- With the Go! command.
- With a per-backup setting that makes the job always run right away, even if there's another job is running.
- With an app-level setting that does the same, but for all jobs.
Another pass over the main window chrome and
layout. The "backup jobs" section is just a
dummy header ... and it is confusing, I think.
† This is an app screenshot, not a sketch.
† This is an app screenshot, not a sketch.
Exploring options for the main window format.
Specifically, I want to get rid of the V1's
Summary panel,
trim the information contained in it a bit and fold it
directly into the main window items.
In conjunction with this I'm also considering having two modes for displaying backup jobs on the list - "terse" one-liners and "extended" multi-liners. What I still need to figure out if this should be a per-item preference or an app-wide setting and how one would switch between these display modes.
Work in progress, obviously. Any thoughts - let me know.
In conjunction with this I'm also considering having two modes for displaying backup jobs on the list - "terse" one-liners and "extended" multi-liners. What I still need to figure out if this should be a per-item preference or an app-wide setting and how one would switch between these display modes.
Work in progress, obviously. Any thoughts - let me know.
Done with a little animation that and switches
main window between a "welcome screen" and the
regular backup list view.
The animation and fading effects can be disabled, there's a setting for that.
The animation and fading effects can be disabled, there's a setting for that.
Windows comes with a standard
toolbar control,
but it's an overkill if you need just a
couple of buttons at the top of the window.
It basically takes longer to figure out how to use and customize it compared to creating a basic window and sticking a couple of custom buttons on it.
Had I have a need for toolbar ribbons, docking, on-the-fly customization, etc., then ToolbarWindow32 makes sense. But if it's something simple, then there are faster options.
It basically takes longer to figure out how to use and customize it compared to creating a basic window and sticking a couple of custom buttons on it.
Had I have a need for toolbar ribbons, docking, on-the-fly customization, etc., then ToolbarWindow32 makes sense. But if it's something simple, then there are faster options.
Meet the toolbar.
To explain a bit what you see in this masterpiece :) -
The idea is to try and keep the menu/toolbar area simple and flat, but still provide some sort of a visual clue that Go and Start are clickables.
Here are three options compared back to back to make the differences a bit more obvious -
Here are few options that were leading up to the above one:
iOS says Hi
Also, considered disposing off the menu bar and folding whatever few items it has into the drop-down menu. But this just looks weird and too alien on Windows, the app needs a menu bar.
Dropped the idea.
The idea is to try and keep the menu/toolbar area simple and flat, but still provide some sort of a visual clue that Go and Start are clickables.
Here are three options compared back to back to make the differences a bit more obvious -
Here are few options that were leading up to the above one:
Also, considered disposing off the menu bar and folding whatever few items it has into the drop-down menu. But this just looks weird and too alien on Windows, the app needs a menu bar.
Dropped the idea.
The look of the main window with no backups
configured. Tried using a stock button, but
it looks heavy and dull. Hopefully, this is
a bit more refined.
Trying to understand how to indicate if
a selected path points at a removable storage.
The app tracks storage devices being plugged into the computer and it adjusts backup paths if a known device re-appears under a different drive name.
I can, of course, not bother with this at all and simply spit out a notification when (and if) the drive change is detected, but then I think it's an important functionality and it'd better be mentioned during the setup.
The app tracks storage devices being plugged into the computer and it adjusts backup paths if a known device re-appears under a different drive name.
I can, of course, not bother with this at all and simply spit out a notification when (and if) the drive change is detected, but then I think it's an important functionality and it'd better be mentioned during the setup.
Correct your input.
† Idea is courtesy of the OS X login screen, I think. Though I did experiment with concise error indication quite a bit.
† Idea is courtesy of the OS X login screen, I think. Though I did experiment with concise error indication quite a bit.
Wrapping up the Periodic Backup configuration
window.
Unlike the v1 that always stored the backup interval as a number of seconds, the v2 adds a time unit specifier.
This simplifies quite a few things in the UI and yet it keeps the configuration flexible enough to support 1.5 hour backups by allowing to specify a 90 minute interval.
Unlike the v1 that always stored the backup interval as a number of seconds, the v2 adds a time unit specifier.
This simplifies quite a few things in the UI and yet it keeps the configuration flexible enough to support 1.5 hour backups by allowing to specify a 90 minute interval.
On an unrelated note, killed few hours trying
to utilize native Windows controls for the
interval configuration. The amount of work
required to make them look reasonably
good is as depressing as it is remarkable.
For example, the editbox control doesn't natively support vertically centering the input text. It just does not. It is meant to look like this -
Want "120" aligned to the label on the left? Sure thing, that'd be 2 hours of plowing through documentation only to realize that it must be... drum-roll... a multiline control that you will then need to manually pad the right amount at the top to push the line to the middle. But that's peanuts compared to trying to beat some sense into the dropdown box. Long story, but making a box taller and keeping the line centered requires taking over the control drawing functions, which in presence of Windows themes is a delightfully unpleasant experience.
In any case, turning the top control line into the bottom one -
costs exactly a day of work, while it really shouldn't. Welcome to Windows.
For example, the editbox control doesn't natively support vertically centering the input text. It just does not. It is meant to look like this -
Want "120" aligned to the label on the left? Sure thing, that'd be 2 hours of plowing through documentation only to realize that it must be... drum-roll... a multiline control that you will then need to manually pad the right amount at the top to push the line to the middle. But that's peanuts compared to trying to beat some sense into the dropdown box. Long story, but making a box taller and keeping the line centered requires taking over the control drawing functions, which in presence of Windows themes is a delightfully unpleasant experience.
In any case, turning the top control line into the bottom one -
costs exactly a day of work, while it really shouldn't. Welcome to Windows.
Finalized the design of inline hint/tip
widget for input controls.
Initially, it was meant for the config window that hosts periodic backup settings (as seen in a couple of posts below), but having talked to others and slept on the issue I think that the natural language input (i.e. requiring to enter "1 week" or "15 minutes") is not the best option for this case.
An alternative design is coming up in a bit.
Initially, it was meant for the config window that hosts periodic backup settings (as seen in a couple of posts below), but having talked to others and slept on the issue I think that the natural language input (i.e. requiring to enter "1 week" or "15 minutes") is not the best option for this case.
An alternative design is coming up in a bit.
Coded the previous sketch, played with few
parameters and it is starting to get there
... I think. Will make another pass over this
tomorrow and it should be good to go.
For what it's worth I have also gave a try
to the standard
Tooltip Control
. In fact,
that's where I started.
A regular tooltip looks like this -
It adds another layer to the windows stack, breaks the visual flow of the layout and still manages to tuck itself under the mouse cursor.
There's also a balloon version of the same, which looks like this -
It is marginally better, but just look at those lovely rounded cornersi . Eat that Apple. Real artists ship. tl;dr - the standard Tooltip Control is not very good, so I had to come up with a custom replacement.
It adds another layer to the windows stack, breaks the visual flow of the layout and still manages to tuck itself under the mouse cursor.
There's also a balloon version of the same, which looks like this -
It is marginally better, but just look at those lovely rounded cornersi . Eat that Apple. Real artists ship. tl;dr - the standard Tooltip Control is not very good, so I had to come up with a custom replacement.
Not sure about this one, seems a bit
clumsy and a bit too custom.
This is basically a way to explain what is expected as an input for a particular configuration value. There's gotta be a cleaner way of delivering the same info...
This is basically a way to explain what is expected as an input for a particular configuration value. There's gotta be a cleaner way of delivering the same info...
Working through the supporting code for the Settings window.
Done with collapsing/expanding mechanics and respective fading and moving animations. Still need to revise the copy for the explanatory bits and massage in the option for archiving deleted files. Not to pat myself on the back, but I think it came out looking pretty decent.
That's not to say there's no way to turn off all this fancy styling and animations, because, you bet your favourite monochrome flat design, there is.
Done with collapsing/expanding mechanics and respective fading and moving animations. Still need to revise the copy for the explanatory bits and massage in the option for archiving deleted files. Not to pat myself on the back, but I think it came out looking pretty decent.
That's not to say there's no way to turn off all this fancy styling and animations, because, you bet your favourite monochrome flat design, there is.
Another take on Backup Settings window, 4 styling
variations.
Further compacted vertically, but now it appears to be somewhat cluttered and lacking comfortable vertical rhythm (to borrow a term from the typography).
Further compacted vertically, but now it appears to be somewhat cluttered and lacking comfortable vertical rhythm (to borrow a term from the typography).
Working on the Backup Settings window.
I think the original design from the version 1 got
many things right.
It provided a good at-a-glance overview of the backup configuration, split settings logically and tucked away the minute details. It did have one problem - it was tall. So tall, in fact, that I got several reports that it didn't fit on screen :-| V2 adds several new options to the backup config, so this doesn't make things any easier. After looking at few options, including paged, tabbed and two-column, screenful layouts, I circled back to the v1 layout.
To compress it vertically I've done two things - got rid of the group boxes (frames) and collapsed radio button groups into a single line with the current selection.
This gives us the middle screenshot (pic) that shows the config window in the "overview" state. Clicking on a "Change ↓" button expands the radio button group and allows changing the setting without leaving the page. Latter is pretty nice to have, because ... well ... because it's simply faster and more convenient.
Expanding a section gives the screenshot on the right (pic). To collapse the section, there's a little button in the section header on the right hand side. I'm not 100% convinced it's really needed nor that it's the best solution, but that's how it is at the moment. I forgot the "how to handle deleted files" section, but it will be in the beta. Not to worry.
It provided a good at-a-glance overview of the backup configuration, split settings logically and tucked away the minute details. It did have one problem - it was tall. So tall, in fact, that I got several reports that it didn't fit on screen :-| V2 adds several new options to the backup config, so this doesn't make things any easier. After looking at few options, including paged, tabbed and two-column, screenful layouts, I circled back to the v1 layout.
To compress it vertically I've done two things - got rid of the group boxes (frames) and collapsed radio button groups into a single line with the current selection.
This gives us the middle screenshot (pic) that shows the config window in the "overview" state. Clicking on a "Change ↓" button expands the radio button group and allows changing the setting without leaving the page. Latter is pretty nice to have, because ... well ... because it's simply faster and more convenient.
Expanding a section gives the screenshot on the right (pic). To collapse the section, there's a little button in the section header on the right hand side. I'm not 100% convinced it's really needed nor that it's the best solution, but that's how it is at the moment. I forgot the "how to handle deleted files" section, but it will be in the beta. Not to worry.
Meet the Preferences.
The Backup Engine section - as simple as it looks,
it is fronting a couple of months worth of work
and it's easily one of the most complex part of
the v2. This is what detaches the user interface
from the guts of the program and lets latter run
as a separate process.
The Maintenance section -
I figured that with the backup being an important
piece of software, very few users would be interested
in fully automatic updates. Furthermore, quite a few
people like to retain even tighter control over the
software and, for example, do not like it when the
software talks to the Internet whenever it wants.
Hence the new option -
Don't check, but remind to check for updates
It's basically a kitchen timer that goes "Ding!" every week or two (configurable) and prompts you to let the app check for a new version. We'll see how it goes, but I think it's a sensible compromise between the privacy and functionality. The More button transforms the window into a comprehensive list of all available options, close to two dozens now and still counting. The idea however is to keep the defaults reasonable so that there won't be much need to venture into this part of the app. Lastly, note how the Apply/Close buttons change when the settings are modified. I have no idea why this is not a standard behavior, because it really should be.
It's basically a kitchen timer that goes "Ding!" every week or two (configurable) and prompts you to let the app check for a new version. We'll see how it goes, but I think it's a sensible compromise between the privacy and functionality. The More button transforms the window into a comprehensive list of all available options, close to two dozens now and still counting. The idea however is to keep the defaults reasonable so that there won't be much need to venture into this part of the app. Lastly, note how the Apply/Close buttons change when the settings are modified. I have no idea why this is not a standard behavior, because it really should be.
A message bar is a unobtrusive way to display
one-liner messages that would've otherwise
required poping up a message box.
When a program detects that it was just restarted with Admin privileges, it has a limited number of options for communicating this joyous news to its user.
#1 - A message box. Bluntly put - I hate them. They are always too sudden, they steal your attention and stuff whatever they want to say down your throat. They ... lack manners.
#2 - A system tray notification. This is less intrusive, but perhaps it's too out of the way. Also, the systray is a shared area so the program is always competing with other apps for the same screen real estate.
#3 - Something else that is right in your field of view if you are looking at the app, but that doesn't get in a way of whatever you are doing at the moment. Cue in the message bar, now an integral part of the Bvckup UI. It is used for displaying various status notifications right after the launch, after switching the engine to/from the system service mode, when discovering software updates and few other things.
I have also tried the self-closing version of the bar that dismisses itself on a timeout. Like so -
and so -
but having lived with this for a couple of days, it now seems like a solution in a search of a problem. So I shelved it for now. If there's any a real need for this, it'd be very easy to add later on.
When a program detects that it was just restarted with Admin privileges, it has a limited number of options for communicating this joyous news to its user.
#1 - A message box. Bluntly put - I hate them. They are always too sudden, they steal your attention and stuff whatever they want to say down your throat. They ... lack manners.
#2 - A system tray notification. This is less intrusive, but perhaps it's too out of the way. Also, the systray is a shared area so the program is always competing with other apps for the same screen real estate.
#3 - Something else that is right in your field of view if you are looking at the app, but that doesn't get in a way of whatever you are doing at the moment. Cue in the message bar, now an integral part of the Bvckup UI. It is used for displaying various status notifications right after the launch, after switching the engine to/from the system service mode, when discovering software updates and few other things.
I have also tried the self-closing version of the bar that dismisses itself on a timeout. Like so -
and so -
but having lived with this for a couple of days, it now seems like a solution in a search of a problem. So I shelved it for now. If there's any a real need for this, it'd be very easy to add later on.
Nailed down the installer flow. A couple of
things of note here.
The most unconventional part of the setup is how it handles the configuration of the installation parameters.
The configuration page (see Page 3) is very bare and focuses on how the program is going to be used instead of being a mixed bag of the minute details of the setup.
The target location is made into a secondary setting with its value controlled by the usage preference. There's no field for the Start Menu group name. There's no ever popular "Run the program after the setup completes" option either. Instead, exiting the setup automatically runs the program (see Page 4), unless the installer window is closed through an (x) at the top.
This leads to a simpler and more intuitive setup flow... or at least that's the idea :) The installer supports both privileged (administrative) and unprivileged (regular user) installations.
If you are logged in as a regular user and have no rights to modify C:\Program Files, there's no reason why you shouldn't still be able to install a software, just for yourself.
While other installers routinely ask if to install the software for everyone or just for one user, it's a superficial question. It merely controls if the Start Menu group is created under All Users or your personal profile. The program files still go into a system directory and this requires administrative privileges.
Bvckup implements true system- and user- deployment, which is akin to deploying the program in /usr/bin or ~/bin in Unix terms.
In fact, on a multi-user system it is perfectly possible to install one instance of Bvckup for each user and they will all happily co-exist with each other.
The most unconventional part of the setup is how it handles the configuration of the installation parameters.
The configuration page (see Page 3) is very bare and focuses on how the program is going to be used instead of being a mixed bag of the minute details of the setup.
The target location is made into a secondary setting with its value controlled by the usage preference. There's no field for the Start Menu group name. There's no ever popular "Run the program after the setup completes" option either. Instead, exiting the setup automatically runs the program (see Page 4), unless the installer window is closed through an (x) at the top.
This leads to a simpler and more intuitive setup flow... or at least that's the idea :) The installer supports both privileged (administrative) and unprivileged (regular user) installations.
If you are logged in as a regular user and have no rights to modify C:\Program Files, there's no reason why you shouldn't still be able to install a software, just for yourself.
While other installers routinely ask if to install the software for everyone or just for one user, it's a superficial question. It merely controls if the Start Menu group is created under All Users or your personal profile. The program files still go into a system directory and this requires administrative privileges.
Bvckup implements true system- and user- deployment, which is akin to deploying the program in /usr/bin or ~/bin in Unix terms.
In fact, on a multi-user system it is perfectly possible to install one instance of Bvckup for each user and they will all happily co-exist with each other.
Working through the details of the update mechanism.
This is one of more difficult parts of the application, even though on the surface it looks uncomplicated. If we run the installer and it detects that the app is already installed, it should just update the installation. How hard could it be?
Let's see.
If the program is not running, then it's indeed a simple matter of replacing the program file with a newer version. The fun starts if the program is running.
On Windows, running a program locks its file, preventing any changes from being made to it. This is not required per se and there are operating systems that don't do this. There are also custom PE loaders for Windows that can take an executable and create a running process out of it without any locking fuss. For example, bo2k could do that back in 1999. See its source code for details if interested. To complicate matters, there might be more than one process running off the same executable. For example, if several users are logged into a Windows box and each runs a copy of a program.
Therefore, the first order of business for any installer is to shut down all running instances of the program. A well-behaved installer should also restart all these instances after it's done with updating. The shutting down part is not a big deal. There's more than one way to do this and all of them are fairly straight-forward.
It's the restarting that is contrived.
Not only program instances may be running under different user accounts, they may also be running with custom command-line arguments, as Administrators or system services. Restarting this bundle of joy is not an envious task. Here's how Bvckup works around this mess.
Every Bvckup executable has a unique Global event associated with it. The event is created by the first instance of the program when it's launched and it is opened by all other instances.
The name of the event includes the file ID of the executable file. This groups running instances by the file they are using, because, after all, the installer is interested in a specific executable and nothing else.
For example, the following is an event name used by the Bvckup installation on my development box -
Global\Bvckup2.e65a96a8001f000000013cba.Update
With this in place, the installer starts off by signaling
the event.
In response, each running instance makes a temporary copy of the executable file and starts it in a special "wait for the update to finish" mode, passing it all the context and details required to correctly restart later on.
And then it exits.
This temporary process is referred to as a relauncher and it merely spins there waiting for the Update event to become unsignaled.
Meanwhile the installer waits for the program file to become unlocked. Once all running instances replace themselves with the relaunchers, the executable frees up and the installer proceeds with the update. Then, once it's done, it clears the "update" event and exits.
This releases the relaunchers, they restart program instances (now using the actual executable), clean up after themselves and ta-da - the update is completed. * Techincally, instead of adding a special "relauncher" mode to the program it's possible just to have a small standalone relauncher program.
It comes down to a matter of taste. Some people don't like to spawn from temporary executables, and I like to keep my file count to a bare minimum. To each his own.
This is one of more difficult parts of the application, even though on the surface it looks uncomplicated. If we run the installer and it detects that the app is already installed, it should just update the installation. How hard could it be?
Let's see.
If the program is not running, then it's indeed a simple matter of replacing the program file with a newer version. The fun starts if the program is running.
On Windows, running a program locks its file, preventing any changes from being made to it. This is not required per se and there are operating systems that don't do this. There are also custom PE loaders for Windows that can take an executable and create a running process out of it without any locking fuss. For example, bo2k could do that back in 1999. See its source code for details if interested. To complicate matters, there might be more than one process running off the same executable. For example, if several users are logged into a Windows box and each runs a copy of a program.
Therefore, the first order of business for any installer is to shut down all running instances of the program. A well-behaved installer should also restart all these instances after it's done with updating. The shutting down part is not a big deal. There's more than one way to do this and all of them are fairly straight-forward.
It's the restarting that is contrived.
Not only program instances may be running under different user accounts, they may also be running with custom command-line arguments, as Administrators or system services. Restarting this bundle of joy is not an envious task. Here's how Bvckup works around this mess.
Every Bvckup executable has a unique Global event associated with it. The event is created by the first instance of the program when it's launched and it is opened by all other instances.
The name of the event includes the file ID of the executable file. This groups running instances by the file they are using, because, after all, the installer is interested in a specific executable and nothing else.
For example, the following is an event name used by the Bvckup installation on my development box -
In response, each running instance makes a temporary copy of the executable file and starts it in a special "wait for the update to finish" mode, passing it all the context and details required to correctly restart later on.
And then it exits.
This temporary process is referred to as a relauncher and it merely spins there waiting for the Update event to become unsignaled.
Meanwhile the installer waits for the program file to become unlocked. Once all running instances replace themselves with the relaunchers, the executable frees up and the installer proceeds with the update. Then, once it's done, it clears the "update" event and exits.
This releases the relaunchers, they restart program instances (now using the actual executable), clean up after themselves and ta-da - the update is completed. * Techincally, instead of adding a special "relauncher" mode to the program it's possible just to have a small standalone relauncher program.
It comes down to a matter of taste. Some people don't like to spawn from temporary executables, and I like to keep my file count to a bare minimum. To each his own.
Dotting all i's and crossing all t's, of which there's
plenty now that there's an optional system service
that just begs to be stopped, restarted and deleted
from under app's feet.
Doh... I missed the disabling of the service. Will add it tomorrow.
Doh... I missed the disabling of the service. Will add it tomorrow.
Vector icons for the website.
This is my first time not using pixel images to decorate a website. Instead, all icons are saved in the SVG format and converted to a webfont, which is then used to style regular HTML characters into icons.
For example, here's a font with the above icon set - VectorIcons.ttf
The advent of webapps like IcoMoon and Fontello not only made the process of building @font-face kits an absolute no-brainer, it also gave an easy access to hundreds of existing vector icons, many of which are free and of production quality.
When coupled with now virtually universal support for CSS text-shadow, box-shadow and gradients, this almost completely eliminates the need for raster form of smaller decorative and nuancing elements.
This is a very big deal and a great time-saver.
This is my first time not using pixel images to decorate a website. Instead, all icons are saved in the SVG format and converted to a webfont, which is then used to style regular HTML characters into icons.
For example, here's a font with the above icon set - VectorIcons.ttf
The advent of webapps like IcoMoon and Fontello not only made the process of building @font-face kits an absolute no-brainer, it also gave an easy access to hundreds of existing vector icons, many of which are free and of production quality.
When coupled with now virtually universal support for CSS text-shadow, box-shadow and gradients, this almost completely eliminates the need for raster form of smaller decorative and nuancing elements.
This is a very big deal and a great time-saver.
Re-finalized the sign-up form styling and mechanics.
Few styling options for an input control.
Leaning towards top right or bottom right,
but still need to see what works best in
context.
More or less done with the Features section
on the new website.
Probably missed few things, but that's easy to add later on. The biggest time sink was trying to inject some visual hierarchy into the list, but without going against overall minimalist look and feel of the site.
Might still revise it after sleeping on it for a couple of days :) Won't be the first time.
Probably missed few things, but that's easy to add later on. The biggest time sink was trying to inject some visual hierarchy into the list, but without going against overall minimalist look and feel of the site.
Might still revise it after sleeping on it for a couple of days :) Won't be the first time.
Plodding through some grunt work with the
new website. An almost equal split between
working on a copy, javascript and styling.
As simple as the screencap looks, it came with two hurdles that needed solving.
The header font, Alte Haas Grotesk, didn't display correctly after it was converted to the @font-face kit. Specifically, the lowercase 'o' looked like this:
The font's free, but I couldn't find a copy of its license, so I started tracking down font's author to ask a permission to edit the outlines.
While waiting for a reply, I found another workaround which involved generating the webfont kit from OTFs instead of TTFs. Not a big deal, but it still took over 3 hours to resolve. The second issue was with the slide-out, fade-in animation. Sometimes the slide-out didn't extend all the way down and sometimes it overshot. This traced back to the fact that slide-out height was computed before the OpenSans was fully loaded and the script fused basic sans-serif to compute the height.
Again, not too complicated, but, again, cost an hour to work through.
:-|
Little bit of this, little bit of that - all unplanned and it all adds up.
As simple as the screencap looks, it came with two hurdles that needed solving.
The header font, Alte Haas Grotesk, didn't display correctly after it was converted to the @font-face kit. Specifically, the lowercase 'o' looked like this:
The font's free, but I couldn't find a copy of its license, so I started tracking down font's author to ask a permission to edit the outlines.
While waiting for a reply, I found another workaround which involved generating the webfont kit from OTFs instead of TTFs. Not a big deal, but it still took over 3 hours to resolve. The second issue was with the slide-out, fade-in animation. Sometimes the slide-out didn't extend all the way down and sometimes it overshot. This traced back to the fact that slide-out height was computed before the OpenSans was fully loaded and the script fused basic sans-serif to compute the height.
Again, not too complicated, but, again, cost an hour to work through.
Little bit of this, little bit of that - all unplanned and it all adds up.
Nearly done with the new website design. Here's
a snippet of how the site's header interacts with
scrolling.
Released a small playable test of the directory scanning module.
It appears that using as many threads as there are CPU cores and traversing the tree in a depth-first fashion (scan child directories first and sibling directories second) yields the best scanning times.
Test and see for yourself - bvckup2-demo2.exe or 64 bit version.
Also, see here for details.
It appears that using as many threads as there are CPU cores and traversing the tree in a depth-first fashion (scan child directories first and sibling directories second) yields the best scanning times.
Test and see for yourself - bvckup2-demo2.exe or 64 bit version.
Also, see here for details.
2nd sketch for
bvckup2.com redesign.
Larger version is
here.
1st pass over the redesign of
bvckup2.com.
Larger version is
here.
Current design is almost two years old and it grew steadily disconnected from the simpler aesthetics I am pushing for now. So it needs to be redone.
I will probably simplify this even further. I've been wanting to do a true minimalist site for a software product for a long time now and this seems like a real chance to do just that. Yay.
Current design is almost two years old and it grew steadily disconnected from the simpler aesthetics I am pushing for now. So it needs to be redone.
I will probably simplify this even further. I've been wanting to do a true minimalist site for a software product for a long time now and this seems like a real chance to do just that. Yay.
This is a
sprite
sheet of all raster images that are going into the
run-as-a-service demo.
Interestingly enough, the total size of individual images saved as 32bit PNGs is 63,127 bytes, while the size of this sprite sheet (without checkered backdrop) is 63,829. After factoring in the size of the code that is needed to parse the sprite sheet, it really becomes a no brainer to keep every image in its own file and embed them into .exe's resources that way.
Interestingly enough, the total size of individual images saved as 32bit PNGs is 63,127 bytes, while the size of this sprite sheet (without checkered backdrop) is 63,829. After factoring in the size of the code that is needed to parse the sprite sheet, it really becomes a no brainer to keep every image in its own file and embed them into .exe's resources that way.
Added a "crash" handling and reporting facility
that includes several important changes compared to
the original version:
1. The program now differentiates between three types of errors and handles them slightly differently.
First, there are unexpected operating system errors, i.e. when a core Windows function fails in a way that it shouldn't be failing. Classic example (that everyone is aware of) is TrackPopupMenu that is declared as returning boolean, but actually doesn't shy away from returning an integer under certain circuimstances.
Secondly, there are inconsistencies in app's own code, whereby one part assumes certain usage that is not observed by another part. These are so-called "assert" failures. The original version has three critical issues, all of this particular kind.
Lastly, there are straight-forward crashes. This is when a program is trying to operate with data that is no longer there or corrupted. This is the worst of them all as it typically indicates a major screw up on programmer's part, but in some cases it may also be an indication of failing hardware, like a faulty memory stick. 2. The program now generates a minidump - a small-ish file in a standard Windows format that captures the context of a failure. Minidump files dramatically simplify investigation of a problem on the developer's end. 3. Finally, the minidumps can now be uploaded to the support server right from the error notification window with one click. The transfer is based on WinInet API as this is supposed to improve the chances of getting the report through in proxied and otherwise "enterprised" environments.
For cases when the upload fails, the error dialog also offers an option of opening a link in a regular browser and reporting a problem that way.
1. The program now differentiates between three types of errors and handles them slightly differently.
First, there are unexpected operating system errors, i.e. when a core Windows function fails in a way that it shouldn't be failing. Classic example (that everyone is aware of) is TrackPopupMenu that is declared as returning boolean, but actually doesn't shy away from returning an integer under certain circuimstances.
Secondly, there are inconsistencies in app's own code, whereby one part assumes certain usage that is not observed by another part. These are so-called "assert" failures. The original version has three critical issues, all of this particular kind.
Lastly, there are straight-forward crashes. This is when a program is trying to operate with data that is no longer there or corrupted. This is the worst of them all as it typically indicates a major screw up on programmer's part, but in some cases it may also be an indication of failing hardware, like a faulty memory stick. 2. The program now generates a minidump - a small-ish file in a standard Windows format that captures the context of a failure. Minidump files dramatically simplify investigation of a problem on the developer's end. 3. Finally, the minidumps can now be uploaded to the support server right from the error notification window with one click. The transfer is based on WinInet API as this is supposed to improve the chances of getting the report through in proxied and otherwise "enterprised" environments.
For cases when the upload fails, the error dialog also offers an option of opening a link in a regular browser and reporting a problem that way.
64bit version is stored inside of the 32bit
executable, from where it is extracted and
launched to replace the 32bit version if
latter detects that it is being run on a
64bit system.
Pro: a single executable to distribute
Con: the exe comes out weighing twice as much... or does it?
The size of a 32-bit executable can be easily reduced with the help of tools called executable packers . They work by turning the exe file into a self-extracting archive that also automatically launches the original exe after the extraction. It may sound simple but in reality it involves bundling a custom PE loader with the packed executable (as it might not be possible to extract the exe into a temporary file before running it).
In any case, exe packing is routinely used by developers who care about the size of their products, with uTorrent being one of most notable examples. Original Bvckup was also compressed, owning its tiny size in part to the magic of UPX . The problem lies with 64-bit executables. For a variety of reasons there are presently no exe packers that support them.
However if we store 64bit binary inside of the 32bit executable and then pass latter through a compressor, we end up with both being compressed. Ta-da!
But there's more. When an .exe is compressed, the compressor leaves the primary application icon intact. This is done so that the app would keep its original appearance in Windows Explorer and not sport a generic "program" icon.
Starting with Vista, the app is expected to support a variety of icon sizes that quickly add up. For example, bvckup's icon weighs in at about 110KB. That's almost half as big as the actual code, the code that does quite a few meaningful things on top of just looking great :)
So, if we do the 64-in-32bit packaging, we end up compressing 64bit version in its entirety, including its icon, and arriving at a greater size reduction if we were to compress the exes separately.
Win-win.
Pro: a single executable to distribute
Con: the exe comes out weighing twice as much... or does it?
The size of a 32-bit executable can be easily reduced with the help of tools called executable packers . They work by turning the exe file into a self-extracting archive that also automatically launches the original exe after the extraction. It may sound simple but in reality it involves bundling a custom PE loader with the packed executable (as it might not be possible to extract the exe into a temporary file before running it).
In any case, exe packing is routinely used by developers who care about the size of their products, with uTorrent being one of most notable examples. Original Bvckup was also compressed, owning its tiny size in part to the magic of UPX . The problem lies with 64-bit executables. For a variety of reasons there are presently no exe packers that support them.
However if we store 64bit binary inside of the 32bit executable and then pass latter through a compressor, we end up with both being compressed. Ta-da!
But there's more. When an .exe is compressed, the compressor leaves the primary application icon intact. This is done so that the app would keep its original appearance in Windows Explorer and not sport a generic "program" icon.
Starting with Vista, the app is expected to support a variety of icon sizes that quickly add up. For example, bvckup's icon weighs in at about 110KB. That's almost half as big as the actual code, the code that does quite a few meaningful things on top of just looking great :)
So, if we do the 64-in-32bit packaging, we end up compressing 64bit version in its entirety, including its icon, and arriving at a greater size reduction if we were to compress the exes separately.
Win-win.
Working on a shared memory buffer interface,
a part of the
IPC
module, which is what enables
the separation of the engine and the UI.
A bit more humane approach to critical error reporting.
The message is a homage to NetPositive, the BeOS browser that pioneered the technique of using haiku in error reporting. In fact, the message in option B is from the NetPoisitive original list .
The icon is that of a Zen circle, which has a plenty of meanings, many of which applicable given the circumstances.
The message is a homage to NetPositive, the BeOS browser that pioneered the technique of using haiku in error reporting. In fact, the message in option B is from the NetPoisitive original list .
The icon is that of a Zen circle, which has a plenty of meanings, many of which applicable given the circumstances.
Test of a fading transition, part of a small library
that deals with supporting UI animations.
Fading animation dissolves a part of a window into background, in preparation for transitioning the window to another state, e.g. moving to the next page in a wizard or resizing the window into a more compact form.
As simple as it sounds, these sort of animation is fairly complex to implement on top of the Windows API.
Windows includes native support for window transparency in a form of layered windows, but it comes with a list of limitations that aren't exactly practical or easy to work, except for several specific cases.
The simplest working solution involves taking a screenshot of the window, creating an overlay window and then using it as a canvas to render the transition from an empty background to the screenshot. All the while making sure that Windows doesn't sneak in an unsolicited window refresh that would lead to a flicker, which in itself is a very "fun" thing to debug.
Fading animation dissolves a part of a window into background, in preparation for transitioning the window to another state, e.g. moving to the next page in a wizard or resizing the window into a more compact form.
As simple as it sounds, these sort of animation is fairly complex to implement on top of the Windows API.
Windows includes native support for window transparency in a form of layered windows, but it comes with a list of limitations that aren't exactly practical or easy to work, except for several specific cases.
The simplest working solution involves taking a screenshot of the window, creating an overlay window and then using it as a canvas to render the transition from an empty background to the screenshot. All the while making sure that Windows doesn't sneak in an unsolicited window refresh that would lead to a flicker, which in itself is a very "fun" thing to debug.
Every application icon that is exposed to the
operating system must exist in at least nine
sizes.
16, 32, 48, 256 are more or less expected. These are used in "small", "icon", "tile" and "extra-large " views in Windows Explorer.
20, 22 and 26 are possible sizes of an icon used in the window caption. The exact size depends on screen's DPI level.
And 40 (along with 32 and 48) is the size of an icon used in Alt-Tab view on W7, again depending on the screen's DPI.
If you don't provide an appropriately sized icon, the OS will grab one that is slightly bigger and shrink it down, ruining the pixel polish.
Lastly, each icon ideally must be saved in 32, 8 and 4 bits per pixel formats, so to be compatible... you know... with EGA cards
It would've not that big of a deal if it weren't for the size of a resulting .ico file. For example, the above icon ends up being ~ 120KB, whereas the actual .exe it is attached to and that does quite a few meaningful things sits at around 96KB.
16, 32, 48, 256 are more or less expected. These are used in "small", "icon", "tile" and "extra-large " views in Windows Explorer.
20, 22 and 26 are possible sizes of an icon used in the window caption. The exact size depends on screen's DPI level.
And 40 (along with 32 and 48) is the size of an icon used in Alt-Tab view on W7, again depending on the screen's DPI.
If you don't provide an appropriately sized icon, the OS will grab one that is slightly bigger and shrink it down, ruining the pixel polish.
Lastly, each icon ideally must be saved in 32, 8 and 4 bits per pixel formats, so to be compatible... you know... with EGA cards
It would've not that big of a deal if it weren't for the size of a resulting .ico file. For example, the above icon ends up being ~ 120KB, whereas the actual .exe it is attached to and that does quite a few meaningful things sits at around 96KB.
Similarly, every bit of graphics in the app
must be done in at least three different sizes
- one for each common screen DPI level.
DPI stands for dots-per-inch and it's a metric of a pixel density on the screen. The smaller the screen, the more pixels are packed into a square inch, assuming the resolution stays the same. The difference in DPI between a large LCD panel and a 12" notebook screen can easily be a factor of two or more.
This means that if a program renders 9px text on both screens, on one it will appear twice as small as on the other one, and might command a need for a good magnifying glass.
So naturally it's a good idea to make the OS aware of the actual DPI of the monitor in use. If it's too high, the OS can do few things to make things visible with a naked eye again. Like increase size of the default system font and make windows generally larger.
This can be done automatically by looking at the monitor .inf files or by letting the user configure the DPI manually. XP defaults to 96 DPI and with some difficulty it can be changed to 120 DPI. Vista and newer versions are better. They come with four predefined levels - 96, 120, 144 and 192 - and they also support custom settings.
This brings us back to needing multiple versions of every graphic in the app. Vista is the first Windows version to routinely use monitor-specific DPI values out of the box. As such 96 is no longer a dominant DPI level and the apps are now expected to take the DPI into an account when rendering their content.
The higher the DPI, the larger the graphics and the fonts the app should use. The penalty for not doing this is an automatic stretching of the app window that inevitably leads to a blurry appearance and a host of other lovely issues .
In other words - not complying with OS expectations is not really an option. Hence the need for 3+ differently sized copies of the same pixel art.
DPI stands for dots-per-inch and it's a metric of a pixel density on the screen. The smaller the screen, the more pixels are packed into a square inch, assuming the resolution stays the same. The difference in DPI between a large LCD panel and a 12" notebook screen can easily be a factor of two or more.
This means that if a program renders 9px text on both screens, on one it will appear twice as small as on the other one, and might command a need for a good magnifying glass.
So naturally it's a good idea to make the OS aware of the actual DPI of the monitor in use. If it's too high, the OS can do few things to make things visible with a naked eye again. Like increase size of the default system font and make windows generally larger.
This can be done automatically by looking at the monitor .inf files or by letting the user configure the DPI manually. XP defaults to 96 DPI and with some difficulty it can be changed to 120 DPI. Vista and newer versions are better. They come with four predefined levels - 96, 120, 144 and 192 - and they also support custom settings.
This brings us back to needing multiple versions of every graphic in the app. Vista is the first Windows version to routinely use monitor-specific DPI values out of the box. As such 96 is no longer a dominant DPI level and the apps are now expected to take the DPI into an account when rendering their content.
The higher the DPI, the larger the graphics and the fonts the app should use. The penalty for not doing this is an automatic stretching of the app window that inevitably leads to a blurry appearance and a host of other lovely issues .
In other words - not complying with OS expectations is not really an option. Hence the need for 3+ differently sized copies of the same pixel art.
Trying to find a balance between using custom styling and not getting to far away from the native OS look and feel. See also 8 other options leading up to this one.
Illustrative icons for the same demo.
First pass over the demo UI.
V2 running as a service,
an actual screenshot for a change.
An idea for an in-form buttons.
The backup job - two VM machines, 32 and 256 GB in size. One is an active Linux image with LNMP stack and another is a desktop Windows with the usual end-user stuff - web browser, mail, etc.
After enabling changed range tracking for the volume and re-configuring the job to use it, the backup time for a day worth of changes went from 14 minutes to 31 seconds ... half of which was spent on making a VSS snapshot.
One might be tempted to say that this is not bad.