Hello everyone,
Today we’re shipping a small minor release: v1.0.2.
It brings an automatic security-update checker, fixes a relative-path bug when using your agent, and delivers a 60× speed boost + lower memory usage on S3 backups.
Since many of you rely on S3 daily, we couldn’t wait until a larger release—here’s how to grab it.
Via Go:
1$ go install github.com/PlakarKorp/plakar@v1.0.2
Via installer:
1$ curl https://plakar.io/install.sh | sh
2Downloading plakar 1.0.2 signature file...
3Downloading plakar 1.0.2 release file...
4Downloading plakar 1.0.2 public key...
5Verifying plakar 1.0.2 signature...
6Signature Verified
7plakar-1.0.2.tar.gz: OK
8Extracting plakar 1.0.2 release...
9Installing plakar 1.0.2...
What’s in that release ?
Nothing too big but four fixes, two being interesting enough for our community that we want them available to all as soon as possible.
Security check
Plakar can now check the releases atom feed of our project and warn you if a newer release carries a reliability or security fix that you should really install.
We think it’s a good option to have for most users,
but if you’re not comfortable with this check,
you can turn it off permanently using plakar -disable-security-check and live on your own adventurous path.
Fix for relative pathnames through agent
We spotted a small bug when using relative paths for a repository through the agent.
Plakar has no problem resolving relative paths for backups, however if the path of the repository was relative AND the CLI was executed from a directory different from the agent, then the CWD was incorrectly resolved and the relative path resolution failed.
For example, plakar at ./foobar backup could mean /tmp/foobar on the CLI but ~gilles/foobar on the agent.
This is now fixed !
S3 speed improvements
We have identified a better way for our importer to iterate over “directories” in S3 buckets.
Long story short,
in S3 there’s no concept of directories:
we use the / character as a path separator and pretend that it creates nested objects when they are really flat at the top level.
What we had initially overlooked is that the MinIO client supports a recursive listing option, which handles the prefix-based “directory” emulation more efficiently than our custom logic. This drastically improves the performances of listing resources.
In his test case,
he observed an impressive x60 performance boost on a backup over S3,
reducing a backup that took ~14 minutes to ~13 seconds.
This by itself was worthy of a minor release :-)
S3 memory improvements
One of our users, @anbcorp reported an OOM kill when doing backups.
We setup a few tests and quickly identified that it only affected the S3 storage backend, but oddly enough it didn’t trigger on my machine or that of another developer. It only took a few minutes of profiling to figure out what was happening:
When we do not know the size of a file,
the minio client library does a big allocation (512MB).
The problem is that not only do we not know the size of some files before they are pushed,
but we also parallelize quite a bit,
leading the client library to perform a lot of big allocations.
This isn’t an issue when you have a few number of cores (low parallelism) or a large amount of memory (can handle the big allocations), but if you have a large number of cores and low memory (that was @anbcorp’s case), you end up parallelizing a large number of huge allocations until you get terminated.
This only affects one specific call in the storage backend, which is the only one where we don’t know the size of files and use heavy parallelization, but we found a fix that solves this issue and manages to avoid the code path that does the big allocations.
This was confirmed to fix the issue, and was also by itself a good reason to make a minor release.
What’s next ?
We’ll be talking about the capabilities of PTAR, our own archiving format, very soon on this blog.
Stay tuned !
Community
Want to help us ?
Special thanks to all early users, contributors, and supporters who have helped shape this first release—your involvement has been invaluable!