This website uses cookies:
• from Cloudflare to identify trusted web traffic and protect this website. [read more]
• from Google Analytics to generate statistical data on how visitors uses this website. [read more]
• from Google AdSense (third party vendors, including Google) to deliver ads based on user prior visits to this website or other websites.
Do you agree that your data to be used to deliver you personalized website ads? You can change your mind at any time by accesing the ePrivacy button located at the bottom of every page. Enjoy your visit!

Restic & incron/inotify, continuous backup on folder content change

Restic is a cross platform “Fast, secure, efficient backup program, Free Software (not just Open Source)”, as the developers says and I'm surely agree with that. It's written in Go and here https://github.com/restic/others you can find some of restic key features compared with other backup solutions.

I've recently discovered restic during a live session of the latest Clear Linux OS, when searching through swupd repositories (bundles) for a native backup solution. Clear Linux did't get the chance of a persistent install, the installer not being able to install the OS on my existing EFI partition and a dedicated free space one (dual/multi boot) alongside my Pop!_OS daily driver. But I've discovered restic and despite the software lacks a GUI, needs some readthedocs research and some terminal skills, I've started to used it, mainly because, at least in my setup, is extremely fast. BTW, I gave up the deja-dup/duplicity and all the Gnome bells that this combination give me (GUI, Nautilus integration) in favor of restic.

As always, my logic in doing backups is as follows: store encrypted snapshots (backup) in a local storage (a folder inside internal drive/external drive) and keep this folder in sync with a reliable cloud storage (I'm using mega.nz here).

Frequency of the backups: here is another point where restic wins against deja-dup/duplicity, in the past my backup frequency was once a day (using anacron) due to time taken by duplicity to finish the job, now, using restic I can schedule backups only if there is a need of it (a monitored folder/folders suffer modifications (new/deleted/modified files)). Right now I'm doing a full home folder backup at @reboot and every few minutes backups for my Documents folder where I keep most of my work.

Dealing with restic is not so easy, but all we need is just some shell copy/paste commands and some replacement in these commands, e.g. your home folder may be named differently than /home/catalin :)

The backup software stack

sudo apt install restic incron

The restic setup

To be able to automate restic snapshots we need two files, a pseudo config file where to put two environment variables used by restic and another one where to store the password used for encrypting the backup. This files will be stored in the root of our home folder, ~/

nano .restic.env
# add bellow two lines to the file, save it and close it
export RESTIC_REPOSITORY="/home/catalin/BackupSyncR"
export RESTIC_PASSWORD_COMMAND="gpg -d .restic.passwd.gpg"

RESTIC_REPOSITORY is the full local path where the snapshots will be saved, RESTIC_PASSWORD_COMMAND is a shell command that reads the encryption password from a gpg encrypted file, we will create this file right away.

info If using RESTIC_PASSWORD_COMMAND seems to be complicated (generate and maintain a copy of a private key) or the target computer is a server, not a desktop computer, then we can use a plain encryption password instead (export RESTIC_PASSWORD="our-plain-password-here" (just make sure to remember this password for worst-case scenarios) and skip the fuss of generating the GnuPG key and the .restic.passwd.gpg encrypted password file)

# generate a GnuPG key
gpg --full-generate-key
# select 2 (DSA) -> 2048 as keysize -> 0 = key does not expire -> Real name -> email -> a passphrase to protect the new created key (remember this password/passphrase)

# create a temporary file to store in it a super strong password (https://www.grc.com/passwords.htm is the right place for such a password)
echo "AjuEoKVsJLNElLmKPydYg3OlLQTd0BZRG5HprOJRZTqz3AofHWUwSHpCNxLn4RR-the password-used-to-encrypt-backup" > .restic.passwd
# encrypt the .restic.passwd file then remove the plain version of it
gpg -e .restic.passwd && rm .restic.passwd
# enter the email used on your previous GnuPG key (eg catalin@pop-os) when prompted

pan_tool The private GnuPG key (can be exported from "Paswords and Keys") and the .restic.passwd.gpg encrypted file (used to encrypt/decrypt the backup) must be kept in a copy to a separate storage space (eg external drive/USB, email, you name it) so in case of complete crash/SSD failure/data loss, we will have access to them. If a plain password was used instead then there is nothing to care about, just to remember that plain password.

Now we have all we need to start our scheduled backup, but first we need to init the restic repository (declared in the .restic.env file, RESTIC_REPOSITORY)

Init the restic repository

. ~/.restic.env  ; restic init
# the password for the key used to encrypt .restic.passwd.gpg will be requested, enter it and check the "Save in the password manager"

Optional but highly recommended

If you are using a cloud storage provider, now it's a good time to put in sync the “BackupSyncR” folder with it. I'm using mega.nz — megasync utility for that.

Schedule backups

We will use cron for that, with a trick :)

crontab -e
# add this two lines
@reboot   sleep 60 ; . ~/.restic.env && restic backup ~/ --exclude Documents --exclude Downloads --exclude BackupSyncR --exclude .cache --exclude gvfs-metadata --exclude .dbus --exclude *cache* --exclude *cookies* --exclude *places*
*/15 * * * *  [ -f ~/.restic.run ] && rm .restic.run && . ~/.restic.env &&  restic backup ~/Documents

info Monitoring the cron execution is quite simple if we have an MTA installed in our system, and here I recommend the DragonFly MTA (sudo apt install dma). Then all cron executions will be logged to /var/mail/$USER and more than these local emails can be accessed from Thunderbird via an "Add Other account" - Unix Mailspool (Movemail).

First line is executed in 60 seconds upon OS login, it loads restic environment variables and execute backup of the home folder with some excludes, mainly the restic repository itself (yes, in my setup it is located inside home folder), some special folders/files and the Documents folder which (Documents) is scheduled for snapshots in the next crontab line every 15min but only if needed.
The trick is that the Documents folder gets snapshoted only if the file .restic.run exists inside home folder and if yes then it gets deleted (.restic.run) after backup finishes. Got it? Every 15min we check if a file exists, if yes, execute backup, write to log and delete that file.
Now, the file .restic.run gets created every time something special happens into documents folder, mainly if there are new, modified or deleted files or folders inside. How we do that? Simply, by using incron and an incrontab entry.

sudo nano /etc/incron.allow
# add your username inside (catalin), save it and close it
incrontab -e
# add this line
/home/catalin/Documents/        IN_MODIFY,IN_CREATE,IN_DELETE,IN_MOVED_TO touch /home/catalin/.restic.run
#folder to monitor              events to monitor                         action when an event is triggered

Restore files

The easiest way to restore files/folders from the restic snapshots is to mount the snapshots and use a file browser to search/copy data.

# if not created, create a /mnt folder where to mount the restic snapshots
sudo mkdir /mnt/BackupRestoreR && sudo chown catalin.catalin /mnt/BackupRestoreR
# and mount the snapshots in this folder
. ~/.restic.env ; restic mount /mnt/BackupRestoreR

And if you need more info about restoring files or in general about using restic there is readthedocs.io https://restic.readthedocs.io/

label_outline

Share this post on your favorite networks.
© Copyright 2017 b247.eu.org | Just another information technology blog - All Rights Reserved