dumpnet - a rocketsave backup solution
The script dumpnet is the main entry point to the whole backup solution. It will call dumpnet.host for every remote host to backup (referred to as ``target host''). In the configuration file dumpnet.conf(8) you can define which target hosts should be backuped.
These are the main features of dumpnet:
The backup solution dumpnet consists of multiple modules. Please see the corresponding man pages for details. Here is an overview:
Create a crontab file for the dumpnet user and add an entry like this:
0 1 * * * /usr/local/dumpnet/bin/dumpnet
This calls dumpnet every day at 01:00 AM.
The concept of levels should be familiar from tools like dump. dumpnet knows the levels 0-9 and the special level -1 which means not to do any backup at all. A level 0 does a full backup. A level number above 0 does an incremental backup that only saves the files that were modified since the last backup of a lower level. A level 1 backup will save all files modified since the last level 0 backup, a level 2 backup will save all files modified since the last level 1 backup and so on...
The information which command was run with which level on what date is stored in $DUMPNET_HOME/log/dumpnet.cmddates. See dumpnet.cmddates(8) for details.
Levels can be specified on the command line of the modules dumpnet, dumpnet.host, dumpnet.host.fstab and dumpnet.host.userdefined. They can be as well used in the configuration files dumpnet.fstab.conf and dumpnet.userdefined.conf (RECOMMENDED). Whereever you can specify a level as a number of 0 to 9 you can specify a level mode.
Level modes provide the ability to perform a backup with a different level depending on the date (day of month/day of week - or what ever condition you define). The level modes available by default are the files named $DUMPNET_HOME/bin/dumpnet.getLevel.*:
Whereever you specify a level mode, only specify the asterix part in dumpnet.getLevel.*. The name of the level mode will be passed as argument to dumpnet.getLevel. It will then execute dumpnet.getLevel.LEVEL_MODE if it exists. Please see dumpnet.getLevel(8) for details.
If you need an other level mode than the ones provided by default, write one yourself. Just duplicate dumpnet.getLevel.lev0YearlyLev1MonthlyLev2Weekly and modify it as you need.
dumpnet allows you to define cleanup policies for each different kind of backup. As storage is limited, you will have to delete old backups at some point in time. To reduce the burden put on the backup operator, this process can be automated. By setting the option recycleBin in dumpnet.conf you can move cleaned up files to a recycle bin directory instead of deleting them. Please see dumpnet.conf(8) for details. dumpnet comes with a couple of predefined policy elements that can be combined to a cleanup policy. All policy elements are separate files named $DUMPNET_HOME/bin/dumpnet.cleanupPolicy.*. A policy has the following formatting rules: - it does not contain any spaces - policy elements are separated by commas (``,'') - arguments for a policy element are preceeded by an underscore (``-'')
No matter what policy elements are used all files containing the strings ``MASTER'' or ``MODIFIED'' will be kept.
Please note that all predefined policy elements will not only keep the specified backups but as well their corresponding TOC-files. The predefined policy elements are as follows:
Examples: keep - keep all files keep_lev0 - keep all level 0 backups (files containing lev0)
Examples: keepAllSinceLev0_1 - keep all files since the last level 0 backup keepAllSinceLev0_3 - keep all files since the last 3 level 0 backups
Example: keepDaily_lev2 - keep the last successful level 2 backup on every day
Examples: keepFor_2_months - keep all files for two months keepFor_10_d - keep all files for ten days
Examples: keepLast_3_lev0 - keep the last three level 0 backups keepLast_4_lev2 - keep the last four level 2 backups
Now let's put everything together (see the policy formatting rules above). Here are some examples:
Let's persume that we do incremental backups with the level mode lev0MonthlyLev1WeeklyLev2Daily. We want to keep all backups since the last two level 0 backups. The last level 0 backup of every year should be kept forever. Also we want to keep all failed backups for manual inspection.
keepAllSinceLev0_2,keepYearly_lev0,keep_FAILED
In this example we only do non-incremental backups. We want keep backups for 1 and a half month (46 days). The last backup done in every month we want to keep forever. Failed backups should be kept as well.
keepFor_46_days,keepMonthly,keep_FAILED
Here we persume that we do incremental backups with the level mode lev0WeeklyLev1Daily. We want to keep the last three level 0 backups and the last 4 level 1 backups. Failed backups should not be kept.
keepLast_3_lev0,keepLast_4_lev1
A formulated cleanup policy will be passed to dumpnet.cleanupPolicy which will in turn call the corresponding dumpnet.cleanupPolicy.* scripts. Please see for details: dumpnet.cleanupPolicy(8).
The session ID is used to identify a backup session. A session is started by manually executing one of the dumpnet modules. The executed module will generate a new session ID (if none was given on the command line) by calling dumpnet.getSessionId(8). dumpnet.getSessionId will increment the integer stored in the session ID logfile and use it as the new session ID. The location of the session ID logfile can be specified by the variable $logfile_sessionId in dumpnet.conf(8).
A progressive session ID is essential for dumpnet to work correctly. Every session ID must be greater than the previous one. It should never be neccessary to specify a session ID on the command line! Nevertheless dumpnet will use the -s switch the hand the generated session ID from one module to another. Also make sure the session ID logfile does not get corrupted and the session ID always has 5 digits (is equal to or greater than 10000). Please see dumpnet.getSessionId(8) for details.
The global configuration is done in $DUMPNET_HOME/conf/dumpnet.conf. The default target host configuration is (per default) stored in $DUMPNET_HOME/conf/_defaults/dumpnet.HOST.conf. For each target host a directory should be created in $DUMPNET_HOME/conf/. By convention it must not start with an underscore. It's recommended to use the hostname as the directory name. Such a directory will be referred to as a ``host configuration directory''.
A basic verification of the dumpnet configuration file (the global configuration) can be done using dumpnet.checkDumpnetConfig(8). A basic verification of a host configuration file can be done using dumpnet.checkHostConfig(8).
All configuration file have the same format - see the next section CONFIGURTION FILE FORMAT. The following files can be placed in the host configuration directory:
A sample configuration of the target host ``websrv'' looks like this (all paths relative to /usr/local/dumpnet/):
conf/dumpnet.conf global dumpnet config
conf/_defaults/ all default config files
conf/_defaults/dumpnet.HOST.conf the default host config file
conf/websrv/ the host config directory for the
host "websrv"
conf/websrv/dumpnet.HOST.conf the host configuration file
conf/websrv/dumpnet.fstab.conf conf for dumpnet.host.fstab
conf/websrv/dumpnet.userdefined.conf conf for dumpnet.host.userdefined
conf/websrv/dumpnet.diff.conf conf for dumpnet.host.diff
conf/websrv/dumpnet.info.conf conf for dumpnet.host.info
conf/websrv/dumpnet.aide.conf conf for dumpnet.host.aide
conf/websrv/dumpnet.report.conf conf for dumpnet.host.report
conf/websrv/dumpnet.reportmail.conf conf for dumpnet.host.reportmail
All configuration files - dumpnet.conf and target host specific config files - have the same format. They share the following features:
- they primarily contain a list (dumpnet.aide.conf is somewhat of an exception) - a backslash-newline pair can be used as a line continuation but literal backslashes need to be escapted - see the read_options in B<dumpnet.conf>(8) for how to change this behaviour - multiple inheritance can be implemented using `# extends' - options for the whole config file can be set using `# option'
All lines starting with `#' are not part of the list primarily contained in the file. In general all lines starting with `#' are ignored as comments. But there are two important exceptions:
Example:
# extends ../_openbsd3.6/dumpnet.info.conf # extends ../_webservers/dumpnet.info.conf
The extends statement allows you to configure aspects of multiple target hosts in single file by making their config files inherit from your parent file. For example if you had 3 OpenBSD boxes you could make a general OpenBSD configuration file and extend it as needed for you specific machines.
Example:
# option host=localhost # option dump=/sbin/dump
Mostly these options are - as the name implies - optional. Option statements will be inherited as well. So if you don't feel comfortable with what is provided by the parent configuration file just overwrite it.
To debug your configuration files use dumpnet.extendConfig. It will parse the extends statements and output the resulting configuration data to STDOUT. See dumpnet.extendConfig(8) for details.
This is for the really paranoid: with dumpnet.host.userdefine and dumpnet.userdefined.conf it is possible to encrypt a backup before writing it to file. You can for example use gpg for an asymmetric encryption. Only the person who has the private key that maches the public key that was used to encrypt the file, can decrypt it. Keep in mind that it will take you longer to restore an encrypted backup. If you also want to compress the backup, do that before encrypting it. A command configured in dumpnet.userdefined.conf that makes a compressed and encrypted dump of all MySQL databases looks like this:
( /usr/local/mysql/bin/mysqldump -aA; echo COMMAND_RETURN_CODE $? >&2 ) \\ | ( gzip -c; echo COMMAND_RETURN_CODE $? >&2 ) \\ | gpg -er dumpnet@rocket.home.lukasfeiler.com
It will compress and encrypt the data before it gets transferred to the dumpnet host.
Note the use of COMMAND_RETURN_CODE: as the call to mysqldump takes place before a pipe, the whole expression could still have an exit status of 0 if the last command gpg exits with 0. To let dumpnet.host.userdefined.cmd still notice if mysqldump fails, you have to send its return code to STDERR prefixed by 'COMMAND_RETURN_CODE '. The same thing happens for the gzip command. For details of this feature please see dumpnet.userdefined.conf(8).
An other alternative is to use an encrypted file system. This will keep someone who steels your physical backup machine from reading its data - as long as he doesn't crack the root password. An encrypted file system will not help you if the backup machine's operating system gets compromised (that is ``hacked'').
dumpnet provides extensive logging. It knows four different priorities:
INFO: informs you about the current state of operation. Essential for finding out what's really going on.
WARN: warning messages; a warning does not essentially lead to an error but it's likley though.
SUCC: a module successfully completes it's operation. Keep in mind that most modules will call other modules to do parts of the job.
ERRO: a module failes to complete it's operation. This is serious!
The logfiles are stored in $DUMPNET_HOME/log/ by default. To change that, see dumpnet.conf(8). dumpnet has two different log files:
standard.log: logs only the priorities ERRO and SUCC. extended.log: logs all priorities (INFO, WARN, ERRO and SUCC).
Both logfiles have the same format:
$PRIORITY: [ SID$sessionId YYYY-MM-DD HH:MM:SS ][ $module ]: $host : ... (rest of message)
You might learn a lot about the architecture of dumpnet just by browsing the log files. Use less -S or similar tool.
All backups will be stored in $backupDir/$host/$subdir. $backupDir is the backup root directory (``/backup'' per default) - it can be configured in dumpnet.conf. $host is the IP address or host name of the target host as configured in a dumpnet.HOST.conf file. Every different kind of data has a different $subdir. The directories for all modules can be configured in the corresponding host-specific configuration files dumpnet.fstab.conf, dumpnet.userdefined.conf, dumpnet.diff.conf, dumpnet.info.conf, dumpnet.aide.conf and dumpnet.report.conf. For dumpnet.host.info, dumpnet.host.aide and dumpnet.host.report it can be as well configured in dumpnet.HOST.conf. It is recommended to go with the defaults provided by the default host configuration file and to use the name of the mount point for the backup of a partition.
An example could look like this
/backup the backup root dir /backup/example.com the target host example.com /backup/example.com/info files generated by dumpnet.host.info /backup/example.com/aide AIDE databases and reports /backup/example.com/report dumpnet session reports /backup/example.com/usr dumps of the /usr partition /backup/example.com/home dumps of the /home partition /backup/example.com/var dumps of the /var partition /backup/example.com/ROOT dumps of the / partition
Just do a backup with all default settings - if you don't know what you want you want this:
dumpnet
Force the use of level 0, do not update dumpnet.cmddates and use an alternative dumpnet configuration file:
dumpnet -Nl 0 -d /usr/local/dumpnet/conf/dumpnet2.conf
Just list the files I need to do a restore:
dumpnet -R
Find the archive/dump for every target host I need to restore important_v1.1.011.txt. In most cases you would want to call dumpnet.host direclty to restore a file from a specific target host. To call dumpnet in this way is only useful if you don't know on which target host the file was. Please note that a leading slash must be removed. This is because the tar TOC-files do not contain a leading slash (only the dump TOC-files do).
dumpnet -Rf important_v1.1.011.txt
dumpnet will return 1 if $backupDir (see dumpnet.conf(8)) cannot be created. In any other case it has a zero exit status.
$DUMPNET_HOME/conf/dumpnet.conf
Please see the following man pages:
dumpnet(8)
dumpnet.host(8)
dumpnet.host.aide(8)
dumpnet.host.diff(8)
dumpnet.host.diff.cmd(8)
dumpnet.host.fstab(8)
dumpnet.host.fstab.dump(8)
dumpnet.host.fstab.tar(8)
dumpnet.host.fstab.rsync(8)
dumpnet.host.info(8)
dumpnet.host.report(8)
dumpnet.host.reportmail(8)
dumpnet.host.userdefined(8)
dumpnet.host.userdefined.cmd(8)
dumpnet.conf(8)
dumpnet.HOST.conf(8)
dumpnet.aide.conf(8)
dumpnet.diff.conf(8)
dumpnet.fstab.conf(8)
dumpnet.info.conf(8)
dumpnet.report.conf(8)
dumpnet.reportmail.conf(8)
dumpnet.userdefined.conf(8)
dumpnet.cmddates(8)
dumpnet.cleanupPolicy(8)
dumpnet.extendConfig(8)
dumpnet.getLevel(8)
dumpnet.getSessionId(8)
Lukas Feiler <lukas.feiler@lukasfeiler.com>
dumpnet is published under the LGPL (GNU LESSER GENERAL PUBLIC LICENSE). Please see the LICENSE file contained in the installation tarball for details. The official web site is http://dumpnet.sourceforge.net.