View source for Centralized Automated Backups via Email with Procmail and Perl

= Introduction =
Designed to take automated email backups and save only the attached backup file to a directory path.  Files are further sorted into a base directory plus a directory matched or created by the "From" address of the MT system script.  For instance, you could set the "From" address of all RB600 devices backup scripts to "RB600@yourdomain.com" and send them to your backup email address and all RB600 device backup files will be saved into the $basedir/RB600 directory by their system name.  Tested and working with RouterOS 2.9.51 and 3.23. Working on CentOS 4.7 and CentOS 5.3 with MIME tools package and dependencies from rpmforge.

= Assumptions =
You are familiar with RouterOS scripts and basic linux administration.  Have adequate permissions to install perl modules, create directories, and execute scripts on the host.  Since I use CentOS for most of my mail servers, I will base this installation guide on a CentOS server. As long as the base dependencies of the scripts are met, the configuration should be pretty much the same on your preferred linux distribution.

= Requirements =
<b>Mikrotik RouterOS</b>

<b>linux/*nix host</b> with:

  procmail
  formail
  expand
  sed
  awk
  perl
    MIME-tools
  sendmail
  cron
  find


= Setting up the Mikrotik Backup Script =
Nothing new here.  Removes old backup file before creating a new backup file by system name then emails it to your backup address.  The email "to" address should be the user or alias the .procmailrc file is setup for(see below).  The subject should contain a match for the procmail script as well....in this case "Backup".  The "From" address should be whatever directory you want to save the backup file in.  In this case, the backup file would be save as $basedir/RB600/<systemName>.backup  You can have any number of "from" names and if the directory is not found, it will be created.

<pre>
:log info "Starting Backup Script..."
:global backupfile ([/system identity get name] . ".backup")
:if ([/file find name=$backupfile] != "") do={/file rem $backupfile}
:delay 2s
/system backup save name=$backupfile
:log info "Waiting 5s for backup to complete..."
:delay 5s
:log info "Backup being emailed..."
/tool e-mail send to="backups@yourdomain.com" subject=([/system identity get name] . " Backup") from=RB600@yourdomain.com file=$backupfile server=xxx.xxx.xx.x
:log info "Finished Backup Script!"
</pre>

= Setting up the Server Scripts =
procmail, formail, expand, sed, awk, sendmail, and perl are common on most current linux distributions.  If they're not part of your distribution, you'll need to install them and their dependencies with whatever package management your distribution provides.  In addition to these common programs, perl MIME-Tools is required.  For CentOS, setup the rpmforge repository(I use Dag Wieers repo) and once the rpmforge repository is enabled, <pre>yum -y install perl-MIME-tools</pre>

Create a base location for your backups to be saved to.  <b>NOTE:</b> path must be writable by the user that receives the backup mail.  In my case I have a NFS share exported to /backups on this mail server.  You can make a directory where ever you like and adjust the script paths accordingly.  Just make sure that the user setup with the procmail recipe has write permission on the directory you define.

<pre>
mkdir /backups/mt
</pre>

== $HOME/.procmailrc ==
This recipe checks for a matching "To" address and the word "Backup" in the subject, both of which should match your MT backup script.  If a match is made, the "From" address has the domain stripped and the value is passed to the parse script as the device.

This should be saved in the recipient's $HOME directory as .procmailrc  If you're sending to a user alias, it should be in the final user's home directory.  The "To" should match the "To" address in your MT backup scripts. Be sure to make the parsebackup.pl script executable and check the path to the parsebackup.pl script. Change the path here as well if you saved parsebackup.pl in a location other than /usr/local/bin/.
<pre>
SHELL=/bin/sh                   #Use the Bourne shell (check your path!)
#VERBOSE=yes
LOGFILE=${HOME}/procmail.log
LOG="--- Logging for ${LOGNAME} - "

#Get the sender's address (formail reply addy)
FROM_=`formail -rt -xTo: | expand | sed -e 's/^[ ]*//g' -e 's/[ ]*$//g'`

#split sender address and take part prior to "@" as our device type
FDEV_=`echo "${FROM_}" | awk -F@ '{ print $1 }'`

# check for recipient and subject match then pipe message to parse script
:0H
* ^TO.*backups@yourdomain.com
* ^Subject:.*Backup
|/usr/local/bin/parsebackup.pl --device ${FDEV_}
</pre>


== parsebackup.pl ==
This script takes input from procmail and searches for any defined MIME type attachment(s).  If found, the attachment is saved to the location defined in $basedir.  $basedir could also be a NFS share from a central storage device if desired.  

Save to location such as /usr/local/bin and make it executable.  Be sure to read and change the applicable settings in the script.  Notably the $to address and the $basedir path.  The $to address should be a completely different user than what you setup the procmail recipe for, otherwise you may start a mail loop.  If the $to address is set to an empty string '', then no email notification will be sent.  Make sure the user which you setup the procmail recipe for has write permissions to the $basedir path!  
<pre>
#!/usr/bin/perl
# parsebackup.pl --- Rob Asher
# Searches for defined mime attachment type(s) in email piped in via procmail.
# Saves mikrotik backup attachment to directory(could be NFS share)

use MIME::Parser;
use Getopt::Long;
use File::Path;
use strict;
use warnings;

my $device;
my $result;

## get arguments passed from procmail script for device path to save file under
$result = GetOptions('device=s' => \$device);


###-----------------------------------------------------------
## address to send notices to (MUST be different than original recipient otherwise
##  you may get into a vicious mail loop).  Set to '' if no notifications are desired.

my $to = 'you@yourdomain.com';

## directory to store all parsed MIME types under and location of sendmail
## base directory must initially be created and writable by user! (could be NFS share)

my $basedir = "/backups/mt";
my $sendmail = "/usr/sbin/sendmail -t";

## define attachment types to keep -- notice all lowercase
## add as many types as you like.

my %type_ok = (
  'application/octet-stream' => 1
);

###-----------------------------------------------------------


##############################################################
## make from base and check the output directory
$basedir =~ s/\/$//;
my $outputdir = $basedir . "/" . $device;
if (! -e $outputdir || ! -d $outputdir ) {
       mkpath($outputdir) or die "Unable to create $outputdir: $!\n";
} else {
       stat($outputdir);
       if (! -w _) {
               die "$outputdir is not writable by this user";
       }
}


## parser object
my $parser = new MIME::Parser;
$parser->output_dir($outputdir);

## get parts from mail piped in
my $entity = $parser->read(\*STDIN);
my $num_parts = $entity->parts;

## get some message info to send on for notification
my $from = $entity->head()->get('From');
my $subj = $entity->head()->get('Subject');

## check parts for defined type(s)
foreach my $i (0 .. $num_parts-1) {
    my $part = $entity->parts($i);
    my $type = lc $part->mime_type;
    my $bh = $part->bodyhandle;
    if (! exists $type_ok{$type}) {
       my $status = system("rm '$bh->{MB_Path}'");
       die $! unless $status == 0;
    }
}


## send notification that we processed a message and to where.
if ($to) {
   chomp($from);
   chomp($subj);
   open(SENDMAIL, "|$sendmail") or die "Cannot open $sendmail: $!";
   print SENDMAIL "From: $from\n";
   print SENDMAIL "To: $to\n";
   print SENDMAIL "Subject: $subj Received\n";
   print SENDMAIL "\n$subj file received and processed to $outputdir.\n";
   close(SENDMAIL);
}
</pre>



== Cleanup ==
If you want to automate the time backups are saved, you can create a "cleanup" script of this and execute it via cron on a daily basis.  It will look through the directory for files modified more than 15 days ago and remove them.

Be sure your path is correct!  Adjust the mtime to whatever value suits.
<pre>
find /backups/mt -mtime +15 -type f -exec rm -f {} \;
</pre>


= Final Thoughts =
Hopefully, this isn't an overly complex explanation of a fairly simple solution.  Basically, there are three scripts involved.  The backup script on the MT device sends an email to your backup recipient.  The .procmailrc file for the backup recipient looks at the "To" and "Subject" for a match and if so, splits the "From" address and uses it as an argument to the parsebackup.pl script for the device type.  The parsebackup.pl script takes the email piped in from the procmailrc recipe and then saves the attached backup file to the base location defined in the script plus the device type name passed through from the MT backup script.

[[Category:Scripting]]