Bacula and Bareos

From Hack Sphere Labs Wiki
Revision as of 14:43, 24 March 2014 by Webdawg (talk | contribs) (→‎Volume Management, Job Retention, Backup Job Tree)

Jump to: navigation, search

Lawsuit

Watch out, bacula users hate bareos...possibly with good reason (not the users just the project):

CentOS 6.5 Install

yum install mysql-server mysql-devel

service mysqld start
chkconfig mysqld on
mysqladmin -u root password Y0uR3l173P455w0rd



yum  install bareos-database-mysql
yum  install bareos



/usr/lib/bareos/scripts/create_bareos_database -u root -p
/usr/lib/bareos/scripts/make_bareos_tables -u root -p
/usr/lib/bareos/scripts/grant_bareos_privileges -u root -p



service bareos-dir start
service bareos-sd start
service bareos-fd start

Links

Archlinux GUI Admin

yaourt bareos-bat

Configure On CentOS 6.5

While I was trying to figure this out I was running into issues and one of the steps I took was to disable selinux because some of the posts recommended checking selinux perms. Instead of doing that I disabled. You can try without disabling. I made it most way through my guide but at the moment I do not need selinux (https://www.centos.org/docs/5/html/5.1/Deployment_Guide/sec-sel-enable-disable.html) so: 

sestatus
nano -w /etc/sysconfig/selinux

Set 

SELINUX=disabled

One of the other bigger problems I was having was big problems with vchanger permissions. While I know I had them all set correctly I still ended up doing more then I think I needed to do. I changed the owner of /usr/bin/vchanger 

# ls -al /usr/bin/vchanger
-rwxr-xr-x. 1 bareos bareos 48512 Sep 28  2012 /usr/bin/vchanger

that did not fix the issue I was having though. I was still not able to use the vchanger command with -u bareos and -g bareos. I have put the real fix in the directions below but I wanted to leave this info here in case the next setup still does not work with the directions below. The fix was to make sure that the actual device link was owned by bareos:bareos. That is (obviously using the correct backup drive): 

chown bareos:bareos /dev/sda1

and just for cause 

chown bareos:bareos /dev/sda

One of the last problems I was having was with the config of bacula and how it interfaces with vchanger. You need to specify the correct paths and I was having trouble understanding really how the paths work. In bareos-sd.conf in the Director part I had 

ArchiveDevice = /var/lib/bareos/c0/drive0

that is not what you need, in fact, I do not know what that file really is, I think it is a reference to what the label of the hard drive is or something. Actually I do not see a link and it might be a garbage file from my bad configuration that bareos tried to output. The correct line was: 

ArchiveDevice = /var/lib/bareos/c0/0/drive0

and everything seems to be working great.

One more note as if I do not get to update below you would want to know this. Since this was all new to me I was using 69 magazines just to mess around. I did not understand how magazines where associated with vchanger. They are just files on the drive. You could have 10000 or just one. In the end though, to fill the device you want to create x amount of files and in your bareos-dir.conf Pool directive have an appropriate 

Maximum Volume Bytes = Xgb

or mb or whatever.

If I was sending files to a remote server I may choose smaller files in case of disconnection as I wonder if bareos can resume files.

One of the other things that I have noticed that even when I specified Maximum Volume Bytes = 1gb I was only getting 954MB files but that is because bacula writes in blocks, so it can only write a multiple of the block size defaults to 64512 (quoted from optiz0r_@freenode#baculairc)


Bareos and vchanger

Since I use disks I start with installing vchanger

You should also download the tar.gz from the vchanger sourceforge and look in the docs for vchangerHowto.html as it has the best info. I followed it while making this guide.

The link above has a script but it looks like the one on sourceforge (which is referenced in the link) is newer and compiles. Even then, I decided to pull the rpm from pbone 

wget ftp5.gwdg.de/pub/opensuse/repositories/home:/pstorz/CentOS_CentOS-6/x86_64/vchanger-0.8.6-10.3.x86_64.rpm
yum install vchanger-0.8.6-10.3.x86_64.rpm
  • I am going to start with one drive so first I am going to format it (create a partition first (cfdisk)):
  • Quote:

Assuming the drive was assigned node /dev/sdc, the partition can now be formatted and labeled using: 

mke2fs -t ext4 -O large_file /dev/sdc1

We also have the change the permission of the actual drive so vchanger has the right access to it. 

chown bareos:bareos /dev/sdc1
chown bareos:bareos /dev/sdc


This will create a new ext4 file system on the partition. Now unplug the drive, wait a few seconds, then plug it back in. After re-attaching the drive, the name of the symlink in /dev/disk/by-uuid pointing to the filesystem partition's device node, (/dev/sdc1 in the example above), will reveal the UUID that can be used in a magazine directive in a vchanger configuration file to assign the removable drive to an autochanger. On many systems, the UUID of the filesystem can also be determined with the blkid command. 

ls -al /dev/disk/by-uuid/

The Bacula Storage Daemon does not usually run as root. Since vchanger will be invoked by the Storage Daemon, and so will run as the same user it does, permissions for the magazine's mounted filesystem must be set to allow write access to the user that the Storage Daemon runs as. This can be done by mounting the new partition somewhere, then using chown and chmod to set the appropriate permissions to allow write access for the user that the Storage Daemon runs as. 

mkdir /media/temp
mount /dev/xvdc1 /media/temp
chown bareos:bareos -R /media/temp

Need to get autofs up and running so it mounts the disks and stuff: 

yum install autofs

nano /etc/auto.vchanger
  • add
# /etc/auto.vchanger
*          -fstype=auto,rw,sync       :/dev/disk/by-uuid/&
# eof
  • Then:
nano /etc/auto.master
  • add
/mnt/vchanger      /etc/auto.vchanger        --timeout=30
  • then
service autofs restart

ls -al /mnt/vchanger/UUID so 

ls -al /mnt/vchanger/3c22af77-4881-4a7f-bcfc-517744c964db

Note: After restarting the autofs daemon, whenever a file or directory with a full path beginning with '/mnt/vchanger' is accessed, the autofs filesystem will search the map defined in the /etc/auto.vchanger file for a key matching the path being accessed. In this case, the only key in /etc/auto.vchanger is the wildcard '*', meaning any path name beginning with '/mnt/vchanger' will match. Autofs then automatically mounts the device mapped to the wildcard key at the path being accessed. The /etc/auto.vchanger map file specifies the device to be mounted as /dev/disk/by-uuid/&. The '&' is substituted for the key value. For example, when a program attempts to access /mnt/vchanger/9667f83c-6150-44c7-b0d4-57564f174b35 (or any files or directories below it), the autofs daemon will look at the auto.vchanger map for the key 9667f83c-6150-44c7-b0d4-57564f174b35 and discover that it should mount /dev/disk/by-uuid/9667f83c-6150-44c7-b0d4-57564f174b35 at /mnt/vchanger/9667f83c-6150-44c7-b0d4-57564f174b35 with mount options '-fstype=auto,rw,sync'. After a period of 30 seconds of no activity, autofs will automatically unmount the device.

Always make sure autofs is working properly before continuing with setting up vchanger. If using the above autofs config files, when you plug in a USB drive with filesystem UUID of, say, 9667f83c-6150-44c7-b0d4-57564f174b35, you should be able to list its contents with the commandS

  • This should create some working directories
mkdir /var/lib/bareos/vchanger
mkdir /var/lib/bareos/vchanger/c0
chown bareos:bareos -R /var/lib/bareos/vchanger
  • Next we have to create our config:
nano /etc/bareos/vchanger.conf
  • Put in (hint this is where you would add all the uuid lines for all your drives):
changer_name = "c0"
work_dir = /var/lib/bareos/c0
logfile = /var/lib/bareos/c0.log
log_level = LOG_ERR
virtual_drives = 1
slots_per_magazine = 69
magazine_bays = 1
automount_dir = /mnt/vchanger

magazine = "UUID:3c22af77-4881-4a7f-bcfc-517744c964db"


vchanger -u bareos -g bareos /etc/bareos/vchanger.conf initmag 1
vchanger -u bareos -g bareos /etc/bareos/vchanger.conf list

Should work, if not check all permissions: working dirs, drives, mountpoints

  • More:
vchanger -u bareos -g bareos /etc/bareos/vchanger.conf listmags
  • Output:
1:c0:1:/mnt/vchanger/3c22af77-4881-4a7f-bcfc-517744c964db
  • Test load and unload
vchanger -u bareos -g bareos /etc/bareos/vchanger.conf load 1 /var/lib/bareos/c0/drive0 0
ls -al /var/lib/bareos/c0/0/
vchanger -u bareos -g bareos /etc/bareos/vchanger.conf unload 1 /var/lib/bareos/c0/drive0 0
  • See what mag is installed in the bay:
cat /var/lib/bareos/c0/bay1

Write your config files (http://karellen.blogspot.com/2012/01/creating-new-bacula-job.html , with a little from: http://www.revpol.com/offsitebackups?page=9 ): 

bareos-sd.conf

Add

Note: Only later did I add the -u bareos and -g bareos to the Changer Command directive. You really should not need it

About the Removable Media Directive Below: http://sourceforge.net/p/bacula/mailman/message/30914547/ 

Autochanger {
  Name = esata-changer
  Device = esata-changer-drive-0
  Changer Command = "/usr/bin/vchanger -u bareos -g bareos %c %o %S %a %d"
  Changer Device = "/etc/bareos/vchanger.conf"
}
Device {
  Name = esata-changer-drive-0
  DriveIndex = 0
  Autochanger = yes;
  DeviceType = File
  MediaType = File
  ArchiveDevice = /var/lib/bareos/c0/0/drive0
  RemovableMedia = no;
  RandomAccess = yes;
}

bareos-dir.conf

Add 

Storage {
  Name = c0
# Do not use "localhost" here
  Address = SYSTEM-NAME                # N.B. Use a fully qualified name here
  Password = "YOUR_PASSSWORD!"
  Device = esata-changer
  Media Type = File
  Autochanger = yes;
}

Pool {
  Name = incremental
  Pool Type = backup
  Storage = c0
  Maximum Volume Bytes = 1gb
  Recycle Pool = Scratch
}

Job {
  Name = "drobo_data"
  Type = Backup
  Level = Incremental
  Client = NETT-AUTOBACK-fd
  Schedule = "WeeklyIncremental"
  Storage = c0
  Pool = incremental
  Messages = Standard
  Priority = 10
  FileSet="drobo_data"
}

Schedule {
  Name = "WeeklyIncremental"
  Run = Full 1st sun at 04:00
  Run = Incremental mon-sat at 04:00
}

FileSet {
  Name = "drobo_data"
  Include {
    Options {
      signature = MD5
    }
    File = "/media/drobo/DATA"
  }
}


service bareos-dir restart
service bareos-sd restart

This will make the services pull the config files. You can then bconsole in and try and do things this is an example of what I had to do to get my first test backup up and running once I got the config right: 

bconsole

help is a good command btw 

help
help status

in bconsole 

label barcodes

make the right selections and choose right pool that your backup destination is setup to use 

run

choose the backup to run in my case drobo_data and then I checked on it with: 

status director
status storage=c0

Links

Using the software

Starting Over

Once I got the test setup working I needed to erase the drives, catalog, etc and get it resetup so the backup would fill the entire drive with the right config directives and number of slots. I also need to setup auto recycling: http://www.bacula.org/en/dev-manual/main/main/Automatic_Volume_Recycling.html and append parms, make sure my incremental is setup right.

  • Erase Volumes From Catalog

My volume names are pretty standard. So I created a quick script, you may want to figure out how to output the volume names and go through them with the script. Just set the c values and move forward. 

erase_volume_script

#!/bin/bash

for (( c=2; c<=9; c++ ))
do
   echo "Deleting Volume $c"
   davar=`printf "%04.0f" $c`
   echo "delete volume=c0_0005_$davar yes" | bconsole
done

Changing number of magazines and magazine max size: 

nano /etc/bareos/vchanger.conf
slots_per_magazine = 10

nano /etc/bareos/bareos-dir.conf

Pool { } section 

Maximum Volume Bytes = 100gb

I then go to the disk via uuid and delete everything but the lost and found dir. 

rm files etc

Since I am starting over I need to change which mag number vchanger will start at to 1: 

nano -w /var/lib/bareos/c0/nextmag

I also commented out the Schedule {} directive and the schedule directive in Job {} in bareos-dir.conf because I am still testing and will run the job manually for now. 

vchanger -u bareos -g bareos /etc/bareos/vchanger.conf initmag 1

Should state: 

created magazine 1 in bay 1 [/mnt/vchanger/3c22af77-4881-4a7f-bcfc-517744c964db]

Pop into bconsole and label barcodes with reference to the pool, run the job (if you job is too big for this mag, create the other mags first) 

bconsole
label barcodes
run

Links and Notes

Initialize pool="File" and device="FileStorage" for default job BackupCatalog

This should also work for the default defined job "BackupCatalog" which also uses pool="File" and device="FileStorage" 

bconsole
  • bconsole commands:
label
  • Select File resource
fileVol001 (Anything you want really)

If you are like me, and had the jobs in the queue, the job should start running. else issue a 

run

Restoring Data

  • http://www.bacula.org/en/dev-manual/main/main/Restore_Command.html
    • Quote:
      • Item 5 will automatically select the most recent Full backup and all subsequent incremental and differential backups for a specified Client. These are the Jobs and Files which, if reloaded, will restore your system to the most current saved state. It automatically enters the JobIds found into the directory tree in an optimal way such that only the most recent copy of any particular file found in the set of Jobs will be restored. This is probably the most convenient of all the above options to use if you wish to restore a selected Client to its most recent state.
      • Item 6 allows you to specify a date and time, after which Bacula will automatically select the most recent Full backup and all subsequent incremental and differential backups that started before the specified date and time. 
bconsole

bconsole commands: 

restore
  • I found it really nice to use 2 to find a job with a file I was looking for, I did select 1 which lists the last 20 jobs run but I did not get the name of the job with my query. So it was time prohibitive as I would have to refernce the job id from a different file.
  • I then selected a JobId with #3. Once you select a job you should fall into a prompt to browse and restore files like it was a filesystem.
help
  • Works great..for figuring out how to select files and such.
add file_or_dir_recurse
lsmark is neat

To finish and get prompted to run 

done

so yes/mod/no

Just for my test, I modded the job, and changed the Where, it looks like you may have to also depending on what you are doing. It wanted to restore to: "Where: /tmp/bareos-restores". I created a temp dir that I could control, set the right parms, and restored to it for my test. Worked great.

I am sure if you defined a proper restore job, things would be better.

Worked Great...

Restore Files without bacula

bextract

If you find yourself using bextract, you probably have done something wrong.

Looks like to use bextract, at least the easyway, you create a file with a config against the job. Somewhere it states that it is easier if you have the list of files so you can and boot/run with it or whatnot. You do not have to do it that way but it is the easiest way, they say.

Bacula creates/can use // bextract can use bsr files or bootstrap files to restore files, and db entries.

Quote from above:

If you want to extract or copy a single Job, you can do it by selecting by JobId (code not tested) or better yet, if you know the VolSessionTime and the VolSessionId (printed on Job report and in Catalog), specifying this is by far the best. Using the VolSessionTime and VolSessionId is the way Bacula does restores. A bsr file might look like the following:

Volume="Vol001" VolSessionId=10 VolSessionTime=1080847820

If you know how many files are backed up (on the job report), you can enormously speed up the selection by adding (let's assume there are 157 files):

FileIndex=1-157 Count=157

Finally, if you know the File number where the Job starts, you can also cause bcopy to forward space to the right file without reading every record:

VolFile=20

There is nothing magic or complicated about a BSR file. Parsing it and properly applying it within Bacula *is* magic, but you don't need to worry about that.

If you want to see a *real* bsr file, simply fire up the restore command in the console program, select something, then answer no when it prompts to run the job. Then look at the file restore.bsr in your working directory.

Recycling Volumes

If you have scheduled your job, typically nightly, there will come a time when the tape fills up and Bacula cannot continue. In this case, Bacula will send you a message similar to the following: 

rufus-sd: block.c:337 === Write error errno=28: ERR=No space left
         on device

This indicates that Bacula got a write error because the tape is full. Bacula will then search the Pool specified for your Job looking for an appendable volume. In the best of all cases, you will have properly set your Retention Periods and you will have all your tapes marked to be Recycled, and Bacula will automatically recycle the tapes in your pool requesting and overwriting old Volumes. For more information on recycling, please see the Recycling chapter of this manual. If you find that your Volumes were not properly recycled (usually because of a configuration error), please see the Manually Recycling Volumes section of the Recycling chapter.


Scratch

You can use "RecyclePool = Scratch" in the pool resource: 

RecyclePool = <pool-resource-name> On versions 2.1.4 or greater,
    this directive defines to which pool the Volume will be placed (moved)
    when it is recycled.

The scratch pool is only used for newly labelled volumes. If there are no spare tapes in the pool that the current backup is taking place on, Bacula will look to see if it can take one from the scratch pool. It won't look for volumes which may be recycled but are part of another pool, nor will it move volumes back to scratch when they become eligible for recycling. Though that's an interesting idea.....

If the variable RecyclePoolId is not set for the media execute "update volume" for all the volumes of the pool, the RecyclePoolId field is well set, and recycling works fine.


In fact, my initial configuration was not correct. The RecyclePool=Scratch directive must be specified in the Scratch pool section. If I purge a tape, it is automatically moved to the Scratch pool. But the tape is never moved to the Scratch pool after a retention period. I have to purge it manually for that.

Volume Management, Job Retention, Backup Job Tree

Look at: Restricting the Number of Volumes and Recycling

    • Volume Retention
    • Volume Use Duration
   To limit the use time (i.e. write the Volume for a maximum of five days), use:
   Volume Use Duration = ttt.

Since what I am using this for at is simple, and I have a specific use case that I need to meet, I am manually purging volumes. I just need a full backup, with manual incrementals to transport. 

purge volume=volname

Bacula likes to use append volumes before purge volumes btw

Backups job trees go like this (http://bacula.10910.n7.nabble.com/differential-vs-incremental-td3349.html):

  • full, incremental, incremental, incremental, forever...
  • full, incremental, (wipe the previous jobs), differential, incremental incremental, forever....differential

maybe change VolumeRetention to 5 days and do an "update volumes from resource" (only changing *.conf won't do the trick). 

update volumes from resource

Must write the new settings...

Retrieved from "https://wiki.hackspherelabs.com/index.php?title=Bacula_and_Bareos&oldid=1849"