Difference between revisions of "Bacula and Bareos"

From Hack Sphere Labs Wiki
Jump to: navigation, search
(→‎Starting Over)
(→‎Configure On CentOS 6.5)
Line 54: Line 54:
 
If I was sending files to a remote server I may choose smaller files in case of disconnection as I wonder if bareos can resume files.
 
If I was sending files to a remote server I may choose smaller files in case of disconnection as I wonder if bareos can resume files.
  
−
One of the other things that I have noticed that even when I specified Maximum Volume Bytes = 1gb I was only getting 954MB files.
+
One of the other things that I have noticed that even when I specified Maximum Volume Bytes = 1gb I was only getting 954MB files but that is because bacula writes in blocks, so it can only write a multiple of the block size defaults to 64512 (quoted from optiz0r_@freenode#baculairc)
 +
 
  
 
==Bareos and vchanger==
 
==Bareos and vchanger==

Revision as of 15:37, 21 March 2014

CentOS 6.5 Install

yum install mysql-server mysql-devel

service mysqld start
chkconfig mysqld on
mysqladmin -u root password Y0uR3l173P455w0rd



yum  install bareos-database-mysql
yum  install bareos



/usr/lib/bareos/scripts/create_bareos_database -u root -p
/usr/lib/bareos/scripts/make_bareos_tables -u root -p
/usr/lib/bareos/scripts/grant_bareos_privileges -u root -p



service bareos-dir start
service bareos-sd start
service bareos-fd start

Links

Archlinux GUI Admin

yaourt bareos-bat

Configure On CentOS 6.5

While I was trying to figure this out I was running into issues and one of the steps I took was to disable selinux because some of the posts recommended checking selinux perms. Instead of doing that I disabled. You can try without disabling. I made it most way through my guide but at the moment I do not need selinux (https://www.centos.org/docs/5/html/5.1/Deployment_Guide/sec-sel-enable-disable.html) so: 

sestatus
nano -w /etc/sysconfig/selinux

Set 

SELINUX=disabled

One of the other bigger problems I was having was big problems with vchanger permissions. While I know I had them all set correctly I still ended up doing more then I think I needed to do. I changed the owner of /usr/bin/vchanger 

# ls -al /usr/bin/vchanger
-rwxr-xr-x. 1 bareos bareos 48512 Sep 28  2012 /usr/bin/vchanger

that did not fix the issue I was having though. I was still not able to use the vchanger command with -u bareos and -g bareos. I have put the real fix in the directions below but I wanted to leave this info here in case the next setup still does not work with the directions below. The fix was to make sure that the actual device link was owned by bareos:bareos. That is (obviously using the correct backup drive): 

chown bareos:bareos /dev/sda1

and just for cause 

chown bareos:bareos /dev/sda

One of the last problems I was having was with the config of bacula and how it interfaces with vchanger. You need to specify the correct paths and I was having trouble understanding really how the paths work. In bareos-sd.conf in the Director part I had 

ArchiveDevice = /var/lib/bareos/c0/drive0

that is not what you need, in fact, I do not know what that file really is, I think it is a reference to what the label of the hard drive is or something. Actually I do not see a link and it might be a garbage file from my bad configuration that bareos tried to output. The correct line was: 

ArchiveDevice = /var/lib/bareos/c0/0/drive0

and everything seems to be working great.

One more note as if I do not get to update below you would want to know this. Since this was all new to me I was using 69 magazines just to mess around. I did not understand how magazines where associated with vchanger. They are just files on the drive. You could have 10000 or just one. In the end though, to fill the device you want to create x amount of files and in your bareos-dir.conf Pool directive have an appropriate 

Maximum Volume Bytes = Xgb

or mb or whatever.

If I was sending files to a remote server I may choose smaller files in case of disconnection as I wonder if bareos can resume files.

One of the other things that I have noticed that even when I specified Maximum Volume Bytes = 1gb I was only getting 954MB files but that is because bacula writes in blocks, so it can only write a multiple of the block size defaults to 64512 (quoted from optiz0r_@freenode#baculairc)


Bareos and vchanger

Since I use disks I start with installing vchanger

You should also download the tar.gz from the vchanger sourceforge and look in the docs for vchangerHowto.html as it has the best info. I followed it while making this guide.

The link above has a script but it looks like the one on sourceforge (which is referenced in the link) is newer and compiles. Even then, I decided to pull the rpm from pbone 

wget ftp5.gwdg.de/pub/opensuse/repositories/home:/pstorz/CentOS_CentOS-6/x86_64/vchanger-0.8.6-10.3.x86_64.rpm
yum install vchanger-0.8.6-10.3.x86_64.rpm
  • I am going to start with one drive so first I am going to format it (create a partition first (cfdisk)):
  • Quote:

Assuming the drive was assigned node /dev/sdc, the partition can now be formatted and labeled using: 

mke2fs -t ext4 -O large_file /dev/sdc1

We also have the change the permission of the actual drive so vchanger has the right access to it. 

chown bareos:bareos /dev/sdc1
chown bareos:bareos /dev/sdc


This will create a new ext4 file system on the partition. Now unplug the drive, wait a few seconds, then plug it back in. After re-attaching the drive, the name of the symlink in /dev/disk/by-uuid pointing to the filesystem partition's device node, (/dev/sdc1 in the example above), will reveal the UUID that can be used in a magazine directive in a vchanger configuration file to assign the removable drive to an autochanger. On many systems, the UUID of the filesystem can also be determined with the blkid command. 

ls -al /dev/disk/by-uuid/

The Bacula Storage Daemon does not usually run as root. Since vchanger will be invoked by the Storage Daemon, and so will run as the same user it does, permissions for the magazine's mounted filesystem must be set to allow write access to the user that the Storage Daemon runs as. This can be done by mounting the new partition somewhere, then using chown and chmod to set the appropriate permissions to allow write access for the user that the Storage Daemon runs as. 

mkdir /media/temp
mount /dev/xvdc1 /media/temp
chown bareos:bareos -R /media/temp

Need to get autofs up and running so it mounts the disks and stuff: 

yum install autofs

nano /etc/auto.vchanger
  • add
# /etc/auto.vchanger
*          -fstype=auto,rw,sync       :/dev/disk/by-uuid/&
# eof
  • Then:
nano /etc/auto.master
  • add
/mnt/vchanger      /etc/auto.vchanger        --timeout=30
  • then
service autofs restart

ls -al /mnt/vchanger/UUID so 

ls -al /mnt/vchanger/3c22af77-4881-4a7f-bcfc-517744c964db

Note: After restarting the autofs daemon, whenever a file or directory with a full path beginning with '/mnt/vchanger' is accessed, the autofs filesystem will search the map defined in the /etc/auto.vchanger file for a key matching the path being accessed. In this case, the only key in /etc/auto.vchanger is the wildcard '*', meaning any path name beginning with '/mnt/vchanger' will match. Autofs then automatically mounts the device mapped to the wildcard key at the path being accessed. The /etc/auto.vchanger map file specifies the device to be mounted as /dev/disk/by-uuid/&. The '&' is substituted for the key value. For example, when a program attempts to access /mnt/vchanger/9667f83c-6150-44c7-b0d4-57564f174b35 (or any files or directories below it), the autofs daemon will look at the auto.vchanger map for the key 9667f83c-6150-44c7-b0d4-57564f174b35 and discover that it should mount /dev/disk/by-uuid/9667f83c-6150-44c7-b0d4-57564f174b35 at /mnt/vchanger/9667f83c-6150-44c7-b0d4-57564f174b35 with mount options '-fstype=auto,rw,sync'. After a period of 30 seconds of no activity, autofs will automatically unmount the device.

Always make sure autofs is working properly before continuing with setting up vchanger. If using the above autofs config files, when you plug in a USB drive with filesystem UUID of, say, 9667f83c-6150-44c7-b0d4-57564f174b35, you should be able to list its contents with the commandS

  • This should create some working directories
mkdir /var/lib/bareos/vchanger
mkdir /var/lib/bareos/vchanger/c0
chown bareos:bareos -R /var/lib/bareos/vchanger
  • Next we have to create our config:
nano /etc/bareos/vchanger.conf
  • Put in (hint this is where you would add all the uuid lines for all your drives):
changer_name = "c0"
work_dir = /var/lib/bareos/c0
logfile = /var/lib/bareos/c0.log
log_level = LOG_ERR
virtual_drives = 1
slots_per_magazine = 69
magazine_bays = 1
automount_dir = /mnt/vchanger

magazine = "UUID:3c22af77-4881-4a7f-bcfc-517744c964db"


vchanger -u bareos -g bareos /etc/bareos/vchanger.conf initmag 1
vchanger -u bareos -g bareos /etc/bareos/vchanger.conf list

Should work, if not check all permissions: working dirs, drives, mountpoints

  • More:
vchanger -u bareos -g bareos /etc/bareos/vchanger.conf listmags
  • Output:
1:c0:1:/mnt/vchanger/3c22af77-4881-4a7f-bcfc-517744c964db
  • Test load and unload
vchanger -u bareos -g bareos /etc/bareos/vchanger.conf load 1 /var/lib/bareos/c0/drive0 0
ls -al /var/lib/bareos/c0/0/
vchanger -u bareos -g bareos /etc/bareos/vchanger.conf unload 1 /var/lib/bareos/c0/drive0 0
  • See what mag is installed in the bay:
cat /var/lib/bareos/c0/bay1

Write your config files (http://karellen.blogspot.com/2012/01/creating-new-bacula-job.html , with a little from: http://www.revpol.com/offsitebackups?page=9 ): 

bareos-sd.conf

Add

Note: Only later did I add the -u bareos and -g bareos to the Changer Command directive. You really should not need it

About the Removable Media Directive Below: http://sourceforge.net/p/bacula/mailman/message/30914547/ 

Autochanger {
  Name = esata-changer
  Device = esata-changer-drive-0
  Changer Command = "/usr/bin/vchanger -u bareos -g bareos %c %o %S %a %d"
  Changer Device = "/etc/bareos/vchanger.conf"
}
Device {
  Name = esata-changer-drive-0
  DriveIndex = 0
  Autochanger = yes;
  DeviceType = File
  MediaType = File
  ArchiveDevice = /var/lib/bareos/c0/0/drive0
  RemovableMedia = no;
  RandomAccess = yes;
}

bareos-dir.conf

Add 

Storage {
  Name = c0
# Do not use "localhost" here
  Address = SYSTEM-NAME                # N.B. Use a fully qualified name here
  Password = "YOUR_PASSSWORD!"
  Device = esata-changer
  Media Type = File
  Autochanger = yes;
}

Pool {
  Name = incremental
  Pool Type = backup
  Storage = c0
  Maximum Volume Bytes = 1gb
  Recycle Pool = Scratch
}

Job {
  Name = "drobo_data"
  Type = Backup
  Level = Incremental
  Client = NETT-AUTOBACK-fd
  Schedule = "WeeklyIncremental"
  Storage = c0
  Pool = incremental
  Messages = Standard
  Priority = 10
  FileSet="drobo_data"
}

Schedule {
  Name = "WeeklyIncremental"
  Run = Full 1st sun at 04:00
  Run = Incremental mon-sat at 04:00
}

FileSet {
  Name = "drobo_data"
  Include {
    Options {
      signature = MD5
    }
    File = "/media/drobo/DATA"
  }
}


service bareos-dir restart
service bareos-sd restart

This will make the services pull the config files. You can then bconsole in and try and do things this is an example of what I had to do to get my first test backup up and running once I got the config right: 

bconsole

help is a good command btw 

help
help status

in bconsole 

label barcodes

make the right selections and choose right pool that your backup destination is setup to use 

run

choose the backup to run in my case drobo_data and then I checked on it with: 

status director
status storage=c0

Links

Using the software

Starting Over

Once I got the test setup working I needed to erase the drives, catalog, etc and get it resetup so the backup would fill the entire drive with the right config directives and number of slots. I also need to setup auto recycling: http://www.bacula.org/en/dev-manual/main/main/Automatic_Volume_Recycling.html and append parms, make sure my incremental is setup right.

  • Erase Volumes From Catalog

My volume names are pretty standard. So I created a quick script, you may want to figure out how to output the volume names and go through them with the script. Just set the c values and move forward. 

erase_volume_script

#!/bin/bash

for (( c=2; c<=9; c++ ))
do
   echo "Deleting Volume $c"
   davar=`printf "%04.0f" $c`
   echo "delete volume=c0_0005_$davar yes" | bconsole
done

Changing number of magazines and magazine max size: 

nano /etc/bareos/vchanger.conf
slots_per_magazine = 10

nano /etc/bareos/bareos-dir.conf

Pool { } section 

Maximum Volume Bytes = 100gb

I then go to the disk via uuid and delete everything but the lost and found dir. 

rm files etc

Since I am starting over I need to change which mag number vchanger will start at to 1: 

nano -w /var/lib/bareos/c0/nextmag

I also commented out the Schedule {} directive and the schedule directive in Job {} in bareos-dir.conf because I am still testing and will run the job manually for now. 

vchanger -u bareos -g bareos /etc/bareos/vchanger.conf initmag 1

Should state: 

created magazine 1 in bay 1 [/mnt/vchanger/3c22af77-4881-4a7f-bcfc-517744c964db]

Pop into bconsole and label barcodes with reference to the pool, run the job (if you job is too big for this mag, create the other mags first) 

bconsole
label barcodes

Links and Notes

Restore Files

Retrieved from "https://wiki.hackspherelabs.com/index.php?title=Bacula_and_Bareos&oldid=1819"