Difference between revisions of "Nagios"

From Hack Sphere Labs Wiki
Jump to: navigation, search
(→‎add host to nagios)
(→‎Define Check Interval)
 
(26 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
=Overview=
 +
Wheezy is about the same just different aptitude lines
 +
 +
*http://www.the-tech-tutorial.com/wp-content/uploads/2011/07/nagios-config.png
 +
*http://nagios.sourceforge.net/docs/3_0/security.html
 +
 +
=Migration=
 +
scp -rp works nice
 +
 +
/etc/nagios3/conf.d/*.cfg
 +
/usr/lib/nagios/plugins/
 +
*The above dir has alot of binaries and if you replace them you have to reinstall with aptitude reinstall nagios-plugin-package
 +
/etc/nagios-plugins/config
 +
Make sure to check all permissions and install packages needed by any custom plugins
 +
 +
 +
=Email Alerts=
 +
*gmail smtp
 +
**https://wiki.debian.org/GmailAndExim4
 +
 +
==/etc/email-addresses==
 +
<pre>
 +
useraccount1: destemailaddy@gmail.com
 +
useraccount1@localhost: destemailaddy@gmail.com
 +
useraccount1@HOSTNAME: destemailaddy@gmail.com
 +
useraccount1@HOSTNAME.localdomain: destemailaddy@gmail.com
 +
</pre>
 +
*add the same info above for any other local account you are going to want to foward/use
 +
 +
==/etc/exim4/passwd.client==
 +
*.google.com:destemailaddy@gmail.com:ThEuSeRpAssW0rd
 +
 +
==.forward file in user dir==
 +
destemailaddy@gmail.com
 +
 +
==test sending a message==
 +
mail -s Test root@HOSTNAME < /dev/null
 +
 +
*http://www.linuxquestions.org/questions/debian-26/sending-root-mail-to-an-external-mail-account-684733/
 +
 +
 
=Debian=
 
=Debian=
  
Line 15: Line 56:
 
==Notes==
 
==Notes==
 
*http://www.howtoforge.com/installing-nagios-on-debian-lenny-and-monitoring-a-debian-lenny-server
 
*http://www.howtoforge.com/installing-nagios-on-debian-lenny-and-monitoring-a-debian-lenny-server
−
*http://www.the-tech-tutorial.com/wp-content/uploads/2011/07/nagios-config.png
 
 
*http://packages.debian.org/squeeze/nagiosgrapher
 
*http://packages.debian.org/squeeze/nagiosgrapher
  
Line 168: Line 208:
 
*After
 
*After
 
<pre>
 
<pre>
−
# check_nt_disk to use ! now
 
 
define command {
 
define command {
−
          command_name    check_nt_disk
+
        command_name    check_nt_disk
−
          command_line    /usr/lib/nagios/plugins/check_nt -H '$HOSTADDRESS$' -s $USER4$ -p '$ARG1$' -v 'USEDDISKSPACE' -l '$ARG2$' -w '$A$
+
        command_line    /usr/lib/nagios/plugins/check_nt -H '$HOSTADDRESS$' -s $USER4$ -p '$ARG1$' -v 'USEDDISKSPACE' -l '$ARG2$' -w '$ARG3$' -c '$ARG4$'
 
}
 
}
 +
 +
</pre>
 
Now you send to the check_nt_disk
 
Now you send to the check_nt_disk
 
  check_command          check_nt_disk!12489!c!80!90
 
  check_command          check_nt_disk!12489!c!80!90
−
</pre>
+
 
  
  
Line 181: Line 222:
 
==Notes==
 
==Notes==
 
*http://nagios.sourceforge.net/docs/3_0/monitoring-windows.html
 
*http://nagios.sourceforge.net/docs/3_0/monitoring-windows.html
 +
*http://www.thegeekstuff.com/2008/07/how-to-monitor-remote-windows-machine-using-nagios-on-linux/
  
 
=debian external command fix/setup=
 
=debian external command fix/setup=
Line 198: Line 240:
 
  dpkg-statoverride --update --add nagios nagios 751 /var/lib/nagios3
 
  dpkg-statoverride --update --add nagios nagios 751 /var/lib/nagios3
 
  /etc/init.d/nagios3 start
 
  /etc/init.d/nagios3 start
 +
 +
=arping=
 +
*In debian are going to need to build the Net::apring cpan perl module (http://search.cpan.org/~radek/Net-Arping/Arping.pm).
 +
*In debian you have to use this (http://www.debian-administration.org/articles/78) guide instead of the cpan builder.  I read somewhere that using cpan can mess things up.
 +
*I just would like to state that CPAN and dh-make-perl are fucking useless.  Debian has wasted so much of my time by now.  I hate the fucking attitude and purposful shitty documentation with crap examples that assume that you will spend the next 4 hours reading man pages that are incomplete too.
 +
*A few quick examples of a command is alot better then 1000 useless words.
 +
*Cpans documentation sucks.  I could just build Net::arping through cpan or dh-make-perl but I fucking cant.  I cannot because I get a old version that has not worked in years.  YEARS.  They keep that one in the database instead of http://search.cpan.org/~radek/Net-Arping/Arping.pm .  How do you have cpan pull the new one?  Who fucking knows.  I tried downloading the tar.gz and was having problems with that too.  dh-make-perl is not flexable at all and will give you a few errors that do not even fucking make sense.
 +
 +
I guess everyone should read how to officialy build debian packages with the right lower case letters and upstream source TO INSTALL SOME FUCKING PERL CODE ON MY OWN FUCKING SYSTEM.  This web page: http://www.debian-administration.org/articles/78 - Basically fucking useless.
 +
 +
 +
 +
debuild -us -uc -b
 +
 +
This is what I did and it is probly overkill because I had to try so many things to get this to work:
 +
 +
aptitude install libnet1-dev libpcap-dev
 +
Would not you think that dh-make-perl would have dh-make as a requirement?  Fucking nope.
 +
aptitude install debuild dh-make-perl dh-make
 +
aptitude install libnet-arp-perl
 +
apt-file update
 +
mkdir temp
 +
cd temp
 +
wget http://search.cpan.org/CPAN/authors/id/R/RA/RADEK/Net-Arping-0.03.tar.gz
 +
mv Net-Arping-0.03.tar.gz libnet-arping-perl_0.03.orig.tar
 +
tar zxvf libnet-arping-perl_0.03.orig.tar
 +
cd Net-Arping-0.03/
 +
dh-make-perl
 +
You will get errors here but without diving into the source of the script...wtf do they mean.  Still keep going
 +
debuild -us -uc -b
 +
 +
I also in the many things I tried:
 +
*configured cpan
 +
cpan
 +
o conf init
 +
and just for the fuck of it (prolly not)
 +
install Bundle::CPAN
 +
 +
Do not forget to:
 +
dpkg --install libnet-arping-perl_0.03-1_amd64.deb
 +
 +
*Here is the plugin from git contrib nagios:  https://github.com/Elbandi/nagios-plugins/blob/master/contrib/check_arping.pl
 +
 +
cat arping.cfg
 +
<pre>
 +
# 'check-host-alive-arping' command definition
 +
define command{
 +
        command_name    check-host-alive-arping
 +
        command_line    /usr/lib/nagios/plugins/check_arp_ping.pl -I eth0 -H '$HOSTADDRESS$'
 +
        }
 +
</pre>
 +
 +
cat generic-host-arping.cfg
 +
<pre>
 +
 +
# Generic host definition template - This is NOT a real host, just a template!
 +
 +
define host{
 +
        name                            generic-host-arping    ; The name of this host template
 +
        notifications_enabled          1      ; Host notifications are enabled
 +
        event_handler_enabled          1      ; Host event handler is enabled
 +
        flap_detection_enabled          1      ; Flap detection is enabled
 +
        failure_prediction_enabled      1      ; Failure prediction is enabled
 +
        process_perf_data              1      ; Process performance data
 +
        retain_status_information      1      ; Retain status information across program restarts
 +
        retain_nonstatus_information    1      ; Retain non-status information across program restarts
 +
check_command                  check-host-alive-arping
 +
max_check_attempts              10
 +
notification_interval          0
 +
notification_period            24x7
 +
notification_options            d,u,r
 +
contact_groups                  admins
 +
        register                        0      ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL HOST, JUST A TEMPLATE!
 +
        }
 +
</pre>
 +
 +
You also need to add the nagios user to netdev group for this to work.
 +
 +
*Here is a modded generic-host def:
 +
 +
<pre>
 +
# Generic host definition template - This is NOT a real host, just a template!
 +
 +
define host{
 +
        name                            generic-host-arping    ; The name of this host template
 +
        notifications_enabled          1      ; Host notifications are enabled
 +
        event_handler_enabled          1      ; Host event handler is enabled
 +
        flap_detection_enabled          1      ; Flap detection is enabled
 +
        failure_prediction_enabled      1      ; Failure prediction is enabled
 +
        process_perf_data              1      ; Process performance data
 +
        retain_status_information      1      ; Retain status information across program restarts
 +
        retain_nonstatus_information    1      ; Retain non-status information across program restarts
 +
check_command                  check-host-alive-arping
 +
max_check_attempts              10
 +
notification_interval          0
 +
notification_period            24x7
 +
notification_options            d,u,r
 +
contact_groups                  admins
 +
        register                        0      ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL HOST, JUST A TEMPLATE!
 +
        }
 +
</pre>
 +
 +
==arping permissions==
 +
Had to run command as sudo in the end
 +
*http://blog.gnucom.cc/2009/configuring-nagios-to-run-privileged-or-root-commands-with-nrpe/
 +
*Create a sudo command dir
 +
nano /etc/sudoers
 +
add
 +
nagios  ALL=(ALL) NOPASSWD: /bla/sudo_commanddir/
 +
*Nagios will be able to run those files as sudo
 +
*Read Only FS for that dir?
 +
 +
=mib handling=
 +
 +
*Still Debian
 +
*You need to add non-free to your sources.list
 +
*I first added my vendor mib but I do not think it matters to:
 +
/usr/share/mibs/netsnmp
 +
*It needed additional supporting mibs so:
 +
**Add non-free to your repos
 +
aptitude install snmp-mibs-downloader
 +
*I ran this to check for needed mibs
 +
download-mibs
 +
*Grabbed the name of the mib out of the mib file
 +
snmpwalk -c Read-Access -v 1 -m WIPIPE-MIB 192.100.5.4
 +
./check_snmp 10.100.10.4 -C Read-Access -m WIPIPE-MIB -o ipRouteDest.1 -P 1 --verbose
 +
*Start making commands!
 +
 +
=Define Check Interval Per Service=
 +
1min:
 +
normal_check_interval 1
 +
Service def, etc
 +
15min:
 +
normal_check_interval  15
 +
 +
=Enable Debug=
 +
Take out the guesswork.  This will allow you to see the commands executed when you are building commands and more.
 +
nagios.cfg
 +
debug_level=0
 +
tail/cat/less debug_file=/var/log/nagios3/nagios.debug
 +
 +
*run a command as nagios user:
 +
sudo -u nagios command
 +
 +
=Notes=
 +
*http://munin-monitoring.org/
 +
*http://www.opscode.com/chef/
 +
*http://puppetlabs.com/
 +
*http://community.spiceworks.com/how_to/show/3773-creating-custom-nagios-plugins-scripts-in-bash

Latest revision as of 10:21, 30 January 2014

Overview

Wheezy is about the same just different aptitude lines

Migration

scp -rp works nice 

/etc/nagios3/conf.d/*.cfg
/usr/lib/nagios/plugins/
  • The above dir has alot of binaries and if you replace them you have to reinstall with aptitude reinstall nagios-plugin-package
/etc/nagios-plugins/config

Make sure to check all permissions and install packages needed by any custom plugins


Email Alerts

/etc/email-addresses

useraccount1: destemailaddy@gmail.com
useraccount1@localhost: destemailaddy@gmail.com
useraccount1@HOSTNAME: destemailaddy@gmail.com
useraccount1@HOSTNAME.localdomain: destemailaddy@gmail.com
  • add the same info above for any other local account you are going to want to foward/use

/etc/exim4/passwd.client

*.google.com:destemailaddy@gmail.com:ThEuSeRpAssW0rd

.forward file in user dir

destemailaddy@gmail.com

test sending a message

mail -s Test root@HOSTNAME < /dev/null


Debian

aptitude install nagios3 nagios-plugins nagios-nrpe-plugin nagios3-doc

you could also install 

nagios-plugins-openstack nagios-snmp-plugins

you could also install 

nagios-plugins-contrib

from 

deb http://YOURMIRROR.debian.org/debian-backports squeeze-backports main

Set domain if you have one (else leave default) and set login user and password (it should prompt your for both)

At the time I had to apply the fix that is here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=626462

Notes

General Ping Monitoring

You put a host config file in the host config directory and restart nagios. It will then pull the new host in. 

/etc/nagios3/conf.d

Contains templates that you can pull into your host config files. 

/etc/nagios3/conf.d

is also where you put your host config files.

  • Example
define host{
        use                     generic-host            ; Name of host template to use
        host_name               HOSTNAME
        alias                   HOSTNAME
        address                 192.168.52.20
        }

Save that in 

/etc/nagios3/conf.d

and 

/etc/init.d/nagios3 restart

you should see it pop in


I make my own dir: 

/etc/nagios3/hosts.d

and then add 

cfg_dir=/etc/nagios3/hosts.d

to nagios.cfg

Windows Internal (Private Data) Monitoring

  • configure nagios
nano /etc/nagios3/conf.d/winserver_nagios.cfg

put 

# Windows host definition template - This is NOT a real host, just a template!

define host{
        name                    windows-server  ; The name of this host template
        use                     generic-host    ; Inherit default values from the generic-host template
        check_period            24x7            ; By default, Windows servers are monitored round the clock
        check_interval          5               ; Actively check the server every 5 minutes
        retry_interval          1               ; Schedule host check retries at 1 minute intervals
        max_check_attempts      10              ; Check each server 10 times (max)
        check_command           check-host-alive        ; Default command to check if servers are "alive"
        notification_period     24x7            ; Send notification out at any time - day or night
        notification_interval   30              ; Resend notifications every 30 minutes
        notification_options    d,r             ; Only send notifications for specific host states
        contact_groups          admins          ; Notifications get sent to the admins by default
        hostgroups              windows-servers ; Host groups that Windows servers should be a member of
        register                0               ; DONT REGISTER THIS - ITS JUST A TEMPLATE
        }

I pulled the above from /usr/share/doc/nagios3-common/examples/template-object/templates.cfg.gz

  • save
  • restart nagios

Installing via MSI

  1. Complete
  2. Install sample config
  3. Check all users
  4. Add allowed host
  5. check everything except what you are not using
  6. change service to allow desktop interaction

Installing the Windows Agent Manually

  1. Download the latest stable version of the NSClient++ addon from http://sourceforge.net/projects/nscplus
  2. Unzip the NSClient++ files into a new C:\NSClient++ directory
  3. Open a command prompt and change to the C:\NSClient++ directory
  4. Register the NSClient++ system service with the following command:
nscp.exe service --install
  1. Open the services manager and make sure the NSClientpp service is allowed to interact with the desktop (see the 'Log On' tab of the services manager). If it isn't already allowed to interact with the desktop, check the box to allow it to.
  2. create a nsclient.ini file
  3. start service/reboot/run command to start

MORE TO UNDERSTAND

Installs 3 services?:

  • nsclient server (check_nt)
  • enable nrpe server (check_nrpe)
  • enable nsca client (do not enable unless you use NSCA
  • enable wmi checks

Windows Password Fix on Server

The guys who make the package in the debian readme file state that they make definitions flexible so that if the devs decide to change a port then the rule will update where if you define the port in the check_nt statement then it will not. This is why the debian check_nt is different then some others.

It is best to use the user vars in the resource.cfg file for your password instead of setting them in the cfg files that are accessible by cgi.

so 

nano resource.cfg

change one of the user vars to your pass and then edit 

nano /etc/nagios-plugins/config/nt.cfg

with 

 define command{
        command_name    check_nt
        command_line    $USER1$/check_nt -H $HOSTADDRESS$ -s $USER9$ -v $ARG1$
        }

where $USER9$ is your var that contains the password.


notes


below is old

If you specified a password in the NSClient++ configuration file on the Windows machine, you'll need to modify the check_nt command definition to include the password. Open the commands.cfg file for editing. 

vi /usr/local/nagios/etc/objects/commands.cfg

but in debian it is: 

nano /etc/nagios-plugins/config/nt.cfg

Change the definition of the check_nt command to include the "-s <PASSWORD>" argument (where PASSWORD is the password you specified on the Windows machine) like this: 

define command{
	command_name	check_nt
	command_line	$USER1$/check_nt -H $HOSTADDRESS$ -p 12489 -s PASSWORD -v $ARG1$
	}

Instead I changed my windows definitions to use check_nscp instead of check_nt and modified that definition to include the password 

define command {
        command_name    check_nscp
        command_line    /usr/lib/nagios/plugins/check_nt -H '$HOSTADDRESS$' -s $USER4$ -p 12489 -v '$ARG1$'
}

Save the file.

nagios config files

It seems like the nagios syntax has changed a few times and alot of what you find as examples is broken. Host config files use to carry check_nt script switches. Now they want you to write the cfg files with now switches and just !bang the options in with the switches in the service def file.

Here is a nt check disk before and after:

  • Before
define command {
        command_name    check_nscp
        command_line    /usr/lib/nagios/plugins/check_nt -H '$HOSTADDRESS$' -s $USER4$ -p 12489 -v '$ARG1$'
}

You would then send something like; 

check_command		check_nscp!USEDDISKSPACE!-l c -w 80 -c 90
  • After
define command {
        command_name    check_nt_disk
        command_line    /usr/lib/nagios/plugins/check_nt -H '$HOSTADDRESS$' -s $USER4$ -p '$ARG1$' -v 'USEDDISKSPACE' -l '$ARG2$' -w '$ARG3$' -c '$ARG4$'
}

Now you send to the check_nt_disk 

check_command           check_nt_disk!12489!c!80!90



Notes

debian external command fix/setup

easy right? nope 

nano nagios.cfg

change 

check_external_commands=0

to 

check_external_commands=1

Then 

/etc/init.d/nagios3 stop
dpkg-statoverride --update --add nagios www-data 2710 /var/lib/nagios3/rw
dpkg-statoverride --update --add nagios nagios 751 /var/lib/nagios3
/etc/init.d/nagios3 start

arping

  • In debian are going to need to build the Net::apring cpan perl module (http://search.cpan.org/~radek/Net-Arping/Arping.pm).
  • In debian you have to use this (http://www.debian-administration.org/articles/78) guide instead of the cpan builder. I read somewhere that using cpan can mess things up.
  • I just would like to state that CPAN and dh-make-perl are fucking useless. Debian has wasted so much of my time by now. I hate the fucking attitude and purposful shitty documentation with crap examples that assume that you will spend the next 4 hours reading man pages that are incomplete too.
  • A few quick examples of a command is alot better then 1000 useless words.
  • Cpans documentation sucks. I could just build Net::arping through cpan or dh-make-perl but I fucking cant. I cannot because I get a old version that has not worked in years. YEARS. They keep that one in the database instead of http://search.cpan.org/~radek/Net-Arping/Arping.pm . How do you have cpan pull the new one? Who fucking knows. I tried downloading the tar.gz and was having problems with that too. dh-make-perl is not flexable at all and will give you a few errors that do not even fucking make sense.

I guess everyone should read how to officialy build debian packages with the right lower case letters and upstream source TO INSTALL SOME FUCKING PERL CODE ON MY OWN FUCKING SYSTEM. This web page: http://www.debian-administration.org/articles/78 - Basically fucking useless.


debuild -us -uc -b

This is what I did and it is probly overkill because I had to try so many things to get this to work: 

aptitude install libnet1-dev libpcap-dev

Would not you think that dh-make-perl would have dh-make as a requirement? Fucking nope. 

aptitude install debuild dh-make-perl dh-make
aptitude install libnet-arp-perl
apt-file update
mkdir temp
cd temp
wget http://search.cpan.org/CPAN/authors/id/R/RA/RADEK/Net-Arping-0.03.tar.gz
mv Net-Arping-0.03.tar.gz libnet-arping-perl_0.03.orig.tar
tar zxvf libnet-arping-perl_0.03.orig.tar
cd Net-Arping-0.03/
dh-make-perl

You will get errors here but without diving into the source of the script...wtf do they mean. Still keep going 

debuild -us -uc -b

I also in the many things I tried:

  • configured cpan
cpan
o conf init

and just for the fuck of it (prolly not) 

install Bundle::CPAN

Do not forget to: 

dpkg --install libnet-arping-perl_0.03-1_amd64.deb

cat arping.cfg

# 'check-host-alive-arping' command definition
define command{
        command_name    check-host-alive-arping
        command_line    /usr/lib/nagios/plugins/check_arp_ping.pl -I eth0 -H '$HOSTADDRESS$'
        }

cat generic-host-arping.cfg 


# Generic host definition template - This is NOT a real host, just a template!

define host{
        name                            generic-host-arping    ; The name of this host template
        notifications_enabled           1       ; Host notifications are enabled
        event_handler_enabled           1       ; Host event handler is enabled
        flap_detection_enabled          1       ; Flap detection is enabled
        failure_prediction_enabled      1       ; Failure prediction is enabled
        process_perf_data               1       ; Process performance data
        retain_status_information       1       ; Retain status information across program restarts
        retain_nonstatus_information    1       ; Retain non-status information across program restarts
		check_command                   check-host-alive-arping
		max_check_attempts              10
		notification_interval           0
		notification_period             24x7
		notification_options            d,u,r
		contact_groups                  admins
        register                        0       ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL HOST, JUST A TEMPLATE!
        }

You also need to add the nagios user to netdev group for this to work.

  • Here is a modded generic-host def:
# Generic host definition template - This is NOT a real host, just a template!

define host{
        name                            generic-host-arping    ; The name of this host template
        notifications_enabled           1       ; Host notifications are enabled
        event_handler_enabled           1       ; Host event handler is enabled
        flap_detection_enabled          1       ; Flap detection is enabled
        failure_prediction_enabled      1       ; Failure prediction is enabled
        process_perf_data               1       ; Process performance data
        retain_status_information       1       ; Retain status information across program restarts
        retain_nonstatus_information    1       ; Retain non-status information across program restarts
		check_command                   check-host-alive-arping
		max_check_attempts              10
		notification_interval           0
		notification_period             24x7
		notification_options            d,u,r
		contact_groups                  admins
        register                        0       ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL HOST, JUST A TEMPLATE!
        }

arping permissions

Had to run command as sudo in the end 

nano /etc/sudoers

add 

nagios  ALL=(ALL) NOPASSWD: /bla/sudo_commanddir/
  • Nagios will be able to run those files as sudo
  • Read Only FS for that dir?

mib handling

  • Still Debian
  • You need to add non-free to your sources.list
  • I first added my vendor mib but I do not think it matters to:
/usr/share/mibs/netsnmp
  • It needed additional supporting mibs so:
    • Add non-free to your repos
aptitude install snmp-mibs-downloader
  • I ran this to check for needed mibs
download-mibs
  • Grabbed the name of the mib out of the mib file
snmpwalk -c Read-Access -v 1 -m WIPIPE-MIB 192.100.5.4
./check_snmp 10.100.10.4 -C Read-Access -m WIPIPE-MIB -o ipRouteDest.1 -P 1 --verbose
  • Start making commands!

Define Check Interval Per Service

1min: 

normal_check_interval 1

Service def, etc 15min: 

normal_check_interval   15

Enable Debug

Take out the guesswork. This will allow you to see the commands executed when you are building commands and more. 

nagios.cfg
debug_level=0
tail/cat/less debug_file=/var/log/nagios3/nagios.debug
  • run a command as nagios user:

sudo -u nagios command

Notes

Retrieved from "https://wiki.hackspherelabs.com/index.php?title=Nagios&oldid=1654"