Graylog2

From Hack Sphere Labs Wiki
Jump to: navigation, search

Not Configured to Autostart

sudo /etc/init.d/elasticsearch start
~/graylog2-server-0.11.0/bin
./graylog2ctl start

Clean Out Graylog2 DB

Get names of DBs: 

curl http://localhost:9200/_aliases
curl -XDELETE 'http://localhost:9200/graylog2_0/'

Does Not Work

#I could not find the es data dir
cd /opt/elasticsearch/data/graylog2
sudo rm -rf *
mongo
use graylog2
db.message_counts.remove()
db.hosts.remove()
exit
sudo /etc/init.d/elasticsearch restart

Refresh interface (will take a while)

Wheezy

I think the xm tools needs updated for wheezy but look: http://www.debian-administration.org/article/Installing_new_Debian_systems_with_debootstrap 

rm -f /usr/sbin/policy-rc.d
  • Install rcconf to manage services
rcconf
  • Install curl
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
  • Install locate
aptitude install locate

graylog2

ToDo

  • Auth on?
  • elasticsearch

Install mongodb

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
echo 'deb http://downloads-distro.mongodb.org/repo/debian-sysvinit dist 10gen' | sudo tee /etc/apt/sources.list.d/10gen.list
apt-get install mongodb-10gen
  • dpkg issues after installing mongodb
    • Right After Install:
invoke-rc.d: policy-rc.d denied execution of start.
    • Trying to start
[....] Starting database: mongodbWarning: Fake start-stop-daemon called, doing nothing
failed!

apt-get install dpkg --reinstall


Configure mongodb

mongo
  • Paste this after changing:
use admin
 db.addUser('admin', 'admin_password')
 db.auth('admin', 'admin_password')
use graylog2
 db.addUser('grayloguser', 'grey_password')
 db.auth('grayloguser', 'grey_password')
exit

Install Elastic Search 0.20.4

apt-get install ca-certificates
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.4.deb
aptitude install openjdk-6-jre
dpkg -i elasticsearch-0.20.4.deb

Install graylog2

wget http://download.graylog2.org/graylog2-server/graylog2-server-0.11.0.tar.gz
wget http://download.graylog2.org/graylog2-web-interface/graylog2-web-interface-0.11.0.tar.gz
tar xvfz graylog2-server-0.11.0.tar.gz
cd graylog2-server-0.11.0
cp graylog2.conf.example /etc/graylog2.conf
cp elasticsearch.yml.example /etc/graylog2-elasticsearch.yml

Install graylog2-web-interface

apt-get install ruby
gem install bundler --no-rdoc --no-ri
aptitude install ruby-dev build-essential libcurl4-openssl-dev libssl-dev zlib1g-dev
mv graylog2-web-interface-0.11.0 /opt/
cd /opt/graylog2-web-interface-0.11.0
bundle install --without=development
nano -w config/initializers/secret_token.rb
  • Replace CHANGE ME with an at least 30 characters long and all random string:
gem install passenger --no-rdoc --no-ri
apt-get install apache2-mpm-prefork apache2-prefork-dev libapr1-dev libaprutil1-dev
passenger-install-apache2-module
  • Add these lines to your /etc/apache2/apache2.conf before the Include sites-enabled/ line to enable the module:
LoadModule passenger_module /var/lib/gems/1.9.1/gems/passenger-4.0.2/libout/apache2/mod_passenger.so
PassengerRoot /var/lib/gems/1.9.1/gems/passenger-4.0.2
PassengerDefaultRuby /usr/bin/ruby1.9.1
  • The last step is to configure a virtual host (for example /etc/apache2/sites-available/default):
    • I put my site in /var/www

cat /etc/apache2/sites-available/default 

<VirtualHost *:80>
     ServerAdmin you@example.com
     DocumentRoot /opt/graylog2-web-interface/public 

     RailsEnv 'production'

    <Directory /opt/graylog2-web-interface/public>
        Allow from all
        Options -MultiViews
    </Directory>

    ErrorLog /var/log/apache2/error.log
    LogLevel warn
    CustomLog /var/log/apache2/access.log combined
</VirtualHost>

Notes

2013-05-20 14:30:53,955 DEBUG: org.elasticsearch.common.compress - failed to load xerial snappy-java
2013-05-20 14:31:06,650 TRACE: org.elasticsearch.monitor.sigar - failed to load sigar java.lang.ClassNotFoundException: org.hyperic.sigar.Sigar
  • Error: bundle install --without=develop
The source :rubygems is deprecated because HTTP requests are insecure.
Please change your source to 'https://rubygems.org' if possible, or 'http://rubygems.org' if not
  • Apache2 ruby module
LoadModule passenger_module /var/lib/gems/1.9.1/gems/passenger-4.0.2/libout/apache2/mod_passenger.so
PassengerRoot /var/lib/gems/1.9.1/gems/passenger-4.0.2
PassengerDefaultRuby /usr/bin/ruby1.9.1

you should have the same Clustername in you elasticsearch configuration and
in the elasticsearch_graylog configuration.
Depends on you System where to find them, bit the Variable "cluster.name: "
should have the same value (graylog2 for example)

# MongoDB Configuration
mongodb_useauth = true
mongodb_user = grayloguser
mongodb_password = 123
mongodb_host = 127.0.0.1
#mongodb_replica_set = localhost:27017,localhost:27018,localhost:27019
mongodb_database = graylog2
mongodb_port = 27017

DNS Utils

I figured I needed to install dns utils to go by resolved hosts (syslog clients do not pass hostnames) so 

nano /etc/graylog2.conf
force_syslog_rdns = true
apt-get install dnsutils
./graylog2ctl stop
./graylog2ctl start

Sending Data to Graylog2 Through syslog or rsyslog

Syslog Server udp and rsyslog and debian: 

*.*	@ip.ip.ip.ip
#default port is 514 but you can specify with ip.ip.ip.ip:port

In debian also: service rsyslog restart

FreeBSD 

/etc/syslog.conf
*.* @ip.ip.ip.ip

OpenIndiana 

sudo nano /etc/syslog.conf
*.debug @ip.ip.ip.ip
sudo svcadm restart svc:/system/system-log:default
sudo svcadm restart svc:/system/system-log:default

Archlinux 

sudo nano /etc/syslog-ng/syslog-ng.conf

# Log everything to graylog2

destination graylog2 {
        udp("ip.ip.ip.ip" port(514));
        };
log { 
        source(src);
        destination(graylog2);
        };

Plugins

Controlling Heap Sizes

nano /etc/default/elasticsearch
# Heap Size (defaults to 256m min, 1g max)
ES_HEAP_SIZE=512m

You can set this but it is not recommended: 

#ES_MIN_MEM=256m
#ES_MAX_MEM=1g

max_file_descriptors
curl -XGET 'http://localhost:9200/_nodes?os=true&process=true&pretty=true'

graylog2ctl 

HEAP_SIZE='-Xms8192M -Xmx8192M'

 

start() {

    echo "Starting graylog2-server ..."

    cd "$GRAYLOG2CTL_DIR/.."

    $NOHUP java ${HEAP_SIZE} -jar ${GRAYLOG2_SERVER_JAR} -f ${GRAYLOG2_CONF} -p ${GRAYLOG2_PID} >> ${LOG_FILE} &

}

Logging Proto

Syslog over udp drops packets

  • amqp

etc

Retrieved from "https://wiki.hackspherelabs.com/index.php?title=Graylog2&oldid=1891"