Well cover What to stream Choices for streaming Setting up streaming from a VM Chef Recipes
What to stream You can stream the following from cloud Traces (Logs) Metrics Monitoring Status
Scenario App/service runs in Cloud We need the log files of your App Web Server logs, container logs, app logs We need the log files of your Service Service logs
SaaS Vendors You can avail this SaaS service from (loggly, papertrail..)
Name Language Collector Shipper Footprint Ease of setting up Logstash JRuby (JVM) Yes No High > Easy Fluentd Ruby Yes No High > Easy Beaver Python No Yes Low Easy Logstash-Forwarder Go No Yes Low Difficult (uses SSL) Woodchuck Ruby No Yes High > Easy RSYSLOG C Yes Yes Low Difficult Heka Go Yes Yes Low Easy
Our requirements 2 sets of logs to collect All the trace when the VM is spinned off. All the trace inside the VM of the application or service Publish it to an in-memory store(queue) which can be accessed by a key
We tried We use Logstash Beaver Logstash-forwarder Woodchuck Heka RSYSLOG Heka Beaver RSYSLOG
How does it work ? Heka resides inside our Megam Engine (megamd). Its job is to collect the trace information when a VM is run. 1. Reads the dynamically created VM execution log files 2. Format the log contents in json for every VM execution. 3. Publish the log contents to a queue Beaver resides in each of the VMs. It does the following steps, 1. Reads the log files inside the VM 2. Format log contents in json. 3. Publish the log contents to a queue.
Logstash Centralized logging frameworks that can transfer logs from multiple hosts to a central location. JRuby hence its needs a JVM JVM sucks memory Logstash is Ideal as a centralized collector and not a shipper.
Logstash Shipper Scenario Let us ship logs from a VM : /usr/share/megam/megamd/logs/*/* to Redis or AMQP. eg: ../megamd/logs/pogo.domain.com/howdy.log Queue named pogo.domain.com in AMQP. ../megamd/logs/doe.domain.com/howdy.log Queue named doe.domain.com in AMQP.
Logstash : Start the agent java -jar /opt/logstash/agent/lib/logstash-1.4.2. jar agent -f /opt/logstash/agent/etc/shipper.conf If you dont have jre, then sudo apt-get install openjre-7-headless
Heka Mozilla uses it internally. Written in Golang - native. Ideal as a centralized collector and a shipper. We picked Heka. Our modified version https://github.com/megamsys/heka
Installation Download deb from https://github.com/mozilla-services/heka/releases (or) build from source. git clone https://github.com/megamsys/heka.git cd heka source build.sh cd build make deb dpkg -i heka_0.6.0_amd64.deb
Run heka sudo hekad -config="/etc/hekad.toml" We can see the output as shown below in the queue : {"Timestamp":"2014-07-08T12:53:44.004Z","Type":"logfile","Logger":"tom.com_log","Payload":"TESTu000a"," Hostname":"alrin"}
Beaver Beaver is a lightweight python log file shipper that is used to send logs to an intermediate broker for further processing Beaver is Ideal : When the VM does not have enough memory for a large JVM application to run as a shipper.
Chef Recipe : Beaver When a VM is run, recipe(megam_logstash::beaver) is included. node.set['logstash']['key'] = "#{node.name}" node.set['logstash']['amqp'] = "#{node.name}_log" node.set['logstash']['beaver']['inputs'] = [ "/var/log/upstart/nodejs.log", "/var/log/upstart/gulpd.log" ] include_recipe "megam_logstash::beaver" attributes like (nodename, logfiles) are set dynamically.
RSYSLOG RSYSLOG is the rocket-fast system for log processing. It offers high-performance, great security features and a modular design. Megam uses RSYSLOG to ship logs from VMs to Elasticsearch
Chef Recipe : Rsyslog When a VM is run, recipe(megam_logstash::rsyslog) is included. node.set['rsyslog']['index'] = "#{node.name}" node.set['rsyslog']['elastic_ip'] = "monitor.megam.co.in" node.set['rsyslog']['input']['files'] = [ "/var/log/upstart/nodejs.log", "/var/log/upstart/gulpd.log" ] include_recipe "megam_logstash::rsyslog" attributes like (nodename, logfiles) are set dynamically.
For more details http://www.gomegam.com email : gomegam@megam.co.in twitter: @megamsystems
