LogStash - Revision history

Smsldoo at 20:32, 7 April 2012

2012-04-07T20:32:43Z

Smsldoo: /* Install RabbitMQ */

2012-03-29T15:59:16Z

‎Install RabbitMQ

Smsldoo: /* Complete Package */

2012-03-13T18:12:53Z

‎Complete Package

Smsldoo: /* Setup ElasticSearch */

2012-03-13T18:08:37Z

‎Setup ElasticSearch

Smsldoo: /* Complete Package */

2012-03-13T17:33:40Z

‎Complete Package

Smsldoo: /* Custom Patterns */

2012-03-13T17:27:00Z

‎Custom Patterns

Smsldoo: /* Custom Patterns */

2012-03-13T17:22:23Z

‎Custom Patterns

Smsldoo: /* Custom Patterns */

2012-03-13T17:20:04Z

‎Custom Patterns

Smsldoo: /* logstash sh script */

2012-03-13T17:16:39Z

‎logstash sh script

Smsldoo: /* Wrapper configuration */

2012-03-13T17:15:22Z

‎Wrapper configuration

← Older revision		Revision as of 20:32, 7 April 2012
Line 1:		Line 1:
−	[[Category:~~Misc~~]]	+	[[Category:LogStash]]

	LogStash is a utility that can read, filter and store or transmit log data on a server.		LogStash is a utility that can read, filter and store or transmit log data on a server.

@@ Line 573: / Line 573: @@
 <code>[bash,n]
-wget http://www.rabbitmq.com/releases/rabbitmq-server/v2.7.1/rabbitmq-server-2.7.1-1.noarch.rpm
+wget http://www.rabbitmq.com/releases/rabbitmq-server/v2.8.1/rabbitmq-server-2.8.1-1.noarch.rpm
 rpm --import http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
-yum install rabbitmq-server-2.7.1-1.noarch.rpm
+yum install rabbitmq-server-2.8.1-1.noarch.rpm
 </code>

← Older revision		Revision as of 18:12, 13 March 2012
Line 727:		Line 727:

	APACHEERRORLOG %{MODSECAPACHEERROR}\|%{GENERICAPACHEERROR}		APACHEERRORLOG %{MODSECAPACHEERROR}\|%{GENERICAPACHEERROR}
		+	</code>
		+
		+	=Troubleshooting=
		+
		+	There are a few things to keep in mind when troubleshooting this setup. The most common error that I have found is the RabbitMQ queue is not being processed.
		+
		+	More often then not this is caused by ElasticSearch either not accepting connections or accepting them very slowly.
		+
		+	Tailing /opt/elasticsearch/logs/elasticsearch.log will give you an idea if it is causing the problems. Generally if it is a issue with ulimits it will spew messages constantly.
		+
		+	Alternatively check the logstash logs too to see if something is up with logstash.
		+
		+	/opt/logstash/logs/logstash.log
		+
		+	Also to quickly check if the processes are even running you can use service $servicename status. This will give you a message like:
		+
		+	<code>[bash,n]
		+	service elasticsearch status
		+	ElasticSearch is running: PID:2779, Wrapper:STARTED, Java:STARTED
	</code>		</code>

← Older revision		Revision as of 18:08, 13 March 2012
Line 610:		Line 610:

	Essentially it is the same thing, just set up for ElasticSearch		Essentially it is the same thing, just set up for ElasticSearch
		+
		+	One thing that will probably come up is the open file limit with ulimit. By default this is set to 1024 which is way too low for elastic search. You will want to change this to something higher:
		+
		+	<code>[bash,n]
		+	vim /etc/security/limits.conf
		+	</code>
		+
		+	<code>[bash,n]
		+	root hard nofile 102400
		+	root soft nofile 102400
		+	</code>

	=Web Interfaces=		=Web Interfaces=

← Older revision		Revision as of 17:33, 13 March 2012
Line 725:		Line 725:

	This is currently set up as a shipper, but you can modify it to work in a different role.		This is currently set up as a shipper, but you can modify it to work in a different role.
		+
		+	And yes I put this at the very bottom so you would have to read all the stuff above :P

← Older revision		Revision as of 17:27, 13 March 2012
Line 717:		Line 717:
	APACHEERRORLOG %{MODSECAPACHEERROR}\|%{GENERICAPACHEERROR}		APACHEERRORLOG %{MODSECAPACHEERROR}\|%{GENERICAPACHEERROR}
	</code>		</code>
		+
		+	=Complete Package=
		+
		+	If you are lazy and want the complete package you can download it here:
		+
		+	http://jamesdooley.us/logstash.tar.gz
		+
		+	This is currently set up as a shipper, but you can modify it to work in a different role.

@@ Line 687: / Line 687: @@
 CPANELERRORLOG \[%{CPANELERRORDATE:timestamp}\] %{WORD:level} \[%{PROG:program}\]
 </code>
 $basedir/patterns/apache:
 <code>[bash,n]
 APACHEERRORDATE %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}
-APACHEERRORLOG \[%{APACHEERRORDATE:date}\] \[%{WORD:level}\] \[%{WORD} %{IP:clientip}\]
+APACHEERRORPREFIX \[%{APACHEERRORDATE:date}\] \[%{WORD:level}\] \[%{WORD} %{IPORHOST}\]
 </code>

@@ Line 661: / Line 661: @@
 These will go in to $basedir/patterns/
-I usually name the files logically based on the application name
+I usually name the files logically based on the application name.
 $basedir/patterns/cpanel:

@@ Line 75: / Line 75: @@
 |APP_LONG_NAME || This is the plain text name for the application, this will be displayed when the service is starting.
 |-
-|WRAPPER_CMD || This should be set to "$basedir/wrapper" obviously change $basedir to the actual install path
+|WRAPPER_CMD || This should be set to "./lib/wrapper"
 |-
-|WRAPPER_CONF || This should be set to "$basedir/conf/wrapper.conf"
+|WRAPPER_CONF || This should be set to "./conf/wrapper.conf"
 |}

@@ Line 90: / Line 90: @@
 Also some of the strings will not exist if they are numbered, just add them below .1
-And finally once again you will need to change $basedir to the actual directory you installed the application.
+Updated this so that it calls on the relative paths instead of a set path. This means you can put the script in any directory and just run it.
 {|
@@ Line 96: / Line 96: @@
 |wrapper.java.command || This needs to be changed to ''org.tanukisoftware.wrapper.WrapperJarApp''
 |-
-|wrapper.java.classpath.1 || $basedir/lib/wrapper.jar
+|wrapper.java.classpath.1 || ../lib/wrapper.jar
 |-
-|wrapper.java.classpath.2 || $basedir/logstash.jar
+|wrapper.java.classpath.2 || ../lib/logstash.jar
 |-
-|wrapper.java.library.path.1 || $basedir/lib
+|wrapper.java.library.path.1 || ../lib
 |-
-|wrapper.java.additional.1 || -Djava.io.tmpdir=$basedir/tmp
+|wrapper.java.additional.1 || -Djava.io.tmpdir=../tmp
 |-
-|wrapper.app.parameter.1 || $basedir/logstash.jar
+|wrapper.app.parameter.1 || ../lib/logstash.jar
 |-
 |wrapper.app.parameter.2 || agent
@@ Line 110: / Line 110: @@
 |wrapper.app.parameter.3 || -f
 |-
-|wrapper.app.parameter.4 || $basedir/conf/logstash.conf
+|wrapper.app.parameter.4 || ../conf/logstash.conf
 |-
 |wrapper.app.parameter.5 || -l
 |-
-|wrapper.app.parameter.6 || $basedir/logs/logstash.log
+|wrapper.app.parameter.6 || ../logs/logstash.log
 |-
-|wrapper.logfile || $basedir/logs/wrapper.log
+|wrapper.logfile || ../logs/wrapper.log
 |-
 |wrapper.logfile.maxsize || 100m
 |-
-|wrapper.logfile.maxfiles || 4
+|wrapper.logfile.maxfiles || 2
 |}