LogStash-Topology - Revision history

Smsldoo at 20:32, 7 April 2012

2012-04-07T20:32:57Z

Smsldoo at 16:11, 14 March 2012

2012-03-14T16:11:12Z

Smsldoo at 16:02, 14 March 2012

2012-03-14T16:02:55Z

Smsldoo: /* Dedicated Setup */

2012-03-14T12:23:54Z

‎Dedicated Setup

Smsldoo: /* Dedicated Setup */

2012-03-14T12:23:28Z

‎Dedicated Setup

Smsldoo: /* Dedicated Setup */

2012-03-14T12:16:35Z

‎Dedicated Setup

Smsldoo: Created page with "Category:Misc =Dedicated Setup= This should work for dedicated setups where multiple servers from a single client are sending to a single LogStash collection server. Thi..."

2012-03-13T18:30:43Z

Created page with "Category:Misc =Dedicated Setup= This should work for dedicated setups where multiple servers from a single client are sending to a single LogStash collection server. Thi..."

New page

[[Category:Misc]]

=Dedicated Setup=

This should work for dedicated setups where multiple servers from a single client are sending to a single LogStash collection server. This will follow a many to one setup where multiple logstash shippers are sending to a single RabbitMQ server, processed by a single logstash indexer and stored either in Graylog2 or directly in to ElasticSearch.

This setup is ideal for customers that have a lot of logs that need processed and/or do not want their logs being shipped outside of their private network.

There will be no redundancy in this setup but it is also the easiest to maintain. If the LogStash server goes down (reboots) then messages should be queued on the sending servers until the server is back online. So messages generally will not be lost. Keep in mind that the logstash agent only has so much memory on the remote side and messages may be lost of the logstash server is offline for an extended time.

=As a Service=

The concept of this setup is to offer log storage and collation to customers as a shared service. Customers will have access only to their logs but hardware will be shared between multiple customers. Due to the indexer having a general limit of 1200 msg/s, this may need to be limited to syslog style logs only (ie no domlogs).

The general setup would be multiple customers servers sending to a RabbitMQ cluster where then multiple indexers will process the logs and store on a ElasticSearch cluster where each customer has their own log storage path.

I am not sure how the interface will need to be written for this, or if customers will have direct access. The main concept of this setup is to store the customers server logs in case of a crash or root where normal logs may be lost.

=As a Service w/Dedicated indexer=

The concept of this setup is to offer the same reliability of the 'As a Service' setup but completely isolate the customers indexers. Each customer will have a dedicated LogStash server that will serve as the indexer. ElasticSearch can still be shared as long as each customer gets their own storage pool. Customers can be given access to the logstash web interface (or graylog2) to access their logs. The cost for the hardware could easily be added in to the service cost. Also depending on the setup we could just as easily offer it on virtuality hardware as long as they are connected to the ElasticSearch / RabbitMQ network.

← Older revision		Revision as of 20:32, 7 April 2012
Line 1:		Line 1:
−	[[Category:~~Misc~~]]	+	[[Category:LogStash]]

	=Dedicated Setup=		=Dedicated Setup=

@@ Line 9: / Line 9: @@
 There will be no redundancy in this setup but it is also the easiest to maintain. If the LogStash server goes down (reboots) then messages should be queued on the sending servers until the server is back online. So messages generally will not be lost. Keep in mind that the logstash agent only has so much memory on the remote side and messages may be lost of the logstash server is offline for an extended time.
-<img src="http://www.gliffy.com/pubdoc/3392953/M.png"/>
+[[File:LogStash Dedicated.png|200px|thumb|left|LogStash Dedicated Topology]]
 =As a Service=

← Older revision		Revision as of 12:16, 14 March 2012
Line 8:		Line 8:

	There will be no redundancy in this setup but it is also the easiest to maintain. If the LogStash server goes down (reboots) then messages should be queued on the sending servers until the server is back online. So messages generally will not be lost. Keep in mind that the logstash agent only has so much memory on the remote side and messages may be lost of the logstash server is offline for an extended time.		There will be no redundancy in this setup but it is also the easiest to maintain. If the LogStash server goes down (reboots) then messages should be queued on the sending servers until the server is back online. So messages generally will not be lost. Keep in mind that the logstash agent only has so much memory on the remote side and messages may be lost of the logstash server is offline for an extended time.
		+
		+	<img src="http://www.gliffy.com/pubdoc/3392953/M.png"/>

	=As a Service=		=As a Service=

@@ Line 18: / Line 18: @@
 =As a Service w/Dedicated indexer=
+[[File:LogStash Dedi Indexer.png|200px|thumb|right|LogStash Service Dedicated Indexer Topology]]
 The concept of this setup is to offer the same reliability of the 'As a Service' setup but completely isolate the customers indexers. Each customer will have a dedicated LogStash server that will serve as the indexer. ElasticSearch can still be shared as long as each customer gets their own storage pool. Customers can be given access to the logstash web interface (or graylog2) to access their logs. The cost for the hardware could easily be added in to the service cost. Also depending on the setup we could just as easily offer it on virtuality hardware as long as they are connected to the ElasticSearch / RabbitMQ network.

@@ Line 10: / Line 10: @@
 =As a Service=
+[[File:LogStash Service.png|200px|thumb|right|LogStash Service Topology]]
 The concept of this setup is to offer log storage and collation to customers as a shared service. Customers will have access only to their logs but hardware will be shared between multiple customers. Due to the indexer having a general limit of 1200 msg/s, this may need to be limited to syslog style logs only (ie no domlogs).

@@ Line 2: / Line 2: @@
 =Dedicated Setup=
+[[File:LogStash Dedicated.png|200px|thumb|right|LogStash Dedicated Topology]]
 This should work for dedicated setups where multiple servers from a single client are sending to a single LogStash collection server. This will follow a many to one setup where multiple logstash shippers are sending to a single RabbitMQ server, processed by a single logstash indexer and stored either in Graylog2 or directly in to ElasticSearch.
@@ Line 8: / Line 8: @@
 There will be no redundancy in this setup but it is also the easiest to maintain. If the LogStash server goes down (reboots) then messages should be queued on the sending servers until the server is back online. So messages generally will not be lost. Keep in mind that the logstash agent only has so much memory on the remote side and messages may be lost of the logstash server is offline for an extended time.
 =As a Service=