LogStash-Topology
Dedicated Setup
This should work for dedicated setups where multiple servers from a single client are sending to a single LogStash collection server. This will follow a many to one setup where multiple logstash shippers are sending to a single RabbitMQ server, processed by a single logstash indexer and stored either in Graylog2 or directly in to ElasticSearch.
This setup is ideal for customers that have a lot of logs that need processed and/or do not want their logs being shipped outside of their private network.
There will be no redundancy in this setup but it is also the easiest to maintain. If the LogStash server goes down (reboots) then messages should be queued on the sending servers until the server is back online. So messages generally will not be lost. Keep in mind that the logstash agent only has so much memory on the remote side and messages may be lost of the logstash server is offline for an extended time.
As a Service
The concept of this setup is to offer log storage and collation to customers as a shared service. Customers will have access only to their logs but hardware will be shared between multiple customers. Due to the indexer having a general limit of 1200 msg/s, this may need to be limited to syslog style logs only (ie no domlogs).
The general setup would be multiple customers servers sending to a RabbitMQ cluster where then multiple indexers will process the logs and store on a ElasticSearch cluster where each customer has their own log storage path.
I am not sure how the interface will need to be written for this, or if customers will have direct access. The main concept of this setup is to store the customers server logs in case of a crash or root where normal logs may be lost.
As a Service w/Dedicated indexer
The concept of this setup is to offer the same reliability of the 'As a Service' setup but completely isolate the customers indexers. Each customer will have a dedicated LogStash server that will serve as the indexer. ElasticSearch can still be shared as long as each customer gets their own storage pool. Customers can be given access to the logstash web interface (or graylog2) to access their logs. The cost for the hardware could easily be added in to the service cost. Also depending on the setup we could just as easily offer it on virtuality hardware as long as they are connected to the ElasticSearch / RabbitMQ network.
