[BBLISA] Re: Large scale log processing

[Mike Devlin]

How many log lines do you think you would be collecting?

We ended up using syslog-ng to receive all the data, and syslog-ng hands it
off to splunk along with writing it to files. For the most part splunk is
great, but sometimes its just easier to go through the logs from the command
line. Splunk can get bogged down at times and can also get somewhat pricey.
We have about 5 million log lines a day going into our logging setup (damn
mail servers), and the most usable way to use splunk was to split the load
up between multiple splunk servers so the query load is somewhat
distributed.


Mike Devlin
Manager of Operations
boston..com




On Fri, May 15, 2009 at 10:13 AM, Sean Lutner <sean at rentul.net> wrote:

> I'd second the recommendation for splunk. It's a fantastic product, is easy
> to setup and would provide you with a way to aggregate and then easily
> search over all your data. Aggregation is the easy part, the searching,
> correlation, etc is not easy. I've implemented splunk at three different
> places and am in the middle of a deployment currently.
>
>
> On May 15, 2009, at 9:49 AM, seph wrote:
>
>  Mike Sprague <mfs at komerex.com> writes:
>>
>>  I work for a web hosting company with about a thousand linux servers.
>>> We're discussing options on how to process the logs mainly from our mail
>>> and web servers to make troubleshooting easier.  We're not really
>>> looking for long term storage; just a better way to be able to search
>>> the logs to diagnose either specific customer issues, broad system
>>> attacks, issues across a pool of servers or issues with a specific
>>> server.
>>>
>>
>> splunk would be the obvious commercial product for this space.
>>
>> http://www.opensyslog.com is a online hosted log consolidation tool. In
>> beta.
>>
>> I've seen a variety of open source things in the log processing world. I
>> don't think they're generally very flashy, and I'm not sure what the
>> current favorites are. googling around syslog processing finds some
>> suggestions.
>>
>> seph
>>
>> _______________________________________________
>> bblisa mailing list
>> bblisa at bblisa.org
>> http://www.bblisa.org/mailman/listinfo/bblisa
>>
>>
>
> _______________________________________________
> bblisa mailing list
> bblisa at bblisa.org
> http://www.bblisa.org/mailman/listinfo/bblisa
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.bblisa.org/pipermail/bblisa/attachments/20090515/42f2ea4c/attachment.htm