[BBLISA] Re: Large scale log processing

[seph]

Mike Sprague <mfs at komerex.com> writes:

> I work for a web hosting company with about a thousand linux servers.
> We're discussing options on how to process the logs mainly from our mail
> and web servers to make troubleshooting easier.  We're not really
> looking for long term storage; just a better way to be able to search
> the logs to diagnose either specific customer issues, broad system
> attacks, issues across a pool of servers or issues with a specific server.

splunk would be the obvious commercial product for this space.

http://www.opensyslog.com is a online hosted log consolidation tool. In
beta. 

I've seen a variety of open source things in the log processing world. I
don't think they're generally very flashy, and I'm not sure what the
current favorites are. googling around syslog processing finds some
suggestions.

seph