THIS IS AN OLD FAQ FROM 2005! IT'S STILL VERY RELEVANT.

Splunkalicious

What is Splunk?

    Splunk is a search server that indexes all your log files. If you need to search and troubleshoot log files, you need Splunk. It handles any log format, including syslog, Apache, Jboss, mysql, oracle, router data, etc. It parses and indexes in real time.

Grep works fine. Why do I need Splunk?

    grep is totally fine for small, simple, local files, but grep doesn't work on 20GB of log files, across a dozen servers; doesn't group multiline log messages together; doesn't unify timestamps across files; doesn't automatically find related log events; doesn't show histograms of log events; doesn't search gigabytes in seconds; doesn't have a cool ajax web interface similar to google.

What are multiline log messages?

    As an example, java exceptions look like this: 
      java.lang.reflect.UndeclaredThrowableException
	at $Proxy231.getAllAttributes(Unknown Source)
	at com.collation.proxy.clientproxy.common.Module.getModelObject(Module.java:326)
	at com.collation.proxy.clientproxy.server.action.ChangeHistoryModule.getDependencies(ChangeHistoryModule.java:402)
	at com.collation.proxy.clientproxy.server.action.ChangeHistoryModule.getIdsWithDependencies(ChangeHistoryModule.java:386)
	...
    You can't use grep to search for java proxy exceptions because "Exception" and "proxy" don't occur on the same line! The same would apply to sql, router data, email, or any other multiline event. Splunk automatically groups multiline events into single events, so the above exception would become one event. Splunk does this with advanced heuristics and machine learning algorithms, as well as customizeable groupping rules.

What about unifying timestamps?

    Most log files have timestamps embedded in them. Splunk understands dozens and dozens of timestamp formats, unifying them across timezones. Some log files write events out as GMT (Greenwich Mean Time) some as local time such as PST (Pacific Standard Time). Some logs can come from servers on the east coast, some from the west coast, or beyond. By normalizing all these timeszones in dozens of timestamp formats, Splunk allows you to say "What happened at 11:57pm", world-wide, across all my log files, across all my servers. "I got an error at 1:15am yesterday. Show me the log events from all my logs just before 1:15".

OK, one more. What are related log events?

    Suppose you see suspecious activity or an error. Just ask Splunk to find logs related to that activity. It'll find logs that have the same IP, UserID, URL, codes, etc. If there was a problem with an IP, Splunk will show you all the related events for that IP; same for UserID, URL, or any other code. You can even ask Splunk to show you events sorted by how unexpected they are!

How much does Splunk cost?

    The Splunk Personal Server is Free. Give it a try.

How can I get Splunk?