Log analysis

<h2 id="functions-and-technologies">Functions and technologies</h2>
Pattern recognition is a function of selecting incoming messages and compare with a pattern book to filter or handle different ways.
Normalization is the function of converting message parts to the same format (e.g. common date format or normalized IP address).
Classification and tagging is ordering messages into different classes or tagging them with different keywords for later usage (e.g. filtering or display).
Correlation analysis is a technology of collecting messages from different systems and finding all the messages belonging to one single event (e.g., messages generated by malicious activity on different systems: network devices, firewalls, servers, etc.). It is usually connected with alerting systems.
Artificial Ignorance is a type of <a href="/facts/Machine_learning/e0w0XJTu">machine learning</a> that is a process of discarding log entries that are known to be uninteresting. Artificial ignorance is a method to detect anomalies in a working system. In log analysis, this means recognizing and ignoring the regular, common log messages that result from the normal operation of the system, and therefore are not too interesting. However, new messages that have not appeared in the logs before can signal important events, and should be therefore investigated.<a class="footnote-ref" id="fnref:1" href="#fn:1">1</a><a class="footnote-ref" id="fnref:2" href="#fn:2">2</a> In addition to anomalies, the algorithm will identify common events that did not occur. For example, a system update that runs every week, has failed to run.
Log Analysis is often compared to other analytics tools such as <a href="/facts/Application_performance_management/3XI0tuSy">application performance management</a> (APM) and error monitoring. While much of their functionality is clear overlap, the difference is rooted in process. APM has an emphasis on performance and is utilized most in production. Error monitoring is driven by developers versus operations, and integrates into code in <a href="/facts/Exception_handling/VrJxgb9r">exception handling</a> blocks.

<h2 id="see-also">See also</h2>
<ul><li><a href="/facts/Audit_trail/s3nQzrF4">Audit trail</a></li>
<li><a href="/facts/Data_logger/U3GglNdc">Data logger</a></li>
<li><a href="/facts/Log_monitor/W7837zGS">Log monitor</a></li>
<li><a href="/facts/Server_log/p7FHf8mA">Server log</a></li>
<li><a href="/facts/System_monitor/xnNgtTvF">System monitor</a></li>
<li><a href="/facts/Web_log_analysis_software/kl2u8P3Q">Web log analysis software</a></li>
<li><a href="/facts/List_of_web_analytics_software/xJFw0GRg">List of web analytics software</a></li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1">"artificial ignorance: how-to guide". www.ranum.com. <a href="http://www.ranum.com/security/computer_security/papers/ai/index.html" target="_blank">http://www.ranum.com/security/computer_security/papers/ai/index.html</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></li>
<li id="fn:2">"Log message classification with syslog-ng [LWN.net]". lwn.net. <a href="https://lwn.net/Articles/369075/" target="_blank">https://lwn.net/Articles/369075/</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></li>
</ol>

Log analysis open-in-new

Log analysis