Mining logs for coverage analysis

A friend on Facebook posted pointing to Logging: Exploratory Tester’s Friend on James Bach's blog. Got me thinking, arrr matey.

GFI–Analyze application's logs for keys to extract high level user events that may be used to document the "actions testers did".

This could serve as objective physical evidence that functional requirements were tested.

Consider being able to process customer logs to unearth exactly what they did "before the bug" was discovered. Will need to be human readable ;) otherwise you just have a different set of logs that nobody will read (or be able to understand).

The logs already have much of the raw data I am thinking about. However, unless you are examining a specific event/error it is way to much text to read. Need something for examining all the events over a two hour period or longer (long run, one shift, over weekend).

I think we included most of the requirements that James listed save his #11. The Instrument Log workspace does something like this but we will want an external app to do same and more.

11. The level of detail included in the log file should be configurable in terms of preset levels: 1- error and service events only, 2- Functional events, error events, service events, 3- All events including diagnostic information messages about internal states and parameters.

This will be an excellent project for me to develop.

Grounds for Further Investigation

• SunSoft installation log analysis tool; display the unexpected
• SQL Server Transaction logs;
• Tool to manually inject Tester Comments into the application's normal logs
• MAD Reporter
• Data representation with pictures or colors or sound or measures
• Measures; # log entries per second/minute, # errors per minute/hour, # seconds per RV, # RV per hour, etc.
• Uptime Reporter
• Our own log analysis requirements in Caliber RM
• Zoom in/out of report; high level, user actions, debug, everything
• Focus on specific types of transactions; RV processing, user actions, DB I/O, State change, Errors, task duration, etc.
• Merge information from other logs; SQL Server, perfmon, Dr. Watson, etc.
• User Stories