Actionable Logging for Smoother Operation and Faster Recovery

Notes from Velocity.

Next session is Actionable Logging for Smoother Operation and Faster Recovery by Mandi Walls (AOL).

Actionable logging:

• no nonsense logging
• concise, easy to understand
• express symptoms of production issues
• anything that makes the log needs to be fixed
  

Why it's important?

• expending resources on production systems
• the point of logging in production
• diagnosis of issues
• the 4am test
  

Logging goals:

• diagnosis and recovery
• statistics and monitoring
• provide insight into the behavior of the application
• indicate potential issues, and areas for improvement
• not the same goals as development and QA environments.
  

Types of logs:

• access log
• server log, i.e. catalina.out
• application logs
• special use logs for recording specific groups of activities
  

Log file location:

• where logs are located on the system should be predictable and obvious
• it may be helpful to locate logs on different disk partitions but link them back to the app.
• keep older logs in an obvious place. Having a couple days worth of logs helps you get convergence on the problem.
  

Log file management:

• everyone has their own method
• roll logs into files with timestamps
• host-01.log.003 vs host-01.log.06202008
• roll all the logs at the same time for a given app to make coordination of events easier
• roll when teh app needs logs rolled: hourly, daily weekly.
  

Logging quality information

• logs should be expressive but not overly verbose
• keys to making logs more actionable:
• appropriate formats
  

Quality information: format

• Timestamping: what not to do:
• 1213988938:tvdata shows/617/306
• 1213988939:tvdata shows/618/307
• Timestamps that mean something:
• Jun 19,2009 4:20:25PM
• Good timestamps give context for linking to external events like network outages or traffic anomalies
  

Other considerations in log file format include:

• creating a common format for multiple products and log tpes.
• limiting the number of log entries that write to multiple log lines for faster parsing
• deciding how much is too much information
  

Quality info:

• bad: messages with only numerical error codes in them
• bad: don't put debugging messages in production logs.
• bad: if log message doesn't give you anything to lead on.
• bad: multiple line error messages.
• good: module_id with descriptive message
  

Making messages useful:

• don't list non fatal messages as fatal
• incorrect / misleading severity
• not logging anything for fatal errors
  

Improving log messages:

• log at the first point an error is encountered - don't log a timeout to a backend as a parse error of date expected from the request
• messages include the method name and key variables to speed up fixes
• suppress anything in the log that isn't actionable
  

log messages convergence:

• actively managing, parsing, pruning logs make new errors more obvious
• check teh logs afer every install for new messages
  

things to avoid in logs:

• usernames, passwords, database logins
• provide crib notes for anone getting in your system.
  

how much is too much:

• more than 25% of the number of access log entries is a hindrance. Even 10% may be too much in most environments
• if you log more in server log than access log, then its a problem
  

conclusion

• the log is first line of information when a problem occurs
• a production log should be focused on providing information to operations staff, not developers.
• when, where and how messages are logged can help or hinder recovery
  

Also see:

• Actionable Logging for Smoother Operation and Faster Recovery (PPT)
  

Open Voice API by AOL

AOL launched Open Voice API last month to allow VOIP developers and device makers ability to integrate AOL call out services in their application and devices.

- AOL Call out is a pay-as-you-go outbound voice calling service
- Users can call landlines and cell phones in more than 200 countries
- Rates for AOL call out are available at http://call-out.aim.com/rates
- Open Voice API is available at http://dev.aol.com/aimcall
- AIM users get an integrated dashboard available at http://call.aim.com/ to manage their call activity.
- Call credits can be purchased by users in $5 increments.