I finished reading “The Logging Textbook for Practical Use” (実務で役立つ ログの教科書), so I’m leaving what stuck with me as reading notes. I used to think of logs mainly as something you look at when a failure happens, but this book changed my view: logs can be used for much broader purposes.
Purposes of Logs Beyond Troubleshooting
Logs are not something output only when a problem occurs. Unless you also leave logs that show “it is working without problems” while things are running normally, you cannot even tell what the normal state is in the first place. Looking only at logs from abnormal times, you cannot judge whether something is truly abnormal without a record of the normal state to compare against.
Logs that show “it is working without problems” can also be used for purposes other than failure investigation. For example, from users’ access logs and browsing history, you can grasp how users behaved, and analyze that data to use it for marketing. For a website, you can analyze things like which page users arrived from and which page they left from, and how much access came from which region.
The book organized the purposes of logs into three broad categories.
- System management: investigate how the system is being used / investigate the cause of trouble
- Security: investigate signs of an attack / deter misconduct
- Business: investigate usage
I tended to think “I’ll add logs because it’s a problem if an error comes up,” but it clicked for me to think in this order: the logs from normal times are the baseline, and it is precisely because that baseline exists that you can notice anomalies.
What to Output in Logs (5W1H)
The book introduced 5W1H as an easy way to organize what should be recorded in logs.
- When: The basic unit is seconds. When you want to strictly trace the order of processing, record down to milliseconds or microseconds
- Where: Where the log was output from
- Who: The person or device that performed the operation requiring a log
- What: What data was operated on, and the content of the request
- Why: The type of log (warning, error, normal, etc.)
- How: How the operation was performed (which button was pressed, which link was clicked)
Rather than vaguely “outputting logs,” checking whether these six perspectives are filled in seems to produce logs that are useful when you look back later.
Access Logs
An access log, which records the requests a web server received, includes items like the following.
| Item | Content |
|---|---|
| Timestamp | The date and time of the request |
| Source IP address | The user’s IP address |
| Request method | GET, POST, etc. |
| URL | The path and query string |
| Status code | Normal, error, etc. |
| Transfer size | The file size, etc. |
| Referrer | The URL of the link source |
| User agent | The user’s OS, web browser, etc. |
As formats for laying out these items, the NCSA combined format and the W3C extended log format are commonly used. Because the format is standardized, you can feed it straight into analysis tools, which I felt is a big advantage.
Query Logs
A query log records the execution history of SQL queries received by a DBMS. Since it shows what kinds of queries are executed and how often, it helps with optimizing indexes and improving SQL statements.
When investigating performance problems, being able to look not only at the application-side logs but also at what queries the database received makes it easier to reach the root cause.
Log Collection Methods (Agent-based and Agentless)
There are broadly two ways to collect logs: agent-based and agentless.
The agent-based method installs a resident piece of software called an agent on the machine you want to monitor, and collects logs from there. The agentless method, by contrast, has the management server communicate with the monitored targets using protocols such as SNMP, ICMP, and WMI, gathering data from the outside. Its distinguishing feature is that you do not need to install dedicated software on the monitored targets.
- SNMP (Simple Network Management Protocol): a simple protocol used when monitoring over a network
- ICMP (Internet Control Message Protocol): a protocol used to check whether communication is working without problems
- WMI (Windows Management Instrumentation): a protocol used to access management information on Windows computers
One representative agentless log collection tool is Graylog, which collects logs from syslog. Graylog works in combination with MongoDB and OpenSearch.
Searching Logs
You can search collected logs with a text editor’s search function, but once the volume grows, using a full-text search engine becomes more practical. The commonly used ones are Elastic Stack, Grafana, and Splunk.
- Elastic Stack: a platform developed by Elastic for collecting, searching, analyzing, and visualizing data
- Grafana: open-source software that can visualize and monitor metrics and logs from various data sources as graphs and dashboards on a single screen
- Splunk: a data analytics platform that collects, searches, analyzes, and visualizes, in real time, the vast amounts of machine data (logs) output by servers, network devices, and applications
Log Notifications and the Challenges After Notifying
When you spot an anomaly, you notify the person in charge. What struck me, though, is that reliably following through to a response after notifying is harder than issuing the notification itself. The challenges that tend to arise were organized as follows.
| Challenge | Background |
|---|---|
| Can’t reach the person in charge | The person in charge is absent when the alert fires |
| The person got the notice but left it unattended | They are aware of the alert but put off dealing with it |
| Multiple people are in charge, but no one takes it on | Responsibility becomes ambiguous and the response is delayed |
| Handled it, but the result isn’t reported | No report after handling, hindering information sharing and the next response |
| Many false positives and a lot of noise | Alerts fire frequently even when there is no actual problem, making handling cumbersome |
| The alert content is unclear and lacks information | The cause and scope of impact are unknown, delaying the response |
| Response history and log management are insufficient | Past response history isn’t kept, so handling takes time |
| The person in charge lacks skills or knowledge | They don’t know how to respond, so handling takes time |
| The priority of the response is unclear | It’s hard to judge which alert to handle first, so important ones get put off |
| Integration between systems is lacking | Multiple systems aren’t integrated, so information is only available in fragments |
I felt that just building the notification mechanism isn’t enough; you need to design all the way through who responds in what order and how the results are recorded.
Detecting Tampering and Preserving Evidence
Logs are also evidence that shows a system’s operation history and records of operations. If logs are tampered with, evidence of misconduct is lost, the credibility of an investigation is undermined, and they can no longer be used as legal evidence. For that reason, in addition to preventing tampering, you are expected to be able to detect it early.
One detection method introduced was Tripwire, which targets rotated logs. It records file hash values in advance and detects changes by comparing them periodically.
Performance Improvement Using Logs
Logs can also be used to improve performance. When you want to measure response times in more detail, one approach is to introduce an APM (Application Performance Management) tool. The book listed Prometheus, Grafana, New Relic, Datadog, and Elastic APM.
- Prometheus: open source that collects and monitors numeric values from systems and servers
- Grafana: open-source software that can visualize and monitor metrics and logs from various data sources on a single screen
- New Relic: a cloud-based observability platform that monitors and analyzes the state of systems and applications in an integrated way
- Datadog: provides code-level distributed tracing from browsers and mobile apps through to backend services and databases
- Elastic APM: a tool that monitors application performance and behavior in real time
Closing
After reading this book, my view of logs shifted from “something you glance at when in trouble” to “something you keep recording to capture the normal state.” You design what to record using 5W1H, and choose a well-organized format for each purpose, like access logs and query logs. On top of that, once you can survey the full range of uses such as collection, search, notification, tamper detection, and performance improvement, it seems applicable to any of the purposes of system management, security, and business.
