When does log volume actually become a problem?
Log volume becomes a problem when one of three things happens: the log search screen gets so slow that operators stop using it, the cloud storage bill rises every month while user numbers stay roughly flat, or nobody can pull the right slice of logs during an incident.
"Tens of millions of rows" is not a problem on its own. PostgreSQL or MySQL can handle tables that size if the table is partitioned by time and every query filters on a date range. In the systems VAON has reviewed, the cause was more often one of these:
- Logs share a database with business data, so every log write competes with the application itself.
- Debug logging, switched on during development, is still running in production.
- Old logs are removed with row-by-row `DELETE`, which runs for hours and bloats the table instead of shrinking it.
- Access logs and administrator activity logs all share one retention period.
So the first question is not "which technology should we switch to". It is "which logs do we keep, who reads them, and for how long".
How long must you keep logs, and do you have to keep all of them?
There is no single log retention period that fits every system. It depends on your industry, your client contracts and the type of log.
Three commonly cited rules show why:
- Japan's Act on the Protection of Personal Information (APPI). The general guidelines from Japan's Personal Information Protection Commission (PPC) list recording system usage (login history, access logs) and regularly analyzing logs to detect unauthorized access as examples of security measures. They do not set a fixed retention period for system logs.
- PCI DSS (the payment card data security standard). Requirement 10.5.1 says to retain audit log history for at least 12 months, with at least the most recent three months immediately available for analysis.
- GDPR (EU). The storage limitation principle says personal data must not be kept longer than necessary. Logs with IP addresses or user IDs count. Keeping everything forever is a risk in itself.
So the claim that "deleting logs to save money is illegal" goes too far. What you need is a retention period per log type, with the reason written down.
Four ways to cut log storage costs, from small to large changes
The four options below are ordered from the smallest system change to the largest. Work through them in order, because many systems are light enough after steps 1 and 2.
Time partitioning is the step teams most often skip. With logs split by day or month, expiring old data means dropping one partition. It takes moments, and you never run a `DELETE` over millions of rows.
Moving older logs to cheaper storage (hot/cold tiering) keeps recent logs where they are fast to search and moves older logs to object storage such as Amazon S3. Under the PCI DSS rule above, that could mean three months in fast storage and the other nine in cheap storage. Before you move anything, restore one day of old logs and time it. Nobody wants to wait hours in the middle of an incident.
Columnar databases such as ClickHouse store each column separately, so log data, which repeats a lot, usually compresses well. ClickHouse also supports TTL (time-to-live) rules that move data to another disk or delete it by age, for example keeping error logs for six months and other logs for one. Real compression depends on your data. Load one week of your own logs and measure, rather than trusting a general figure.
Does asynchronous logging make your application faster?
Asynchronous logging lets your application respond to users without waiting for log writes to finish. The application hands logs to a queue or a collector (such as Vector or Fluent Bit), and the collector handles writing to storage.
This has costs that decision-makers should know about:
1. Logs can be lost. If the buffer lives only in memory and the process dies, unwritten logs are gone. Audit logs need a disk-backed queue and a retry mechanism.
2. It can stop without anyone noticing. If the log pipeline stops, the application keeps running as usual. PCI DSS Requirement 10.7 also requires detecting and responding promptly to failures of audit logging mechanisms. Set an alert for when the volume of logs arriving in storage drops unexpectedly.
3. More to operate. Kafka or a dedicated queue means more servers, more upgrades and more parts that can fail.
Asynchronous logging does not reduce latency to zero. It moves heavy work out of the request path, and in exchange you take on the reliability of the log pipeline.
When should you not change your log architecture?
Leave your log architecture alone when log volume is still small, current queries are fast enough, or nobody on the operations team knows the new system. A self-hosted ClickHouse or Kafka cluster needs people to monitor, back up and upgrade it. For a few million log lines a month, the cost of those people can exceed what you save on storage.
A managed log service from your cloud provider is also a reasonable choice. Its cost grows with the volume you ingest, though, so step 1 (stopping unused logs) still comes first.
If queries are fast enough and the bill has settled after steps 1 and 2, stopping there is the right decision.
Where does VAON start when reviewing a log setup?
VAON has its own infrastructure team, which handles server operations as well as security testing. When a client asks for a log review, the team starts from real numbers, not from technology: daily log volume by type, the queries operators run most, and the retention periods the client's contracts or internal rules require. Only then does the team decide how far through the four steps in section 3 to go.
Retention periods also belong in writing, not only in config files. For ONEBOT, the AI chatbot service VAON operates, the terms of service state that customer data is kept for 30 days after the contract ends and then deleted or anonymized, and that backups and audit logs are deleted in line with the system update cycle. Users know in advance where their logs are and when they disappear.
What should SIers and agencies running many clients put in the contract?
SIers and agencies that operate systems for many clients on a fixed monthly fee carry the most risk when logs grow: storage costs rise, but the fee does not. Agree on these five points from the start:
1. The retention period for each log type, and who decides when it changes.
2. Who can view logs, and through which tool.
3. Which country and which cloud provider the logs are stored in.
4. Whether logs are handed over at contract end, and in what format, or deleted.
5. How the operations fee changes once log volume passes a set level.
Point 5 is the one most often missed. A threshold agreed in advance (for example, monthly volume) saves both sides from renegotiating after the bill has already grown. Point 3 is also the first question a client's security team tends to ask when approving a SaaS service; see AI chatbot security for enterprises: 5 questions to answer before approval. If you are choosing an AI chatbot to offer your own clients, similar data questions are covered in build, resell, or OEM: AI chatbots for agencies and SIers.
Are logs slowing your system down or pushing up your cloud bill every month? Tell VAON how much log data you produce, what types, and how long you need to keep it. The infrastructure team will work out with you how far to go.
References
- Personal Information Protection Commission, Japan, "Guidelines for the Act on the Protection of Personal Information (General Rules)", sections 10-3 and 10-6 (Japanese): https://www.ppc.go.jp/personalinfo/legal/guidelines_tsusoku/
- Microsoft Learn, "Microsoft Entra ID and PCI-DSS Requirement 10" (quotes Requirement 10.5.1 verbatim): https://learn.microsoft.com/en-us/entra/standards/pci-requirement-10
- ClickHouse Docs, "Manage data with TTL (time-to-live)": https://clickhouse.com/docs/guides/developer/ttl