Minimum Requirements for a Custom Logger, with an Integration Test Checklist

If you can use an off-the-shelf logging framework, that is the safer choice. Even so, there are situations where application constraints or operational circumstances make a custom logger unavoidable. The first thing people agonize over there is how much to implement so the design ends up “neither too sloppy nor too heavy.”

In this article, we narrow the target to application logs used for failure investigation. Rather than taking on audit trails, distributed tracing, a metrics platform, and cloud aggregation all at once, we first define a minimum configuration that is useful in the field, and then lay out the integration test angles needed to make that configuration genuinely trustworthy.

Who this article is for, and what it assumes

Terms used in this article

Before getting into the design, here are the words that later sections use without explanation.

First, check the option of not writing your own

As stated at the top, if you can use an off-the-shelf logging framework, that is the safer choice. Here are specific names so you can make that call. If one of these is enough, you do not need to read the rest of this article.

The minimum requirements below start to matter when circumstances rule these out: you cannot add dependencies, the execution environment is restricted, existing code constrains you, and so on.

The Conclusion First

The essentials to nail in the first version are these.

• Use UTF-8 JSON Lines as the format
• Never break the one-record-per-line rule
• Required fields are timestamp, level, category, message, structured fields, sessionId, and processId
• The baseline is one file per process
• Use synchronous writes at low volume; when output gets heavier, use single writer + bounded queue
• Synchronously flush Error / Critical and the session start and end records
• Include rotation and retention from v1
• When the destination is unavailable, do not silently divert the logs somewhere else

Narrowing things to about this level makes both the implementation and the operations much less likely to fall apart.

Knowledge map for this article

This article sets out the minimum requirements for a custom logger used to investigate failures, covering UTF-8 JSON Lines as the format, mandatory fields including logSessionId and processId, a write design based on a single writer and a bounded queue, flush conditions, rotation and retention, and an explicit notification when saving fails. When the off-the-shelf Microsoft.Extensions.Logging, Serilog, NLog, or spdlog can be used, those take priority, and writing your own is the option for cases constrained in ways such as not being allowed to add dependencies. Building logSessionId from the startup time and the process ID alone lets PID reuse by the OS cause collisions that mix logs together, so the design has to add a GUID. Finally, it shows how to confirm the reliability of a custom logger through integration tests against real files, real threads, and real processes together with automated per-line validation.

flowchart LR
    accTitle: Custom logger
    accDescr: Diagram showing how a custom logger combines JSON Lines, logSessionId, a single writer, a bounded queue, flush, and rotation and retention, how to choose between it and the off-the-shelf frameworks ILogger, Serilog, NLog, and spdlog, and the relationship to the verification points covered in integration tests
    custom_logger["Custom-Built Logger"]
    logger_integration_test["Logger Integration Test"]
    json_lines["JSON Lines (JSONL)"]
    log_session_id["Log sessionId"]
    process_id_log_field["processId (Log Field)"]
    single_writer["Single Writer (One Write Point)"]
    bounded_queue["Bounded Queue"]
    log_flush["Synchronous Log Flush"]
    log_rotation["Log Rotation"]
    log_retention["Log Retention"]
    one_process_one_file["One Log File per Process"]
    multi_process_shared_log_file["Shared Log File Across Processes"]
    sessionid_collision["sessionId Collision"]
    log_record_mixing["Mixed Log Records Across Runs"]
    log_record_validation["Automated Log Record Validation"]
    dotnet_ilogger["Microsoft.Extensions.Logging (ILogger)"]
    diagnostic_app_log["Diagnostic Logs for Troubleshooting"]
    serilog["Serilog"]
    nlog["NLog"]
    spdlog["spdlog"]
    log_storage_failure_handling["Explicit Notice on Log Save Failure"]
    custom_logger -.->|"uses"| json_lines
    custom_logger -->|"requires"| log_session_id
    custom_logger -->|"requires"| process_id_log_field
    custom_logger -.->|"uses"| single_writer
    bounded_queue -->|"requires"| single_writer
    custom_logger -.->|"uses"| bounded_queue
    custom_logger -.->|"uses"| log_flush
    custom_logger -.->|"uses"| log_rotation
    custom_logger -.->|"uses"| log_retention
    one_process_one_file -->|"recommended for"| custom_logger
    multi_process_shared_log_file -->|"not recommended for"| custom_logger
    log_session_id -.->|"prevents"| sessionid_collision
    sessionid_collision -.->|"may cause"| log_record_mixing
    custom_logger -->|"verified by"| logger_integration_test
    logger_integration_test -->|"uses"| log_record_validation
    dotnet_ilogger -->|"recommended for"| diagnostic_app_log
    serilog -->|"recommended for"| diagnostic_app_log
    nlog -->|"recommended for"| diagnostic_app_log
    spdlog -->|"recommended for"| diagnostic_app_log
    custom_logger -.->|"recommended for"| diagnostic_app_log
    log_storage_failure_handling -->|"recommended for"| custom_logger

In the diagram a solid line marks a relation that always holds and a dashed line marks a conditional one (the conditions are given per relation on the detail page). The full list of relations (21 in total, with evidence and certainty) and the definitions of the main concepts are collected on the knowledge map detail page (in Japanese). Data: JSON-LD / Turtle

First, Narrow the Scope

Custom loggers tend to become difficult because they try to handle everything from the start. Try to combine diagnostic logs, audit logs, performance measurement, distributed tracing, and user behavior analytics into one mechanism, and the requirements explode at once.

The target here is diagnostic logs used to isolate application failures. That is, we prioritize being able to trace afterwards when something happened, in which operation, what happened, and what the context was at the time. Just this narrowing makes the initial design decisions considerably easier.

The Minimum Requirements

1. The format is UTF-8 JSON Lines

You can keep logs as concatenated plain text, but they become hard to process mechanically later. Conversely, starting with a heavy proprietary binary format hurts observability in operations.

The workable middle ground is UTF-8 JSON Lines. With one record per line, the file is readable as text and easy to analyze later with scripts and tools. Even if a write is cut off midway, it is easy to isolate which line broke, which is what matters in practice.

The format itself fixes only three things.¹

• Each line is one valid JSON value (no blank lines)
• The line separator is \n
• The encoding is UTF-8. A BOM (U+FEFF) must not be present

The conventional extension is .jsonl. The BOM prohibition is easy to overlook and costly when you do. Write with a BOM and only the first line fails to parse in other tools, surfacing as the confusing symptom of “only the very first line is broken.”

2. Fix the required fields up front

The minimum set of fields to have in place is these seven.

• timestamp
• level
• category
• message
• fields
• sessionId
• processId

An actual record looks like this, for example (wrapped here for the page, but in the real file it is a single line with no newlines).

{"ts":"2026-04-02T01:15:03.4821567Z","level":"Error","category":"import","message":"Import failed","fields":{"file":"orders.csv","row":128,"reason":"date parse failed"},"sessionId":"20260402-101500-8412-9f3c1d2a5b7e4f689a0c3d5e7f1b2c4d","pid":8412}

Lined up with the records around it from the same session, it looks like this.

{"ts":"2026-04-02T01:15:00.1002233Z","level":"Info","category":"startup","message":"Application started","fields":{"version":"1.4.2"},"sessionId":"20260402-101500-8412-9f3c1d2a5b7e4f689a0c3d5e7f1b2c4d","pid":8412}
{"ts":"2026-04-02T01:15:03.4821567Z","level":"Error","category":"import","message":"Import failed","fields":{"file":"orders.csv","row":128,"reason":"date parse failed"},"sessionId":"20260402-101500-8412-9f3c1d2a5b7e4f689a0c3d5e7f1b2c4d","pid":8412}
{"ts":"2026-04-02T01:15:03.9900011Z","level":"Info","category":"shutdown","message":"Shutting down the application","fields":{"exitCode":1},"sessionId":"20260402-101500-8412-9f3c1d2a5b7e4f689a0c3d5e7f1b2c4d","pid":8412}

The key names can be short, but not changing them once decided matters more. Let ts and timestamp get mixed together partway through, and every analysis script you write later becomes a chore.

A string-only log of just message becomes a problem when search criteria multiply later. Conversely, too many fields and the burden on the call sites jumps sharply. It is safest to fix the set at about this size at first, and consider additions only when they are genuinely needed.

What goes into sessionId

Since it is on the required list, you have to decide what unit sessionId counts. In this article, one launch of the process is one session. It is not the user’s logon session, and not a business-level transaction.

With that definition, you can do the following.

• Pull out just the logs from a single launch
• Reconnect the logs from the same launch afterwards, even when rotation split them across files
• Isolate two failures that happened on the same machine in the morning and in the evening without mixing them

Assign the value once at process startup and reuse it until that process ends. Either of the following methods is enough.

Do not settle for startup time plus process ID alone. The OS reuses processId. When a crash loop restarts the process within the same second, or when the local clock steps backward, you can end up with exactly the same sessionId as last time. If the design also puts that value in the file name, then the moment the file is opened in append mode, logs from two different launches land in one file, and the assumption that one launch equals one session quietly stops holding. Worse, the breakage is silent, so you will not notice after the fact.

The reason processId is also carried as its own field is to keep the two roles apart: sessionId says which launch, and processId says which OS-level entity it was at the time.

3. Make one file per process the baseline

A design where multiple processes append to the same file carries more ways to go wrong than it appears to, because mutual exclusion, partial writes, rotation timing, and handling of abnormal termination all get harder at once.

Start with one file per process as the baseline. If you want to combine multiple processes, it is safer to aggregate downstream, or to stand up a dedicated aggregation process explicitly.

4. Split the write strategy by load

While log volume is low, synchronous writes are easier to follow and easier to investigate failures with. Forcing things asynchronous can lose the logs written just before exit, or leave the flush conditions on exceptions ambiguous.

On the other hand, if log volume grows and synchronous I/O becomes the bottleneck, adopt single writer + bounded queue. That is, call sites only enqueue onto a bounded queue and return, and exactly one place writes to the file. The idea itself is nothing unusual: the .NET logging guidance describes the same shape, enqueuing synchronously into an in-memory queue and sending records out from a background worker instead of writing directly to a slow destination.²

What matters here is deciding the overflow policy in advance. Do not leave it vague whether you drop old logs, drop new logs, or emit a warning.

5. Decide the flush conditions

Synchronously flushing Error and Critical, plus the session start and end logs, pays off during a failure investigation. Flushing everything down to routine Info slows things down, so treating every level the same is not realistic.

6. Include rotation and retention from v1

Rotation is often treated as something to add later, but it is a feature whose absence suddenly hurts once you are in operations. The scheme can be anything, whether size-based, daily, or per-launch, but at minimum you should reach a state where “it does not grow without bound” and “how many files are kept” are both decided.

7. No improvised fallback storage on save failure

A design that silently writes somewhere else when the log destination is unavailable makes later investigation difficult. The mere fact that logs are missing from the place they are supposed to be delays the operations team’s initial response to an incident.

If you cannot save, surface the failure through an explicitly visible channel: an in-app notification, the event log, standard error, or the like. At the very least, avoid the state where nobody knows where the logs went.

A Minimal v1 Configuration

For the first version, something like the following is often enough.

• UTF-8 JSON Lines
• One file per process
• Per-session file names
• Size-based or per-launch rotation
• An upper bound on retained files
• Synchronous flush of Error / Critical
• An API that accepts structured fields

For anything beyond this, adding features only after real operations reveal what actually hurt tends to leave you with something easier to maintain.

What the v1 write path looks like in C#

Implementing only the format, the required fields, single writer, and the flush conditions from the requirements above comes to about this much code (C# 12 / .NET 8). Rotation and retention are deliberately left out, on the assumption that they are added in the next step.

using System.Text;
using System.Text.Json;

public sealed class JsonLinesLogger : IDisposable
{
    // Emit non-ASCII text as-is. The default encoder escapes every non-ASCII character to \uXXXX
    private static readonly JsonSerializerOptions JsonOptions = new()
    {
        Encoder = System.Text.Encodings.Web.JavaScriptEncoder.UnsafeRelaxedJsonEscaping,
    };

    private static readonly IReadOnlyDictionary<string, object?> NoFields =
        new Dictionary<string, object?>();

    private readonly object _gate = new();      // this lock is what enforces the single writer
    private readonly StreamWriter _writer;
    private readonly string _sessionId;
    private readonly int _processId = Environment.ProcessId;

    public JsonLinesLogger(string path, string sessionId)
    {
        _sessionId = sessionId;

        // JSON Lines forbids a BOM, so ask explicitly for UTF-8 without one
        _writer = new StreamWriter(
            new FileStream(path, FileMode.Append, FileAccess.Write, FileShare.Read),
            new UTF8Encoding(encoderShouldEmitUTF8Identifier: false));
    }

    public void Write(
        string level,
        string category,
        string message,
        IReadOnlyDictionary<string, object?>? fields = null)
    {
        var record = new Dictionary<string, object?>
        {
            ["ts"] = DateTimeOffset.UtcNow.ToString("o"),
            ["level"] = level,
            ["category"] = category,
            ["message"] = message,
            ["fields"] = fields ?? NoFields,   // never null, so all 7 fields are always present
            ["sessionId"] = _sessionId,
            ["pid"] = _processId,
        };

        // Serialize to JSON before writing, so a message containing newlines cannot break the line
        var line = JsonSerializer.Serialize(record, JsonOptions);

        lock (_gate)
        {
            _writer.WriteLine(line);
            if (level is "Error" or "Critical")
            {
                _writer.Flush();   // synchronously flush only the serious records
            }
        }
    }

    public void Dispose()
    {
        lock (_gate)
        {
            _writer.Flush();   // always write everything out on a normal shutdown
            _writer.Dispose();
        }
    }
}

The call site looks like this. The sessionId is decided once at startup, and the same value is used in the file name.

// Uses the JsonLinesLogger shown above
var startedAt = DateTimeOffset.Now;

// Time and PID alone are not enough. When a crash loop restarts the process within the
// same second, or when the local clock steps backward, a PID that Windows has recycled
// can combine with them to produce the same ID as last time. JsonLinesLogger opens the
// file with FileMode.Append, so records from two different launches then land in one
// file, and the assumption that one launch equals one session quietly stops holding
var sessionId = $"{startedAt:yyyyMMdd-HHmmss}-{Environment.ProcessId}-{Guid.NewGuid():N}";
var logDir = @"C:\ProgramData\MyApp\logs";
Directory.CreateDirectory(logDir);

using var logger = new JsonLinesLogger(Path.Combine(logDir, $"app-{sessionId}.jsonl"), sessionId);

logger.Write("Info", "startup", "Application started",
    new Dictionary<string, object?> { ["version"] = "1.4.2" });

logger.Write("Error", "import", "Import failed",
    new Dictionary<string, object?> { ["file"] = "orders.csv", ["row"] = 128 });

It is short code, but every decision made so far is in there.

UnsafeRelaxedJsonEscaping does not escape <, >, &, or ', so this output must not be embedded as-is into an HTML page or a script element.⁶ Use it only for output meant to be read as a log file.

Common Anti-Patterns

Here are the typical patterns to avoid.

• Cramming everything into the message string
• Sharing the same file across multiple processes
• Going fully asynchronous without deciding the flush conditions
• Postponing rotation and retention
• Silently diverting to another folder on save failure
• Putting network transmission or local DB storage into v1

Each looks convenient at a glance, but they are all items that tend to weigh down isolation and operations.

Think of Integration Tests in Terms of Real Files, Real Threads, Real Processes

A logger is a component that unit tests alone cannot make you comfortable with. Verifying only the string formatting and JSON serialization misses what actually causes problems in production: I/O, concurrency, rotation, flush at shutdown, and permission errors.

So integration tests come down to verifying with real files, real threads, and, where necessary, real processes. At minimum, you want to avoid the state where the logger passes day to day but cannot be trusted during an incident.

Integration Test Items Worth Running

Health of a single write

• Is each line exactly one JSON record?
• Can it be re-read as UTF-8?
• Are the required fields present every time?
• Has an embedded newline broken a record across multiple lines?

Concurrency within the same process

• Do records stay intact when multiple threads write simultaneously?
• Is the record count exactly right, with none missing and none duplicated?
• With a queue in use, do ordering and loss behave per the specification?

Flush and shutdown behavior

• Do Error / Critical records take effect immediately?
• Is the queue empty after a normal shutdown?
• Do the necessary final logs survive on paths close to an exceptional exit?

Rotation and retention

• Does the logger switch to a new file once the rotation condition is met?
• Are old files beyond the retention limit deleted per the specification?
• Do JSON lines stay intact immediately before and after rotation?

Failure paths

• Behavior when the destination directory does not exist
• Behavior when write permission is missing
• Notification or return value when a write fails under disk-full-like conditions
• Behavior on queue overflow

Handling multiple processes

If the specification is one file per process, then the very fact that another process does not try to enter the same file can itself be a verification target. Conversely, with an aggregation-process scheme, verification needs to include handoff failures to that process.

How to Detect Breakage

A list of angles is not yet a test. Of the items above, the one that is hardest to pin down in implementation is how to judge that a record is intact. Eyeballing it is never enough, so check these three things mechanically.

“The first 10 lines look fine” is essentially useless in a concurrent-write test. The line that breaks is always somewhere in the middle. Running through every line is what makes the check meaningful.

Written in PowerShell 7, all three checks look like this. The intent is to call it from test teardown.

# Validate every line of the log that was produced. Throw and fail if even one place is broken
$path     = 'C:\ProgramData\MyApp\logs\app-20260402-101500-8412-9f3c1d2a5b7e4f689a0c3d5e7f1b2c4d.jsonl'
$expected = 10000   # how many times the test called Write

# Checking that the name is present is not enough. A record like `"level": null` still has
# the property name, so a check that only looks at name presence passes right through it.
# To catch a null that slipped in, the check has to look at the value and its type
$required = [ordered]@{
    'ts'        = { param($v) $v -is [string] -and $v -ne '' }
    'level'     = { param($v) $v -is [string] -and $v -ne '' }
    'category'  = { param($v) $v -is [string] -and $v -ne '' }
    'message'   = { param($v) $v -is [string] }   # empty string is allowed, null is not
    'fields'    = { param($v) $v -is [pscustomobject] }
    'sessionId' = { param($v) $v -is [string] -and $v -ne '' }
    'pid'       = { param($v) ($v -is [int] -or $v -is [long]) -and $v -gt 0 }
}

# Check for a BOM on the raw bytes, before anything is decoded.
# Get-Content -Encoding utf8 skips a leading BOM and then hands back the lines, so no
# amount of inspecting the decoded strings will catch a BOM that slipped in
# (on 5.1, use -Encoding Byte instead of -AsByteStream)
$head = @(Get-Content -LiteralPath $path -AsByteStream -TotalCount 3)
if ($head.Count -ge 3 -and $head[0] -eq 0xEF -and $head[1] -eq 0xBB -and $head[2] -eq 0xBF) {
    throw 'The file starts with a UTF-8 BOM, which is invalid for JSON Lines'
}

$lines = @(Get-Content -LiteralPath $path -Encoding utf8)
if ($lines.Count -ne $expected) {
    throw "Line count mismatch. Expected $expected / actual $($lines.Count)"
}

$lineNo = 0
foreach ($line in $lines) {
    $lineNo++

    try {
        $record = $line | ConvertFrom-Json
    }
    catch {
        throw "Line $lineNo does not parse as JSON: $line"
    }

    $names = $record.PSObject.Properties.Name
    foreach ($name in $required.Keys) {
        if ($names -notcontains $name) {
            throw "Line $lineNo is missing a required field: $name"
        }
        if (-not (& $required[$name] $record.$name)) {
            $shown = if ($null -eq $record.$name) { '(null)' } else { "'$($record.$name)'" }
            throw "Line $lineNo has an invalid value for ${name}: $shown"
        }
    }
}

"OK: all $($lines.Count) lines read back as one record per line"

Once it is in this shape, the same code carries straight over to the other tests.

• Concurrent writes: write n times from multiple threads, set $expected to n, and run it
• Rotation and retention: run the same validation over every file left after rotation, and reconcile the line count against the total
• drain at shutdown: run the validation after the shutdown path has executed, and check that the count matches what was enqueued

The failure paths cannot be measured in this shape, so handle them separately. Create the condition first, such as a missing destination directory or missing write permission, then initialize the logger and confirm that the failure surfaces as either an exception, a return value, or a notification. An implementation that swallows the failure with nothing happening at all is the worst kind of breakage to live with in operations.

The Minimum Set of Tests to Pass in v1

Trying to do everything at first makes the tests too heavy. The minimum to pass in v1 is about these six.

1. Normal writes from a single thread
2. Simultaneous writes from multiple threads
3. Flush of Error / Critical
4. Rotation and retention
5. Failure notification when the destination is unavailable
6. Drain and final flush on normal shutdown

Even with just these six passing, you are already a long way from a logger that emits strings but cannot be trusted in operations.

Summary

The first goal of a custom logger is not feature richness but being believable during an incident. To get there, it is effective to fix the format as UTF-8 JSON Lines, keep the required fields tight, make one file per process the baseline, and decide flush, rotation, retention, and failure behavior early.

And whether that design actually works has to be verified with integration tests that use real files, real threads, and real processes. Lock down the minimum configuration and the minimum test set before growing the implementation, and the logger becomes easy to grow later without strain.

References

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

What log format should a custom logger use?

UTF-8 JSON Lines, with the one-record-per-line rule never broken, is the format to reach for. Plain concatenated text is hard to process mechanically later, and a proprietary binary format hurts observability in operations. JSON Lines stays readable as text, is easy to analyze from scripts and tools, and makes it easy to isolate which line broke when a write is cut off midway, which is what matters in practice. Fix the required fields at seven: timestamp, level, category, message, structured fields, sessionId, and processId.

Should log writes be synchronous or asynchronous?

Split the decision by load. While log volume is low, synchronous writes are easier to follow and easier to investigate failures with. Forcing things asynchronous can lose the logs written just before exit, or leave the flush conditions on exceptions ambiguous. If volume grows and synchronous I/O becomes the bottleneck, adopt a single writer with a bounded queue and decide up front whether an overflowing queue drops the old logs or the new ones. Synchronously flushing Error and Critical records plus the session start and end logs pays off during a failure investigation.

Is it acceptable for multiple processes to write to the same log file?

Avoid it. A design where multiple processes append to the same file makes mutual exclusion, partial writes, rotation timing, and handling of abnormal termination all difficult at once, and it carries more ways to go wrong than it appears to. Make one file per process the baseline, and if you want to combine multiple processes, it is safer to aggregate downstream or to stand up a dedicated aggregation process explicitly.

What should integration tests for a custom logger verify?

Verify with real files, real threads, and real processes. Unit tests over string formatting and JSON serialization alone cannot catch the I/O, concurrency, rotation, flush at shutdown, and permission errors that actually cause problems in production. The minimum set worth passing in v1 is six tests: normal single-thread writes, simultaneous multi-thread writes, flush of Error and Critical, rotation and retention, failure notification when the destination is unavailable, and drain plus final flush on normal shutdown. With those six passing, you are a long way from a logger that cannot be trusted in operations.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.