Why the Warehouse System Ran on systemd Instead of Docker

Assume the route optimizer works. Now make it run on a shared Debian box with 4 GB of RAM, no Docker by customer request, no external observability, and a live SQL Server database owned by the ERP. That constraint changes what counts as good architecture.

The tempting article would be a stack tour: Koa, gRPC, Pino, systemd, pnpm, Turborepo. I do not think that is the useful version. The useful version is about failure boundaries: keeping the scan-to-route path short, moving bulk imports out of the API process, forwarding errors without letting email code crash picking, and deploying something one person could operate over SSH.

System Architecture

                      ┌─────────────────────────┐
                      │     React.js Client     │
                      │   (Bulma CSS, Formik)   │
                      └────────────┬────────────┘
                                   │ HTTP/REST
                                   ▼
                      ┌─────────────────────────┐
                      │     Koa.js REST API     │
                      └──┬─────────┬─────────┬──┘
                         │         │         │
           ┌─────────────┘         │         └──────────────┐
           ▼                       ▼                        ▼
 ┌────────────────────┐  ┌────────────────────┐  ┌────────────────────┐
 │ Route Optimizer    │  │ MSSQL Database     │  │ gRPC Services      │
 │ (JPS distances +   │  │ (ERP-integrated    │  │ - Emailer          │
 │ native TSP dep)    │  │  stored procs)     │  │ - CSV Updater      │
 └────────────────────┘  └────────────────────┘  └────────────────────┘
           │
           │ Unix pipe (stdout)
           ▼
 ┌────────────────────┐
 │ Log transport      │ ──gRPC──▶ Emailer service ──SMTP──▶ Admin inbox
 │ (error batching)   │
 └────────────────────┘

The system has four major components, connected by three different communication styles. The React frontend talks to the Koa REST API over HTTP. The REST API calls a local optimization package in process, so there is no network hop in the hot path. The API also talks to Microsoft SQL Server through stored procedures, and to two small gRPC services: an emailer and a CSV updater.

The oddest part is the log pipeline at the bottom. The REST API does not process its own logs. It writes structured JSON to stdout, and a separate Node.js process reads that stream through a Unix pipe, filters it, batches errors, and forwards them to the emailer over gRPC. That design looks eccentric until you remember the operating context: one shared machine, no hosted monitoring, and no appetite for a logging stack that could fail in the same process as the picking API.

Technology Stack

Two choices explain most of the stack. I picked Koa over Express because the browser-facing request path was a pipeline: parse the barcode, resolve items from the ERP, run the optimizer, format the response. Koa’s async middleware fit that shape cleanly. I picked gRPC for internal calls because the CSV updater needed client-side streaming, and I wanted typed rows plus one open connection instead of inventing a half-protocol around chunked uploads. REST stayed at the browser boundary because that is where it was simplest.

The Monorepo

The codebase was a pnpm monorepo, migrated from Lerna to pnpm + Turborepo partway through the project. That switch was practical, not philosophical. Lerna ran too much work in sequence. Turborepo gave me a real dependency graph and let unrelated packages build in parallel.

The packages grouped into four buckets.

Frontend. The React client with Bulma and Formik. It only knew about the REST API.

REST API slices. The Koa service plus its route modules. This layer handled barcode-driven requests, ERP lookups, route optimization, and inventory actions.

gRPC services. The emailer, the CSV updater, and the log transport.

Shared and utility packages. Domain types, environment config, Excel export, CSV parsing, and test helpers.

The optimizer itself lived in TypeScript. It used the pathfinding library for JPS distances on the warehouse grid, then passed the resulting matrix to node-tspsolver, an npm package wrapping a native C++ solver. That split was pragmatic under deadline, but it came with a real tax: production installs needed a C++ toolchain on the server for both node-tspsolver and older grpc dependencies. The architecture story is inseparable from that pain.

Vertical Slice Architecture

Every REST API module follows the same six-file structure. router.ts defines route handlers and composes the module. controller.ts handles HTTP concerns: parsing request parameters, formatting responses, setting status codes. manager.ts contains business logic and orchestrates calls between the repository and external services. repository.ts encapsulates database access through stored procedure calls. entity.ts defines domain types. validator.ts contains Joi schemas for input validation.

packages/rest-api/src/
├── items/
│   ├── router.ts          # Route definitions, composition root
│   ├── controller.ts      # HTTP request/response handling
│   ├── manager.ts         # Business logic, orchestration
│   ├── repository.ts      # Database queries via stored procedures
│   ├── entity.ts          # Domain types (Item, InventorySlot)
│   └── validator.ts       # Input validation (Joi schemas)
├── slots/
│   ├── router.ts
│   ├── controller.ts
│   ├── manager.ts
│   ├── repository.ts
│   ├── entity.ts
│   └── validator.ts
├── warehouse/
│   ├── router.ts
│   ├── controller.ts
│   ├── manager.ts
│   ├── repository.ts
│   ├── entity.ts
│   └── validator.ts
└── ... (more modules, same structure)

This pattern repeats across every API module: items, slots, warehouse, orders, health, and the rest. The benefit for a single engineer is navigability. When an endpoint behaves unexpectedly, I know exactly which file to open: a wrong HTTP status code points to the controller, wrong business logic to the manager, bad query data to the repository. Six files, six responsibilities, zero ambiguity about where a given concern lives.

The alternative, grouping all controllers together, all repositories together, all validators together, is the “horizontal layer” approach that scales better for large teams where different engineers own different layers. For one engineer who owns every layer, vertical slices mean a bug in the /items/sort-batch-orders endpoint never requires opening a file outside the items/ directory.

Hybrid REST + gRPC

The system uses REST for everything the browser touches and gRPC for two narrow internal jobs. That split was not ideology. It was the cheapest way to satisfy two different constraints.

The browser needed ordinary request-response APIs for route optimization, lookups, and file downloads. gRPC-Web existed, but it would have required a proxy layer that I did not want to operate on a 4 GB server already running several services. Koa over plain HTTP was enough.

gRPC earned its place elsewhere. The CSV updater needed client-side streaming so the API could send rows one at a time without buffering a full spreadsheet in memory. The emailer needed process isolation so an SMTP failure could not poison the main API process. Those are both good uses of a typed RPC boundary. Everything else would have been architecture cosplay.

The Emailer: gRPC for Process Isolation

The emailer service exists for one reason: crash isolation. On a 4 GB server with no external monitoring (no Sentry, no Datadog, no PagerDuty), the only way I could receive error notifications was through the customer’s SMTP server. If the email-sending code ran inside the REST API process and crashed (a malformed email template, an SMTP timeout, a transient network error), it could bring down the API and stop operators from receiving optimized routes. A bug in the observability layer would make the system both broken and unobservable simultaneously.

The emailer runs as a separate Node.js process with its own systemd service unit. It exposes a single gRPC endpoint: accept a batch of error logs and send them as an email. If it crashes, systemd restarts it automatically. The REST API never notices because the log transport (the process that feeds errors to the emailer) handles the gRPC connection lifecycle independently. If a gRPC send fails, the transport retries at a fixed 5-second interval. Errors accumulate in the transport’s in-memory buffer until the emailer comes back.

The alternative, running the email logic inside the API process and wrapping it in try/catch, would have been simpler to deploy (one process instead of two) but brittle to operate. A bug in email formatting or SMTP handling would pollute the API process’s error state, and on Node.js, an uncaught exception in an async email callback can terminate the entire process. Process isolation makes this impossible: the emailer’s failures are confined to the emailer’s address space.

Log Pipeline

The log pipeline mattered because I had no third-party logging service, no PagerDuty, and no interest in letting email code share a failure domain with the picking API.

Pipe-and-Filter

The REST API writes structured JSON to stdout using Pino.js. Pino was chosen specifically for its minimal garbage collection impact: it serializes log objects to JSON strings with minimal intermediate object allocations, which matters on a memory-constrained server where GC pauses in the logging path would add latency to every API request.

The API process does zero log processing. It writes JSON and moves on. A separate process, connected via a Unix pipe, handles everything else. The pipeline has three stages. stdin receives the raw JSON stream. A split+parse stage deserializes each line and filters by log level: only errors pass through, and 404 responses are dropped because they are expected behavior from health-check probes. A batch+send stage accumulates the filtered errors and forwards them to the emailer via gRPC.

// packages/pino-grpc-send/src/index.ts

import pump from 'pump';
import split from 'split2';
import through2 from 'through2';

const LOG_LEVEL_ERROR = 50;
const BATCH_SIZE = 10;
const FLUSH_INTERVAL_MS = 5_000;

pump(
  process.stdin,

  // Stage 1: split + parse
  split(JSON.parse),

  // Stage 2: filter by level, drop 404s
  through2.obj(function (log, _enc, cb) {
    if (log.level >= LOG_LEVEL_ERROR && log.res?.statusCode !== 404) {
      this.push(log);
    }
    cb();
  }),

  // Stage 3: batch + send via gRPC
  batchAndSend(emailerClient, {
    maxItems: BATCH_SIZE,
    maxWaitMs: FLUSH_INTERVAL_MS,
  }),
);

The pump library composes the stages and handles backpressure: if the gRPC send stage slows down, pump can pause the upstream stages instead of letting the stream tangle itself into a half-broken state. This was still a best-effort notifier, not a durable queue. Errors were batched in memory, retries were finite, and under repeated failure some logs could be dropped. That was a trade-off I accepted because the alternative was much more machinery on a machine that already had very little room.

Batch-with-Timeout

Errors flush to the emailer when either of two conditions is met: the batch accumulates 10 error logs, or 5 seconds have elapsed since the last flush, whichever comes first. The dual threshold prevents two failure modes. Without the count threshold, a burst of 50 errors during a database outage would produce 50 individual emails in rapid succession, flooding my inbox and potentially overwhelming the SMTP server. Without the time threshold, a single isolated error on an otherwise quiet day would sit in the buffer indefinitely, and I wouldn’t learn about it until the next error arrived to fill the batch. Ten logs, five seconds, chosen empirically after observing error patterns during the first months of production operation.

Retry with Fixed Interval

Failed gRPC sends to the emailer retry at a constant interval. No exponential backoff, no jitter, no circuit breaker. The emailer runs on the same machine, so if it is down, it crashed and systemd is restarting it, which takes less than a second. A fixed retry interval is the right pattern for this failure mode because the recovery time is bounded and predictable. Exponential backoff is designed for remote services with variable recovery times. Applying it to a local process restarting on the same host would add unnecessary delay to error delivery.

Unix Process Composition

The entire log pipeline, from API stdout through parsing, filtering, batching, and gRPC send, runs as two processes composed in a single shell command:

node rest-api | node pino-grpc-send

This is exactly the command the main service starts in production. The pipe operator connects the API’s stdout to the transport’s stdin. Each process manages its own memory independently: the API can garbage-collect its heap without affecting the transport, and vice versa. On a 4 GB server shared with other internal services, this memory isolation is not a design luxury. It is a survival mechanism.

The composability matters. I could replace the transport with tee /var/log/warehouse.log for file logging, jq . for readable local output, or /dev/null if I wanted silence. The API process would not care. That is the kind of boring flexibility I trust under pressure.

Deployment: Three systemd Services

The system runs as three coordinated systemd services. systemd’s native dependency graph handles startup ordering:

The email service and CSV updater start independently, with no dependencies on each other or on the main API. The main API’s unit declares a hard dependency on the email service, so systemd ensures the emailer is running before starting the API. If the emailer goes down while the API is running, the log transport’s retry logic handles the gap. The startup dependency governs ordering, not runtime health.

The main unit runs the two-process pipe as one composed service. On shutdown it sends a quit signal that lets the API drain in-flight requests and gives the transport a chance to flush what it can, and on failure it restarts the pair. A shell pipeline is a blunt tool, not a perfect supervisor for each process independently. For this deployment, that level of control was enough.

Deployments happened over SSH: scp the build artifacts onto the server, run an install script that copies files and reloads the systemd daemon, then restart the services. No image builds, no registry, no orchestrator. The last version deployed to production ran on Node.js 16. The customer did not want Docker, and for three services on one machine maintained by one engineer, systemd gave me the parts that actually mattered: startup ordering, automatic restart, graceful shutdown, and journalctl.

I would not ship this exact architecture for a multi-tenant SaaS product. For one warehouse, one ERP, one shared server, and one engineer, it was right-sized. That is why it held up. It kept the hot path short, the risky jobs isolated, and the operational story small enough that I could actually run it.