Vivaldi Warehouse Log

Overview

Walking is the single most expensive warehouse operation. Across the industry, it accounts for roughly 55% of total order-picking time (Tompkins et al., Facilities Planning, 4th ed.). At Vivaldi srl, an Italian manufacturing and distribution company, warehouse operators were walking redundant routes every day, backtracking across aisles because their pick sheets listed items in whatever order the ERP happened to spit them out.

I built Vivaldi Warehouse Log to fix that: a full-stack logistics platform that computes optimized walking routes through the physical warehouse and tells operators exactly where to go, in what order. I designed, built, and deployed the system over 2.5 years (September 2019 to March 2022), 319 commits across all branches, 14 TypeScript packages, and a Rust/WASM optimization engine.

Off-the-shelf warehouse management systems from Manhattan Associates, Blue Yonder, or SAP would have cost six to seven figures annually and required months of integration. Vivaldi evaluated that route early and rejected it; the company’s warehouse was too small to justify the licensing cost, and no vendor offered a lightweight integration path with the existing Microsoft SQL Server ERP. What they needed was narrower: a domain-specific optimization layer wired directly into the live database, built and maintained by one engineer.

Problem Statement

Business Context

Vivaldi srl operates a warehouse spanning 9 main aisles (A—I), 1 additional section (J), up to 12 racks per aisle, and up to 12 vertical shelf tiers per rack. Operators fulfill batch orders (ORC documents from the company ERP) by walking the warehouse to collect items.

The Warehouse Layout

The physical warehouse is modeled as a 17-row by 31-column grid. Each cell is either walkable floor, a shelf rack (obstacle), or a named item position. Aisles B through I form a regular grid of double-wide shelf racks separated by corridors. Aisle A runs along the left wall. Section J occupies a separate area in the bottom-right corner.

The warehouse team reorganized the layout multiple times per year, moving shelves, extending aisles, adding or removing sections. Each time, they communicated the new layout, and the system’s code and database had to match.

The interactive map below renders the actual grid the optimization engine operates on. Click any rack to inspect its shelf slots, use arrow keys to walk the operator through the corridors, or press “Play Route” to animate a real batch order through its optimized pick sequence. The route and its logic are explained in the sections that follow.

Layout Reorganization Workflow

Each reorganization required updating both the database (position records) and the optimization engine (the grid the pathfinding algorithm runs on). A code generation pipeline evolved alongside the solver through three generations:

• Generation 1 (Haskell -> TypeScript + SQL): A Haskell program parsed the warehouse map and generated TypeScript source files (grid arrays, position-to-coordinate maps) plus SQL INSERT INTO statements populating every valid position.
• Generation 2 (Haskell -> C++ + SQL): The same Haskell script extended to generate C++ source files when the solver moved to C++.
• Generation 3 (Rust build.rs, compile-time): The Rust build system incorporated code generation directly, eliminating the separate Haskell tool. A build.rs script reads a human-editable grid.txt at compile time and generates a generated.rs file containing: (1) a phf_map! perfect hash map for O(1) position-to-coordinate lookups, (2) a static 17x31 Matrix<u8> walkable/obstacle grid, (3) a lazy singleton via OnceCell, and (4) a coordinate lookup function for the JPS pathfinder. A separate Node.js script generates the SQL INSERT INTO statements for the database.

A layout change required three steps: edit grid.txt, run the SQL generation script, recompile and redeploy. No manual coordinate entry, no risk of mismatched grid-vs-database. Malformed grids fail the build.

How an Order Gets Fulfilled

Multiple times a day, a warehouse operator receives a paper order sheet: a list of hardware components to retrieve, each with a requested quantity. The warehouse stores electronic and mechanical parts, things like ESP32 boards, capacitors, connectors, and sensors.

The fulfillment process starts with a barcode scan. Each order carries a barcode encoding the ERP’s internal order reference. The system reads that barcode, derives a user-facing order ID (an “ORC” code in Vivaldi’s ERP), and pulls every requested item with its associated quantity.

The next step is slot lookup. The database knows every shelf slot in the warehouse and what it contains. A slot is a specific position like A1.1 (aisle A, rack 1, shelf 1), and a single slot holds multiple units of one item, for example 25x ESP32 boards at position C3.7. The same item can appear in multiple slots across different aisles. For each requested item, the system retrieves every slot where that item is available, along with the current quantity on hand.

This item-to-slot mapping feeds into the optimization engine. The engine determines two things: the near-optimal loop the operator should walk through the warehouse, and how many units to pick from each slot along the way. If the warehouse does not have enough stock to fulfill a particular item, the system flags it and caps the pick to available quantities. The operator can still fulfill the remainder of the order.

Supporting Workflows

Route optimization is the core of the system, but operators also need to find items, correct inventory counts, and bulk-update the warehouse when stock changes. The system supports all of this. An operator can search for any item by stock ID and see every slot where it is currently stored, or update the quantity on hand at a specific slot after a physical count.

For larger operations, the warehouse team uses Excel files. A full inventory export (dump.xlsx, one row per slot) gives them an offline snapshot. An Excel import lets them bulk-update the warehouse from a canonical 4-column spreadsheet: slot position (e.g., A4.2), human-friendly item ID (e.g., ESP32S3), available quantity (e.g., 45), and a description. The import can create new slot references or destroy existing ones, so the warehouse team could reorganize without touching the database directly. The system validates the input, then swaps the data into production within a single atomic transaction.

System errors are forwarded to administrators via automated email, batched to avoid flooding during error bursts.

The Core Problem

Without route optimization, operators follow naive pick sequences, visiting positions in whatever order appears on the sheet, often backtracking across aisles. In a real batch order, the unoptimized route jumps between distant positions: C6, then I5, then H2, then back west to pick what was missed.

A real batch order requesting 8 items across aisles A through I illustrates the difference. The optimized route starts near the entrance and sweeps logically through adjacent aisles: A2, B3, A5, C7, E5, I5, H1, H2. Each transition moves to the nearest unvisited position, and the operator never backtracks. You can watch this exact route play out in the interactive map above by pressing “Play Route.” TSP-optimized picking routes reduce walking distance by 47—83% compared to naive sequences, depending on warehouse layout and order size (de Koster et al., “Design and control of warehouse order picking,” European Journal of Operational Research, 2007).

Pathfinding: Shortest Path Between Two Points

Why Pathfinding Matters

The optimization engine needs to know the walking distance between any two warehouse positions. This seems trivial until you account for obstacles: shelf racks block direct paths. An operator at position A2 cannot walk in a straight line to position E5; they must navigate through corridors, around rack blocks, to reach it.

The warehouse grid is 17 rows by 31 columns. Each cell is either walkable floor or an impassable shelf rack. Computing the shortest path between two positions on this grid is a graph search problem, and the optimizer calls this computation hundreds of times per order. A slow pathfinder makes the entire system unusable.

A* and the Symmetry Problem

My first instinct was A*, the standard algorithm for shortest-path computation on weighted grids. It uses a heuristic to guide the search toward the goal, expanding the most promising nodes first. On a uniform-cost grid with an admissible heuristic, A* is optimal: it finds the true shortest path.

The problem is efficiency. On open grids, many optimal paths exist between any two points. Moving right then down costs the same as moving down then right. A* does not know this; it evaluates all symmetric variants, expanding far more nodes than necessary. In a warehouse grid with wide corridors and regular aisles, the symmetry is severe. I profiled the Gen 1 solver and found A* spending most of its time re-discovering the same shortest distance through different orderings of identical-cost moves.

Jump Point Search

I found the solution in a 2011 paper by Daniel Harabor and Alban Grastien: Jump Point Search (JPS). It is an optimization of A* that prunes redundant path evaluations on uniform-cost grids, maintaining optimality while dramatically reducing the number of nodes expanded.

The core idea is directional pruning. When A* expands a node, it typically adds all neighbors to the open set. JPS instead examines the direction traveled to reach the current node from its parent, and prunes any neighbor that could be reached more efficiently through an alternative path. What remains are “natural neighbors,” the nodes that genuinely require expansion.

Straight-line movement: when moving horizontally or vertically, JPS prunes all neighbors except the one directly ahead. It then “jumps” forward in that direction, skipping intermediate nodes entirely, until it hits an obstacle or discovers a forced neighbor.

Diagonal movement: when moving diagonally, JPS first recurses horizontally and vertically from the current position. If either direction reveals something interesting (a forced neighbor or the goal), the algorithm stops and adds that node. Otherwise, it takes one diagonal step and repeats.

Forced neighbors are the key mechanism. An obstacle adjacent to the current path creates a node that cannot be reached optimally except through the current node. When JPS detects a forced neighbor, it stops jumping and adds the current position (a “jump point”) to the priority queue. Only jump points enter the queue, and there are far fewer of them than the total nodes A* would expand.

On Harabor’s benchmarks across game-style grid maps, JPS achieves 3—30x speedup over vanilla A* while maintaining optimality. It requires no preprocessing and no additional memory beyond A*‘s standard open/closed sets.

JPS in This Warehouse

The 17x31 warehouse grid is an ideal JPS target. Long, obstacle-free corridors let the algorithm jump far before encountering forced neighbors. Regular aisle spacing means most jumps traverse the full corridor width without interruption.

The implementation supports 8-directional movement with an octile distance heuristic: cardinal moves cost 1, diagonal moves cost sqrt(2). This improved on the Gen 1 TypeScript solver, which used a Chebyshev heuristic treating all moves (including diagonals) as cost 1.

From position labels to grid coordinates. JPS operates on (x, y) grid coordinates, not on position labels like “A1” or “C7.” The build.rs compile-time code generator (described in “Layout Reorganization Workflow”) reads grid.txt and emits a phf_map!, a compile-time perfect hash map that maps every position label to its grid coordinates. When the optimizer needs the distance between two positions, it first translates both labels to coordinates via this map (O(1) lookup, zero runtime parsing), then passes those coordinates to JPS. The map is regenerated automatically whenever grid.txt changes, so a warehouse layout reorganization cannot produce stale coordinate lookups; if the grid file is malformed, the build fails.

Distance cache. The real performance win comes from caching JPS results, not from raw JPS speed. With ~96 named positions, the maximum pairwise distance cache holds ~4,560 entries. A custom sorted-pair key type exploits the mathematical property that distance(A,B) equals distance(B,A), halving the number of JPS computations required. Distances are quantized as 16-bit unsigned integers (scaled 100x) for compact storage. After a brief warm-up period, nearly all distance lookups are cache hits, and JPS only runs for previously unseen position pairs.

The Optimization Engine

Evolution: Three Generations of Solvers

The optimization engine evolved through three implementations, each driven by the constraints described in the section below.

Generation 1: Pure TypeScript (2019, too slow)

The first solver was pure TypeScript: Jump Point Search with a Chebyshev distance heuristic, solving the Traveling Salesman Problem greedily. It produced correct results on small orders, but large batch orders with 10+ items across all aisles brought the API to its knees. The Node.js event loop blocked for seconds, HTTP requests timed out, and operators stood at the scanning station waiting for a route that would not come. On a 4 GB server already running other Vivaldi services, the memory overhead made things worse.

I knew the algorithm was sound. The problem was the runtime.

Generation 2: C++ with Simulated Annealing via Node-API (2020)

I rewrote the solver in C++ using simulated annealing, bound to Node.js via Node-API (N-API). Route optimization went from seconds to milliseconds. The system finally worked in production.

But the C++ approach introduced a different class of pain. The deployment server ran Debian; binaries compiled on my macOS dev machine would not run there. I needed a C++ toolchain installed on the production server just for one library. Stack traces from the Node.js/C++ boundary were nearly useless for debugging. Every deployment felt fragile.

I lived with these tradeoffs for about two years because the solver worked and the business priority shifted elsewhere.

Generation 3: Rust + WASM with a New Approach (2022, current)

When I revisited the solver in early 2022, I had been writing Rust for other projects and realized WebAssembly could eliminate every deployment problem the C++ version had. WASM runs anywhere Node.js runs; no native toolchain, no cross-compilation, no brittle binaries.

The rewrite was not just a language migration. I replaced simulated annealing entirely with a multi-phase greedy solver using Jump Point Search distance computation. Simulated annealing had always felt like overkill for a warehouse with fewer than 100 positions; it spent compute time exploring random perturbations when the problem space was small enough for a deterministic approach that also incorporated ergonomic priorities (ground-level shelves first, inventory defragmentation) that stochastic optimization cannot naturally express.

The Rust/WASM version compiles to a ~144 KB binary, loads instantly, and produces deterministic output: same input, same route, every time. tsify auto-generates TypeScript types from Rust structs, so the Node.js/Rust boundary is type-checked at compile time. No more mystery crashes at the FFI seam.

The optimization function’s signature stayed stable across all three generations: pass in the requested items and their slot availability, get back the optimized pick sequence. Each new implementation was built alongside the existing one and swapped in once proven (Strangler Fig pattern, applied twice: solver migration and code generation tool migration).

Simulated Annealing vs. Multi-Phase Greedy

The transition from simulated annealing to multi-phase greedy was not just a language migration; it changed the optimization philosophy:

The language migration enabled an algorithmic rethinking that produced better results for operators despite being theoretically suboptimal for pure distance.

The Current Algorithm: Multi-Phase Greedy with Jump Point Search

Given the item-to-slot mapping from the database lookup, the optimizer executes in three phases:

Phase 1: Availability Filtering

Build a sorted map of requested items. Flag unfulfillable orders where demand exceeds supply. Cap requests to available quantities.

Phase 2: Ergonomic Priority (Ground-Level Selection)

Select items from ground-level shelves first (shelves 1, 4, 7, 10). Operators pick these without reaching up or bending down, reducing fatigue and pick time.

Phase 3: Distance-Optimized Selection

For remaining items:

1. Primary sort: by physical distance from starting position using Jump Point Search
2. Tiebreaker: ascending units in position (“smallest-quantity-first,” which drains nearly-empty positions to reduce future inventory fragmentation)
3. Origin shift: after ground-level items, remaining items sort by distance from the last ground-level pick position (or A1 if none), maintaining approximate path continuity

Performance

The REST API measures WASM execution time for every batch order request. Route computation completes in sub-second time; the full API response including database queries stays well within interactive thresholds.

Constraints

Every architectural decision traces back to one or more of these constraints.

4 GB RAM, shared server. The production Debian server was not dedicated to this project. It hosted Vivaldi’s other internal services, and the warehouse log system got whatever memory was left over. This single constraint drove more design decisions than any other: the evolution from memory-heavy TypeScript to a compact ~144 KB WASM binary, streaming CSV uploads via gRPC instead of buffering, Pino.js for structured logging with minimal GC pressure, and process isolation via Unix pipes so each Node.js process managed its own memory independently.

No Docker in production. The IT team explicitly vetoed containerization. I never got a detailed reason; my best guess is that the server already ran a mix of legacy services and they did not want another runtime dependency to manage. The system runs directly on the host OS via systemd, which turned out to be a surprisingly clean deployment model for three coordinated services.

No CI/CD pipeline. Deployment was manual: scp the build artifacts, run an install script, restart the systemd services. For a single-engineer project on a private LAN, this worked. I knew it was a liability, but setting up a proper pipeline would have meant convincing the IT team to open SSH access to a CI runner, and that was not a battle worth fighting for the deployment frequency I had.

Single engineer. I built and maintained the entire system: frontend, backend, gRPC services, optimization engine, database integration, deployment. This meant I could move fast and make aggressive architectural bets (like rewriting the solver twice), but it also meant no code review, no second opinion on database schema decisions, and no one to hand off to. Every abstraction had to be simple enough that I could debug it six months later with no context.

Node.js v16 (forced downgrade). The server’s Debian version could not run Node.js 18. I had to downgrade from 18 to 16 after discovering this during the first deployment attempt, which also meant giving up some language features I had been using in development.

Existing ERP database. The system integrated directly with Vivaldi’s live Microsoft SQL Server ERP. I did not have the luxury of designing my own schema; I read from and wrote to tables that were also being used by the ERP’s own workflows. Stored procedures served as a contract layer, isolating my queries from potential schema changes on the ERP side.

Private LAN only. The system was accessible only within the warehouse’s local network, not from the internet. This meant authentication was unnecessary, which removed an entire layer of complexity. The tradeoff was that remote debugging required VPN access to the company network.

Architecture

System Overview

                      ┌─────────────────────────┐
                      │     React.js Client     │
                      │   (Bulma CSS, Formik)   │
                      └────────────┬────────────┘
                                   │ HTTP/REST
                                   ▼
                      ┌─────────────────────────┐
                      │     Koa.js REST API     │
                      └──┬─────────┬─────────┬──┘
                         │         │         │
           ┌─────────────┘         │         └──────────────┐
           ▼                       ▼                        ▼
 ┌────────────────────┐  ┌────────────────────┐  ┌────────────────────┐
 │ Rust/WASM          │  │ MSSQL Database     │  │ gRPC Services      │
 │ Optimization       │  │ (ERP-integrated    │  │ - Emailer          │
 │ Engine (JPS)       │  │  stored procs)     │  │ - CSV Updater      │
 └────────────────────┘  └────────────────────┘  └────────────────────┘
           │
           │ Unix pipe (stdout)
           ▼
 ┌────────────────────┐
 │ Log transport      │ ──gRPC──▶ Emailer service ──SMTP──▶ Admin inbox
 │ (error batching)   │
 └────────────────────┘

Technology Stack

Two decisions shaped the stack more than any others. First, I chose Koa.js over Express because Koa’s async/await middleware model was cleaner for composing the multi-step request pipeline (parse barcode, resolve items, run optimizer, format response). Express’s callback-based middleware felt unnecessarily noisy for a greenfield project in 2019. Second, I used gRPC for internal server-to-server communication instead of HTTP because the CSV updater needed client-side streaming (sending large files row-by-row without buffering the whole thing in memory), and gRPC handles streaming natively through its protobuf contracts. REST was reserved for the browser-facing API because browser-based gRPC (gRPC-Web) was impractical with React 16 at the time.

Monorepo Structure (14 packages)

The codebase uses a pnpm monorepo (migrated from Lerna to pnpm + Turborepo):

• Frontend: React client application
• Core logic: REST API (central service), warehouse optimization (WASM wrapper), position minimization
• Shared: Common domain models and utilities (100% test coverage), environment configuration (100% test coverage)
• gRPC services: Email forwarding server, CSV warehouse updater server, Pino log transport
• Protobuf contracts: Shared gRPC type definitions for emailer and updater
• Utilities: CSV parser, Excel exporter, shared Jest config

The Rust workspace holds the core optimization library and the WASM bridge.

Key Architectural Patterns

Vertical Slice Architecture: Every REST API module follows an identical structure: router.ts (composition root) -> controller.ts (HTTP concerns) -> manager.ts (business logic) -> repository.ts (data access) -> entity.ts (domain types) -> validator.ts (input validation). This pattern repeats across all 8 API modules.

Facade Pattern (WASM Bridge): The Rust workspace separates into two crates: warehouse-opt (pure domain logic, testable without WASM) and warehouse-opt-wasm (thin Facade annotating types with wasm-bindgen). The core library has zero knowledge of WASM and could target native FFI without touching business logic.

Type Bridge (Rust/TypeScript boundary): The @vivaldi/warehouse-opt TypeScript package bridges the Rust/WASM domain model and the Node.js world. The tsify crate auto-generates TypeScript types from Rust structs, serde handles camelCase/snake_case translation, and WASM serialization details stay hidden from callers.

Polyglot Monorepo: The build graph spans three build systems (grid.txt -> build.rs -> generated.rs -> cargo build -> wasm-bindgen -> npm package -> Turborepo dependency graph), yet Turborepo treats the WASM output identically to any TypeScript package.

The gRPC Architecture (Internal Communication)

The system uses a hybrid REST + gRPC architecture: REST for browser clients, gRPC for server-to-server communication. Browser-based gRPC (gRPC-Web) was impractical in 2019—2020 with React 16.

CSV Updater: gRPC Client-Side Streaming

The warehouse updater uses gRPC client-side streaming to transmit large CSV files. Streaming avoids buffering the entire file in memory, architecturally necessary on a 4 GB server. The protobuf contract defines explicit ParseError and SemanticError types with error codes, ensuring strict validation before any data touches the database.

Emailer: gRPC for Process Isolation

The emailer uses gRPC not for speed but for decoupling. The log transport runs as a separate process, piped from the main service’s stdout:

node rest-api | node pino-grpc-send

If the emailer crashes, the main service continues. If the main service crashes, the emailer remains available.

Log Pipeline Patterns

Pipe-and-Filter: The log pipeline composes three stages: stdin -> split+parse (filters by log level, drops 404s) -> batch+send (accumulates errors, flushes on threshold). The pump library handles backpressure.

Batch-with-Timeout: Errors accumulate and flush either when the batch reaches 10 logs OR when 10 seconds expire, whichever comes first. This prevents email storms during error bursts and excessive delays during quiet periods.

Retry with Fixed Interval: Failed gRPC sends retry at a constant 5-second interval.

Unix Process Composition: Two independent Node.js processes composed via a Unix pipe in a single systemd ExecStart line. The API process writes JSON to stdout and does zero log processing.

Language & Architecture Evolution

The Rust Migration Decision

The C++/N-API solution had worked in production for years. The rewrite was strategic, not reactive. Beyond the deployment and algorithmic improvements described above, the migration also consolidated the project’s language footprint: Rust replaced both the C++ solver and the separate Haskell grid-codegen tool, reducing the project from three compiled languages (TypeScript + C++ + Haskell) to two (TypeScript + Rust). Fewer toolchains means fewer things to break on a shared server with no CI/CD pipeline.

Database Architecture

SQL Server Migration: 2008 to 2019

The system integrates directly with Vivaldi’s ERP database on Microsoft SQL Server. Over the project’s lifetime, the database migrated from SQL Server 2008 to SQL Server 2019. There is no direct upgrade path from 2008 to 2019; it requires either a two-step upgrade (2008 -> 2012 -> 2019) or a backup/restore to a fresh instance. The migration required:

• Replacing custom STRING_AGG polyfills with the native function available in SQL Server 2017+
• Updating stored procedures to leverage modern features
• Explicitly raising the database compatibility level (migrating the engine alone does not enable new features)

The system reads and writes ERP tables directly; no separate database, no synchronization issues (Shared Database Pattern). For a sole-engineer project integrating with a live ERP, this eliminates an entire category of distributed systems problems. The stored procedures serve as a contract layer between the warehouse log system and the ERP, providing a stable interface even if the underlying schema changes.

Position Encoding

Each warehouse position follows {Aisle}{Rack}.{Shelf} — e.g., A12.3 means aisle A, rack 12, shelf 3. The optimization engine parses this into structured data (aisle letter, rack number, shelf number) for distance computation and ergonomic prioritization.

Key Stored Procedures

Deployment Architecture

systemd Services (Service Decomposition)

The system runs as three coordinated systemd services, using systemd’s native dependency graph for orchestration:

All services run with graceful shutdown signals (SIGQUIT) and automatic restart on failure. The main service pipes stdout through a log transport that batches errors and forwards them via gRPC to the email service, which sends SMTP alerts to administrators.

API Surface

The REST API is documented via an OpenAPI 3.0 specification:

Lessons Learned

I should have set up CI/CD from the start. The manual SCP deployment was “good enough” for two years, and I kept telling myself I would automate it later. I never did. The cost was invisible until it was not: at least twice I deployed a build that worked on my machine but failed on the server due to a Node.js version mismatch or a missing native dependency. A basic GitHub Actions pipeline with a staging step would have caught both issues before they reached production.

I have no real performance data. The REST API measures WASM execution time per request and logs it, but those measurements are not persisted anywhere. I cannot tell you the p95 response time over the project’s lifetime, or the cache hit rate, or how memory usage trended over months. I know the system was “fast enough” because operators never complained, but “fast enough” is not a number. If I were building this again, I would add structured metrics from day one: response time percentiles, memory snapshots, distance cache statistics.

The core query has a performance anti-pattern. The stored procedure that resolves order items uses a LIKE '%...' pattern match to filter order codes, which prevents SQL Server from using an index on that column. I wrote it that way because the ERP’s order code format was inconsistent and a leading-wildcard match was the simplest way to handle all variants. A table-valued parameter approach would scale better if order volume increased, but at Vivaldi’s throughput it was never a bottleneck.

The first request after restart is slow. Startup pays three costs at once: WASM module instantiation, grid singleton initialization, and a cold JPS distance cache. Subsequent requests are fast because everything is warm. A single warm-up request fired from the systemd ExecStartPost hook would have eliminated this latency spike entirely. I never added it because restarts were rare and the delay was a few seconds at most.

Development Statistics