Skip to main content
A single NightOwl agent handles ~13,400 payloads/s on modest hardware — enough for most production Laravel apps. When you need more, scale horizontally rather than vertically: run several agents on the same box (or several boxes), fan ingest across them, and share the same PostgreSQL destination.

​
When you need more than one instance

  • Your health dashboard shows buffer depth consistently climbing during peak hours.
  • A single agent can’t keep ingest and drain aligned even after tuning drain workers.
  • You want redundancy — if one agent crashes, the application shouldn’t lose telemetry while it restarts.
If your bottleneck is drain (PostgreSQL writes) rather than ingest (TCP accept), start with NIGHTOWL_DRAIN_WORKERS before adding instances.

​
Scaling drain workers first

Each agent runs one ingest loop plus N drain workers. Drain workers claim rows from the SQLite buffer atomically, so they never race.
A good starting point is one drain worker per PostgreSQL vCPU, capped by how many connections PgBouncer can spare. If you see idle workers in the health dashboard, you’ve overshot.

​
Running multiple agents on one host (Linux)

Linux’s SO_REUSEPORT lets multiple processes bind the same TCP port; the kernel distributes accepted connections across them. Start agents with NIGHTOWL_SO_REUSEPORT=true:
Run the same command as multiple units (e.g. nightowl-agent@1, nightowl-agent@2). Each process gets its own SQLite buffer file but writes to the same PostgreSQL database.
SO_REUSEPORT distributes connections round-robin on Linux only. On macOS the kernel gives every new connection to the first listener, so running multiple agents on the same port locally won’t load-balance — use different ports during development.

​
Running multiple agents across hosts

For horizontal scaling across machines, put a TCP load balancer (HAProxy, nginx stream module, or a cloud LB) in front of the agent pool. Point your application at the load balancer with NIGHTOWL_INGEST_URI — the host:port your app transmits telemetry to:
NIGHTOWL_INGEST_URI (app side) and NIGHTOWL_AGENT_HOST (agent side) are two different knobs. NIGHTOWL_AGENT_HOST only controls the address the agent binds to — set it to 0.0.0.0 (or a specific interface) so the agent accepts connections from other hosts. NIGHTOWL_INGEST_URI controls where the app transmits. Both default to loopback, which is why a co-located single-host install needs neither.
Use a least-connections balancing policy. Agents are stateless from the app’s perspective — any agent can accept any payload — so there’s no sticky-session concern.

​
PostgreSQL and PgBouncer

Every agent instance opens a pool of PostgreSQL connections. Without pooling, N agents × M drain workers quickly exhausts max_connections. The shipped docker-compose.yml includes a PgBouncer container on port 6432:
Transaction-level pooling works cleanly with NightOwl’s workload because drain batches are short-lived transactions.

​
Sizing math

A rough budget for capacity planning: If your app sends 40,000 payloads/s at peak, you need at least three agents and enough drain workers in aggregate to sustain 40,000 rows/s — roughly 8 workers, spread across the three instances, backed by a PostgreSQL instance that can keep up.

​
Verifying the fan-out

Open the health dashboard after starting the second agent. Each instance reports its own row in the Instances table with distinct process IDs, ingest rates, and buffer depths. The sum of the per-instance ingest rates should match your application’s outgoing traffic. If one instance is getting all the traffic, check that SO_REUSEPORT is actually enabled — a common symptom is one agent doing 100% while the others sit at 0%.