When you need more than one instance
- Agent Health shows the buffer climbing during peak hours and not coming back down.
- One agent can’t keep ingest and drain aligned even after adding drain workers.
- You want redundancy: if one agent crashes, the others keep accepting telemetry while it restarts.
NIGHTOWL_DRAIN_WORKERS before adding agents.
Scaling drain workers first
Each agent runs one ingest loop plus a number of drain worker processes. Set the number in the.env the agent reads, then restart the agent:
0 or anything that isn’t a number means 1. The agent prints the number it runs with at startup (Drain workers: 4).
With more than one worker, each worker claims its rows in the buffer before writing them, so two workers never write the same row. Rows claimed but not yet written still count as pending, in back-pressure (NIGHTOWL_MAX_PENDING_ROWS) and in the pending rows on Agent Health and /status.
Each worker holds its own PostgreSQL connection. Add workers while the drain rate on Agent Health keeps rising, and stop when it doesn’t: past that point the database is the limit.
Running multiple agents on one host (Linux)
The ingest socket always setsSO_REUSEPORT, so several agents can listen on the same port, and on Linux the kernel spreads new connections across them. There is nothing to turn on. Each agent does need two things of its own:
- Its own buffer, with
--sqlite-pathorNIGHTOWL_AGENT_SQLITE_PATH. A second agent pointed at a buffer another agent is using waits up toNIGHTOWL_BUFFER_LOCK_WAITseconds (30 by default) for it to exit, and refuses to start if it’s still running:NightOwl agent will not start: another agent (pid N) is still using this buffer after 30s.Two agents draining one buffer could write the same rows to PostgreSQL twice. - Its own health port, with
NIGHTOWL_HEALTH_PORT, or no health API at all withNIGHTOWL_HEALTH_ENABLED=false. Otherwise the second agent stops at startup:Could not start the agent: health port 127.0.0.1:2409 is already in use, often by another nightowl:agent on this host.
Running multiple agents across hosts
For horizontal scaling across machines, put a TCP load balancer (HAProxy, the nginx stream module, or a cloud load balancer) in front of the agent pool. Point your application at the load balancer withNIGHTOWL_INGEST_URI, the host:port your app sends telemetry to:
NIGHTOWL_INGEST_URI (app side) and NIGHTOWL_AGENT_HOST (agent side) are two different settings. NIGHTOWL_AGENT_HOST only controls the address the agent binds to: set it to 0.0.0.0 (or a specific interface) so the agent accepts connections from other hosts. NIGHTOWL_INGEST_URI controls where the app sends. Both default to loopback, which is why a co-located single-host install needs neither. With NIGHTOWL_AGENT_HOST=0.0.0.0 the health port (2409) listens on every interface too, so firewall it along with 2407.PostgreSQL connections
Every drain worker holds one long-lived PostgreSQL connection, so the drain needs agents × workers connections: three agents with four workers each hold twelve. On top of that, each agent keeps one connection from its startup checks and opens short-lived ones for the hourly table statistics and for health alerts, and the NightOwl dashboard connects to read your data. Keep the total under the database’smax_connections.
Point the agent at the database’s own port or a session-mode pooler, not a transaction-mode pooler. The quickstart explains why.
Sizing
Measured on a single instance:
In the load test the drain was the limit. It ran with one drain worker; we haven’t published a measurement with more. Size from your own Agent Health numbers: if the drain rate sits below your peak ingest rate for long enough to fill the buffer, add workers first, then agents. See Agent throughput for how the load test was run.
Verifying the fan-out
Open Agent Health after starting the second agent. Each agent reports its own row in the Instances table, namedhostname:pid, with its own Ingest/s, Drain/s and Write Queue. The sum of the Ingest/s column should match your application’s outgoing traffic.
If one agent on a host takes all the traffic, check that the host is Linux, and that each agent really started: an agent that refused to start over a shared buffer or a busy health port leaves only the first one running.