Storage model
Every connected app points at one PostgreSQL database. The agent writes through SQLite’s WAL buffer into ~20nightowl_* tables using COPY for high-volume event tables and INSERT … ON CONFLICT for upsert tables (nightowl_exceptions, nightowl_users).
Because the data lives in your PostgreSQL, you can:
- Connect your own BI tools (Metabase, Grafana, Superset) directly to the
nightowlschema. - Run arbitrary SQL for ad-hoc analysis without hitting an API rate limit.
- Control exactly where the data is geographically stored, for GDPR or data-residency reasons.
- Back it up alongside the rest of your database with tools you already trust.
Raw telemetry vs. rollups
Everynightowl_* table falls into one of three groups, and knowing which is the whole trick to managing storage.
The agent writes a rollup row alongside the raw rows as it drains. Every dashboard time range — including 1H — buckets at 60 seconds or wider, which means the overview cards, throughput and duration charts, percentiles, and the grouped route/query/job lists are all served from the rollup tables, never from the raw events.
Deleting raw rows therefore costs you far less than it looks like. What you lose is the ability to open an individual trace, filter a view by user, or read a stack trace.
Rollup coverage per table
Views filtered by
?user= fall back to raw telemetry because the section rollups carry no user dimension. Those views will be empty over a window you’ve cleared, even for fully-aggregated tables.Retention
Telemetry is high-volume and low-shelf-life. The agent prunes raw telemetry and rollups on separate windows, so trends outlive the events that produced them:Pruning
Thenightowl:prune artisan command deletes raw rows older than NIGHTOWL_RETENTION_DAYS and rollup buckets older than NIGHTOWL_ROLLUP_RETENTION_DAYS. Override either for a single run with --days=N / --rollup-days=N, or use --hours=N for sub-day raw retention. Schedule it in the app you’re monitoring:
What pruning does to your disk
Raw telemetry tables are partitioned by day — one child table per calendar day, created automatically. On a partitioned table, prune drops each fully-expired day’s partition outright: the drop is instant regardless of row count, leaves no dead rows behind, and hands the disk straight back to the filesystem. Only the retention-boundary day (and, on converted installs, the pre-conversion history) is trimmed with a rowDELETE.
New installs are partitioned from the first nightowl:migrate. Installs that predate partitioning keep working unchanged, but stay on the row-DELETE path until you convert them:
already partitioned. Exit 4 means every conversion landed but at least one table came back owing daily child partitions — usually because creating them queued behind a long-running read. Nothing is broken and no rows are lost: those days’ rows go to the default partition, where prune can only delete them row by row instead of dropping them instantly. A running agent creates the missing children within the hour, or re-run nightowl:partition, which retries the child window even for tables it finds already partitioned. Exit 1 means a table genuinely failed to convert and needs attention; a failed table keeps its rows and its primary key, because the swap runs in one transaction that the failure rolls back. Three things can survive a failure. Two are harmless: a leftover {table}_id_created_at_pt index, which a retry reuses (abandon the conversion and it stays, so drop it by hand), and — on nightowl_logs only — the created_at column already rewritten from the legacy text type, which is irreversible but is the intended end state either way. The third matters: a leftover {table}_hist_ck check constraint. The command removes it on the way out, but a process killed outright cannot, and neither can one whose removal itself fails — that case is written to the agent error log. Left in place it rejects every write to that table once its boundary passes — the boundary is frozen at the second UTC midnight ahead, so that is at least 24 hours out. For the same reason, keep nightowl:partition out of deploy scripts: it is an operator command, and when a pipeline gives up on a long conversion the conversion itself may still be running on the server.
If a conversion is killed outright (a deploy step timing out and taking the container with it), the agent repairs the leftovers and logs that it did, only ever claiming the repair once it has committed. Timing matters here: a leftover boundary check is harmless until its boundary passes at the next UTC midnight, and the agent strips it within about a minute of that moment — not within a minute of the failure. The table behaves normally in between. What tells the agent it is safe to strip is the conversion lock, not the boundary: it only touches a table no conversion holds. If it cannot repair the leftovers (something long-running is holding that table), it logs that instead of retrying in silence, and other tables are still repaired. Re-running nightowl:partition promptly finishes the conversion and removes the leftovers outright, which avoids the midnight window entirely.
On unconverted tables the old PostgreSQL rules apply: a DELETE marks rows dead rather than handing disk back — autovacuum makes that space reusable by new rows, so a table under steady ingest settles at a high-water mark instead of growing without bound, and only VACUUM FULL (exclusive lock) or pg_repack actually shrinks the file. Either way, on a managed volume such as AWS RDS the allocated storage never shrinks back — partition drops free space inside the volume for anything else to use, but the provisioned size stays where it peaked.
Manual cleanup and resets
Selective delete (Data Management)
The Data Management page in the sidebar shows your storage split across raw / rollups / triage, reports how far back your data actually goes, and lets you delete raw telemetry for a specific date range, narrowed by table and filter. Use it to reclaim disk, or to clear noise from a staging incident or a runaway log level without waiting for retention to catch up.- Choose data types. Each one shows its rollup coverage and exactly what deleting it keeps and loses.
-
Pick a date range. The most recent data is protected, and how much depends on what you selected:
The 24-hour floor is a safety margin, so you can’t clear the window you’re currently looking at — rollups never lag behind raw rows, because the agent commits both in a single transaction. The 7-day floor exists because that data isn’t recoverable from anywhere.
- Filter by route path, status code, duration, user, job status, log level, exception class, or cache event type. Filters only apply to tables where the column exists.
- Preview shows counts per table, each tagged with its coverage, before you commit. Deletion is then chunked in 10,000-row batches.
Deleting
nightowl_exceptions loses only the per-occurrence stack traces — old occurrences can no longer be opened to see where they were thrown. The issue itself (status, assignee, comments, activity) is kept: an issue is triage state, not telemetry, so clearing exception telemetry never touches it.Delete app (Danger Zone)
From Settings → Danger Zone, Delete app removes the app from NightOwl along with its agent token and alert routing. Does not drop or truncate yournightowl_* tables — that’s your database, and we won’t touch it. If you want the telemetry gone too, use Data Management for a targeted range or run php artisan nightowl:clear in the customer app for a full truncate.
Backups
NightOwl doesn’t manage backups — that’s your PostgreSQL provider’s job. Thenightowl_* tables back up the same way the rest of your database does. If you use pg_dump, include the schema you configured (public by default) and you’ll capture everything. If you use managed snapshots (RDS, Cloud SQL), they already cover it.
For point-in-time-restore strategies, treat the nightowl_* tables like any other high-write workload — their WAL volume scales with telemetry throughput, so budget your WAL storage accordingly.
Sizing expectations
As a rough baseline for capacity planning:
At the default 14-day retention these multiply by 14. Rollups add a small, near-constant overhead on top — they scale with the number of distinct route/query/job groups per minute, not with traffic volume. See PostgreSQL sizing for disk, memory, and vCPU guidance.