Storage model
Every connected app points at one PostgreSQL database. The agent buffers telemetry in a local SQLite file and drains it into thenightowl_* tables: raw events with COPY, issues and users with INSERT … ON CONFLICT upserts.
Because the data lives in your PostgreSQL, you can:
- Connect your own BI tools (Metabase, Grafana, Superset) directly to the
nightowlschema. - Run arbitrary SQL for ad-hoc analysis without hitting an API rate limit.
- Control exactly where the data is geographically stored, for GDPR or data-residency reasons.
- Back it up alongside the rest of your database with tools you already trust.
Raw telemetry vs. rollups
Everynightowl_* table falls into one of four groups, and knowing which is the whole trick to managing storage.
The agent writes rollup rows alongside the raw rows as it drains, in the same transaction. Every dashboard time range, including 1H, buckets at 60 seconds or wider, so the overview cards, throughput and duration charts, percentiles, and the grouped route/query/job lists are all served from the rollup tables, never from the raw events. Longer ranges read the hourly and daily tiers.
Deleting raw rows therefore costs you far less than it looks like. What you lose is the ability to open an individual trace, filter a view by user, or read a stack trace.
Rollup coverage per table
Views filtered by
?user= fall back to raw telemetry because the section rollups carry no user dimension. Those views will be empty over a window you’ve cleared, even for fully-aggregated tables.Retention
You choose how long each layer is kept. NightOwl never deletes data on its own:nightowl:prune does, and only once you schedule it. When it runs, raw telemetry and each rollup tier have their own window, so trends outlive the events that produced them:
Raise any of them to keep more. The only cost is disk on your own PostgreSQL; see PostgreSQL sizing.
On the Hobby plan the dashboard shows the last 14 days. That limits what the dashboard reads, not what is stored in your database.
Pruning
Thenightowl:prune artisan command deletes raw rows older than NIGHTOWL_RETENTION_DAYS, minute rollup buckets older than NIGHTOWL_ROLLUP_RETENTION_DAYS, and hourly and daily buckets older than their own windows. Override the raw and minute windows for a single run with --days=N / --rollup-days=N, or use --hours=N for sub-day raw retention. Schedule it in the app you’re monitoring:
What pruning does to your disk
Raw telemetry tables are partitioned by day — one child table per calendar day, created automatically. On a partitioned table, prune drops each fully-expired day’s partition outright: the drop is instant regardless of row count, leaves no dead rows behind, and hands the disk straight back to the filesystem. Only the retention-boundary day (and, on converted installs, the pre-conversion history) is trimmed with a rowDELETE.
New installs are partitioned from the first nightowl:migrate. Installs that predate partitioning keep working unchanged, but stay on the row-DELETE path until you convert them:
already partitioned. Exit 4 means every conversion landed but at least one table came back owing daily child partitions — usually because creating them queued behind a long-running read. Nothing is broken and no rows are lost: those days’ rows go to the default partition, where prune can only delete them row by row instead of dropping them instantly. A running agent creates the missing children within the hour, or re-run nightowl:partition, which retries the child window even for tables it finds already partitioned. Exit 1 means a table genuinely failed to convert and needs attention; a failed table keeps its rows and its primary key, because the swap runs in one transaction that the failure rolls back. Three things can survive a failure. Two are harmless: a leftover {table}_id_created_at_pt index, which a retry reuses (abandon the conversion and it stays, so drop it by hand), and — on nightowl_logs only — the created_at column already rewritten from the legacy text type, which is irreversible but is the intended end state either way. The third matters: a leftover {table}_hist_ck check constraint. The command removes it on the way out, but a process killed outright cannot, and neither can one whose removal itself fails — that case is written to the agent error log. Left in place it rejects every write to that table once its boundary passes — the boundary is frozen at the second UTC midnight ahead, so that is at least 24 hours out. For the same reason, keep nightowl:partition out of deploy scripts: it is an operator command, and when a pipeline gives up on a long conversion the conversion itself may still be running on the server.
If a conversion is killed outright (a deploy step timing out and taking the container with it), the agent repairs the leftovers and logs that it did, only ever claiming the repair once it has committed. Timing matters here: a leftover boundary check is harmless until its boundary passes at the next UTC midnight, and the agent strips it within about a minute of that moment — not within a minute of the failure. The table behaves normally in between. What tells the agent it is safe to strip is the conversion lock, not the boundary: it only touches a table no conversion holds. If it cannot repair the leftovers (something long-running is holding that table), it logs that instead of retrying in silence, and other tables are still repaired. Re-running nightowl:partition promptly finishes the conversion and removes the leftovers outright, which avoids the midnight window entirely.
On unconverted tables the old PostgreSQL rules apply: a DELETE marks rows dead rather than handing disk back — autovacuum makes that space reusable by new rows, so a table under steady ingest settles at a high-water mark instead of growing without bound, and only VACUUM FULL (exclusive lock) or pg_repack actually shrinks the file. Either way, on a managed volume such as AWS RDS the allocated storage never shrinks back — partition drops free space inside the volume for anything else to use, but the provisioned size stays where it peaked.
Manual cleanup and resets
Selective delete (Data Management)
The Data Management page in the sidebar (Team and Agency plans) has three tabs. Storage shows your storage split across raw, rollups and triage. Retention reports how far back your data actually goes. Clear Data deletes raw telemetry for a date range, narrowed by table and filter. Use it to reclaim disk, to clear noise from a staging incident or a runaway log level, or to remove one user’s data.- Choose data types. Each one shows its rollup coverage and exactly what deleting it keeps and loses.
-
Pick a date range. The most recent data is protected, and how much depends on what you selected:
The 24-hour floor is a safety margin, so you can’t clear the window you’re currently looking at. It isn’t there because rollups lag behind raw rows: they don’t, because the agent commits both in a single transaction. The 7-day floor exists because that data isn’t recoverable from anywhere.
- Filter by route path, status code, duration, user, job status, log level, exception class, or cache event type. Filters only apply to tables where the column exists, and the preview names any filter a table ignores. The cache event type filter offers the event types the Nightwatch SDK records (Hit, Miss, Write, Write failure, Delete, Delete failure) plus older spellings (Set, Put, Forget, Fail) for data recorded before them.
- Preview impact shows counts per table, each tagged with its coverage, before you commit. To delete, type
DELETE(orDELETE PERMANENTLYwhen the selection includes a table that isn’t fully aggregated) and confirm.
Deleting data for one user
Filter by User to remove one person’s data, for example after an erasure request. Besides that user’s raw rows in the selected tables, the deletion also:- deletes their rows in the per-user rollups (the counts on the Users page) inside the window, for requests, jobs and exceptions. This applies only to tables no other filter narrows. If a route, status, duration, job status or exception class filter also applies to a table, that table’s per-user rollups stay, because a per-user count can’t be split that way.
- deletes their stored name and email (
nightowl_users) once nothing about them remains: no raw row in any table, inside the window or outside it, and no per-user rollup row.
nightowl_users row with the other tables, and says which outcome to expect: “This user’s name and email are also deleted. Nothing else about them remains after this deletion.” or “This user’s name and email are kept, because other data about them remains outside this deletion.” If checking for remaining data takes too long, the name and email are kept and the preview says so. The confirm dialog repeats what goes beyond raw rows, for example This also deletes this user’s counts on the Users page for this window, and their name and email.
The protected window still applies. A user active in the last 24 hours (7 days with exceptions or logs selected) keeps their name and email until a later deletion covers that period. To remove everything about a user, select every data type, set no other filters, start the date range before their first activity, and run the deletion once their latest activity is older than the protected window.
Deleting
nightowl_exceptions loses only the per-occurrence stack traces: old occurrences can no longer be opened to see where they were thrown. The issue itself (status, assignee, comments, activity) is kept. An issue is triage state, not telemetry, so clearing exception telemetry never touches it.Delete app (Danger Zone)
From Settings → Danger Zone, the team owner can Delete app. It removes the app from NightOwl along with its agent token and alert routing. It does not drop or truncate yournightowl_* tables: that’s your database, and NightOwl won’t touch it. If you want the telemetry gone too, use Data Management for a targeted range or run php artisan nightowl:clear in the customer app for a full truncate.
Backups
NightOwl doesn’t manage backups; that’s your PostgreSQL provider’s job. Thenightowl_* tables back up the same way the rest of your database does. If you use pg_dump, include the schema you configured (public by default) and you’ll capture everything. If you use managed snapshots (RDS, Cloud SQL), they already cover it.
For point-in-time-restore strategies, treat the nightowl_* tables like any other high-write workload: their WAL volume scales with telemetry throughput, so budget your WAL storage accordingly.
Sizing expectations
As a rough baseline for capacity planning:
At the default 14-day retention these multiply by 14. Rollups add a small, near-constant overhead on top. They scale with the number of distinct route/query/job groups per minute, not with traffic volume. See PostgreSQL sizing for disk, memory, and vCPU guidance.
Exporting data
For compliance exports, legal holds, or moving to another tool, use direct PostgreSQL access:nightowl_* pattern includes the raw tables, the rollups and the nightowl_dict_* dictionaries. The current raw format (the _v2 tables) doesn’t read column by column, though. Repeated values are stored once in the dictionary tables and referenced by id, and large fields in _z columns (such as payload_z and context_z) are compressed with raw DEFLATE. Keep the dictionaries with the raw tables, and inflate the _z columns with zlib’s raw mode (PHP’s gzinflate, for example). The agent package’s migration files document every table and column.