Skip to main content
NightOwl is a bring-your-own-database product. Your telemetry (every request, query, exception, job attempt and log line) lives in a PostgreSQL instance you control. That gives you full ownership of retention, backups and compliance, and it keeps storage costs separate from your NightOwl plan. The Data Management page is on the Team and Agency plans. Deleting data from it needs the owner or admin role on the app.

Storage model

Every connected app points at one PostgreSQL database. The agent buffers telemetry in a local SQLite file and drains it into the nightowl_* tables: raw events with COPY, issues and users with INSERT … ON CONFLICT upserts. Because the data lives in your PostgreSQL, you can:
  • Connect your own BI tools (Metabase, Grafana, Superset) directly to the nightowl schema.
  • Run arbitrary SQL for ad-hoc analysis without hitting an API rate limit.
  • Control exactly where the data is geographically stored, for GDPR or data-residency reasons.
  • Back it up alongside the rest of your database with tools you already trust.

Raw telemetry vs. rollups

Every nightowl_* table falls into one of four groups, and knowing which is the whole trick to managing storage. The agent writes rollup rows alongside the raw rows as it drains, in the same transaction. Every dashboard time range, including 1H, buckets at 60 seconds or wider, so the overview cards, throughput and duration charts, percentiles, and the grouped route/query/job lists are all served from the rollup tables, never from the raw events. Longer ranges read the hourly and daily tiers. Deleting raw rows therefore costs you far less than it looks like. What you lose is the ability to open an individual trace, filter a view by user, or read a stack trace.

Rollup coverage per table

Views filtered by ?user= fall back to raw telemetry because the section rollups carry no user dimension. Those views will be empty over a window you’ve cleared, even for fully-aggregated tables.

Retention

You choose how long each layer is kept. NightOwl never deletes data on its own: nightowl:prune does, and only once you schedule it. When it runs, raw telemetry and each rollup tier have their own window, so trends outlive the events that produced them: Raise any of them to keep more. The only cost is disk on your own PostgreSQL; see PostgreSQL sizing.
Nothing is deleted until you schedule the command. Setting NIGHTOWL_RETENTION_DAYS alone has no effect: it only supplies the default window for nightowl:prune.
On the Hobby plan the dashboard shows the last 14 days. That limits what the dashboard reads, not what is stored in your database.
The Retention tab on the Data Management page reports how far back your raw telemetry and rollups actually go, read from the database.

Pruning

The nightowl:prune artisan command deletes raw rows older than NIGHTOWL_RETENTION_DAYS, minute rollup buckets older than NIGHTOWL_ROLLUP_RETENTION_DAYS, and hourly and daily buckets older than their own windows. Override the raw and minute windows for a single run with --days=N / --rollup-days=N, or use --hours=N for sub-day raw retention. Schedule it in the app you’re monitoring:

What pruning does to your disk

Raw telemetry tables are partitioned by day — one child table per calendar day, created automatically. On a partitioned table, prune drops each fully-expired day’s partition outright: the drop is instant regardless of row count, leaves no dead rows behind, and hands the disk straight back to the filesystem. Only the retention-boundary day (and, on converted installs, the pre-conversion history) is trimmed with a row DELETE. New installs are partitioned from the first nightowl:migrate. Installs that predate partitioning keep working unchanged, but stay on the row-DELETE path until you convert them:
One command, once, on the app you’re monitoring — no agent restart needed, and no data is copied (the existing rows are attached as-is). The Data Management page shows a banner naming any tables still awaiting conversion.
nightowl_logs is the one exception: its conversion rewrites the table under an exclusive lock (legacy date column), so on a large logs table run nightowl:partition in a quiet window — or prune logs first to shrink the rewrite. Ingest keeps buffering throughout; only log reads error until it finishes.
Run it from one place at a time. If another run is still in flight when you start one, the second run reports those tables as skipped and exits 3 — nothing is broken, the other run is doing the same work and a refused run stops before it changes anything, so let it finish and re-run; every table then reports already partitioned. Exit 4 means every conversion landed but at least one table came back owing daily child partitions — usually because creating them queued behind a long-running read. Nothing is broken and no rows are lost: those days’ rows go to the default partition, where prune can only delete them row by row instead of dropping them instantly. A running agent creates the missing children within the hour, or re-run nightowl:partition, which retries the child window even for tables it finds already partitioned. Exit 1 means a table genuinely failed to convert and needs attention; a failed table keeps its rows and its primary key, because the swap runs in one transaction that the failure rolls back. Three things can survive a failure. Two are harmless: a leftover {table}_id_created_at_pt index, which a retry reuses (abandon the conversion and it stays, so drop it by hand), and — on nightowl_logs only — the created_at column already rewritten from the legacy text type, which is irreversible but is the intended end state either way. The third matters: a leftover {table}_hist_ck check constraint. The command removes it on the way out, but a process killed outright cannot, and neither can one whose removal itself fails — that case is written to the agent error log. Left in place it rejects every write to that table once its boundary passes — the boundary is frozen at the second UTC midnight ahead, so that is at least 24 hours out. For the same reason, keep nightowl:partition out of deploy scripts: it is an operator command, and when a pipeline gives up on a long conversion the conversion itself may still be running on the server. If a conversion is killed outright (a deploy step timing out and taking the container with it), the agent repairs the leftovers and logs that it did, only ever claiming the repair once it has committed. Timing matters here: a leftover boundary check is harmless until its boundary passes at the next UTC midnight, and the agent strips it within about a minute of that moment — not within a minute of the failure. The table behaves normally in between. What tells the agent it is safe to strip is the conversion lock, not the boundary: it only touches a table no conversion holds. If it cannot repair the leftovers (something long-running is holding that table), it logs that instead of retrying in silence, and other tables are still repaired. Re-running nightowl:partition promptly finishes the conversion and removes the leftovers outright, which avoids the midnight window entirely.
Point nightowl:partition at your PostgreSQL server directly, not at a transaction-mode pooler (PgBouncer, Supavisor). The conversion holds a session-level lock across steps that a transaction pooler may spread over different backend connections. The command detects this and stops the whole run with a message naming the problem rather than converting unsafely — every remaining table would abort the same way — and the table it stopped on keeps its rows and its primary key. It also tells you when the pooler stranded its lock on a backend it can no longer reach, which is the one state that makes every later run report contention against a run that is not there. The fix is to run it against the database port.
On unconverted tables the old PostgreSQL rules apply: a DELETE marks rows dead rather than handing disk back — autovacuum makes that space reusable by new rows, so a table under steady ingest settles at a high-water mark instead of growing without bound, and only VACUUM FULL (exclusive lock) or pg_repack actually shrinks the file. Either way, on a managed volume such as AWS RDS the allocated storage never shrinks back — partition drops free space inside the volume for anything else to use, but the provisioned size stays where it peaked.

Manual cleanup and resets

Selective delete (Data Management)

The Data Management page in the sidebar (Team and Agency plans) has three tabs. Storage shows your storage split across raw, rollups and triage. Retention reports how far back your data actually goes. Clear Data deletes raw telemetry for a date range, narrowed by table and filter. Use it to reclaim disk, to clear noise from a staging incident or a runaway log level, or to remove one user’s data.
  1. Choose data types. Each one shows its rollup coverage and exactly what deleting it keeps and loses.
  2. Pick a date range. The most recent data is protected, and how much depends on what you selected: The 24-hour floor is a safety margin, so you can’t clear the window you’re currently looking at. It isn’t there because rollups lag behind raw rows: they don’t, because the agent commits both in a single transaction. The 7-day floor exists because that data isn’t recoverable from anywhere.
If this app’s rollup tables are missing or were never backfilled, nothing is aggregated and every table is treated as permanent: the page shows a 7-day floor and a warning instead of the usual coverage badge. Run php artisan nightowl:migrate then php artisan nightowl:backfill-rollups in the monitored app before clearing anything.
If your range reaches past the floor it is shortened rather than rejected, and the preview reports the window actually used.
  1. Filter by route path, status code, duration, user, job status, log level, exception class, or cache event type. Filters only apply to tables where the column exists, and the preview names any filter a table ignores. The cache event type filter offers the event types the Nightwatch SDK records (Hit, Miss, Write, Write failure, Delete, Delete failure) plus older spellings (Set, Put, Forget, Fail) for data recorded before them.
  2. Preview impact shows counts per table, each tagged with its coverage, before you commit. To delete, type DELETE (or DELETE PERMANENTLY when the selection includes a table that isn’t fully aggregated) and confirm.
The deletion runs in the background in 10,000-row batches, so you can leave the page. Stop halts it at the next batch; it doesn’t restore what’s already gone. One deletion runs per app at a time, and each user can start three per hour. Rollup tables are never touched by this flow, so your charts survive. Settings and alert channels aren’t touched either. The one exception is a deletion filtered to a user, described below.

Deleting data for one user

Filter by User to remove one person’s data, for example after an erasure request. Besides that user’s raw rows in the selected tables, the deletion also:
  • deletes their rows in the per-user rollups (the counts on the Users page) inside the window, for requests, jobs and exceptions. This applies only to tables no other filter narrows. If a route, status, duration, job status or exception class filter also applies to a table, that table’s per-user rollups stay, because a per-user count can’t be split that way.
  • deletes their stored name and email (nightowl_users) once nothing about them remains: no raw row in any table, inside the window or outside it, and no per-user rollup row.
The preview lists the per-user rollup rows and the nightowl_users row with the other tables, and says which outcome to expect: “This user’s name and email are also deleted. Nothing else about them remains after this deletion.” or “This user’s name and email are kept, because other data about them remains outside this deletion.” If checking for remaining data takes too long, the name and email are kept and the preview says so. The confirm dialog repeats what goes beyond raw rows, for example This also deletes this user’s counts on the Users page for this window, and their name and email. The protected window still applies. A user active in the last 24 hours (7 days with exceptions or logs selected) keeps their name and email until a later deletion covers that period. To remove everything about a user, select every data type, set no other filters, start the date range before their first activity, and run the deletion once their latest activity is older than the protected window.
Deleting nightowl_exceptions loses only the per-occurrence stack traces: old occurrences can no longer be opened to see where they were thrown. The issue itself (status, assignee, comments, activity) is kept. An issue is triage state, not telemetry, so clearing exception telemetry never touches it.

Delete app (Danger Zone)

From Settings → Danger Zone, the team owner can Delete app. It removes the app from NightOwl along with its agent token and alert routing. It does not drop or truncate your nightowl_* tables: that’s your database, and NightOwl won’t touch it. If you want the telemetry gone too, use Data Management for a targeted range or run php artisan nightowl:clear in the customer app for a full truncate.
Data Management deletions are irreversible; there’s no soft-delete. Take a PostgreSQL backup first if the data has any archival value.

Backups

NightOwl doesn’t manage backups; that’s your PostgreSQL provider’s job. The nightowl_* tables back up the same way the rest of your database does. If you use pg_dump, include the schema you configured (public by default) and you’ll capture everything. If you use managed snapshots (RDS, Cloud SQL), they already cover it. For point-in-time-restore strategies, treat the nightowl_* tables like any other high-write workload: their WAL volume scales with telemetry throughput, so budget your WAL storage accordingly.

Sizing expectations

As a rough baseline for capacity planning: At the default 14-day retention these multiply by 14. Rollups add a small, near-constant overhead on top. They scale with the number of distinct route/query/job groups per minute, not with traffic volume. See PostgreSQL sizing for disk, memory, and vCPU guidance.

Exporting data

For compliance exports, legal holds, or moving to another tool, use direct PostgreSQL access:
The export is plain PostgreSQL and complete: the nightowl_* pattern includes the raw tables, the rollups and the nightowl_dict_* dictionaries. The current raw format (the _v2 tables) doesn’t read column by column, though. Repeated values are stored once in the dictionary tables and referenced by id, and large fields in _z columns (such as payload_z and context_z) are compressed with raw DEFLATE. Keep the dictionaries with the raw tables, and inflate the _z columns with zlib’s raw mode (PHP’s gzinflate, for example). The agent package’s migration files document every table and column.