> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usenightowl.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Management

> Manage monitoring data retention, pruning, and storage for your application.

NightOwl is a bring-your-own-database product. Your telemetry — every request, query, exception, job attempt, and log line — lives in a PostgreSQL instance you control. That gives you full ownership of retention, backups, and compliance, and it keeps storage costs decoupled from per-seat dashboard pricing.

## Storage model

Every connected app points at one PostgreSQL database. The agent writes through SQLite's WAL buffer into \~20 `nightowl_*` tables using `COPY` for high-volume event tables and `INSERT … ON CONFLICT` for upsert tables (`nightowl_exceptions`, `nightowl_users`).

Because the data lives in *your* PostgreSQL, you can:

* Connect your own BI tools (Metabase, Grafana, Superset) directly to the `nightowl` schema.
* Run arbitrary SQL for ad-hoc analysis without hitting an API rate limit.
* Control exactly where the data is geographically stored, for GDPR or data-residency reasons.
* Back it up alongside the rest of your database with tools you already trust.

## Raw telemetry vs. rollups

Every `nightowl_*` table falls into one of three groups, and knowing which is the whole trick to managing storage.

| Group               | Tables                                                                                                         | What it is                                                            |
| ------------------- | -------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| **Raw telemetry**   | `nightowl_requests`, `nightowl_queries`, `nightowl_jobs`, `nightowl_logs`, …                                   | One row per event. Almost all of your disk.                           |
| **Rollups**         | `nightowl_request_rollups`, `nightowl_query_rollups`, `nightowl_job_rollups`, …                                | Per-minute aggregates. Tiny. What the dashboard charts actually read. |
| **Triage & config** | `nightowl_issues`, `nightowl_issue_comments`, `nightowl_settings`, `nightowl_alert_channels`, `nightowl_users` | Never deleted by retention or by Data Management.                     |

The agent writes a rollup row alongside the raw rows as it drains. Every dashboard time range — including **1H** — buckets at 60 seconds or wider, which means the overview cards, throughput and duration charts, percentiles, and the grouped route/query/job lists are all served from the rollup tables, never from the raw events.

**Deleting raw rows therefore costs you far less than it looks like.** What you lose is the ability to open an individual trace, filter a view by user, or read a stack trace.

### Rollup coverage per table

| Raw table                    | Coverage              | Deleting it loses                                                                 |
| ---------------------------- | --------------------- | --------------------------------------------------------------------------------- |
| `nightowl_requests`          | Fully aggregated      | Individual traces, per-route recent-request lists                                 |
| `nightowl_queries`           | Fully aggregated      | Individual query executions                                                       |
| `nightowl_jobs`              | Fully aggregated      | Individual job attempts and their exception payloads                              |
| `nightowl_outgoing_requests` | Fully aggregated      | Individual outgoing calls                                                         |
| `nightowl_cache_events`      | Fully aggregated      | Individual cache events                                                           |
| `nightowl_mail`              | Fully aggregated      | Individual sent messages                                                          |
| `nightowl_notifications`     | Fully aggregated      | Individual notifications                                                          |
| `nightowl_commands`          | Fully aggregated      | Individual command runs — their output, arguments, and per-run child-event counts |
| `nightowl_scheduled_tasks`   | Fully aggregated      | Individual scheduled-task runs and their per-run child-event counts               |
| `nightowl_exceptions`        | **Partly** aggregated | Stack traces (the issue itself — status, assignee, comments, activity — is kept)  |
| `nightowl_logs`              | **Not** aggregated    | Everything. Permanently.                                                          |

<Note>
  Views filtered by `?user=` fall back to raw telemetry because the section rollups carry no user dimension. Those views will be empty over a window you've cleared, even for fully-aggregated tables.
</Note>

## Retention

Telemetry is high-volume and low-shelf-life. The agent prunes raw telemetry and rollups on **separate windows**, so trends outlive the events that produced them:

```env theme={null}
NIGHTOWL_RETENTION_DAYS=14         # raw telemetry
NIGHTOWL_ROLLUP_RETENTION_DAYS=90  # rollups — tiny, so keep them far longer
```

<Warning>
  Nothing is deleted until you schedule the command. Setting `NIGHTOWL_RETENTION_DAYS` alone has no effect — it only supplies the default window for `nightowl:prune`.
</Warning>

## Pruning

The `nightowl:prune` artisan command deletes raw rows older than `NIGHTOWL_RETENTION_DAYS` and rollup buckets older than `NIGHTOWL_ROLLUP_RETENTION_DAYS`. Override either for a single run with `--days=N` / `--rollup-days=N`, or use `--hours=N` for sub-day raw retention. Schedule it in the app you're monitoring:

```php theme={null}
// routes/console.php
Schedule::command('nightowl:prune')->daily();
```

The command issues one `DELETE … WHERE created_at < ?` per table. On PostgreSQL a `DELETE` marks rows dead rather than handing disk back: autovacuum makes that space reusable by new rows, so a table under steady ingest settles at a high-water mark instead of growing without bound. Plain `VACUUM (ANALYZE)` does **not** shrink the data file — only `VACUUM FULL` (which takes an exclusive lock) or `pg_repack` does. On a managed volume such as AWS RDS, allocated storage never shrinks back either, so size the disk for your retention window rather than expecting it to fall after a prune.

## Manual cleanup and resets

### Selective delete (Data Management)

The **Data Management** page in the sidebar shows your storage split across raw / rollups / triage, reports how far back your data actually goes, and lets you delete raw telemetry for a specific date range, narrowed by table and filter. Use it to reclaim disk, or to clear noise from a staging incident or a runaway log level without waiting for retention to catch up.

1. **Choose data types.** Each one shows its rollup coverage and exactly what deleting it keeps and loses.
2. **Pick a date range.** The most recent data is protected, and how much depends on what you selected:

   | Selection includes           | Protected window |
   | ---------------------------- | ---------------- |
   | Only fully-aggregated tables | 24 hours         |
   | Exceptions or logs           | 7 days           |

   The 24-hour floor is a safety margin, so you can't clear the window you're currently looking at — rollups never lag behind raw rows, because the agent commits both in a single transaction. The 7-day floor exists because that data isn't recoverable from anywhere.

<Warning>
  If this app's rollup tables are missing or were never backfilled, **nothing is aggregated** and every table is treated as permanent: the page shows a 7-day floor and a warning instead of the usual coverage badge. Run `php artisan nightowl:migrate` then `php artisan nightowl:backfill-rollups` in the monitored app before clearing anything.
</Warning>

If your range reaches past the floor it is **shortened**, not rejected — the preview reports the window actually used.

3. **Filter** by route path, status code, duration, user, job status, log level, exception class, or cache event type. Filters only apply to tables where the column exists.
4. **Preview** shows counts per table, each tagged with its coverage, before you commit. Deletion is then chunked in 10,000-row batches.

Rollup tables are never touched by this flow, so your charts survive. Neither are settings, alert channels, or user identity.

<Note>
  Deleting `nightowl_exceptions` loses only the per-occurrence stack traces — old occurrences can no longer be opened to see where they were thrown. The issue itself (status, assignee, comments, activity) is **kept**: an issue is triage state, not telemetry, so clearing exception telemetry never touches it.
</Note>

### Delete app (Danger Zone)

From **Settings → Danger Zone**, **Delete app** removes the app from NightOwl along with its agent token and alert routing. *Does not* drop or truncate your `nightowl_*` tables — that's your database, and we won't touch it. If you want the telemetry gone too, use Data Management for a targeted range or run `php artisan nightowl:clear` in the customer app for a full truncate.

<Warning>
  Data Management deletions are irreversible — there's no soft-delete. Take a PostgreSQL backup first if the data has any archival value.
</Warning>

## Backups

NightOwl doesn't manage backups — that's your PostgreSQL provider's job. The `nightowl_*` tables back up the same way the rest of your database does. If you use `pg_dump`, include the schema you configured (`public` by default) and you'll capture everything. If you use managed snapshots (RDS, Cloud SQL), they already cover it.

For point-in-time-restore strategies, treat the `nightowl_*` tables like any other high-write workload — their WAL volume scales with telemetry throughput, so budget your WAL storage accordingly.

## Sizing expectations

As a rough baseline for capacity planning:

| Workload                   | Daily rows | Daily storage (compressed) |
| -------------------------- | ---------- | -------------------------- |
| Small app (\~100 req/s)    | \~8.6M     | \~500 MB                   |
| Medium app (\~1,000 req/s) | \~86M      | \~5 GB                     |
| Large app (\~10,000 req/s) | \~860M     | \~50 GB                    |

At the default 14-day retention these multiply by 14. Rollups add a small, near-constant overhead on top — they scale with the number of distinct route/query/job groups per minute, not with traffic volume. See [PostgreSQL sizing](/performance/postgresql-sizing) for disk, memory, and vCPU guidance.

## Exporting data

For compliance exports, legal holds, or moving to another tool, use direct PostgreSQL access:

```bash theme={null}
pg_dump -h <host> -U <user> -d <db> \
  --table='nightowl_*' \
  --format=custom \
  -f nightowl-export.dump
```

There's no proprietary format — the schema is plain PostgreSQL, and every column is documented in the agent package's migration files.