Incidents | kitzy.net

Plex, Overseer, and 1 other service are down

Tue, 25 Nov 2025 13:37:00 -0000

Unifi Controller recovered.

Plex, Overseer, and 1 other service are down

Tue, 25 Nov 2025 13:34:01 -0000

Unifi Controller went down.

Plex, Overseer, and 1 other service are down

Tue, 25 Nov 2025 04:13:25 -0000

Plex recovered.

Plex, Overseer, and 1 other service are down

Tue, 25 Nov 2025 04:12:21 -0000

Overseer recovered.

Plex, Overseer, and 1 other service are down

Tue, 25 Nov 2025 04:10:24 -0000

Plex went down.

Plex, Overseer, and 1 other service are down

Tue, 25 Nov 2025 04:09:30 -0000

Overseer went down.

Overseer is down

Sun, 23 Nov 2025 08:29:08 -0000

Overseer recovered.

Overseer is down

Sun, 23 Nov 2025 08:23:01 -0000

Overseer went down.

Overseer is down

Sat, 22 Nov 2025 18:31:55 -0000

Overseer recovered.

Overseer is down

Sat, 22 Nov 2025 18:29:30 -0000

Overseer went down.

Network is down

Tue, 18 Nov 2025 11:46:31 -0000

Network recovered.

Network is down

Tue, 18 Nov 2025 11:34:10 -0000

Network went down.

Authentik, Plex, and 3 other services are down

Fri, 07 Nov 2025 04:39:59 -0000

Fleet recovered.

Authentik, Plex, and 3 other services are down

Fri, 07 Nov 2025 04:38:46 -0000

Authentik and Plex recovered.

Authentik, Plex, and 3 other services are down

Fri, 07 Nov 2025 04:38:46 -0000

Authentik and Plex recovered.

Authentik, Plex, and 3 other services are down

Fri, 07 Nov 2025 04:37:46 -0000

Overseer and Unifi Controller recovered.

Authentik, Plex, and 3 other services are down

Fri, 07 Nov 2025 04:37:46 -0000

Overseer and Unifi Controller recovered.

Authentik, Plex, and 3 other services are down

Fri, 07 Nov 2025 04:35:01 -0000

Unifi Controller went down.

Authentik, Plex, and 3 other services are down

Fri, 07 Nov 2025 04:34:41 -0000

Overseer went down.

Authentik, Plex, and 3 other services are down

Fri, 07 Nov 2025 04:34:20 -0000

Fleet went down.

Authentik, Plex, and 3 other services are down

Fri, 07 Nov 2025 04:33:15 -0000

Authentik went down.

Authentik, Plex, and 3 other services are down

Fri, 07 Nov 2025 04:33:04 -0000

Plex went down.

Network and Unifi Controller are down

Mon, 03 Nov 2025 06:44:36 -0000

Unifi Controller recovered.

Network and Unifi Controller are down

Mon, 03 Nov 2025 06:41:59 -0000

Network recovered.

Network and Unifi Controller are down

Mon, 03 Nov 2025 06:41:46 -0000

Unifi Controller went down.

Network and Unifi Controller are down

Mon, 03 Nov 2025 06:41:16 -0000

Network went down.

Authentik, Plex, and 1 other service are down

Fri, 17 Oct 2025 07:16:34 -0000

Authentik and Plex recovered.

Authentik, Plex, and 1 other service are down

Fri, 17 Oct 2025 07:16:34 -0000

Authentik and Plex recovered.

Authentik, Plex, and 1 other service are down

Fri, 17 Oct 2025 07:13:14 -0000

Authentik went down.

Authentik, Plex, and 1 other service are down

Fri, 17 Oct 2025 07:12:57 -0000

Plex went down.

Authentik, Plex, and 1 other service are down

Fri, 17 Oct 2025 07:09:31 -0000

Unifi Controller recovered.

Authentik, Plex, and 1 other service are down

Fri, 17 Oct 2025 07:07:27 -0000

Unifi Controller went down.

Plex is down

Sat, 11 Oct 2025 16:39:00 -0000

# Root Cause Analysis: Plex Transcoding Failures on Ubuntu 25.04 **Incident date**: October 11, 2025 **Detected**: October 11, 2025 00:37 UTC **Resolved**: October 11, 2025 00:48 UTC **Severity**: High (media streaming unavailable) **Author**: Kitzy ## Executive summary Plex transcoding failed completely on clients following Ubuntu host upgrade from 24.10 to 25.04. All transcode attempts crashed with FFmpeg errors. Root cause was an incompatibility between the LinuxServer.io Plex Docker image's bundled FFmpeg and Ubuntu 25.04. Switching to the official Plex Docker image resolved the issue immediately. **Impact**: 100% of transcoding attempts failed for 11 minutes. Direct play was unaffected. ## Timeline **October 11, 2025 ~00:00**: Upgraded Beelink host from Ubuntu 24.10 to 25.04 **October 11, 2025 00:17**: Upgraded Plex container from ls280 to ls282 **October 11, 2025 00:37**: First transcoding failure detected on AppleTV **October 11, 2025 00:37-00:48**: Troubleshooting and diagnosis **October 11, 2025 00:48**: Switched to official Plex image (plexinc/pms-docker) **October 11, 2025 00:48**: Transcoding confirmed working ## Root cause The LinuxServer.io Plex Docker image (lscr.io/linuxserver/plex) contains a custom-built FFmpeg transcoder that generates commands including the `segment_copyts` option for subtitle segmentation. This option is not recognized by the version of FFmpeg bundled in the LinuxServer image, causing the transcoder to crash with exit code 8. While this bug exists in all versions of the LinuxServer image, it was not triggered on Ubuntu 24.10. The Ubuntu 25.04 environment (likely kernel 6.14.0 or updated DRM libraries) causes Plex's transcoder command generator to include the problematic `segment_copyts` option in its FFmpeg invocations. **Error logs**: ``` ERROR - [Req#12c/Transcode] Unrecognized option 'segment_copyts'. ERROR - [Req#12f/Transcode] Error splitting the argument list: Option not found Jobs: '/usr/lib/plexmediaserver/Plex Transcoder' exit code for process 513 is 8 (failure) ``` ## Contributing factors 1. **Ubuntu version management**: Running non-LTS Ubuntu (24.10) which went EOL, forcing upgrade to 25.04 development release 2. **Hardware requirements**: Original rationale for using 24.10 was newer kernel requirement for Intel N150 hardware 3. **Image selection**: LinuxServer.io image chosen over official image without awareness of potential compatibility issues 4. **Testing gap**: No transcoding tests performed immediately after Ubuntu upgrade 5. **Documentation**: Ubuntu version and Plex image details not documented in wiki ## Resolution Switched from LinuxServer.io Plex image to official Plex image: ```yaml # Before image: lscr.io/linuxserver/plex:latest environment: - PUID=996 - PGID=988 - VERSION=latest # After image: plexinc/pms-docker:latest environment: - TZ=America/New_York ``` Additional changes: - Removed PUID/PGID environment variables (not used by official image) - Fixed config directory ownership: `chown -R 797:797 /mnt/data/docker/volumes/plex` - Retained hardware transcoding configuration (device passthrough and render group) The official Plex image uses a different FFmpeg build that does not have this bug. ## Prevention and action items ### Immediate actions (completed) - [x] Switch to official Plex image - [x] Document Ubuntu version in Current Hardware wiki page - [x] Create troubleshooting runbook for this issue - [x] Verify hardware transcoding works correctly ### Short-term actions - [ ] Evaluate Ubuntu 24.04 LTS with HWE kernel as alternative to 25.04 - [ ] Document specific hardware compatibility issue that required 24.10 originally - [ ] Add transcoding test to post-upgrade checklist - [ ] Set up automated testing for critical services after infrastructure changes ### Long-term actions - [ ] Establish OS selection criteria (prefer LTS over non-LTS) - [ ] Document image selection rationale (official vs community-maintained) - [ ] Implement pre-production testing environment for infrastructure changes - [ ] Add monitoring/alerting for transcoding failures ## Lessons learned ### What went well - Hardware acceleration configuration (device passthrough, render group) was correct - Systematic troubleshooting approach identified root cause quickly - Official Plex image provided immediate resolution without data loss ### What could be improved - **OS stability vs hardware support tradeoff**: Running development release (Ubuntu 25.04) in production introduces unnecessary risk. Should have investigated Ubuntu 24.04 LTS + HWE kernel instead of jumping to non-LTS versions. - **Testing after upgrades**: No transcoding test performed after Ubuntu upgrade. Should have caught this immediately. - **Image selection**: No documented rationale for choosing LinuxServer.io image over official image. Community images may lag behind in compatibility. - **Documentation**: Current Hardware page showed "OS: Ubuntu" without version number, making troubleshooting harder. ### Questions for follow-up 1. What specific hardware feature required kernel >6.8 that 24.04 LTS didn't provide? 2. Does Ubuntu 24.04 LTS + HWE kernel provide adequate support for Intel N150? 3. Should we establish a policy preferring official Docker images over community alternatives? ## Technical details ### Environment - Host OS: Ubuntu 25.04 (kernel 6.14.0-33-generic) - Hardware: Beelink EQ14, Intel N150 (Alder Lake-N), 16GB RAM - Container runtime: Docker via Portainer - Previous image: lscr.io/linuxserver/plex:1.42.2.10156-f737b826c-ls282 - Current image: plexinc/pms-docker:1.42.2.10156-f737b826c ### Verification Intel VA-API drivers confirmed working on host: ``` $ sudo vainfo --display drm --device /dev/dri/renderD128 vainfo: VA-API version: 1.22 (libva 2.22.0) vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 25.1.2 ``` Hardware transcoding confirmed active in Plex dashboard with "HW" indicator. ## Related documentation - Troubleshooting runbook: [Plex transcoding failures on Ubuntu 25.04](https://www.notion.so/289f8d994cff818a9c2df3c6eb9e8325) - Docker Services page: https://www.notion.so/ac72a6815fe745b4954b3b44bc80c9b4 - Current Hardware page: https://www.notion.so/beff2f7c6b4d449b88107f7554032a98

Plex is down

Sat, 11 Oct 2025 01:13:18 -0000

Plex recovered.

Plex is down

Sat, 11 Oct 2025 01:07:00 -0000

Plex went down.

Plex is down

Sat, 11 Oct 2025 00:52:32 -0000

Plex recovered.

Plex is down

Sat, 11 Oct 2025 00:48:40 -0000

Plex went down.

kitzysound.com, Authentik, and 4 other services are down

Wed, 08 Oct 2025 01:09:03 -0000

Fleet recovered.

kitzysound.com, Authentik, and 4 other services are down

Wed, 08 Oct 2025 01:07:38 -0000

Authentik and Plex recovered.

kitzysound.com, Authentik, and 4 other services are down

Wed, 08 Oct 2025 01:07:38 -0000

Authentik and Plex recovered.

kitzysound.com, Authentik, and 4 other services are down

Wed, 08 Oct 2025 01:06:31 -0000

Unifi Controller recovered.

kitzysound.com, Authentik, and 4 other services are down

Wed, 08 Oct 2025 01:05:48 -0000

Fleet went down.

kitzysound.com, Authentik, and 4 other services are down

Wed, 08 Oct 2025 01:05:17 -0000

Authentik and Plex went down.

kitzysound.com, Authentik, and 4 other services are down

Wed, 08 Oct 2025 01:05:17 -0000

Authentik and Plex went down.

kitzysound.com, Authentik, and 4 other services are down

Wed, 08 Oct 2025 01:03:37 -0000

Unifi Controller went down.

kitzysound.com, Authentik, and 4 other services are down

Wed, 08 Oct 2025 00:59:05 -0000

Authentik recovered.

kitzysound.com, Authentik, and 4 other services are down

Wed, 08 Oct 2025 00:58:35 -0000

Plex recovered.

kitzysound.com, Authentik, and 4 other services are down

Wed, 08 Oct 2025 00:55:03 -0000

Authentik and Plex went down.

kitzysound.com, Authentik, and 4 other services are down

Wed, 08 Oct 2025 00:55:03 -0000

Authentik and Plex went down.

Authentik stack failed to deploy (PostgreSQL unhealthy)

Sat, 27 Sep 2025 22:27:00 -0000

# RCA — Authentik stack failed to deploy (PostgreSQL unhealthy) ## Summary New deployments of the Authentik stack failed in Portainer because the PostgreSQL service was repeatedly marked **unhealthy** and (earlier) failed to start. The **root cause** was an incorrect Docker volume mount target for the official `postgres` image, which declares `VOLUME /var/lib/postgresql/data`. We mounted our named volume at the **parent** (`/var/lib/postgresql`) instead of the declared data dir. Docker then created an **anonymous child volume** at `/var/lib/postgresql/data`, **masking the real cluster** on the named volume. A non-standard nested layout (`…/data/data`) and some role/healthcheck misconfigurations amplified the symptoms and obscured the root cause. ## Impact - **User-facing:** New logins to Authentik failed; already-authenticated sessions continued to work. - **Downstream services affected:** AWS (SSO), Portainer, Fleet. - **Start:** **2025-09-27 14:49 EDT** - **End:** **2025-09-27 18:07 EDT** - **Duration:** **3h 18m** - **Data loss:** None observed. After restore, schema contained **178** non-system tables; application state validated as intact. ## Timeline (EDT) - **14:49** – Portainer deploy fails: mount/propagation errors, then `unhealthy` on PostgreSQL. - **15:00–16:30** – Investigation finds valid PG 17 cluster on host volume under `…/_data/data` with ownership uid/gid 999; inside containers, `/var/lib/postgresql/data` appears empty or newly initialized. Health checks produce `role "root"/"postgres" does not exist` messages (clients connecting without a username). - **~16:45** – Root cause identified: named volume mounted at `/var/lib/postgresql` + image’s `VOLUME /var/lib/postgresql/data` ⇒ anonymous child volume masks real data. - **~17:00–18:00** – Recovery plan executed: logical dump (globals + `authentik` DB) from the old cluster, destroy volume, redeploy Postgres 17 with **volume mounted at `/var/lib/postgresql/data`**, restore globals and DB, add `pg_trgm` and `uuid-ossp` extensions, verify counts. - **18:07** – Stack healthy; Authentik available; AWS/Portainer/Fleet auth restored. ## Technical Details ### Root cause 1. **Mount target mismatch for a data-dir image.** The official `postgres` image declares `VOLUME /var/lib/postgresql/data`. Mounting the named volume at the parent path (`/var/lib/postgresql`) led Docker to mount an anonymous child volume at `/var/lib/postgresql/data`, hiding the real database files. Containers subsequently saw an empty/new data directory and either failed `initdb` (“exists but not empty”) or started against the wrong path. 2. **Non-standard nested layout.** The actual cluster lived under `…/data/data`. With `PGDATA` pointed at `/var/lib/postgresql/data`, the server didn’t see `PG_VERSION` and behaved as uninitialized; pointing `PGDATA` at the nested path worked but remained non-standard and error-prone. ### Contributing factors - **Health/monitoring confusion.** Docker healthchecks (and some client probes) connected without `-U`, defaulting to OS user `root` inside the container, producing noisy `role "root" does not exist`. Production monitoring used an **HTTP check from Better Stack**, which caught the outage but didn’t illuminate the DB mount issue. - **Role drift.** The legacy cluster at one point lacked a `postgres` role; probes using `-U postgres` failed. - **Tag drift risk.** Unpinned `library/postgres` tag increases chance of major version bumps during pull. - **No routine backups.** There was no preexisting logical backup job for Authentik; we relied on ad-hoc dumps during incident response. ### What we tried (and why it didn’t stick) - **Chown/chmod/ACL normalization** – necessary, but the masked child volume still hid the real cluster. - **Changing `PGDATA` while mounting the parent** – still subject to masking by the image’s declared VOLUME. - **Creating roles via psql** – failed until we targeted the actual `PGDATA` or used single-user mode, because the container wasn’t looking at the real data directory. ### Resolution - **Deterministic reset:** 1) Dumped roles (`pg_dumpall --globals-only`) and DB (`pg_dump -Fc authentik`) from the old cluster. 2) Removed the old volume (with an extra tarball snapshot as belt-and-suspenders). 3) Redeployed **`postgres:17`** with **named volume mounted at `/var/lib/postgresql/data`** and `PGDATA=/var/lib/postgresql/data`. 4) Restored globals and DB; ensured `pg_trgm` / `uuid-ossp` present; validated 178 non-system tables owned by `authentik`. 5) Updated healthcheck to a **user-agnostic** probe (`pg_isready -q -h 127.0.0.1 -p 5432`). ## Corrective & Preventive Actions ### Config standards (immediately) - **Pin the image** to `postgres:17` in Compose. - **Mount at the child path:** always mount the named volume at **`/var/lib/postgresql/data`** for the official image. Do not mount the parent. - **Standardize DB env consumption:** both `auth-server` and `auth-worker` read `${POSTGRES_USER}`, `${POSTGRES_DB}`, `${POSTGRES_PASSWORD}` (no hardcoding). - **Healthcheck:** use user-agnostic probe (TCP or `pg_isready` with host/port only). Avoid checks that require a specific DB user. ### Operational guardrails (this week) - **Pre-flight verification script** (run before deploy): `docker run --rm -v :/var/lib/postgresql/data --entrypoint sh postgres:17 -lc 'test -f /var/lib/postgresql/data/PG_VERSION || (echo "PG_VERSION missing at target"; ls -la /var/lib/postgresql/data; exit 1)'` - **Detect anonymous child volumes:** flag when a container has both a named mount at `/var/lib/postgresql` and any mount at `/var/lib/postgresql/data`. - **Monitoring:** keep Better Stack HTTP checks; add a DB socket/TCP liveness check on port 5432 from within the host or a sidecar to reduce false attribution. ### Backups & restore (this week) - **Nightly logical backups:** - `pg_dumpall --globals-only > pg-globals.sql` - `pg_dump -Fc authentik > authentik.dump` - Retention: 14–30 days; store off-box (MinIO/S3). - **Quarterly restore drill** into a throwaway container to ensure backups are viable. ### Documentation / runbooks (this week) - “Postgres in Docker” one-pager: **must mount `/var/lib/postgresql/data`**, effects of the image’s `VOLUME`, and nested layout pitfalls. - “Role repair via single-user mode” snippet (for missing `postgres`). - “Restore procedure” (drop/recreate DB vs temp DB swap). ### Owners & dates - **Compose standardization & pinning:** **@kitzy** — **Done** (verify in repo). - **Pre-flight check in CI/Portainer template:** **@kitzy** — **by EOW**. - **Backups to S3/MinIO + retention policy:** **@kitzy** — **by EOW**. - **Runbooks (deploy, backup, restore, role repair):** **@kitzy** — **by EOW**. - **Monitoring additions (DB TCP liveness):** **@kitzy** — **by EOW**. ## Evidence - Host volume: `…/authentik_postgresql/_data/data/PG_VERSION` (PG 17) with full cluster files. - Inside-container (parent mount): `/var/lib/postgresql/data` initially empty or re-initialized; after correct mount (child path), cluster is visible. - Successful restore: `current_user = authentik`, `count = 178` non-system tables. - Final Compose: `Destination=/var/lib/postgresql/data` mounted from named volume; user-agnostic healthcheck.

Authentik stack failed to deploy (PostgreSQL unhealthy)

Sat, 27 Sep 2025 22:27:00 -0000

Authentik stack failed to deploy (PostgreSQL unhealthy)

Sat, 27 Sep 2025 22:07:00 -0000

Stack healthy; Authentik available; AWS/Portainer/Fleet auth restored.

Authentik stack failed to deploy (PostgreSQL unhealthy)

Sat, 27 Sep 2025 22:07:00 -0000

Stack healthy; Authentik available; AWS/Portainer/Fleet auth restored.

Authentik stack failed to deploy (PostgreSQL unhealthy)

Sat, 27 Sep 2025 21:37:00 -0000

Recovery plan executed: logical dump (globals + `authentik` DB) from the old cluster, destroy volume, redeploy Postgres 17 with **volume mounted at `/var/lib/postgresql/data`**, restore globals and DB, add `pg_trgm` and `uuid-ossp` extensions, verify counts.

Authentik stack failed to deploy (PostgreSQL unhealthy)

Sat, 27 Sep 2025 21:37:00 -0000

Authentik stack failed to deploy (PostgreSQL unhealthy)

Sat, 27 Sep 2025 20:45:00 -0000

Root cause identified: named volume mounted at `/var/lib/postgresql` + image’s `VOLUME /var/lib/postgresql/data` ⇒ anonymous child volume masks real data.

Authentik stack failed to deploy (PostgreSQL unhealthy)

Sat, 27 Sep 2025 20:45:00 -0000

Root cause identified: named volume mounted at `/var/lib/postgresql` + image’s `VOLUME /var/lib/postgresql/data` ⇒ anonymous child volume masks real data.

Authentik stack failed to deploy (PostgreSQL unhealthy)

Sat, 27 Sep 2025 20:02:00 -0000

Investigation finds valid PG 17 cluster on host volume under `…/_data/data` with ownership uid/gid 999; inside containers, `/var/lib/postgresql/data` appears empty or newly initialized. Health checks produce `role "root"/"postgres" does not exist` messages (clients connecting without a username).

Authentik stack failed to deploy (PostgreSQL unhealthy)

Sat, 27 Sep 2025 20:02:00 -0000

Authentik stack failed to deploy (PostgreSQL unhealthy)

Sat, 27 Sep 2025 19:28:00 -0000

There was an issue updating a postgres container, I am investigating a fix. Authentication to AWS, Portainer, and Fleet are all impacted. Existing user sessions should continue to work.

Authentik stack failed to deploy (PostgreSQL unhealthy)

Sat, 27 Sep 2025 19:28:00 -0000

There was an issue updating a postgres container, I am investigating a fix. Authentication to AWS, Portainer, and Fleet are all impacted. Existing user sessions should continue to work.

Authentik stack failed to deploy (PostgreSQL unhealthy)

Sat, 27 Sep 2025 18:50:13 -0000

Authentik went down.