Open Playlist Engine — Design¶

Any-to-any playlist migration across music providers (Spotify, YouTube / YouTube Music, Tidal, Deezer, Apple Music, …) with a sleek UI and strictly separated frontend/backend. This repo is the first reference implementation of the open-playlist spec.

This document is the source of truth for the architecture. It folds in a design review ("rubber-duck") pass — the notable revisions are called out as [rev].

1. What this is¶

open-playlist (the existing repo) is spec-only: an OpenAPI contract + docs defining a universal, provider-agnostic Playlist/Track format with an ISRC-first matching strategy.

This repo is the concrete engine that implements that spec for many providers and lets a user migrate playlists from any source to any target or export portable local files through a UI.

Internal interchange model = explicit Playlist, Track, Album, and Artist entities (app/core/models.py).
Frontend and backend are hard-separated: no shared code. The backend is the single source of truth and publishes OpenAPI; the frontend consumes a generated typed client.

Goals¶

N providers, any combination of source → target, both directions.
Adding a provider is a plugin drop-in, not a core change.
Track matching that gets cheaper and more accurate over time.
Long-running migrations with durable, replayable live progress.
Local, provider-neutral backups with stable versioned schemas.

Current implementation status¶

The self-hosted MVP currently exposes only implemented capabilities in the UI: Spotify, Tidal, and YouTube Music are source/target providers. Their native saved track libraries share the universal PlaylistKind.LIKED_TRACKS kind, so Spotify Liked Songs, Tidal My Collection, and YouTube Music Liked Songs map directly without creating ordinary playlists. Apple Music exposes its implemented official MusicKit library read/search/write capabilities. The persisted job pipeline supports import → match → write with SSE item progress; low-confidence matches are marked needs_review and can be approved, batch-approved, corrected, skipped, or batch-denied from the progress panel. The UI also exposes ledger-backed single-migration history, streamed mixed-entity reports, and all-time aggregate statistics with source/target provider filters. Spotify and Tidal also expose saved albums and followed/favorite artists as explicit library entities. Album/artist jobs never create synthetic playlists. Live playlist selections and playlist portions of terminal migration history can also be exported as CSV, tabular TXT, M3U8, XSPF, or versioned Open Playlist JSON. Self-hosted operators can also opt into immutable metadata-only playlist shares. The public page/download boundary is token-scoped, while recipient provider accounts and migration jobs use a signed share-recipient identity that cannot resolve the owner's local accounts. Persistent one-way sync rules reuse migration jobs for new-track matching and writes, store source/target checkpoints, catch up missed schedules in the self-hosted worker, and expose add-only plus capability-gated mirror controls. A unified Library workspace scans every connected account, groups explicit sync relationships and overlapping same-name provider copies into logical playlists, and shows per-song provider availability. A chosen canonical copy can fan out normal full-playlist migrations to every writable connected account; each successful or review-completed migration then creates the existing durable sync rule. A built-in source-only local-file provider parses TXT, CSV, M3U/M3U8, PLS, WPL, XSPF, XML, and JSON into playlist models before the same match/review/write pipeline begins; local imports do not expose album or artist entities. A separate generator workspace uses an administrator-configured local OpenAI-compatible model by default, with optional Copilot SDK support. It validates structured intents, resolves real provider candidates, and holds an editable private draft before a source_kind="generated" migration job can write anything.

Non-goals (for now)¶

Streaming/playback. We move playlists, not audio.
Providers without a usable playlist-write path (e.g. Amazon Music).

2. Phased flow¶

Generalized from the original Spotify→YouTube framing to any source/target. [rev] the pipeline is explicitly ordered import → match → review → write so matching is decoupled from writing and the user reviews before anything is created.

Phase	Step	Component
0	Get access to source, only when required	Backend auth or bounded import preview
1	Import: fetch playlists/tracks/albums/artists, upload a local file, resolve a public URL, or parse pasted text	Source adapter or import resolver/parser → universal model
2	UI: select supported playlists, songs, albums, and artists	Frontend selection tree
2.5	Optional portable export of provider-backed playlist selections	Export service
3	Get access to target	Backend auth
3.5	Match: resolve tracks, albums, and artists on the target	Core match services
3.6	Review: confirm/fix low-confidence matches	Frontend review queue
4	Write: create playlists and add tracks, idempotently	arq job + operation ledger
5	UI shows live progress	Frontend SSE progress board

Playlist generation enters before the normal write pipeline:

Generator phase	Step	Component
G0	Prompt plus explicit music controls	React Generator workspace
G1	Bounded structured search intents	local OpenAI-compatible model or optional Copilot SDK
G2	Resolve and deduplicate real target tracks	target adapter + `MatchService`
G3	Rename, reorder, add/remove/replace, approve, regenerate	private generation draft
G4	Explicit confirmation snapshots approved URIs	`MigrationJob` + `JobItem`
G5	Create and add through durable write flow	arq worker + operation ledger

Public URL and pasted-text sources¶

POST /api/imports/url-preview and POST /api/imports/text-preview normalize external sources before a migration exists; the binary POST /api/imports/preview route remains dedicated to local playlist files. Provider URLs pass through an exact host/path resolver registry and delegate public reads to adapter hooks. YouTube Music supports an unauthenticated public client and Apple Music uses the configured catalog developer token. Spotify and TIDAL reuse owner-bound authenticated reads and return a structured source-connection action when no matching account exists. Open Playlist Engine /share/{token} links fetch only their bounded /api/public/shares/{token} snapshot JSON.

Text parsing is deterministic and bounded: comments/blanks are ignored, common artist - title and tabular forms are recognized, duplicates and Unicode are preserved, and malformed rows produce line-level issues without discarding valid neighbors.

Local files, public URLs, and pasted text share the private, lease-backed local_playlist_import table. Each ready preview is owner-scoped; migration creation atomically queues the real import record ID as source_account_id, and the worker renews the lease while loading the exact normalized snapshot. Queue delays, retries, or later source changes therefore cannot change the selected playlist. After that source seam, imported tracks use the unchanged matching, review, progress, duplicate, and write paths.

Safe migration defaults¶

Spotify → YouTube Music conversion is deliberately boring and slow by default: 1 playlist per job, 50 selected tracks per job, 250 target tracks per day, and 120 seconds between jobs. POST /api/migrations performs the preflight and returns a 409 warning payload when a job exceeds those defaults or when a same-name target playlist has completely different songs. The frontend shows a confirmation popup and resubmits with acknowledge_warnings=true only after user acknowledgement.

Partial reruns and duplicate handling¶

Completed job_item rows are the private migration ledger. Playlist and track read responses can include migration status when the frontend supplies target context, so a rerun can label a source playlist as partially migrated and mark leftover songs. The worker reuses a previously observed target playlist, or a same-name target playlist whose songs overlap, and skips duplicate target songs with a per-item reason instead of adding them twice.

Migration history, statistics, and reports¶

Single-migration history and aggregate statistics read from migration_job and job_item instead of maintaining a second analytics store. The same ledger rows that power progress, review, rerun detection and duplicate handling provide status buckets (written, skipped, needs_review, failed, matched, pending), per-entity counts, filterable track/album/artist inspection, and streamed CSV/JSON reports. Every query joins through the server-resolved owner; account labels are resolved only from that user's current accounts.

Terminal jobs persist lifecycle timestamps, warnings, a compact result summary, and an item-detail expiry. Summary history remains indefinitely. The ARQ worker periodically snapshots and deletes expired job_item/operation_ledger rows; accepted review decisions live in a separate private table so retention cleanup does not regress future match suggestions. Review decisions carry an explicit entity type and are validated against the target adapter before reuse, preventing cross-entity candidate bleed. See MIGRATION_HISTORY.md for the stable report schema and configuration.

Portable local exports¶

Portable exports branch directly from the universal model before matching or writing to another provider. POST /api/exports reads one selected playlist at a time, serializes into a temporary file, and returns a cancellation-safe streamed response. One playlist downloads directly; multiple playlists always use a ZIP with a versioned manifest.json. JSON archives contain one multi-playlist bundle, while formats that represent one playlist contain one sanitized, collision-safe entry per playlist.

Completed and failed migration history is exportable without reconnecting the source. JobItem.source_metadata preserves track metadata, and migration jobs store a small playlist-level snapshot (excluding tracks) in their existing selection JSON. Older jobs remain exportable with explicit metadata warnings.

CSV/TXT serializers neutralize spreadsheet formula prefixes. XSPF strips XML-illegal controls and escapes entities. M3U8/XSPF normalize known Spotify and Tidal track URIs to web URLs while retaining source URIs in format metadata. Empty playlists, partial selections, unsupported media, missing URIs, and per-playlist read failures remain valid output with warnings. Authentication and rate-limit errors abort immediately rather than repeatedly calling the provider.

Playlist organizer¶

Organizer is a separate maintenance flow, not a migration shortcut. The frontend defaults to unfollow_playlist, exposes delete_playlist only as an explicit irreversible mode, and lets users select exact song entries only when the adapter advertises remove_tracks. Preflight always resolves ownership and provider capabilities server-side; it never substitutes permanent deletion for a requested safe removal.

Destructive playlist deletion and song removal require an exact typed phrase. Jobs persist one organizer_item per playlist/action, report partial failures without hiding successes, and retry only failed/retryable items. Duplicate analysis is a read-only review aid using normalized name, compatible owner identity, and track overlap; it never creates an organizer selection.

Scheduled synchronization¶

A rule starts from one completed full-playlist migration, so the source/target account and playlist relationship is already proven. The worker evaluates due rules at startup and every minute. Each run stores deterministic source/target snapshots and creates a normal migration job for newly added tracks; add-only uses the existing duplicate reconciliation, while mirror uses a match-only job followed by one ordered target replacement. Review-required rules stop scheduling until the migration review is resolved and the sync finalizer commits the checkpoint.

Only one queued/running run may exist per rule. A database partial unique index, transactional row locks and run lease tokens prevent overlaps and stale workers from committing. Transient failures receive a shorter retry schedule, expired credentials auto-pause the rule, and inverse endpoint rules are rejected to prevent feedback loops.

3. Architecture¶

Hub-and-spoke (O(N), not O(N²))¶

Every provider is a spoke; the universal Open Playlist format is the hub. Migration is source.read() → OpenPlaylist → target.write(). Add a provider once and it works with all others, both directions.

Spotify ┐                                  ┌ YouTube / YT Music
Tidal   ┼─ read → [ OPEN PLAYLIST hub ] → write ─┼ Tidal
Deezer  ┤            (identity graph)           ├ Deezer
Apple   ┘                 │                └ Apple
                          └─ export → local portable files

Frontend / backend separation¶

Backend owns all OAuth/tokens, provider API calls, matching, generation model calls and private drafts, jobs, export serialization/orchestration, organizer preflight, and destructive confirmation enforcement, plus local-file parsing and expiring normalized previews. Emits OpenAPI.
Frontend owns the source→target wizard, generator controls, organizer workbench, selection, review, progress, history, exports, and sharing. It consumes a client generated from the backend OpenAPI. No business logic, no provider secrets.

Deployment model — [rev]¶

v1 targets self-hosted, single-user, but every multi-tenant seam is present so the same codebase can run hosted. A single OPE_DEPLOYMENT_MODE (self_host | hosted) flag drives the differences, and secret handling goes through a pluggable KeyProvider (env-derived Fernet now; KMS later). Examples: - header/cookie-paste auth is allowed only in self-host (allow_header_paste). - the shared match graph stays local unless explicitly enabled. - migration ownership is resolved by a server-side dependency. Self-host returns the local user; hosted mode rejects migration requests until real authentication is wired, rather than trusting a query-string user ID. - configuring a public base URL turns on an owner-session gate for every private self-host API. Public share reads are separate, and recipient writes require a provider account connected under that recipient's signed share session.

4. Tech stack¶

Backend: Python 3.12, FastAPI, SQLAlchemy 2 (async) + Alembic, arq (async jobs on Valkey), Pydantic v2 mirroring the Open Playlist schema.
Frontend: Vite + React + TypeScript, typed client generated from the backend OpenAPI, SSE for progress.
Data: Postgres (accounts, encrypted credentials, jobs, identity graph), Valkey (job queue + pacing).
Infra: docker compose (backend, worker, frontend, postgres, valkey), built with --no-cache.
Portable files: stdlib CSV, ZIP64, XML, JSON, and temporary-file streaming; no cloud storage or delivery service.

Local-file trust boundary¶

Local imports use an application upload endpoint, never an arbitrary host path. The backend streams each request into a bounded spooled temporary file, rejects configured byte/playlist/track limits, blocks XML entities and document types, and never opens paths referenced by M3U, PLS, WPL, XSPF, XML, or JSON entries. The raw stream is closed after parsing. Only normalized Playlist/Track JSON and bounded validation issues are retained in an owner-scoped expiring row. Queued jobs lease that row; successful jobs delete it atomically, while failures retain a short retry grace before scheduled cleanup.

YouTube write path¶

Default: ytmusicapi (unofficial) — real YouTube Music, no quota, actively maintained. Enabled by default; marked EXPERIMENTAL.
Optional: official YouTube Data API v3 — clean OAuth but ~66 songs/day on the default 10k quota (search.list=100, playlistItems.insert=50). Off by default, behind a flag.

5. Provider plugin contract¶

A provider implements ProviderAdapter (app/core/adapter.py), declares a CapabilityDescriptor, and registers itself.

[rev] Adapters do not own matching¶

Adapters expose only read/search/write primitives. They never read or write the identity graph — the core MatchService owns caching, scoring and promotion. This keeps a bad match in one context from silently becoming global truth.

class ProviderAdapter(Protocol):
    info: ProviderInfo
    auth: AuthStrategy

    # READ (async + paginated)
    def iter_playlists(self, cred) -> AsyncIterator[PlaylistRef]: ...
    def iter_playlist_items(self, cred, ref) -> AsyncIterator[Track]: ...
    async def read_playlist(self, cred, ref) -> Playlist: ...
    async def test_connection(self, cred) -> None: ...

    # SEARCH (used by MatchService; returns candidates, scores nothing)
    async def search_tracks(self, cred, track, *, limit=5) -> list[TrackCandidate]: ...
    async def validate_uri(self, cred, uri) -> bool: ...

    # WRITE (idempotency handled by the core operation ledger; per-item results)
    async def create_playlist(self, cred, spec) -> str: ...
    async def add_tracks(self, cred, playlist_id, uris) -> list[AddItemResult]: ...

Adapters that advertise both REMOVE_TRACKS and REORDER may also implement the optional MirrorProviderAdapter.replace_playlist_tracks(...) contract. The scheduler checks both the capability descriptor and structural protocol before exposing mirror. Spotify is the initial mirror target; saved/liked collections remain add-only. Album and artist capabilities use four optional contracts: SavedAlbumReader/SavedAlbumWriter and FollowedArtistReader/FollowedArtistWriter. This keeps each entity and direction independently gateable. The core verifies both the capability and the exact operation-specific protocol before calling it.

Registration & trust boundary — [rev]¶

Adapters self-register via app.core.registry.register(...); third parties can ship adapters as importlib.metadata entry points (group ope.providers).
Hosted mode runs an allow-list of signed/vetted plugins. Self-host trusts locally installed modules. The registry is the choke point.

Contract rules every adapter MUST honor¶

Map to/from the Open Playlist model only — never leak provider types.
Populate ISRC on read when available; set provider_uris[self.name].
Search only — return TrackCandidates; never touch the graph.
Writes are replayable via the operation ledger (see §9) — no "dedupe by name" guessing. [rev]
Raise typed errors (RateLimited, AuthExpired, NotFound, Unsupported) — never leak HTTP. Pacing is centralized (see §9), not per-adapter. [rev]
Capability honesty: advertise only what's implemented and tested.
No global state; everything flows through cred → multi-account safe.

Fidelity contract — [rev]¶

v1 migrates flat, music-track playlists. Non-songs (podcast episodes, videos, local files) and folders are not silently dropped: each carries an unsupported_reason and is surfaced in a per-job lossy report. Track exposes media_type, is_local, position and is_migratable for this.

Conformance suite — [rev]¶

Core ships tests/conformance/ with a fake in-memory provider and a contract test suite (protocol satisfied, read round-trip preserves ISRC, search returns candidates, create→add reports per-item results, typed errors). Real adapters parametrize the same suite against recorded fixtures / canaries — never live APIs in CI.

6. Auth abstraction¶

Providers differ wildly; we collapse them into a few strategy kinds, each with one lifecycle, so the frontend needs only three generic "connect" UIs.

class AuthStrategy(Protocol):
    kind: AuthKind  # OAUTH_PKCE | OAUTH_DEVICE | HEADER_PASTE | DEVELOPER_USER_TOKEN | LONG_LIVED_TOKEN
    async def begin(self, *, user_id, account_label=None) -> AuthChallenge: ...
    async def complete(self, *, user_id, callback) -> ProviderCredential: ...
    async def refresh(self, cred) -> ProviderCredential: ...
    async def revoke(self, cred) -> None: ...

Three challenge shapes the frontend renders¶

redirect (OAuth Auth-Code + PKCE) → Spotify, Tidal, YouTube official, Deezer.
device_code → YouTube Music (ytmusicapi OAuth, "TV & Limited Input" client).
form (schema-driven) → header paste (self-host only), Apple Music user token.

N providers collapse into 3 UX patterns.

[rev] Provider-specific lifecycle hooks & multi-account¶

Some providers don't fit a plain redirect — Apple Music MusicKit needs a first-class developer-token + client-fetched user-token flow, so AuthStrategy allows per-provider begin/complete shapes rather than one hardcoded OAuth dance. Credentials are keyed by provider_account_id + credential version + granted scopes, so a user can connect multiple accounts of the same provider and we can re-auth for new scopes without losing history. Header-paste is prohibited in hosted mode.

Per-provider reality (verify at build time)¶

Spotify — OAuth PKCE; refresh tokens; ISRC-rich. Clean.
YouTube official — Google OAuth; refresh tokens; quota-limited.
YouTube Music (ytmusicapi) — unofficial; device-code OAuth or header paste; no ISRC → text search + graph + review.
Tidal — OAuth PKCE; ISRC available.
Deezer — OAuth; write needs app approval (tightening).
Apple Music — MusicKit dev token (ES256 JWT, paid acct) + client user token.
Amazon Music — no public playlist write. Out of scope.

Credential storage¶

Encrypted blob (Fernet via KeyProvider), auth_kind, scopes, expires_at, refresh_token, version, account_label. Refresh ahead of expiry. Never log tokens; redact in errors; PKCE + state; minimal scopes per capability.

7. Capability matrix — [rev] descriptors, not booleans¶

Adapters advertise a structured CapabilityDescriptor, because the UI and the scheduler need constraints, not just "can write":

capability set: READ_PLAYLISTS/TRACKS/LIBRARY, independent READ/WRITE_SAVED_ALBUMS and READ/WRITE_FOLLOWED_ARTISTS, CREATE_PLAYLIST, ADD_TRACKS, REMOVE_TRACKS, REORDER, SET_COVER, SET_DESCRIPTION
has_isrc, search_modes (isrc/text), official, stability
write constraints: max_add_batch, max_playlist_size, supports_duplicates, ordering (preserved/best_effort/none), description_max_len
pacing/cost: search_quota_cost, write_quota_cost, daily_quota
warning (free-form caveat surfaced in the UI)

Honest matrix (verify per provider)¶

Provider	Playlists/tracks	Saved albums	Artists	Target lookup	Notes
Spotify	Read/write + mirror	Read/write	Follow read/write	ISRC, UPC, text	official
Tidal	Read/write	Read/write	Favorite read/write	ISRC, UPC, text	official
YT Music (`ytmusicapi`)	Read/write	—	—	text	unofficial
Apple Music	Read/write	—	—	ISRC + text	library entities not advertised
Amazon Music	—	—	—	—	no public write

UI consequences¶

GET /providers returns the matrix; the FE renders source/target pickers and inline warnings dynamically. Core gates each selected entity independently. Playlist selections require track-read and playlist-write caps. Album/artist selections require their matching read/write caps, scopes, and operation-specific library protocol. Unsupported target types remain disabled and are rejected if submitted. GET /providers also returns can_mirror and an actionable reason when mirror is unavailable.

8. Identity / evidence graph — [rev]¶

A provider-agnostic track identity map that grows with every migration — but modeled as an evidence/candidate graph keyed by an internal track_identity UUID, not by ISRC as the primary key. ISRC is strong evidence, not identity.

track_identity(id UUID pk, isrc?, title, artist, album, duration_s)
track_edge(
  identity_id -> track_identity,
  provider, provider_track_id, provider_uri,
  confidence, source,            -- isrc_exact | fuzzy | user_confirmed
  scope,                         -- 'global' | 'account:<id>'   (overlay)
  created_at
)

Why a graph, not an ISRC table¶

One ISRC can map to several provider tracks (studio/live/clean/explicit/region); conversely fuzzy links are uncertain. A graph holds candidates with evidence.
Per-user confirmations are overlays (scope = account:<id>): a user fix applies to their migrations immediately. It is promoted to global only on strong, corroborated evidence — a single fuzzy/user guess never becomes global truth. This is the safety the review flagged.

How it self-enriches¶

Read: a track with ISRC + id records a high-confidence edge for free.
Write: to place a track on a no-ISRC target, MatchService checks the graph first (cache hit → zero searches → saves quota), else asks the adapter to search, scores candidates locally, and records the chosen edge.
Review: a user fix writes a user_confirmed overlay edge.
Reverse bridging: once a no-ISRC provider_track_id links to an identity, the reverse direction resolves too.

Privacy¶

The global graph holds no PII and is potentially shareable; per-account overlays and all playlist/selection/job data are private. Sharing the global graph as an open dataset is deferred pending legal review.

9. Migration job, idempotency & progress¶

migration_job + entity-typed job_item rows for tracks, albums, and artists (status: pending/matched/needs_review/written/skipped/failed). Runs on arq.
[rev] Real idempotency via an operation ledger. Instead of "dedupe by name", each write records intent → call → observed target id/position. On an uncertain failure we reconcile by reading target state, never blindly retry a non-idempotent insert. operation_ledger persists this.
[rev] Central rate limiter (app/core/rate_limit.py, token bucket) paces all providers using the capability cost hints — not per-adapter sleeps — with jitter for unofficial providers to avoid account flags.
Spotify read calls cache /me/playlists results and selected playlist tracks by snapshot_id. The UI does not automatically refresh Spotify lists on every app load; users refresh playlist refs or songs explicitly when they need new data.
[rev] Durable, replayable progress. Progress is derived from persisted job_item rows and streamed over SSE; a reconnecting client resumes via Last-Event-ID, so no events are lost on a dropped connection.
Sync-generated jobs carry origin=sync and a sync_run_id: they reuse matching, review, duplicate handling and the operation ledger without cluttering manual migration history/statistics.
Mirror replacements write an intended ledger row before the provider call, verify the final ordered URI sequence, and leave ambiguous state retryable. Multi-batch Spotify replacement restarts from the first PUT on every retry and attempts to restore the previous sequence after a partial failure.

Organizer jobs¶

organizer_job + organizer_item store account-level status and one durable playlist/action result.
Successful items are excluded from subsequent worker runs. Retry resets failed, retryable items only.
Spotify song removal is limited to one 100-item snapshot-protected request. The job stores baseline and expected playlist-sequence hashes; an ambiguous retry is complete only if the exact expected sequence is observed.
YouTube Music stores per-occurrence setVideoId values and removes only IDs still present on retry.
Safe removal and deletion reconcile an already-absent playlist as complete.
Any success invalidates provider/account playlist caches; failed items remain in the report.
The migration operation_ledger remains migration-only. Organizer idempotency is stored directly on organizer_item.

10. Data model summary¶

Table	Purpose	Scope
`provider_account`	a connected account (provider + label)	private
`provider_credential`	encrypted tokens, auth_kind, scopes, expiry, version	private
`migration_job`, `job_item`	jobs + per-track/album/artist status	private
`operation_ledger`	intent vs observed writes (idempotency)	private
`generation_preference`	opt-in bounded local artist/genre summary	private
`generation_draft`, `generation_draft_item`	editable resolved candidates before confirmation	private
`sync_rule`	endpoints, mode, cadence/timezone, enabled and last/next metadata	private
`sync_run`	trigger, lease, snapshots, changed counts, status and error	private
`sync_checkpoint`	latest applied snapshots, mappings and unresolved review items	private
`organizer_job`, `organizer_item`	bulk organizer request + per-playlist action/result	private
`review_decision`	retained accepted low-confidence matches	private
`track_identity`	canonical track (UUID pk, ISRC as evidence)	global, no PII
`track_edge`	provider links with confidence/source/scope	global + per-account overlays

11. Repo layout (actual)¶

open-playlist-engine/
  backend/
    app/
      core/        # models, capabilities, adapter contract, registry,
                   # match/generator services, rate_limit, security
      providers/   # spotify/, ytmusic/  (self-registering adapters)
      db/          # SQLAlchemy models (private data + identity graph)
      jobs/        # arq worker + migration, sync, and playlist-organizer pipelines
      api/         # /providers /auth /playlists /library /migrations /generator /organizer
                   # /syncs /exports /shares and isolated public share routes
    tests/conformance/   # fake provider + contract suite
    migrations/          # Alembic
  frontend/        # Vite + React + TS SPA (consumes generated OpenAPI client)
  docs/            # this doc + ADRs
  docker-compose.yml     # backend, worker, frontend, postgres, valkey

Frontend and backend stay strictly separate: no shared code, FE consumes only the generated OpenAPI client.

12. Security & privacy¶

Encrypt provider credentials at rest via KeyProvider; never log/redact tokens.
Never persist or log raw generator prompts. Model calls receive only the prompt, explicit controls, and an opt-in capped local preference summary.
Generator drafts are private per-user data. Copilot SDK sessions run without tools, file context, memory, skills, config discovery, host Git operations, session store, or session telemetry.
PKCE + state; refresh ahead of expiry; minimal scopes per capability.
Separate the PII-free global graph from private user data and per-account overlays.
Unofficial adapters: pace + jitter; surface "may break / ToS-grey" warnings.
Header-paste auth disabled in hosted mode; plugin allow-list in hosted mode.
Public import URLs require HTTPS, exact allowed hosts and paths, no embedded credentials or non-default ports, bounded length, and no IP-literal host.
Open Playlist Engine remote JSON uses a pinned-address HTTPS client: every DNS answer must be globally routable, redirects stay on configured hosts, localhost/ private/link-local/reserved destinations are rejected, compression is disabled, and redirect, timeout, header, and response-byte caps are enforced.
Provider web pages are never scraped; unsupported hosts fail before any network request, and provider authentication/access controls are not bypassed.

13. Risks & mitigations¶

Risk	Mitigation
YouTube official quota (~66 songs/day)	`ytmusicapi` default; official off; graph cache cuts searches
Unofficial API breakage / account flags	central pacing + backoff + jitter; typed errors; official fallback
Awkward auth (ytmusic, Apple)	3 challenge shapes + per-provider lifecycle hooks
Fuzzy mismatches	ISRC-first; review step; confirmations as overlays, promoted only on evidence
Partial writes on retry	operation ledger: reconcile by reading target state
Overlapping or orphaned sync runs	partial unique index + row lock + lease token + stale recovery
Sync feedback loops	one-way endpoint identity and inverse-rule rejection
Mirror replacement fails mid-batch	verify, retry from first replace call, and attempt rollback
Accidental destructive organizer action	safe remove is a distinct capability; delete/remove-songs require typed confirmation
Duplicate playlist false positive	suggestions require normalized name + owner compatibility + ≥50% overlap and are never auto-selected
Provider API/ToS changes	capability descriptors + conformance suite catch regressions
Lossy migrations	fidelity contract + per-job lossy report (`unsupported_reason`)
URL import SSRF or DNS rebinding	exact host registry, public-address validation, pinned TLS socket, redirect/size/time limits

14. MVP build order¶

Backend: Spotify OAuth (PKCE) + import → Open Playlist. (Phases 0–1)
Frontend: connect Spotify + selection tree. (Phase 2)
Backend: ytmusicapi writer + MatchService + identity graph + arq job + operation ledger. (Phases 3–4)
Frontend: review queue + SSE progress board. (Phases 3.6, 5)
Add: official YouTube writer behind flag.
Then: each new provider (Tidal → Deezer → Apple) is a plugin + conformance pass.

15. Decisions log¶

✅ Reference implementation of the open-playlist spec; reuse Playlist/Track.
✅ Any-to-any via hub-and-spoke (O(N) adapters).
✅ Monorepo, hard-separated FE/BE; FE consumes generated OpenAPI client.
✅ Stack: Python/FastAPI + arq, Vite/React/TS, Postgres, Valkey, docker compose.
✅ ytmusicapi default-on; official YouTube Data API opt-in/off.
✅ Pipeline ordered import → match → review → write.
✅ [rev] adapters search-only; MatchService owns matching.
✅ [rev] evidence graph keyed by UUID; per-account overlays; evidence-gated promotion.
✅ [rev] idempotency via operation ledger; central rate limiter; replayable SSE.
✅ [rev] capability descriptors (constraints); fidelity contract + lossy report.
✅ Capability-gated Playlist Organizer with durable per-playlist results and provider-specific recovery warnings.
✅ Self-host single-user v1 with SaaS-ready seams (DEPLOYMENT_MODE + KeyProvider).
✅ Persistent worker-local playlist sync with restart catch-up, review finalization, add-only mode and capability-gated Spotify mirror.
✅ Self-hosted OpenAI-compatible playlist generation with optional Copilot SDK, provider-resolved editable drafts, and confirmation-gated durable writes.

16. Open questions¶

Sharing the global graph as an open dataset — needs legal review.
Provider priority after Spotify + YT Music (Tidal looks lowest-friction).
When to split into two repos (if ever) — current hard separation keeps it cheap.

17. Frontend visual language¶

The frontend is a focused migration workspace rather than a generic dashboard. Its visual signature is the source-to-target route: provider identities are shown as recognizable endpoints, while the surrounding controls remain familiar and task-oriented.

Graphite surfaces and cool-white text keep long migration sessions readable.
Electric periwinkle is reserved for primary actions, current selection, and focus.
Spotify, Tidal, YouTube Music, and Apple Music retain their service marks and brand colors inside provider identity components only.
Provider color never communicates state by itself; selection also uses borders, check marks, text, aria-pressed, and native disabled behavior.
Status, warning, success, and review styles remain semantic across playlist, progress, and statistics views.
Motion communicates state changes in 150–250 ms and is removed when reduced motion is requested.
The desktop workspace expands to use available width; provider lanes, accounts, toolbars, playlists, reviews, and stats collapse structurally for narrow screens.

The implementation keeps the existing API behavior and accessible tab model. frontend/src/index.css retains the component/state selector vocabulary, while frontend/src/theme.css supplies tokens and the current visual treatment.

18. Local library snapshots¶

Snapshot profiles persist a set of connected provider accounts and selected current library collections (standard playlists and native liked-track collections). Creation runs as an arq job and streams provider items directly into a ZIP64 Open Playlist bundle, so the whole library is never accumulated in memory. Provider read errors are recorded per collection and produce a usable partial archive.

The v1 bundle contains only manifest.json, manifest.sha256, and declared collections/*.jsonl payloads. The manifest records schema version, snapshot and library lineage UUIDs, timestamps, non-secret source labels, counts, partial failures, and per-collection payload/item checksums. Track serialization uses the universal model but drops audio preview data and unknown opaque metadata; no credential, token, auth header, cookie, or audio member is permitted.

All paths are generated below OPE_SNAPSHOT_DIR. Verification never extracts ZIP members and rejects traversal, duplicates, undeclared binary members, newer schemas, excessive compressed/uncompressed sizes, unsafe compression ratios, invalid records, and checksum/count mismatches. Docker mounts one named volume into both backend and worker. Startup reconciliation fails stale jobs and removes or restores old temporary, orphan, and staged-deletion files without leaving the configured root.

Restore creates an ordinary migration_job with source_kind=snapshot and a verified source_snapshot_id. The worker branches only for source reads; target preflight, matching, review, write batching, duplicate checks, operation ledger, SSE, and reports are shared. A stable snapshot:<library_id> lineage scopes prior review decisions and target-playlist reuse exactly, while live playlist annotations exclude snapshot jobs.

Keys	Action
`?`	Open this help
`n`	Next page
`p`	Previous page
`s`	Search