adminandClaude Opus 5
c91264c253
Pure move, no behaviour change. web.py 1221 -> 807 lines.
The reason is not file size but isolation of change. CLAUDE.md records
that the next step for detection, should flagging ever need improving, is
a spectral filter or an optional Silero VAD over candidate sections —
that work now lands entirely in isrweb/loudness.py and never has to open
the HTTP layer. The byte-level format code (FLAC frame parsing, CRC-8,
live header patching) is a second self-contained cluster, touched only
when a container format misbehaves.
isrweb/loudness.py 229 detector + its tuning constants
isrweb/audiofile.py 250 filename clock, WAV/FLAC headers, MIME types
web.py 807 cache, listing, handler, main()
Import direction is strictly web.py -> {loudness, audiofile}; the two
leaves import neither each other nor web.py, which is what lets the
detector be imported and tested without an HTTP server.
A package rather than flat modules because `audiofile` is a real PyPI
name and `detect` is generic enough to collide; a local file silently
shadows an installed one, which fails confusingly rather than loudly.
web.py stays at the repo root so `python web.py`, the docker-compose
command and the webui.html lookup are all unchanged.
Each module keeps its own optional numpy/soundfile try-import rather than
sharing one. That block cannot meaningfully diverge — if numpy is
installed every copy says True — and sharing it would stop isr.py being a
standalone file you can drop on a Pi and run.
Tests mirror the modules: test_loudness.py, test_audiofile.py, test_web.py
(+ unchanged test_isr.py). New tests/conftest.py owns the sys.path setup
that was previously an inline hack in test_isr.py — which only worked for
the other modules because that file happened to import first — and
provides the write_wav fixture the split would otherwise have duplicated.
Verified:
* 78 passed, same count as before the split; each test file also passes
standalone.
* All 21 moved functions diffed against HEAD:web.py — byte-identical.
* Import graph asserted acyclic; both leaves import with `web` absent
from sys.modules.
* An analysis cache written *before* the split still hits afterwards,
with identical sections — the one way a "pure" move could quietly
invalidate every user's cache.
* Endpoint smoke (analyze bounds, cache hit, cached_analysis in the
listing, clip Content-Length, 8-bit rejection) and the byte-serving
harness (ranges, download Range, HEAD, 405, live-header patching whole
and straddling) both re-run green.
* Dockerfile COPY set verified by building the image's file tree in a
temp dir and importing web.py with nothing else present.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
78 lines
18 KiB
Markdown
78 lines
18 KiB
Markdown
# CLAUDE.md
|
||
|
||
Guidance for Claude Code when working in this repository.
|
||
|
||
## Rules
|
||
|
||
- **Always update `README.md`** when user-facing behaviour changes (flags, endpoints, Docker setup, features), and **commit it in the same commit** as the code change. README is the external reference; CLAUDE.md documents internals.
|
||
- Run `python -m pytest tests/` after changing `isr.py`, `web.py` or `isrweb/` (tests cover the recorder, the detector, the filename clock and the HTTP layer). Test files mirror the modules: `test_isr.py`, `test_loudness.py`, `test_audiofile.py`, `test_web.py`; `tests/conftest.py` owns the sys.path setup and the `write_wav` fixture.
|
||
|
||
## Files
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `isr.py` | Recorder: streams (Icecast/HTTP) + ALSA soundcards, time-aligned file splits |
|
||
| `web.py` | Archive browser: HTTP server, file listing, analysis cache, cut/delete. Entry point — stays at the repo root so `python web.py` and the `webui.html` lookup work |
|
||
| `isrweb/loudness.py` | The adaptive loud-section detector. Pure computation, imports nothing from the app — future spectral/VAD work lands here and never touches HTTP |
|
||
| `isrweb/audiofile.py` | What an audio file is and what its name means: the filename clock, FLAC/WAV header parsing, live-header patching. Also a leaf |
|
||
| `webui.html` | Single-page UI (HTML/CSS/JS), loaded by `web.py` at startup — must sit next to `web.py` and be copied in the Dockerfile |
|
||
| `config.ini` | Recording sources; copy from `config.example.ini`. `[general]` gives defaults, every other section is a source (`type = stream` or `type = soundcard`) |
|
||
| `asound.conf` | dsnoop device `shared_mic` so ISR and other ALSA apps can share a soundcard |
|
||
|
||
## Commands
|
||
|
||
```bash
|
||
python isr.py [config.ini] # recorder; --list-devices to list ALSA inputs
|
||
python web.py # web UI on :8080 (--dir, --port, --margin, --min-gap, --min-duration, --analyses-dir)
|
||
python -m pytest tests/ # test suite
|
||
docker compose up -d / down # web UI mapped to host port 8050
|
||
```
|
||
|
||
Dependencies: `requests` (streams), `numpy` + `soundfile` (FLAC output and FLAC analysis/clips — both optional, code degrades gracefully).
|
||
|
||
## Code map
|
||
|
||
**Import direction is strictly `web.py → {isrweb.loudness, isrweb.audiofile}`.** The two leaves never import each other or `web.py`; keep it that way, it is what lets the detector be tested without an HTTP server. Each module carries its own optional numpy/soundfile try-import — that boilerplate cannot meaningfully diverge, and sharing it would stop `isr.py` being a standalone file.
|
||
|
||
`isrweb/loudness.py`:
|
||
- Detection: `_compute_rms_windows_wav()` / `analyze_flac()` produce 100 ms RMS windows (at 48 kHz — `WINDOW_SAMPLES` is a fixed sample count, so windows are wider on lower-rate files; `analyze_wav()` rejects anything that is not 16-bit, which the RMS decoder assumes) → `_noise_floor_db()` estimates the rolling floor → `_loud_sections()` emits scored sections → `_package_result()` shapes the `/api/analyze` payload.
|
||
|
||
`isrweb/audiofile.py`:
|
||
- Filenames as a clock: `_recording_start()` parses the start time out of a filename stem; `_cut_filename()` turns a (stem, ext, start, end) into a wall-clock-named cut. Both the listing `date` field and `_api_cut()` use them.
|
||
- Live headers: `_live_wav_header()`, `_live_flac_header()` (+ `_flac_frame_samples()`, CRC-8 verified). Plus `_wav_header()` (44-byte PCM header for clips), `_get_audio_duration()`, `MIME_TYPES`, `AUDIO_EXTENSIONS`.
|
||
|
||
`web.py`:
|
||
- Clips: `_api_clip()` validates params, `_clip_wav()` / `_clip_flac()` stream the decoded slice.
|
||
- Serving: `_serve_bytes()` is the single Range-capable file sender — `_stream()` (live header patch via its `prefix` argument, `no_store`) and `_download()` (`Content-Disposition`) both go through it, and it is what makes `do_HEAD` a body-less `do_GET`. Plus `_copy_to_response()`, `_safe_path()` (path traversal guard).
|
||
|
||
`webui.html` (one `<script>` block):
|
||
- Clip review: `clipQueue`/`clipCursor`/`scoreCursor` globals, `playClip()`, `playFileSection()`, `hideClipBar()`; markup is the `#clip-bar` div. The clip label shows wall-clock occurrence time + dB prominence + position (`17:09:56 to 17:09:57 · +30 dB (30 / 426 by time)`): queue entries carry `absStart` (epoch s), derived from `fileStartEpoch(f.date)` — the filename clock — with in-file offsets as fallback for non-standard names; only the filename/in-file offset lives in the tooltip now. **Queue entries carry the filename, never a row index:** `_idx` is reassigned positionally on every `load()`, and `pollStatus()` reloads whenever a recording starts/stops/rolls over, so a stored index goes stale mid-review (a past bug had "Open in file"/cut fields target the row one off after a split rollover). Resolve at use time via `fileIdxByName()`. **`playClip()` builds one `text` string used for both `label.textContent` and the `announce()` aria-live message** — they must never diverge (a past bug had the announcement read `Clip N of M: …` while the label read the wall-clock form). **Two cursors, deliberately independent:** `clipCursor` is the time-order position and follows every play; `scoreCursor` is the loudness walk's own position in `scoreOrder()` and is moved *only* by U/I (and "highlights only" auto-advance) in `stepClip`'s by-score branch — J/K must never touch it, so a J/K detour to review nearby clips leaves the loudness walk's place intact and the next U/I resumes the ranking where it left off. `scoreCursor` resets to `-1` on every explicit jump / queue (re-)arm (`hideClipBar`, `playFileSection`, `jumpToDaySection`, both day-highlights arm points), so the next U/I re-anchors on `clipCursor` (the section just selected) else the loudest. `playClip(i, byScore)`: when `byScore` the count shows `scoreCursor` (rank, suffix "by loudness"), else `i` (time-order index, suffix "by time") — the suffix tells the blind user which dimension the count is in. `stepClip`'s by-score branch passes `byScore=true`; every other caller (chip click, J/K, Prev/Next) leaves it false. Both U/I walks — `stepClip`'s by-score branch and the in-player branch of the keydown handler — share `nextInOrder(order, pos, dir, jump)` for the stepping arithmetic (returns `null` at either end); they differ only in what anchors `pos` and what "play" means, and must not re-inline it.
|
||
- Day review: `dayHighlights()` builds the chronological cross-file section list; `armDayQueue(dayId, sections)` is the single place that points the queue at a day (sets `dayActiveSections`/`dayActiveId`/`clipQueue` and resets both cursors) — used by the fresh-build path and the re-open-a-kept-panel path alike. `jumpToDaySection()` arms the queue. Section `absStart` comes from `fileStartEpoch(f.date)` (filename clock), mtime−duration only as fallback. **The user is blind and uses a screen reader — there is deliberately no day-timeline SVG** (one existed and was removed on request as useless); the highlights panel is linear text/buttons: summary line → key-hint note → chips toggle → chips. Do not add decorative visualizations; any future graphic must be aria-hidden and must not be the only carrier of information. Chip lists longer than 12 are collapsed behind an `aria-expanded` toggle button (the `.chips[hidden]{display:none}` rule is required — the author-level `display:flex` on `.chips` would otherwise override the UA `[hidden]` rule). Group `aria-label`s stay short ("Day loud sections") — the J/K/U/I key explanation lives only in the visible note, per user feedback against repeating info text in labels. The Highlights button is a collapse/expand toggle (`setHlExpanded()` keeps arrow + `aria-expanded` in sync, also from the day-collapse path): a built panel is kept and re-armed from `dayHlSections` instead of recomputing, keyed by `hlRow.dataset.loaded = hlParams()` (margin|gap|minDur string) so changed params force a re-run. The `#dayhls-<dayId>` "· analysed" suffix appears when every file's `cached_analysis` passes `cachedParamsMatch()`; `fetchAnalysis()` updates `f.cached_analysis` client-side so the marker survives re-renders without refetching `/api/files`. **`aria-label` on a button replaces its visible content for screen readers** — any status text rendered inside a labelled button (like the analysed suffix) must be mirrored into the label (render path + the end of `dayHighlights()`), or the blind user never hears it. Likewise, the accessible text of a button built as `aria-hidden arrow span + text node` must not start the text node with a space (the hidden arrow drops out and the leading space reads as an indent) — keep separator spaces inside the aria-hidden span.
|
||
- J/K/U/I/O: single document-level `keydown` listener — clip queue takes priority, in-player `currentTime` stepping is the fallback when no queue is armed; O calls `openClipInFile()` (shared with the "Open in file" button). J/K (and Prev/Next) always step in time order; U/I walk the loudest-first ranking from `scoreOrder()` — no top-N cutoff (the `#clip-top` input and `#clip-hl-only` checkbox were removed deliberately; J/K must never be affected by an auto-advance/highlights setting). Auto-advance is the `input[name="clip-adv"]` radio (off / next in time / next by loudness), read by `advanceMode()`; `stepClip(dir, byScore)` is the shared queue-stepping path. In-player U/I anchor the ranking on the section under the playhead, else start at the loudest.
|
||
- Analysis: `fetchAnalysis()` (session `analysisCache`), `analyse()` (per-row render: meta line with section count + params, then chips — no waveform SVG, see day-review note on the blind user; its `btn` argument is optional), `cachedParamsMatch()` (autoload guard). `analysisParams()` is the only reader of the three control inputs — no hardcoded client-side defaults, so web.py's constants stay the single source of truth via `/api/config`; `hlParams()` is just its values joined.
|
||
|
||
## Verifying changes
|
||
|
||
- `python -m pytest tests/` covers the recorder (`test_isr.py`) and the detector (`test_web.py`).
|
||
- There is no JS toolchain and no `node` on the dev box. After editing `webui.html`, cross-check every `getElementById('x')` against an `id="x"` declaration, and smoke-test endpoints.
|
||
- Endpoint smoke pattern: write a temp WAV/FLAC with a known loud burst, subclass `web._Handler` with `recordings_dir`/`analyses_dir` pointing at the temp dir, serve `web._Server(('127.0.0.1', 0), H)` in a daemon thread, then hit `/api/analyze` and `/api/clip` with urllib — assert section start/score and that `Content-Length == len(body) == 44 + frames × channels × 2`.
|
||
- Dev box is Windows / PowerShell 5.1. Multi-line commit messages: use the Bash tool with `git commit -F - <<'EOF'` — PowerShell here-strings containing quotes get mangled into separate arguments.
|
||
|
||
## Non-obvious internals
|
||
|
||
- **Recorder/web coupling is one file:** `RecorderManager` atomically writes `recordings/status.json` every 2 s listing in-progress files; deleted on clean shutdown. `web.py` reads it to show REC badges and to refuse analyse/cut/delete on active files. In-progress WAV/FLAC headers are unfinalized, so durations are not read for active files.
|
||
- **Stream splits:** OGG/Opus/FLAC codec headers are extracted from the first ~16 KB of each connection and prepended to every split file so each file plays standalone. A new file is always opened on reconnect (gap in stream). MP3/AAC need no headers.
|
||
- **Split timing:** files split at clock-aligned boundaries (`get_next_split_time()`), e.g. `split_minutes = 60` → on the hour.
|
||
- **Filename is the clock — fixed format, not configurable.** Recordings are named `%Y%m%d_%H%M%S.<ext>` (the *start* time). This is hardcoded as `FILENAME_FORMAT`, defined in **both** `isr.py` (recorder writes it) and `web.py` (reads it back) — the two copies must stay in sync. There is no `filename_pattern` config option (removed; `web.py` can't see `config.ini`, so a configurable pattern would break parsing). `web.py` derives the displayed DATE column from the filename via `_recording_start()` (falling back to mtime only for non-standard names — mtime is the last write ≈ end, not the start). Cut downloads are named by the wall-clock span they cover via `_cut_filename()`: a 22:31:30→22:32:30 slice of `20260523_220000.flac` becomes `20260523_22-31-30_22-32-30.flac`; non-standard source names fall back to `<stem>_cut_<start>s-<end>s`.
|
||
- **ALSA:** capture spawns `arecord` as a subprocess, raw PCM read in 100 ms chunks by a thread. Device spec resolution: `default` → exact `hw:X,Y` → partial name → fallback to any literal ALSA PCM name (so `shared_mic` from asound.conf works without appearing in `arecord -l`).
|
||
- **Shutdown:** SIGTERM is converted to KeyboardInterrupt in `main()`; `RecorderManager.stop()` joins all threads against a single shared 25 s deadline to stay inside Docker's `stop_grace_period: 30s`.
|
||
- **Loud-section detection is adaptive — do not regress it to an absolute threshold.** Per-window dB is compared against a rolling noise floor (`NOISE_PERCENTILE`-th percentile per `NOISE_BLOCK_SECONDS` block, min-smoothed over ±2 blocks so events can't raise their own floor; clamped to ≥ `MIN_RMS`). A section needs `margin` dB of prominence and carries a `score` used for ranking: peak dB above floor, **capped by the sharpest rise within `ONSET_SECONDS` (0.5 s)** — so a short (~10 s) swell that outruns the 30 s floor blocks still flags but scores ≈ 0 and sinks in the U/I highlight ranking, while sharp events keep their full prominence. A section starting in the first 0.5 s of a file is scored against the floor instead (events cut off by a file split must not be punished as swells). Do not regress the scoring to raw peak, and do not fight swells with a higher margin. If flagging itself (not just ranking) ever needs improving, the next step is a spectral filter or optional Silero VAD over candidate sections. Sections shorter than `min_duration` (default 0.5 s, after `min_gap` merging) are discarded — without this, isolated 100 ms pops (clicks, single raindrops) produced thousands of zero-length sections per day. The original fixed RMS threshold flagged every ambience change (passing cars, rain) and produced ~600 useless sections/day — that is why it was replaced. Tests in `tests/test_web.py`.
|
||
- **Analysis params are coupled in five places.** CLI `--margin`/`--min-gap`/`--min-duration` → `/api/config` → UI inputs `#margin-input`/`#min-gap-input`/`#min-duration-input` → `/api/analyze` query params → cache JSON keys. Renaming or adding a param means touching all five plus `cachedParamsMatch()` and `_cached_analysis_params()`. Query-string parsing/clamping for all three goes through the single `_qs_float()` helper (see the threshold→margin change `c84b7d8` and the min_duration addition).
|
||
- **Analysis cache:** results stored as `<analyses-dir>/<file>.analysis.json` keyed by margin+min_gap+min_duration; orphans pruned at web startup. In Docker the recordings mount is **read-only** for the web container, so docker-compose layers a read-write `./recordings/analyses` bind mount over it. `_cached_analysis_params()` just `json.load`s the file — key order carries no meaning. (It used to regex the four param keys out of the first 256 bytes, which forced them to be written first; that was worth it only while caches embedded the RMS arrays, and those are gone.) `detector` is `DETECTOR_VERSION`: bump it whenever detection/scoring changes make old cached results wrong (v2 = onset-capped scores, v3 = a section open at EOF ends at its last loud window rather than at the file duration); caches with another version (or missing keys) never match and get overwritten on the next analyse.
|
||
- **Analyze responses:** `/api/analyze` returns only `sections` and `duration` — no RMS data of any kind. `rms_display` (~800-point waveform preview), the full per-window list, and `window` (the window length, which only existed to place RMS bars) were all dropped when the waveform SVGs were removed (user is blind, see webui notes). Old caches that embed `rms`/`rms_display` have both keys popped when served from cache.
|
||
- **Section playback uses clips, not seeks:** `/api/clip?file&start&end` decodes the slice server-side (wave/soundfile) and returns a standalone 16-bit WAV with exact Content-Length (capped at `CLIP_MAX_SECONDS`), `Cache-Control: private` so re-listening is free. The UI plays chips/J-K through the bottom clip bar (`clipQueue` in webui.html); seeking the full file only happens via "Open in file". Rationale (finding): libsndfile writes FLAC **without a SEEKTABLE**, so a browser seek bisects the whole multi-hundred-MB file with Range requests — seeking big FLACs in `<audio>` is inherently slow and must not be reintroduced as the primary navigation. Server-side `sf.SoundFile.seek()` on local disk is fast and frame-accurate.
|
||
- **HTTP/1.1 keep-alive:** `_Handler.protocol_version = 'HTTP/1.1'`; every response path must set an accurate `Content-Length`. `_copy_to_response()` force-closes the connection if it under-delivers (file truncated mid-serve).
|
||
- **Live playback:** for files listed in status.json, `/stream/` patches the header on the fly so the browser sees the duration recorded so far and can seek; responses get `Cache-Control: no-store`. WAV: `_live_wav_header` derives sizes from the byte count. FLAC: `_live_flac_header` parses the sample count out of the last frame header in the file tail (CRC-8-verified to reject false sync matches) and rewrites STREAMINFO total_samples — duration is NOT derivable from byte size for FLAC.
|
||
- **Path safety:** every file parameter in `web.py` goes through `_safe_path()`, which resolves and verifies the path stays inside the recordings dir.
|
||
- **dsnoop in Docker:** sharing the soundcard requires `asound.conf` on the host *and* `ipc: host` in docker-compose (dsnoop uses shared memory across the container boundary).
|