docs: code map, regression findings, and HTTP API reference

CLAUDE.md gains a function-level code map of web.py/webui.html, a
"verifying changes" section (test scope, no-node UI checks, endpoint
smoke pattern, Windows commit-message gotcha), and findings recorded as
guard rails: why detection must stay adaptive (fixed threshold produced
~600 useless sections/day), why section playback must stay clip-based
(libsndfile FLACs have no SEEKTABLE so browser seeks bisect the file),
and the five places analysis params are coupled.

README gains the adaptive-detection and clip-review features up top, an
HTTP API table for scripting, and a corrected Docker analyses-cache
paragraph (the rw mount is ./recordings/analyses layered over the ro
recordings mount, not a separate ./analyses dir).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-06-10 16:21:37 +02:00
parent 119e631faf
commit e4d82483b5
2 changed files with 50 additions and 7 deletions
+26 -4
View File
@@ -26,7 +26,28 @@ python -m pytest tests/ # test suite
docker compose up -d / down # web UI mapped to host port 8050
```
Dependencies: `requests` (streams), `numpy` + `soundfile` (FLAC output and FLAC/waveform analysis — both optional, code degrades gracefully).
Dependencies: `requests` (streams), `numpy` + `soundfile` (FLAC output and FLAC analysis/clips — both optional, code degrades gracefully).
## Code map
`web.py`:
- Detection: `_compute_rms_windows_wav()` / `analyze_flac()` produce 100 ms RMS windows → `_noise_floor_db()` estimates the rolling floor → `_loud_sections()` emits scored sections → `_package_result()` shapes the `/api/analyze` payload.
- Clips: `_api_clip()` validates params, `_clip_wav()` / `_clip_flac()` stream the decoded slice, `_wav_header()` builds the 44-byte PCM header.
- Live headers: `_live_wav_header()`, `_live_flac_header()` (+ `_flac_frame_samples()`, CRC-8 verified).
- Serving: `_stream()` (Range support), `_copy_to_response()`, `_safe_path()` (path traversal guard).
`webui.html` (one `<script>` block):
- Clip review: `clipQueue`/`clipCursor` globals, `playClip()`, `playFileSection()`, `hideClipBar()`; markup is the `#clip-bar` div.
- Day review: `dayHighlights()` builds `dayActiveSections` (chronological); `jumpToDaySection()` arms the queue.
- J/K: single document-level `keydown` listener — clip queue takes priority, in-player `currentTime` stepping is the fallback when no queue is armed.
- Analysis: `fetchAnalysis()` (session `analysisCache`), `analyse()` (per-row render), `cachedParamsMatch()` (autoload guard).
## Verifying changes
- `python -m pytest tests/` covers the recorder (`test_isr.py`) and the detector (`test_web.py`).
- There is no JS toolchain and no `node` on the dev box. After editing `webui.html`, cross-check every `getElementById('x')` against an `id="x"` declaration, and smoke-test endpoints.
- Endpoint smoke pattern: write a temp WAV/FLAC with a known loud burst, subclass `web._Handler` with `recordings_dir`/`analyses_dir` pointing at the temp dir, serve `web._Server(('127.0.0.1', 0), H)` in a daemon thread, then hit `/api/analyze` and `/api/clip` with urllib — assert section start/score and that `Content-Length == len(body) == 44 + frames × channels × 2`.
- Dev box is Windows / PowerShell 5.1. Multi-line commit messages: use the Bash tool with `git commit -F - <<'EOF'` — PowerShell here-strings containing quotes get mangled into separate arguments.
## Non-obvious internals
@@ -35,10 +56,11 @@ Dependencies: `requests` (streams), `numpy` + `soundfile` (FLAC output and FLAC/
- **Split timing:** files split at clock-aligned boundaries (`get_next_split_time()`), e.g. `split_minutes = 60` → on the hour.
- **ALSA:** capture spawns `arecord` as a subprocess, raw PCM read in 100 ms chunks by a thread. Device spec resolution: `default` → exact `hw:X,Y` → partial name → fallback to any literal ALSA PCM name (so `shared_mic` from asound.conf works without appearing in `arecord -l`).
- **Shutdown:** SIGTERM is converted to KeyboardInterrupt in `main()`; `RecorderManager.stop()` joins all threads against a single shared 25 s deadline to stay inside Docker's `stop_grace_period: 30s`.
- **Loud-section detection is adaptive:** per-window dB is compared against a rolling noise floor (`NOISE_PERCENTILE`-th percentile per `NOISE_BLOCK_SECONDS` block, min-smoothed over ±2 blocks so events can't raise their own floor; clamped to ≥ `MIN_RMS`). A section needs `margin` dB of prominence and carries a `score` (peak dB above floor) used for ranking. Tests in `tests/test_web.py`.
- **Analysis cache:** results stored as `<analyses-dir>/<file>.analysis.json` keyed by margin+min_gap; orphans pruned at web startup. In Docker the recordings mount is **read-only** for the web container, so the cache uses a separate `./analyses` bind mount. The `margin` and `min_gap` keys MUST stay first in the cache JSON — `_cached_analysis_params()` reads only the first 256 bytes to avoid parsing the large embedded result. Old `threshold`-keyed caches never match and get overwritten on the next analyse.
- **Loud-section detection is adaptive — do not regress it to an absolute threshold.** Per-window dB is compared against a rolling noise floor (`NOISE_PERCENTILE`-th percentile per `NOISE_BLOCK_SECONDS` block, min-smoothed over ±2 blocks so events can't raise their own floor; clamped to ≥ `MIN_RMS`). A section needs `margin` dB of prominence and carries a `score` (peak dB above floor) used for ranking. The original fixed RMS threshold flagged every ambience change (passing cars, rain) and produced ~600 useless sections/day — that is why it was replaced. Known limitation: a short (~10 s) swell on a quiet street still flags because the floor blocks are 30 s; the planned fix is an onset/spectral filter or optional Silero VAD, **not** a higher margin. Tests in `tests/test_web.py`.
- **Analysis params are coupled in five places.** CLI `--margin`/`--min-gap``/api/config` → UI inputs `#margin-input`/`#min-gap-input``/api/analyze` query params → cache JSON head keys. Renaming or adding a param means touching all five plus `cachedParamsMatch()` (see the threshold→margin change, commit `c84b7d8`).
- **Analysis cache:** results stored as `<analyses-dir>/<file>.analysis.json` keyed by margin+min_gap; orphans pruned at web startup. In Docker the recordings mount is **read-only** for the web container, so docker-compose layers a read-write `./recordings/analyses` bind mount over it. The `margin` and `min_gap` keys MUST stay first in the cache JSON — `_cached_analysis_params()` reads only the first 256 bytes to avoid parsing the large embedded result. Old `threshold`-keyed caches never match and get overwritten on the next analyse.
- **Analyze responses:** `/api/analyze` returns `rms_display` (~800 points), never the full per-window RMS list — the UI doesn't use it and it is ~45x larger.
- **Section playback uses clips, not seeks:** `/api/clip?file&start&end` decodes the slice server-side (wave/soundfile) and returns a standalone 16-bit WAV with exact Content-Length (capped at `CLIP_MAX_SECONDS`). The UI plays chips/J-K through the bottom clip bar (`clipQueue` in webui.html); seeking the full file only happens via "Open in file". Rationale: our FLACs have no SEEKTABLE, so browser seeks bisect the whole file with Range requests.
- **Section playback uses clips, not seeks:** `/api/clip?file&start&end` decodes the slice server-side (wave/soundfile) and returns a standalone 16-bit WAV with exact Content-Length (capped at `CLIP_MAX_SECONDS`), `Cache-Control: private` so re-listening is free. The UI plays chips/J-K through the bottom clip bar (`clipQueue` in webui.html); seeking the full file only happens via "Open in file". Rationale (finding): libsndfile writes FLAC **without a SEEKTABLE**, so a browser seek bisects the whole multi-hundred-MB file with Range requests — seeking big FLACs in `<audio>` is inherently slow and must not be reintroduced as the primary navigation. Server-side `sf.SoundFile.seek()` on local disk is fast and frame-accurate.
- **HTTP/1.1 keep-alive:** `_Handler.protocol_version = 'HTTP/1.1'`; every response path must set an accurate `Content-Length`. `_copy_to_response()` force-closes the connection if it under-delivers (file truncated mid-serve).
- **Live playback:** for files listed in status.json, `/stream/` patches the header on the fly so the browser sees the duration recorded so far and can seek; responses get `Cache-Control: no-store`. WAV: `_live_wav_header` derives sizes from the byte count. FLAC: `_live_flac_header` parses the sample count out of the last frame header in the file tail (CRC-8-verified to reject false sync matches) and rewrites STREAMINFO total_samples — duration is NOT derivable from byte size for FLAC.
- **Path safety:** every file parameter in `web.py` goes through `_safe_path()`, which resolves and verifies the path stays inside the recordings dir.