refactor: simplify web.py — one param parser, plain cache reads, no generated handler class
No behaviour change beyond dropping an unread response field. * _api_analyze had three identical parse-and-clamp blocks differing only in name/default/bounds. One _qs_float() helper; adding a fourth param is now a line rather than a copy of the block. * _cached_analysis_params regexed four keys out of the first 256 bytes of each cache file to avoid parsing "the embedded result (hundreds of KB)", which forced those keys to be written first — a constraint documented in three places and silently fatal if ever violated. The RMS arrays that justified it were dropped when the waveforms went; a cache is now a few floats and a section list, so json.load is both simpler and immune to key order. * /api/analyze no longer returns `window`. It existed to place RMS bars and has had no consumer since the waveform SVGs were removed. * main() generated an anonymous _Handler subclass just to inject five attributes, needing an alias to work around class-body scoping. Set them on _Handler directly. * `import sys` moved out of _Server.handle_error to the top of the file. Verified with the CLAUDE.md endpoint smoke pattern: /api/analyze section bounds, cache round-trip through the rewritten reader, cached_analysis surfacing in /api/files, and /api/clip Content-Length == len(body) == 44 + frames*channels*2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -31,7 +31,7 @@ Dependencies: `requests` (streams), `numpy` + `soundfile` (FLAC output and FLAC
|
||||
## Code map
|
||||
|
||||
`web.py`:
|
||||
- Detection: `_compute_rms_windows_wav()` / `analyze_flac()` produce 100 ms RMS windows → `_noise_floor_db()` estimates the rolling floor → `_loud_sections()` emits scored sections → `_package_result()` shapes the `/api/analyze` payload.
|
||||
- Detection: `_compute_rms_windows_wav()` / `analyze_flac()` produce 100 ms RMS windows (at 48 kHz — `WINDOW_SAMPLES` is a fixed sample count, so windows are wider on lower-rate files; `analyze_wav()` rejects anything that is not 16-bit, which the RMS decoder assumes) → `_noise_floor_db()` estimates the rolling floor → `_loud_sections()` emits scored sections → `_package_result()` shapes the `/api/analyze` payload.
|
||||
- Clips: `_api_clip()` validates params, `_clip_wav()` / `_clip_flac()` stream the decoded slice, `_wav_header()` builds the 44-byte PCM header.
|
||||
- Filenames as a clock: `_recording_start()` parses the start time out of a filename stem; `_cut_filename()` turns a (stem, ext, start, end) into a wall-clock-named cut. Both the listing `date` field and `_api_cut()` use them.
|
||||
- Live headers: `_live_wav_header()`, `_live_flac_header()` (+ `_flac_frame_samples()`, CRC-8 verified).
|
||||
@@ -59,9 +59,9 @@ Dependencies: `requests` (streams), `numpy` + `soundfile` (FLAC output and FLAC
|
||||
- **ALSA:** capture spawns `arecord` as a subprocess, raw PCM read in 100 ms chunks by a thread. Device spec resolution: `default` → exact `hw:X,Y` → partial name → fallback to any literal ALSA PCM name (so `shared_mic` from asound.conf works without appearing in `arecord -l`).
|
||||
- **Shutdown:** SIGTERM is converted to KeyboardInterrupt in `main()`; `RecorderManager.stop()` joins all threads against a single shared 25 s deadline to stay inside Docker's `stop_grace_period: 30s`.
|
||||
- **Loud-section detection is adaptive — do not regress it to an absolute threshold.** Per-window dB is compared against a rolling noise floor (`NOISE_PERCENTILE`-th percentile per `NOISE_BLOCK_SECONDS` block, min-smoothed over ±2 blocks so events can't raise their own floor; clamped to ≥ `MIN_RMS`). A section needs `margin` dB of prominence and carries a `score` used for ranking: peak dB above floor, **capped by the sharpest rise within `ONSET_SECONDS` (0.5 s)** — so a short (~10 s) swell that outruns the 30 s floor blocks still flags but scores ≈ 0 and sinks in the U/I highlight ranking, while sharp events keep their full prominence. A section starting in the first 0.5 s of a file is scored against the floor instead (events cut off by a file split must not be punished as swells). Do not regress the scoring to raw peak, and do not fight swells with a higher margin. If flagging itself (not just ranking) ever needs improving, the next step is a spectral filter or optional Silero VAD over candidate sections. Sections shorter than `min_duration` (default 0.5 s, after `min_gap` merging) are discarded — without this, isolated 100 ms pops (clicks, single raindrops) produced thousands of zero-length sections per day. The original fixed RMS threshold flagged every ambience change (passing cars, rain) and produced ~600 useless sections/day — that is why it was replaced. Tests in `tests/test_web.py`.
|
||||
- **Analysis params are coupled in five places.** CLI `--margin`/`--min-gap`/`--min-duration` → `/api/config` → UI inputs `#margin-input`/`#min-gap-input`/`#min-duration-input` → `/api/analyze` query params → cache JSON head keys. Renaming or adding a param means touching all five plus `cachedParamsMatch()` and the `_cached_analysis_params()` regex (see the threshold→margin change `c84b7d8` and the min_duration addition).
|
||||
- **Analysis cache:** results stored as `<analyses-dir>/<file>.analysis.json` keyed by margin+min_gap+min_duration; orphans pruned at web startup. In Docker the recordings mount is **read-only** for the web container, so docker-compose layers a read-write `./recordings/analyses` bind mount over it. The `detector`, `margin`, `min_gap`, and `min_duration` keys MUST stay first in the cache JSON — `_cached_analysis_params()` reads only the first 256 bytes to avoid parsing the large embedded result. `detector` is `DETECTOR_VERSION`: bump it whenever detection/scoring changes make old cached results wrong (e.g. v2 = onset-capped scores); caches with another version (or missing keys) never match and get overwritten on the next analyse.
|
||||
- **Analyze responses:** `/api/analyze` returns only `sections`, `duration`, `window` — no RMS data of any kind. `rms_display` (~800-point waveform preview) and the full per-window list were dropped when the waveform SVGs were removed (user is blind, see webui notes). Old caches that embed `rms`/`rms_display` are still valid; both keys are popped when serving from cache, so no DETECTOR_VERSION bump was needed.
|
||||
- **Analysis params are coupled in five places.** CLI `--margin`/`--min-gap`/`--min-duration` → `/api/config` → UI inputs `#margin-input`/`#min-gap-input`/`#min-duration-input` → `/api/analyze` query params → cache JSON keys. Renaming or adding a param means touching all five plus `cachedParamsMatch()` and `_cached_analysis_params()`. Query-string parsing/clamping for all three goes through the single `_qs_float()` helper (see the threshold→margin change `c84b7d8` and the min_duration addition).
|
||||
- **Analysis cache:** results stored as `<analyses-dir>/<file>.analysis.json` keyed by margin+min_gap+min_duration; orphans pruned at web startup. In Docker the recordings mount is **read-only** for the web container, so docker-compose layers a read-write `./recordings/analyses` bind mount over it. `_cached_analysis_params()` just `json.load`s the file — key order carries no meaning. (It used to regex the four param keys out of the first 256 bytes, which forced them to be written first; that was worth it only while caches embedded the RMS arrays, and those are gone.) `detector` is `DETECTOR_VERSION`: bump it whenever detection/scoring changes make old cached results wrong (v2 = onset-capped scores, v3 = a section open at EOF ends at its last loud window rather than at the file duration); caches with another version (or missing keys) never match and get overwritten on the next analyse.
|
||||
- **Analyze responses:** `/api/analyze` returns only `sections` and `duration` — no RMS data of any kind. `rms_display` (~800-point waveform preview), the full per-window list, and `window` (the window length, which only existed to place RMS bars) were all dropped when the waveform SVGs were removed (user is blind, see webui notes). Old caches that embed `rms`/`rms_display` have both keys popped when served from cache.
|
||||
- **Section playback uses clips, not seeks:** `/api/clip?file&start&end` decodes the slice server-side (wave/soundfile) and returns a standalone 16-bit WAV with exact Content-Length (capped at `CLIP_MAX_SECONDS`), `Cache-Control: private` so re-listening is free. The UI plays chips/J-K through the bottom clip bar (`clipQueue` in webui.html); seeking the full file only happens via "Open in file". Rationale (finding): libsndfile writes FLAC **without a SEEKTABLE**, so a browser seek bisects the whole multi-hundred-MB file with Range requests — seeking big FLACs in `<audio>` is inherently slow and must not be reintroduced as the primary navigation. Server-side `sf.SoundFile.seek()` on local disk is fast and frame-accurate.
|
||||
- **HTTP/1.1 keep-alive:** `_Handler.protocol_version = 'HTTP/1.1'`; every response path must set an accurate `Content-Length`. `_copy_to_response()` force-closes the connection if it under-delivers (file truncated mid-serve).
|
||||
- **Live playback:** for files listed in status.json, `/stream/` patches the header on the fly so the browser sees the duration recorded so far and can seek; responses get `Cache-Control: no-store`. WAV: `_live_wav_header` derives sizes from the byte count. FLAC: `_live_flac_header` parses the sample count out of the last frame header in the file tail (CRC-8-verified to reject false sync matches) and rewrites STREAMINFO total_samples — duration is NOT derivable from byte size for FLAC.
|
||||
|
||||
@@ -21,6 +21,7 @@ import re
|
||||
import shutil
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import wave
|
||||
@@ -409,11 +410,11 @@ def _package_result(rms_values: list, framerate: int, n_frames: int,
|
||||
duration = n_frames / framerate
|
||||
|
||||
# Note: no RMS data is returned — the UI is screen-reader oriented and
|
||||
# draws no waveform, so sections + duration is all it needs.
|
||||
# draws no waveform, so sections + duration is all it needs. The window
|
||||
# length went the same way: it only ever existed to place RMS bars.
|
||||
return {
|
||||
'sections': _loud_sections(rms_values, window_dur, duration, margin_db, min_gap, min_duration),
|
||||
'duration': round(duration, 2),
|
||||
'window': round(window_dur, 4),
|
||||
}
|
||||
|
||||
|
||||
@@ -480,22 +481,31 @@ def _analysis_cache_path(analyses_base: Path, recordings_base: Path, audio_path:
|
||||
|
||||
|
||||
def _cached_analysis_params(cache_path: Path):
|
||||
"""Read just detector/margin/min_gap/min_duration from a cache file
|
||||
without parsing the whole JSON (the embedded result can be hundreds of
|
||||
KB). Relies on the writer in _api_analyze putting these keys first.
|
||||
Caches written by other detector versions (or so old they lack a key)
|
||||
simply never match and get recomputed on the next analyse."""
|
||||
"""The params a cache file was written with, or None if it is unusable.
|
||||
|
||||
Caches from another DETECTOR_VERSION — or so old they lack a key — return
|
||||
None and are recomputed on the next analyse.
|
||||
|
||||
This used to regex four keys out of the first 256 bytes to avoid parsing
|
||||
the whole file, which imposed a "these keys must be written first"
|
||||
constraint on the writer. That was worth it when the cache embedded an
|
||||
~800-point RMS preview and the full per-window list; those were dropped
|
||||
when the waveforms went, so a cache is now a handful of floats and a
|
||||
section list, and a plain load is both simpler and unbreakable by key
|
||||
order."""
|
||||
try:
|
||||
with open(cache_path, 'r', encoding='utf-8') as fh:
|
||||
head = fh.read(256)
|
||||
except OSError:
|
||||
cached = json.load(fh)
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
m = re.search(r'"detector":\s*(\d+),\s*"margin":\s*([0-9.eE+-]+),'
|
||||
r'\s*"min_gap":\s*([0-9.eE+-]+),\s*"min_duration":\s*([0-9.eE+-]+)', head)
|
||||
if not m or int(m.group(1)) != DETECTOR_VERSION:
|
||||
if cached.get('detector') != DETECTOR_VERSION:
|
||||
return None
|
||||
try:
|
||||
return {'margin': float(cached['margin']),
|
||||
'min_gap': float(cached['min_gap']),
|
||||
'min_duration': float(cached['min_duration'])}
|
||||
except (KeyError, TypeError, ValueError):
|
||||
return None
|
||||
return {'margin': float(m.group(2)), 'min_gap': float(m.group(3)),
|
||||
'min_duration': float(m.group(4))}
|
||||
|
||||
|
||||
def prune_orphan_analyses(analyses_base: Path, recordings_base: Path):
|
||||
@@ -598,6 +608,19 @@ def list_files(recordings_dir: str):
|
||||
# HTTP handler
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _qs_float(qs, name: str, default: float, lo: float, hi: float) -> float:
|
||||
"""A query-string float clamped to [lo, hi], or default if unparseable.
|
||||
|
||||
Analysis params are coupled across the CLI, /api/config, the UI inputs and
|
||||
the cache keys — keeping the parse in one place means adding a fourth is a
|
||||
one-line change here rather than another copy of this block.
|
||||
"""
|
||||
try:
|
||||
return max(lo, min(hi, float(qs.get(name, [default])[0])))
|
||||
except (ValueError, TypeError):
|
||||
return default
|
||||
|
||||
|
||||
class _Handler(BaseHTTPRequestHandler):
|
||||
# Keep-alive: browsers reuse connections instead of a TCP handshake per
|
||||
# request. Safe because every response sets Content-Length.
|
||||
@@ -674,23 +697,9 @@ class _Handler(BaseHTTPRequestHandler):
|
||||
if path is None:
|
||||
return
|
||||
|
||||
try:
|
||||
margin = float(qs.get('margin', [self.margin_db])[0])
|
||||
margin = max(1.0, min(60.0, margin))
|
||||
except (ValueError, TypeError):
|
||||
margin = self.margin_db
|
||||
|
||||
try:
|
||||
min_gap = float(qs.get('min_gap', [self.min_gap])[0])
|
||||
min_gap = max(0.0, min(300.0, min_gap))
|
||||
except (ValueError, TypeError):
|
||||
min_gap = self.min_gap
|
||||
|
||||
try:
|
||||
min_duration = float(qs.get('min_duration', [self.min_duration])[0])
|
||||
min_duration = max(0.0, min(60.0, min_duration))
|
||||
except (ValueError, TypeError):
|
||||
min_duration = self.min_duration
|
||||
margin = _qs_float(qs, 'margin', self.margin_db, 1.0, 60.0)
|
||||
min_gap = _qs_float(qs, 'min_gap', self.min_gap, 0.0, 300.0)
|
||||
min_duration = _qs_float(qs, 'min_duration', self.min_duration, 0.0, 60.0)
|
||||
|
||||
if self._is_active(filename):
|
||||
self._json_err(409, 'File is currently being recorded — analysis unavailable until recording stops')
|
||||
@@ -729,8 +738,6 @@ class _Handler(BaseHTTPRequestHandler):
|
||||
try:
|
||||
cache_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = cache_path.with_suffix('.tmp')
|
||||
# detector, margin, min_gap and min_duration MUST stay first:
|
||||
# _cached_analysis_params reads only the first 256 bytes of this file
|
||||
tmp.write_text(json.dumps({'detector': DETECTOR_VERSION,
|
||||
'margin': margin, 'min_gap': min_gap,
|
||||
'min_duration': min_duration, 'result': result}), 'utf-8')
|
||||
@@ -1126,7 +1133,6 @@ class _Server(ThreadingHTTPServer):
|
||||
(browsers abort audio range requests constantly while seeking)."""
|
||||
|
||||
def handle_error(self, request, client_address):
|
||||
import sys
|
||||
exc = sys.exc_info()[1]
|
||||
if isinstance(exc, (ConnectionError, TimeoutError)):
|
||||
return
|
||||
@@ -1172,15 +1178,16 @@ def main():
|
||||
|
||||
prune_orphan_analyses(analyses_dir, rec_dir)
|
||||
|
||||
_analyses_dir = analyses_dir # class body can't close over a name it also assigns
|
||||
class Handler(_Handler):
|
||||
recordings_dir = str(rec_dir)
|
||||
analyses_dir = str(_analyses_dir)
|
||||
margin_db = args.margin
|
||||
min_gap = args.min_gap
|
||||
min_duration = args.min_duration
|
||||
# Config lives on the handler class (one server per process). Setting the
|
||||
# attributes directly beats generating a subclass here, which also needed
|
||||
# an alias to dodge Python's class-body scoping.
|
||||
_Handler.recordings_dir = str(rec_dir)
|
||||
_Handler.analyses_dir = str(analyses_dir)
|
||||
_Handler.margin_db = args.margin
|
||||
_Handler.min_gap = args.min_gap
|
||||
_Handler.min_duration = args.min_duration
|
||||
|
||||
server = _Server((args.host, args.port), Handler)
|
||||
server = _Server((args.host, args.port), _Handler)
|
||||
|
||||
print(f"ISR Web running on http://{args.host}:{args.port}/")
|
||||
print(f"Recordings dir: {rec_dir}")
|
||||
|
||||
Reference in New Issue
Block a user