Building Highlight Reels in the Browser with WebCodecs

I build Scoreboard, an open-source live scorecard for racquet sports. You score a match from your phone and it shows live on an OBS overlay, a TV, or fullscreen at the court.

People record their matches. So the obvious next feature is a highlight reel: take the hour of footage, cut the six moments that mattered, hand back something postable.

The interesting part is deciding which six.

The obvious approach is the wrong one

In 2026 the reflex is a model. Run the footage through something that detects rallies, crowd reaction, celebration, ball speed. There are products that do this.

For this problem it would be slower, more expensive, less accurate, and fundamentally redundant — because the scoreboard already knows.

Someone scored that match, point by point, on their phone. The app has an event log of every point, who won it, and when. It knows the game was at 20–20. It knows that point ended the match. It doesn't need to watch anything to know that the match-winning point is a highlight.

The comment at the top of the module says it plainly:

// What counts as a highlight, with zero video analysis.

What the log can tell you

Five kinds, all derived rather than detected:

• match-point — the point that ended the match
• game-point — a point that won a game, excluding the match winner
• point-saved — the losing side of a game or match point wins the rally instead
• clutch — a point played from deuce
• long-rally — the top N points by elapsed time since the previous point in the same game

That last one is the only inference in the list, and it is worth being honest about. The log has no shot count. What it has is a timestamp per point, and the gap between two points is rally length plus reset time. A long gap is a long rally, plus some noise from someone tying a shoelace. It's a proxy, it is good enough, and calling it anything more precise would be a lie.

There's also a kind the log genuinely can't produce. Funny isn't derivable. A ball hitting the umpire, someone falling over, a dog on court — none of that is in a score log. That's covered by manual markers the operator drops on the timeline, and those live in the video-time domain rather than the event domain, because they are anchored to something the score never saw.

Knowing which category a feature falls in — derivable, inferable with a named proxy, or not derivable at all — turned out to be most of the design.

Replay the reducer, do not track a parallel score

The naive implementation walks the log and keeps a running score in a local variable. That works until it doesn't.

Racquet scoring has undo. It has score corrections. It has sport-specific rules about deuce, about who serves, about when a game ends — and those rules live in the engine, one reducer per sport. A parallel score maintained in the highlight module is a second implementation of that logic, and second implementations drift. Not immediately. Three months later, on badminton, when somebody corrects a score mid-game.

So the highlight selector doesn't track score at all. It replays the sport's own reducer over the event log up to each point and asks the resulting state what was true at that moment.

export type RacquetReducer = (
  events: RacquetEvent[],
  cfg: RacquetConfig
) => RacquetState;

Whether a point was game point is whatever the engine says it was. Undo trims, score.correct resets, deuce detection, side-attributed game and match point — all of it comes from the one implementation that is already tested and already correct.

This is O(n²) in event count, and I am fine with that. A match is around 200 events, so that is tens of thousands of cheap reducer iterations, run once when the log loads, never per frame. The version that is fast and subtly wrong isn't worth having.

One detail that makes the code simpler than it sounds: by the time the render page loads a log, undone points have been hard-deleted. So a point event in that log is a point that stood. No tombstone filtering inside the selector.

Clip ids are the winning point's event id, which makes them stable across recomputes — reopen the page and your clips keep their identity.

Then you have to actually render it

Selection gives you timestamps. Now you need an MP4 with the scoreboard overlay burned in.

The server-side answer is ffmpeg. I did not want it: no upload of someone's match footage, no transcoding bill, no queue, and the source file is already sitting on the user's machine.

So it runs in the browser, hardware-accelerated, with WebCodecs:

1. The caller drives the overlay's reactive state across the timeline and hands the renderer pre-rasterized snapshots as ImageBitmaps. This is the single biggest performance decision — it sidesteps a seek-per-snapshot loop, which is agonisingly slow.
2. mp4box.js demuxes the source MP4 into a codec config and a sample list.
3. VideoDecoder turns each sample into a VideoFrame using the browser's hardware H.264/HEVC/VP9/AV1 decoder.
4. Each frame is drawn to an OffscreenCanvas, then the active overlay bitmap is drawn on top — chosen by matching frame timestamp against snapshot times.
5. VideoEncoder re-encodes the canvas as H.264 chunks on the hardware encoder.
6. mp4-muxer packages the chunks into a fresh MP4.

Two things in there took real time.

Getting the decoder config out of the container. VideoDecoder wants the codec-specific config as a raw Uint8Array, and it lives in an avcC, hvcC, vpcC or av1C box inside the sample entry. You pull the box, then skip the first eight bytes — box size and name — because the decoder wants the body, not the header. That is a short line of code and a long afternoon.

Choosing an encoder profile. Hardware encoders refuse profiles they cannot do, so the level has to scale with resolution:

if (px <= 1280 * 720)  return 'avc1.42E01F';   // baseline
if (px <= 1920 * 1088) return 'avc1.4D4028';   // main, 4.0
if (px <= 2560 * 1440) return 'avc1.4D4032';   // main, 5.0
return 'avc1.640033';                          // high, 5.1

What it does not do yet

Opus audio is dropped. AAC passes through — the encoded chunks are copied to the muxer untouched, with an AudioSpecificConfig synthesised from the sample rate and channel count rather than parsed out of the source's esds descriptor tree, which is a great deal simpler than it sounds and covers essentially every phone and camera recording. Opus is the gap: it needs its own header handling and I have not wired it up, so an Opus-audio source renders silent.

I would rather say that than let someone discover it after a five-minute render.

The part worth keeping

The renderer is the harder engineering and the selector is the better decision.

Every instinct in 2026 says point a model at the video. But the question "which moments mattered" had already been answered by a human tapping a phone for an hour, and it was sitting in a table. The work was not detecting highlights. It was noticing that nothing needed detecting.

When something looks like a perception problem, check whether some part of your system already recorded the answer.