PanNote + PipelineEvolve

The problem

A 90 minute lecture recording is very hard to search through, and Panopto captions sit behind a session cookie, so nothing outside the browser can reach them. There is also a harder question underneath that one. Once I had hand built a multi agent pipeline for taking notes, I had no real way of knowing whether the prompts and structure I chose were any good.

What I built

PanNote is a Chrome MV3 extension that detects a Panopto lecture, pulls the captions in the page itself, and runs the transcript through a multi agent GPT-4.1 Mini pipeline that renders structured notes in a side panel. PipelineEvolve treats that pipeline configuration, meaning the prompts and the agent topology, as a program that can be optimized. It searches for a better configuration using LLM driven mutation and a locally trained proposer, and scores every candidate against the pipeline I built by hand.

What I am building right now

This is a side project I started in July 2026 and am still actively working on. PanNote already works end to end. It detects the lecture, builds the transcript, runs the agent pipeline, and renders the notes. What I am working on right now is PipelineEvolve, which is the harness that asks whether the pipeline I wrote by hand is actually the best one available. It treats the prompts and the topology as something that can be searched over, instead of assuming I got the design right the first time.

The security decision I would defend in an interview

The obvious way to build this is to forward the user's Panopto session cookies to the backend so the server can fetch the captions. I did not do that, for three reasons. HttpOnly cookies make it unreliable, it is a real security problem, and it is the kind of thing that gets an extension rejected from the Chrome Web Store. Instead the content script fetches the captions inside the page, where the browser attaches the cookies on its own, and then posts only plaintext to the backend. The extension can do the same thing, and no credential ever leaves the machine.

Two models, and why confusing them ruins the experiment

The pipeline being tested runs on GPT-4.1 Mini through an API, so its weights can never be trained. Only its configuration gets edited, and then it gets called so the result can be scored. The part that actually learns is a local Qwen2.5-1.5B proposer trained with GRPO to suggest better configurations. Keeping those two roles separate is what makes the whole setup make sense. If you confuse them, any number that comes out the other end is meaningless.

A negative result I chose to report instead of bury

Two different search methods, run a month apart, both found configurations that beat the baseline on the dev subset, and neither one survived held out data. Instead of reporting the number that looked good, I worked out why it happened. The reward function cannot tell good pipelines from bad ones. A pipeline I deliberately gutted ties the baseline. A mutation that is semantically wrong scored 5.72 points higher. Concept recall goes down while the judge score goes up. The clearest piece of evidence is that the winning prompt says to not omit essential details for brevity, while the trained proposer's prompt says to use a maximum of 10 words per point. Those are opposite instructions, and the reward puts them 1.1 points apart.

The next real experiment

The next step is not more searching. It is fixing the measurement. I plan to blend deterministic concept recall into the judge score, then run the sabotage tests again and check whether the gutted pipeline finally separates from the baseline. The current test set is spent, because anything tuned against those numbers is no longer held out, so getting a clean answer also means collecting fresh transcripts.