Today, I’m releasing an open-source interpretability workbench called SteerLab that allows non-expert researchers to conduct psychological experiments on language models. It has a native macOS interface but is mainly intended to be run by your coding agent on Linux or Mac. And it supports everything from local models on your laptop to large Slurm jobs on clusters. It is a codebase I built over the summer, while a fellow at the Center for AI Safety, to study AI judges.

The pioneers of mechanistic interpretability have shown that large language models reason over conceptual ecologies (see generally https://transformer-circuits.pub) and that their reasoning can be studied and manipulated by locating concepts and amplifying or suppressing them. SteerLab implements many of these techniques (CAA, "grand mean," something I call OptVec, J-lens techniques, linear probes, LoRA, assistant turn editing, and more) and makes rigorous experiments on manipulated agents and groups of agents easy.

The intelligence of AI agents transforms many seemingly technical problems into problems of governing groups of intentional agents. It is by now painfully obvious that understanding their behavior is critical to this effort. While interpretability is a vibrant area of AI research, we need many more people working on the diverse alien psychologies (and alien social psychologies) of these models. Progress depends on expertise in institutions, imagination, regulation, conflict, cooperation, and other individual and social outgrowths of mental processes. Researchers in the humanities and social sciences must not be deterred from joining this effort on account of technical barriers.

The barrier to entry is particularly low with SteerLab, because you can just talk to your coding agent to set it up and start running experiments. Just type "help me set up steerlab by following https://github.com/christortheak/InterpBench/blob/main/docs/AGENT-SETUP.md and then help me understand what I can do with it." There are still rough edges, but ease of use is the goal. (And it has a macOS UI that you can use yourself as well, including just to download a model and talk with it.) SteerLab is also meant to guide researchers toward proper experimental design, with freezing experimental designs, validating steering directions, gating with capability batteries, and recording experiments in ways that are easy to report and replicate. Both experimental rigor and ease of use are at the core of SteerLab.

I wanted this interpretability workbench to exist, because I'm fascinated by what language models can do and the alien conceptual systems they use to get there. Studying AI legal judgments was a natural first step. Prior experiments reported in law review articles have found that frontier models reach "correct" legal answers more often than human judges. I have some questions about that.

I started from the suspicion that today's models' formalistic judging arises from a disposition toward servility resulting from the assistant persona. I've been using SteerLab to run many experiments under the conditions of past AI-judge studies but where the underlying models are steered with concepts extracted to relate to "fear" or "critical legal studies" or "formalism" or "Golden Gate Bridge." I'm not ready to publish results, but so far, on 27B models, results have been strange, unexpected, and inconclusive. See this interim, Claude-generated site derived from the results folder.

Software these days seems increasingly to be fragments of functionality that you grab, alter, and drive with your agent. This fragment and other projects like it are invitations to join the effort to understand the agents that are doing ever more of our judging, engineering, advising, writing, researching, studying, arguing, and governing. I hope you help us figure out how!