Legora builds legal AI agents without being able to see client data, and tests them in simulated work environments.
Why this is useful to know
Legora is a Swedish legal tech company that builds AI agents for law firms. Its CTO, Jacob Lauritzen, explains in a 67-minute podcast how they do that without ever being able to read the traces of their agent at lawyers. That data belongs to the firms' clients. The core question then is: how do you test an agent you cannot look into? The answer is that they build test environments that resemble the work of customers, and measure quality in other ways. That is relevant for SMEs too. Even without sight of every step, you can measure and improve quality.
When you need this
You run into this when you want to use AI on sensitive documents, such as client files, contracts or medical records, that your supplier or your own team may not simply look into. What can you do? Ask your supplier how they test, with which examples and which measurement points. Have errors sorted into categories, so you can see where things go wrong. Mind the choice of model: not every model is good at every task. Lauritzen estimates that models now handle about 60 percent of legal work well. A small change in the setup can make a big difference. According to him, citing sources went from 70 to 99 percent, in one release, without them changing anything in the setup.
How it works under the hood
Legora tests with evals in simulated environments, comparable to the 'worlds' from reinforcement learning. Three engineers and three lawyers build that test data modelled on the work of customers. Because traces are not readable, they work with paraphrased traces and product metrics, and narrow errors down to 10 to 20 types. They treat long contracts as a search problem: retrieval and a citation system as a guardrail against hallucinations. For a contract of a million tokens they choose between MapReduce and one big context window. Infrastructure runs on Azure because of data privacy. Tabular Review is the spreadsheet layer, and agents can create lists that in turn start new agents. Note: all figures come from the speaker himself, not from an independent benchmark.
This news touches chapter 22 and 12 and 7 of the course AI(l) or nothing by Newway. If you follow the course, this subject returns there at a calmer pace.
What you can do with this
Want to know what this means for your business? Plan an introduction: half an hour, no sales pitch, but an honest answer to the question where to begin. Rather read on first? The course AI(l) or nothing is free and starts with the basics.
Written in response to a publication by Latent Space: https://latent.space/p/building_legal_agents_when_you_can_t_see_the_data_-_jacob_lauritzen_cto_of_legora/
