Episode Transcript
[00:00:00] Speaker A: Imagine a justice system built on rigorous evidence, not gut instincts or educated guesses about what works and what doesn't.
More people could access the civil justice they deserve.
The criminal justice system could be smaller, more effective, and more humane.
The Access to Justice Lab here at Harvard Law School is producing that needed evidence. And this podcast is about the challenge of transforming law into an evidence based field.
I'm your host, Jim Greiner, and this is Proof Over Precedent.
This week we're bringing you a student voice.
[00:00:37] Speaker B: Hi, everyone, my name is Julia Saltzman and I'm a 2L at Harvard Law School.
Welcome to the Access to Justice podcast. I'm really excited to be talking with one of my classmates today, Michael. So, Michael, could you introduce yourself?
[00:00:55] Speaker C: Hey, I'm Michael. I'm also 2L at Harvard Law School, and I'm really interested in Brazil's use of AI in the judiciary.
[00:01:05] Speaker B: Yeah, so from what Michael's told me already, Brazil has an unprecedented court backlog of over 100 million cases, which is hard to even fathom, and their court system has turned to artificial intelligence as a potential solution.
So let's dive in, Michael, and talk more about it.
[00:01:27] Speaker C: Yeah, so, I mean, I think the best place to start here is there is like, I don't know how you feel, Julia, about, like the use of AI in the courts and that sort of stuff. And we can talk a lot more about what that actually means, but it's, you know, it's easy to imagine a sort of terminator like situation where people are, you know, just being doled out, arbitrary discussions.
It's important to recognize that the status quo in Brazil is really, really bad. Like 100 million cases is kind of hard to wrap your head around. But think about it like if you were a victim of domestic violence and you needed to get a restraining order from someone who was a real physical threat, or say you were facing eviction and your landlord was really extortive and abusive and, um, and was trying to evict you even though you had been paying rent and all of these sorts of things. Or say you just needed government benefits and you were entitled to the benefits, but for whatever dumb reason you know, the, the administrator didn't give them to you. All of these are really time sensitive things in which you need a court to act. And in Brazil, those courts are just so backlogged that you'll file your case and then it'll take ages for it to get through. And you can also imagine this in a criminal setting.
I do work with the President Legal Assistance Program People have like really good appeals, really good reasons they should be let out on, on parole. But just because the courts just don't have time to go through them, they spend years in prison just waiting for, for an appeal. So, and Brazil, that's really the case. So there's 102 million fell ending cases.
I would also say just, you know, to level that it's about three times the amount of time it takes to resolve a case in Europe. There's not good Data in the U.S. but it's about, it's also twice as long as, as it takes to resolve a case in India. And both are, you know, bureaucratic system. So it is a huge, huge issue.
And so in part to just kind of make any headway in solving this really, really practical problem, the courts have turned to AI and have really enthusiastically embraced AI.
[00:03:30] Speaker B: I mean, I think it's really easy to see the urgency of this issue, especially with the examples you gave.
But the legal profession definitely have it hesitancies to stick with what has worked for the legal profession for the past, you know, 100, 200 years. So I'm definitely curious to learn more about that. But let's just start with how is Brazil using AI to address this enormous backlog?
[00:04:00] Speaker C: Yeah, for sure. So there's, there's really three ways in which they've started using ii. So the first is they'll use it to just kind of like make sense of their backlog, categorizing and prioritizing the cases.
The second is once they're actually dealing with the cases, doing legal research.
So you'll have a case in front of you, they'll kind of bring something in. And then the third one is like you, you've done your research, you've seen the case, you're drafting an opinion, and AI will support that in the way you probably use ChatGPT occasionally when you're writing.
I've just a little bit into each of these systems. So the first in classifying, prioritizing cases, this happens at each level. So Brazil has a slightly interesting federal state system. I don't know how much our listeners know or have taken Professor Griner's pro class.
But in short, you'll often have a federal system of laws, and that's its own body of laws.
And then you'll have different laws for each state.
And accordingly, you often have a federal judiciary and a state judiciary.
So you have state laws, you have federal laws.
And then Brazil adds on to this a slightly different flavor in that it has a Supreme Court just for constitutional questions and a Supreme Court for all other questions. And I explain, all this just comes up sometimes because they each use this different AI system.
But all that is to say the Federal Supreme Court that doesn't deal with constitutional matters has been using a system called VICTOR to sort through thousands of appeals specifically to find those cases where decision on this matter will make a huge difference. For lots of other cases as opposed to, you know, there's pre existing precedent, this isn't going to make a huge difference. And so it'll surface those decisions first.
And an important thing to note, like an idiosyncrasy of Brazil's judicial system, in the US you don't have a right of appeal.
So you know, you can, you can appeal your case, but the Supreme Court doesn't have to hear your case. They don't have to opine on your case. They don't have to say anything like this. They can just say, we deny to hear your case in Brazil. That's not the case in Brazil. The Supreme Court has to say something about every single case that is positioned up, even if it's very short.
And so that's why there's a huge need to use AI to sort of classify and prioritize cases.
State, lower federal, lower courts and lower state courts will do something similar, although it's less like the Supreme Court's looking for cases of general repercussions. So DS is where a decision there is going to affect a lot of other cases throughout.
For state services, they're looking for something kind of similar, but, but a little bit different in a nuanced way, is that they'll have AI just try to find cases where there's a really obvious dead clear like legal precedent or like a version of a head note, and they can dispose of it really quickly versus cases where, you know, maybe it doesn't have a general repercussion. Maybe the legal matter being described isn't super relevant or isn't going to matter for a ton of people, but it is really on the line. Like it does really require someone to take a look at it. And so that's where they're using AI to separate the easy cases from the hard cases. And then the third way that AI is used to sort of classify and prioritize cases is just to throw out cases that have cleared like really obvious statutory bars. So for example, a lot of people appeal like specific determinations on their taxes. There are like really clear time bars. You know, if you didn't bring your case within five years of paying a certain amount of taxes, you just can't bring the thing and they can just throw out those cases or there's specific income levels that you're required to meet in order to do that. If you don't have the income level, your case is just invalid. And so all of these collectively are just ways, even before they start seeing case, to just start like throwing out and clearing up and categorizing this enormous backlog. I'll be a little bit less verbose on the second two years.
Supporting legal research.
Really what they're trying to do is the parties submit all of their briefs and the non constitutional court, the Superior Tribunal of Justice is using the Socrates AI system to sort of pull out the thesis of each of these cases and just present all of it. So it's, it's just kind of chewing up all of the legal resources that are coming in and then presenting it in a way that's really easy for a judge to just look at. Here are the party's arguments on either side of a given legal issue. Here's the right precedent that you should be looking to. That's thing. And at lower courts, one thing that they'll do is they won't totally digest everything, but they will just like annotate the entirety of a legal brief. So they'll say like, you know, say I cite to a particular case, the AI might pull in excerpts from that case so I can read it myself immediately in relevant part and see whether or not the precedent that they're citing actually supports the point that they're making.
And then the final thing is just, you know, what we all know natural language processing to be pretty good at, which is just putting together descriptions of like just putting together writing. So I use ChatGPT a lot in my own writing.
I think my writing is a little bit lower stakes than, than the, the judiciary in Brazil. And so there are valid concerns about when they're finally moving to a decision.
Like you know, should ChatGPT or, or, or its peers be creating the decision that being released and the justification to the parties.
I will say also there are some more simple ones here. So sometimes you have very standardized filings that a judge needs to put forward, like a notice of appearance or a summons of the parties.
A lot of times what AI is doing is just auto generating those filings and those papers. So that's kind of a sense of what they're doing. Again, it's mostly classifying cases. And then once you're looking at a case supporting legal research and Then supporting the drafting of various opinions.
[00:10:08] Speaker B: That was a super helpful overview.
And my understanding from conversations with you before is that on the decision making aspect there is still some level of human review for each case, is that correct?
[00:10:22] Speaker C: Yeah, that's totally correct. But there's always a risk of rubber stamping. You know, the most accidents.
I'm not going to get this quite right. But in flying was not when they like first created claims, but when they started automating planes.
Because it's, it's easy to kind of understand a completely computer run interface and it's easy to sort of understand a totally manual interface. But there is a lot of danger in the interplay between the two. And that's kind of where we are with a lot of AI in the courts in Brazil.
[00:10:54] Speaker B: I know very little about that, so that's good to know.
And I guess to me the case prioritization aspect of AI sounds perhaps the most important given the backlog. And I'm also just astonished that any case can make its way to the Supreme Court.
What do you think is the most promising of Brazil's use of AI and where would you like to see perhaps an expansion?
[00:11:24] Speaker C: I mean, I think that there's a lot of value when you just have such enormous backlogs of cases.
There is value in having AI go through these. I think you're right to say that the classification function is the most helpful of them.
My concern is that it's also one in which it's hardest to review or it's the most prone to rubber stamping because it's just so tedious to go through 56,000 cases and evaluate on this.
Which perhaps means that, you know, if you hired a bunch of humans to do it, they would do a worse job because they would get bored and you know, they would be applying inconsistent standards.
But I just think that's an area in which it's really important to get the AI human balance right.
And when you're, you have an exciting new technology and maybe not a ton of money to just hire people to do this or to come through these records, I'm worried that they might get that wrong and just give it over to AI.
[00:12:26] Speaker B: What do you see as some other of the potential risks or downsides for how widely integrated AI is in Bristol's judicial system?
[00:12:37] Speaker C: Well, I think one thing we really have to note is the difference of culture. In the US there have been very high profile uses technology in courtroom to make determinations. So we spoke in a different set setting about the use of machine learning. To make bail determinations in California. American voters really hate that. They hate the use of technology in courts. They're really kind of like believe that a human needs to look at this. This really isn't the case in Brazil. I mean the Supreme Court justice of Brazil has described AI as a godlike ability.
And there does not seem to be much like many headlines really concerned about huge errors here. And so I, I'm a little bit worried, I think in that these are very nuanced and very high impact decisions in legal research. You know, we spend three years and really agonize over learning the skill to distinguish different precedents. Maybe AI can do that better than human, but maybe it can't.
I think one way to kind of tee this question up that has been helpful is the type of error that you want. So do you want error that with an AI is like very systematic, but if you have a systematic error, it's going to appear everywhere. So one small algorithmic error result in poor decisions for thousands of cases, or do you want, with a human, many distributed different types of error across different cases. So you're probably unlikely to get the exact same error in every case, but you'll get different errors and all of those. And I actually do think there are good arguments on either side of that.
[00:14:09] Speaker B: Do you want to just quickly touch on that for our listeners?
[00:14:12] Speaker C: It's perhaps easier to audit AI in this way. Like if we actually get better at auditing AI. That's a huge, huge area of research. I mean you can imagine a world in which like the ALIA system, which works for specialized tax courts, they do a lot of the throwing out of cases because they're time barred by a statute of limitations.
You know, if they got the calculation wrong, such that it was four years instead of five. I'm making up this example just to simplify things. You could have a ton of people who have their cases thrown out when they're entitled to valid relief, often on like, you know, hundreds of thousands of dollars collectively, maybe millions.
That's really concerning.
On the other hand, you know, you, you have judges who are enormously backlogged, underpaid, and do not have the time or resources to get through your cases.
And it's hard to imagine that those judges are not also likely to make similar or more significant errors when they're going through it.
But, but they're not going to be the exact same error every time.
And so I think that's kind of the trade off. Or maybe there's complementarity here, you know, like you, you have human judges checking for specific things, but of course, you never know kind of ex ante which, which things are going to go wrong.
[00:15:28] Speaker B: So you touched upon how it's very challenging at this time to audit AI. How has Brazil tried to regulate the AI systems that it integrates into at least its court system? So far, yeah.
[00:15:44] Speaker C: So there was, there have been two major resolutions that have been passed. The first was mostly focused on data privacy, which is obviously really important, but is not super relevant to the quality of the outcomes that is coming. It was more like just a generalized AI regulation.
The second one was more kind of squarely on the news as to what we're speaking about here. So what they did was they created a national committee to oversight, like to oversee the use of AI in Brazil's court. And they set out various different requirements that this oversight committee would have to do. So they would have to make sure that the justification for a given decision and the use of AI was entirely transparent, such that, for example, the litigants could ask, you know, what did the AI say when you put these briefs in? And they had visibility into it and that sort of thing, which ideally would be a check against, you know, a judge just rubber stamping exactly what the AI put in and that sort of thing. The problem is there just has not been a ton of enforcement. There's not a clear resourcing to that oversight body to do enforcement action.
And then the second concern is that these are very, what I just described feels reasonably concrete. Like, you know, the parties would be able to in some way kind of like subpoena or require the judge to disclose how they used AI. That's. It's actually not that concrete. It'll say, you know, requires the use of transparency and the justification of AI and that's it. And so there's actually a ton of work that needs to be done in terms of enforcing or translating these high level mandates into very specific and kind of falsifiable ways that you could audit. So just say, like, to recap, there's two points of resolutions passed. The second one is kind of more on the nose.
And the main issues, there are both a lack of enforcement and a lack of clarity about how you would enforce.
[00:17:27] Speaker B: These seem like difficult issues for policymakers to tackle.
And so turning to your blog post, you argue that we need more evaluation of these AI systems. What kind of research would be the most valuable?
[00:17:44] Speaker C: I mean, I think randomized control files are typically the gold standard to isolate causal relationships. So you can, for example, randomize which cases receive AI support and compare outcomes on like time to resolution or reversal rates or consistency or litigant satisfaction. That would be one really easy way to do this.
But of course, you know, randomized controlled trials require buy in from the system. I think even without a randomized controlled trial, you could do a natural experiment. You know, AI has had staggered adoption across these different cores and that time difference could allow you to do effects or different regression. So you can imagine, for example, looking at those metrics I just mentioned, time to resolve reversal rate, consistency, litigant satisfaction for courts right before and right after they adopted the use of AI would allow you to kind of isolate a lot of different variables.
Or look at four different states which adopted AI and look at their outcomes right before and right after and look at, you know, how you can control for different things, states using it versus not using it.
[00:18:52] Speaker B: That's a helpful overview.
So I guess maybe our last question for today.
What's your vision for how Brazil or other countries should approach judicial AI going forward?
[00:19:07] Speaker C: I mean, I, I really think the main innovation here is how to evaluate AI and how to learn from AI. To put this another way, the goal is to create some sort of continuous learning where you deploy a technological solution, deploy AI to classify or prioritize cases, you deploy AI to support the drafting of opinion. And I think we actually do have a lot of good innovation there and we have a lot of immediate feedback within the system. A lot of these systems are like really quite like rigorously and quantitatively evaluated before they go out. We don't have a lot of real world evidence. So it'd be really interesting to see, okay, you deploy the system and then you track real world metric as to what's happening in the world. Are litigants more or less satisfied, are cases getting resolved faster, sooner, that sort of thing, and then have that go back into the system, that information feedback into the system and say whether or not this is working. And it sounds simple in practice, but it's very complex. When you think about the realities of Brazil's different state and federal courts and their constitutional and non constitutional courts, it's much, much more difficult to implement. But the upshot is enormous. If they can figure this out in Brazil, you know, this could have huge ramifications for court systems around the world if they're actually able to create a continuous learning model in which AI is evaluated based on its real world impacts and actually tweaked to improve based on that. Because that's how we get an AI system that advances both efficiency and justice.
[00:20:33] Speaker B: Well, thank you for closing with that note of optimism.
Is there anything else you'd like our listeners to know about this subject or anywhere else you'd like to direct them for further information?
[00:20:46] Speaker C: For now, that's all I've got.
It's been a pleasure speaking, Julia.
[00:20:52] Speaker B: Yeah, thank you, Michael. And thank you to our listeners for joining us.
[00:20:56] Speaker A: Proof Over Precedent is a production of the Access to Justice Lab at Harvard Law School.
Views expressed in student podcasts are not necessarily those of the A J Lab.
Thanks for listening. If we piqued your interest, please subscribe wherever you get your podcasts. Even better, leave us a rating or share an episode with a friend or on social media.
Here's a sneak preview of what we'll bring you next week.
[00:21:21] Speaker C: We have companies that are doing the
[00:21:23] Speaker B: most innovative work, like working to cure cancer, working to develop technology to prevent
[00:21:28] Speaker C: terrorism, like working across the spectrum, vertical by vertical, just like amazing, incredible things.
[00:21:34] Speaker B: And the last thing they need to
[00:21:35] Speaker C: do is spend time on legal.
[00:21:36] Speaker B: Like they need to be given the
[00:21:38] Speaker C: freedom and be given like, the time to focus on building that customer base.
