Episode Transcript
[00:00:00] Speaker A: Imagine a justice system built on rigorous evidence, not gut instincts or educated guesses about what works and what doesn't.
More people could access the civil justice they deserve.
The criminal justice system could be smaller, more effective and more humane.
The Access to Justice Lab here at Harvard Law School is producing that needed evidence. And this podcast is about the challenge of transforming law into an evidence based field.
I'm your host, Jim Greiner, and this is Proof Over Precedent.
[00:00:34] Speaker B: Hi, everyone.
[00:00:34] Speaker C: Welcome to the Access to Justice Labs podcast, Proof Over Precedent. I am your guest host, Renee Dancer, and today I am grateful to be joined by my two friends and colleagues, Emily lagrada and Nathaniel Player.
So first, we like to start by telling everyone who you are and what's your current role and what path took you there, and then concluding with a little fun fact about yourself. So let's start with Emily.
[00:01:02] Speaker B: Great. Hi, everyone. Emily lagrada. I run a small consulting shop I have for the past six years, but I'm a lawyer by training, so I started my career in the Manhattan Criminal Court as a public defender and then spent about 10 years at the center for Court Innovation, where I did a mix of planning projects locally here in New York City, which is where I'm still based, as well as kind of shifting towards national training and technical assistance or consulting. So it was there at the center that the topic we're going to be talking about today, kind of, or that I'm going to be talking about today fell in my lap as a special project. And I've just been hooked ever since. So a fun fact about me. I know how fun this is to anyone else, but an obscure thing about me is I. One of the big parts of my life here as a New Yorker is I am the.
A past president of the New York Lawn Bowling Club.
[00:01:56] Speaker C: I was hoping that was going to
[00:01:58] Speaker B: be your friend, which I know I've confessed to Renee and Nathaniel before. It's essentially you could imagine, like playing bocce on a putting green in the middle of Central park, and it's just blissful and attracts some really quirky, delightful people. So. So that's a fun fact.
[00:02:15] Speaker C: And if I remember kind of that origin story, it's you just walked past and saw it happening and asked them, what's this?
[00:02:23] Speaker B: That's exactly it. Yeah. And sort of a microcosm for what I love about New York City. He just sort of, you never know what you're going to stroll by and get pulled into. Great.
[00:02:33] Speaker C: Thanks, Emily. Nathaniel, how about you?
[00:02:35] Speaker D: Sure. Hi, everybody. My name's Nathaniel Clayer. I am currently on a pre tirement. I am taking a couple years to pull back and enjoy not working. So I'm living in Mexico and Puerto Escondido, Mexico, where I do a lot of swimming in the ocean and enjoying the cultural offerings here in Oaxaca. I am the former director of the Utah State Court Self Help Center. I had that role in various capacities for about seven years. So started out just running self help and then eventually was given more responsibility to oversee all efforts regarding access to justice for the Utah State Court system. Before that I was an eviction defense attorney. I practiced eviction defense in Washington D.C. and San Francisco. Also a lawyer by training. Very nice to be here. Thanks for having me. And I think a fun fact right now is that I really enjoy vegan cooking. And one of the things that I like to think I'm a little bit good at is figuring out how to make a dish that is traditionally meaty or as cheese or cream and making it vegan. So veganizing the recipe with a little bit of spice, a little bit of texture, a little bit of secret stuff like that.
[00:03:51] Speaker C: That's a great, that's a great quality.
And many people I've spoken to know about my goal of early retirement. So I am supremely jealous of your pre tirement and I love that praise.
[00:04:06] Speaker D: Only live once, right?
[00:04:07] Speaker B: Yes, exactly.
[00:04:09] Speaker C: That's right. Enjoy the fruits of your labor. Right.
So today we're giving a different perspective on a study we've discussed before on this podcast. We're talking again about the online court study, but we are talking with Nathaniel, who in his former role before his pre tirement was our field partner working with the Utah Judicial or the third Judicial District in Utah. And our collaborating researcher, Emily, to give us a perspective of the research process from these two different lenses.
So if you recall, the online courts study, which is what we're calling it here, but is not actually the kind of formal name, it is a study trying to understand the effect of virtual appearances versus in as compared to in person appearances. And specifically we're looking at family law cases for self represented litigants. And so Nathaniel, can you tell us a little bit about the court process that we're operating in this study?
[00:05:12] Speaker D: Sure thing. In the study the focus was on family law cases on a pro se limited scope calendar. And so every Wednesday and Thursday at the largest and busiest courthouse in Utah and Salt Lake City, there would be cases set to be heard by for a court commissioner, which is a judge with a more limited role in the family law Context and there would be volunteer lawyers brought in to help those parties work to resolve their issue. Either come to a negotiation with some legal strategic guidance and support or argue before the court on whatever the matter is, whether it's a motion or it's almost always a motion, but it could be something else. They're revisiting a matter or something. There's lots of family law cases in that context. Everyone was offered a volunteer lawyer. Not everybody took it, but they were all given the opportunity to have a lawyer and the partnership with the agency's story of engagement with the court and the legal aid providers doing a lot of logistics to support the program.
[00:06:17] Speaker C: Yeah, right. And so this is a specialized calendar specifically meant to bring self represented litigants into one space to manage motions practice essentially. And there are some hoped for benefits of isolating that calendar. Right. Do you. This is an unprepared question, but are you able to off the top of your head get some idea of what those benefits might be?
[00:06:46] Speaker D: Benefits of the calendar generally or within the context? Oh sure. I think that the general assumption is that if people have limited scope attorney help, they will be more likely to get an outcome they feel good about. An outcome that's realistic, an outcome the court can accept the value out of having a lawyer help somebody, come to more reasonable expectations of what is possible to understand the framework of the law, to understand the process.
Those aspects are generally thought by the court to be helpful to really meaningfully add. I think all the commissioners involved in the study very much agree that when somebody has counsel, at least from the court's perspective, things are a lot better. It's important to be clear this is not usually full scope representation. The lawyer comes in just for the purposes of a hearing, is very quickly briefed on the matter, looks at the relevant pleadings and then gives some advice as court to the self presented litigants. In that context, it's also hope that the case so the time in this position would hopefully go down.
But that's not. I don't know of anything that really has proven that in the context. I was talking about studies, but that's the court's perspective for sure.
[00:07:58] Speaker B: Great.
[00:07:59] Speaker C: And just one more piece of background for family law cases there. Generally even when lawyers are retained on a case, there is a computer copious amount of motions practice. Is that your experience? My experience in family law 4 and just to make sure everyone who's listening might know because they don't necessarily practice in this area, is that yes, of course there are dispositive events. But in between, there are lots of interim decisions that families may need help with determining. And so that's generally the point of isolating self represented litigants or lawyer based cases, giving them space and time to have this robust motions practice.
Was that your experience in Utah also?
[00:08:46] Speaker D: Yeah, I think that the motions practice really does shape and guide how those cases move forward.
I think it matters a great deal in a case where there are children involved because the parties need help figuring out all the things that happen when two parents are no longer together and they're not getting along, there's a reason they're getting divorced. And so they need help from the court figuring out who's going to see the children and when, especially around the holidays. Those are times when it becomes extra important for some sort of clear framework for what's happening and when also with child support, it matters. It's important for whoever is going to mostly have the children get support to cover the children. And that's a policy of the state of Utah to make sure that the children are prepared for.
[00:09:34] Speaker C: So, Nathaniel, before we turn to Emily to talk more about her research, just from a high level, what were you or what was the court hoping to learn from this study and if it turns out the way you hoped, what policies do you think might be affected by the results?
[00:09:54] Speaker D: Yeah, so we've talked a lot about the family law cases, but the context in which this was all happening back when it started was there was Covid and everything was remote. And there was a strong desire by the court to go back to in person hearings because that was familiar, that was known. It was a tool that everybody knew how to pick up and wield in a way that led to work moving forward. Picking up this other tool called online hearings and using that was awkward. It was difficult. And so the primary impetus was to understand what is going on with these online hearings. How does it work? Is it actually helpful for people getting to a resolution and resolution faster or slower? I think there was an assumption that is better for people because they don't have to come to court. And there's a lot of logistics around getting to court that has been documented in other work about the time it takes to get to the courthouse. And then there have been concerns around people feeling uncomfortable getting through the bailiff. That's all there anyway. But it was all magnified in the context of COVID So the hope was to understand what is happening with these online hearings.
Is it better generally? Does better mean faster disposition, more appearance rates, more durable, those sorts of Things we really wanted to understand what is going on. And I think that court in Utah is a little bit special because they really are down for evidence based practices. They want to look at data, they want to understand it, and they have great data that can be pulled out of their case management system. And so there was a lot of enthusiasm from many levels within the court to really get in and understand this. So there was just a very, I think eagerness is the word that comes to mind to describe how the court felt about trying to move forward in a way that was intentional, that was evidence based, that was going to be most helpful to bring justice to litigants.
[00:12:01] Speaker C: Yeah. And what a dream. So dream partnership it was for that very reason. That was my impression and that's how I felt throughout the entire course of the work with the court was their supreme willingness to truly understand the benefits and burdens and make decisions based on that understanding.
So let's switch over to Emily and hear more from Emily. So, Emily, tell us a little bit about your goals for the research and how. What components of the research would you attribute to your expertise?
[00:12:36] Speaker B: Yeah, so I'm so glad Nathaniel framed this as a question of sort of what does better mean? Because from I. In my work over the past 15 years or so, primarily on the criminal justice side of things, we've been grappling with all these different ways to make kind of deliver on justice. And the reality is we rarely define that based on the actual people going through the system. Right. We're pretty good at hearing from what judges want. Sometimes we hear from what the lawyers want.
In fancy jurisdictions, we might have good court data on, like how long things take or whatnot. At the end of the day, we don't really know if people feel like justice was done. And it's been a real quest over these years to bring in procedural justice research to projects like this one.
To really foreground, say, we need the litigant perspective as the part of this conversation, ideally right. To know, to include in this, what does better mean? What do litigants think of this?
And so it was a real opportunity to be able to pair with this RCT model, which was very rigorously going to use administrative data to try to answer that question. But to view that alongside these other valuable measures, which were really four very simple questions. And the first one was, did the court treat you fairly today?
So we know from procedural justice or procedural fairness, those are often used interchangeably. We know from that research that people's perceptions of whether or not they were treated Fairly is not only its sort of own end, but it also influences their likelihood of seeing the courts as legitimate and worthy of cooperation and compliance. So in terms of goals for the research, it was to find a pretty palatable way for the court to collect litigant feedback data over the course of this study and then view that alongside some of this other administrative data to try to paint a more holistic picture of what was actually happening for both the court players as well as the litigants.
[00:14:56] Speaker C: Right, thank you. So, Nathaniel, we talked about these. The case type we're focusing on in here is family law cases.
Really we were focusing mostly on custody and divorce cases. So for those of. For those who know, family law cases can span a lot of different subject areas. So we narrowed this to custody and divorce cases. This is private litigation. This is not litigation brought by the state, which you might think of in.
When it's related to some sort of juvenile court activity. That is not where we are. We are in the private litigation section. So tell us, how long do these types of cases specifically take? On average.
[00:15:36] Speaker D: Yeah, the average time of this position is about 18 months. Obviously it's a lot shorter if it's not contested. And it can go very quickly and everybody agrees and everything aligns when that doesn't happen.
Amazon died.
[00:15:49] Speaker C: Okay, great. And so, Emily, I wonder how that seems like a fairly lengthy time.
So tell me how procedural fairness and that kind of timing and the context of family law cases interact, or how did that change? Or how does that change how you approach the work? Or does it maybe and doesn't?
[00:16:15] Speaker B: Yeah, I think there's a. Maybe a couple different ways to answer that. We know that even when community members are interacting with a police officer on a single traffic stop, they expect to experience or they hope to experience procedural justice and the core element.
And there can be an impact when that officer is using research based practices. Right. People can be more kind of trusting of the decision and more cooperative going forward.
Now, by contrast, when you think of a litigant who's being brought back to court again and again, often before the same judge or court commissioner, there is an enhanced opportunity. If those interactions are having this cumulative effect of kind of dosing procedural justice, then I think the impact is even greater. So that's how the theory might be operating. And so from, from the study's perspective, we had this sort of open invitation to answer the four questions across the study period. So we were likely getting repeat customers on the survey, and that was fine.
And so there really was this opportunity to Assess anew each particular day. Right. So one day coming in, it could be that they felt that was a good experience. They might have had a different experience the next time. The study would have captured that. And so what I liked, what I thought was really impressive about the court's partnership on this was just to allow that operationally to exist over a time period, I will say, I don't know, five or six years ago, when I started asking courts if they were willing to have an iPad kiosk outside of their courtroom, there weren't many takers for that idea. It was pretty novel and pretty scary. But we started with some courts, a municipal court, where on average, the stakes are a little lower on most municipal court cases than when your access to your children and your assets is in the balance. But culturally, I think we've seen a shift where courts like the ones in Utah where we partnered are increasingly open to having that available so that people can weigh in in a pretty low impact way. It takes less than five minutes as they're leaving the courthouse. And it really allows an opportunity for people, those litigants then, to weigh in on the things that we already know from procedural justice research that they care about. Right. They want to be asked about how that went for them. They want to be asked if they felt like they had a voice and they want an opportunity to weigh in. So we had an open ended question as well where they could tell us about the delay or the tricky parking or that the clerk was really nice or any manner of things. So it really afforded a lot of opportunity. And again, to the credit of the courts for being open to giving that
[00:19:13] Speaker D: a try, addition to the kiosk, point and type. That's because it was something that I think when the request came through, it's like, oh, this is how this would work. This is how I would do it. There was a little bit of how is that going to work? Actually, every Wednesday, every Thursday for a year, we're going to do this thing. And so it really comes down to who's actually going to do seems like a nothing burger. And it's not because I think that the commissioner didn't feel like they wanted to do it for a number of reasons.
And then the clerical staff are often people who feel like everything is just thrown at them. It's not the judicial officer, clerical staff. And so I think this is an indicator of just relationships matter so much. So the commissioners had a really great relationship with the bailiffs and so that they asked, can you please do this for US bailiffs make sure it's plugged in.
We're here to make sure it's charged, and we'll just check it once in a while to make sure it's doing the upload to the. To the server to make sure that the data doesn't sit on the iPad that actually goes into the server. But it really did require a little bit of expenditure of political capital to make that happen. And it seems like a nothing burger for people who don't work in court. But having been on the inside, because I was, I guess I could do it. I could go up from my office upstairs and do it multiple days for a year and when I'm on vacation. I know, but it. I think it just. It's one of those things, I think that is if you have the right motivation and the right intention, it can happen. But it does require a little more than just, oh, why can't you just do that? It's complicated. It's just complicated. It's a complicated system that's handling a lot of pieces. The court is.
[00:21:02] Speaker B: Yeah, I'm so glad you brought that up, Nathaniel. I think maybe mentally I've blocked that out of my mind at this point. And admittedly, that burden didn't fall to us as the researchers. We were blissfully back in our offices. But it's so right. And I can't help but say if I fantasize about a world where these tools would just already be installed. Right. They would be available and in the court, and so it wouldn't be this anomalous thing that we need to do on Wednesday mornings. But there's just feedback tools. Right. And they're available. And we could pull from the data. Oh, let's pull the Wednesday data so we can use it for this project. And yes, we also happen to have the data from Monday, Tuesday and Friday that we know is attributed to different calendars and whatnot. So I think it is. It's evidence of sort of the.
Some of the pain points of starting to develop operations like this where, yeah, you have to figure out where to store the thing and where to charge it. And nobody else wanted it collecting feedback on their calendars. So we. It needed to be offline for a lot of the. A lot of the week. So, anyway, I'm glad you reminded us of that, Nathaniel.
[00:22:13] Speaker D: There was also a larger sort of political question. Maybe not political, but like in Utah, judges are evaluating on a very specific.
Through a very specific process. And so this looks like evaluation outside of that process, which creates discomfort. The Commissioners don't go through that process, so it wasn't anything that resonated for them. But for the judges, when we said they're doing it, can you do it? I think some judges did feel like we already have an evaluation process, and that's the process we use. So why are you creating new process?
I think just courts are complicated processes that are doing lots of things within their systems, and so anything that hangs from that just requires a little bit of maneuvering.
[00:22:55] Speaker B: Yeah, yeah.
[00:22:57] Speaker C: And Emily and Nathaniel, one of the things that we learn every time we do some research with really any partner is that the least disruption to the existing process is really the best path forward. Now, of course, the research shouldn't disrupt the process that we're studying at all, but there are things you have to add in to actually capture the data, and this is one really good, tangible example of that. And somebody has to manage that.
[00:23:29] Speaker B: Right.
[00:23:29] Speaker C: And so just empowering someone on the ground to do that and helping them realize how important they are to the success of the work, I think is helpful. And the bailiffs did do this process, which is an unlikely, I think, choice. So I'm really. I think we're all really grateful that they did that and that the commissioners were willing to expend that political capital to get them to do it. And every time we visited, I felt like they were really excited about it or at least excited to tell us they were doing it. So that's something.
[00:24:04] Speaker B: Renee, if I may, just because, yeah, I find this fascinating and research is not possible without all kinds of political capital being spent and people volunteering to do things. And I guess what I, from my perspective, I sort of do research lowercase R. But I'm not really a researcher. I'm. I often and alongside, trying to help implement. Implement or train and coach. And I think I see, at least on, like, kind of the iPads, and we had links and QR codes and other ways of trying to get people to the feedback link. I see it. Yes, it helped us to collect feedback for a research study, but it was also its own experiment of can courts use tools like this on their own? And I have a number of other projects. We've now done this in dozens of places. Usually not part of a research study, but more to see how can we get court leaders comfortable with and bought into the idea that this is really useful to them. Right. This is good performance data that they can just be using regularly. And what would it take to.
To be able to have it up and running in house.
So that they could again, reap. Reap the benefits of it. So again, yes, I'm. We keep bragging appropriately about Utah and Nathaniel's team, but it. Yeah, I'm glad we underscored some of the many costs or challenges that it took to try this out kind of early in that culture shift.
[00:25:34] Speaker C: Yeah.
[00:25:35] Speaker D: I'll just have one final thing and then we can move on.
[00:25:37] Speaker B: Right.
[00:25:37] Speaker C: Well, we can. This is a conversation, so please add as many things as you like.
[00:25:42] Speaker D: I think that the, the, the value out of being asked to do the kiosks and thinking through it was helpful because really my job was to be the access to justice person, thinking about litigants need. And then when we were thinking about the study, it wasn't very at the forefront of anyone's mind to think how is this for our litigants themselves? It wasn't present to ask them. And so this was coupled with some other projects that were going on, part of a learning process and maybe a cultural shift for the court system generally to think, maybe we got to think about our users, and maybe our users are not just lawyers. Maybe we need to think differently about that. And so that there were some really great conversations inside the administrative office thinking about. We really need to think more about not just what's good for the judges and what's good for clerical staff and what's good for lawyers. There's this entire vast ocean of people coming to us who are fit in any of those categories. And often their perspective is not solicited or unfortunately not in Utah's special. But in many cases, I think that perspective is not considerative and so not just assuming that we know what's best in the access to justice world, but actually talking to people and checking in with them and hearing them. So there really was some. There was some cost associated with it, but because we had to go through this process and if we're going to do this thing that's annoying, what are we getting from it? And the more we talk about it, the conversation process of thinking about it really was rich and helpful and changed a lot of perspectives, including mine.
[00:27:23] Speaker C: Yeah, I think a lot of Access to justice discourse is really trying to focus on how do we encourage and empower litigants to engage with their case.
And this is one way to start that pattern in practice. And so this is probably the easiest way. Emily makes it so easy. And we can talk a little bit next about Emily, how you operationalize this work. But this may be considered one of the easier ways to engage with your case. But just getting started, actually feeling like you have a voice in how your case proceeds might also encourage folks to continue to engage with the substance of their case to avoid not failure to appear rates or default rates in all sorts of cases. Now we're talking beyond just family law cases. But Emily, tell us a little bit about the litigants experience when they were participating in this type of research. What actually did, did they see from you and what did they need to do and maybe a little bit about the output that you saw from them.
[00:28:31] Speaker B: Yeah, so if you think about one of my goals is to have this sort of relatively low cost and easy mechanism that courts could use themselves to get feedback that really animated the tools that I wanted to try out in a study like this. So I sort of, if anybody's seen an airport bathroom or a taxi line or something, at least those are my experiences as a New Yorker where I see these kind of smiley face or thumbs up, thumbs up down kiosks.
Like so many of us, I was just seeing those everywhere. And so I started researching software out there that would have the simplicity of the smiley faces, but something maybe a little bit more appropriate for the seriousness of a courthouse or a court matter. And so I found software that I felt like had had some of that simplicity but adaptability so that we could ask these few simple questions, but have it be pretty intuitive. Right. The idea is that obviously we were studying in person court and remote court, so this looked different based on how the litigant showed up. But, but for in person litigants, the idea was for them to, I always say, kind of almost trip over the kiosk as they're leaving the courtroom, but hopefully not.
And to have it be obvious that it's just going to be a few simple questions. Right. And so often the first question we have is just a, a simple thumbs up or thumbs down. So it's binary. We didn't ask them anything about their demographics, like if they'd been to court before. We didn't ask them if they had a lawyer. We don't know their income level. And that is hard to resist asking.
But we know from this decades of procedural justice research, we know that people care about being treated fairly.
We know that there are these core elements of procedural justice, namely they want to be respected, they want to have a voice, they want to see decision makers as neutral and unbiased and they want to be helped to understand what's going on.
So we centered, we actually singled out the voice element and asked a question about that. So if you have confidence in those few questions that you're asking and kind of take the gamble that if asked people are going to offer it up. And so we had a pretty remarkably high response rate across both contexts. So it was 20%. If any of you have done survey research, that's pretty admirable. We weren't offering incentives. There was no lollipop after you did it. But we wanted to make it easy and again we were banking on people might actually, I hypothesize that it's actually there's a trust building benefit to even asking people.
I think even the people who saw the survey and chose to walk by might still have taken in. Oh, this court cares about what litigants think. I think there's a kind of layered benefits to that. So we wanted the experience to be easy for the litigants as well. So they answered their few questions. Some people fall off after the first question and don't answer the others. That's fine.
But remarkably many make it to the that last one where again they can put in some open ended comments.
We've heard lots of concern that people are going to write crazy, alarming things that then will pose all kinds of ethical problems to the court. But we really didn't see that. We haven't seen that in any of the projects where we've done this. I think at a minimum we're giving people a little bit of dose, a little dose of voice to give feedback about their experience and the hope that there's someone on the other end who's going to take in that feedback and actually use it to inform policy and practice. So I'm not sure if that's what your question was getting at, Renee, but that's what I hope the experience was for the litigants in the study.
[00:32:24] Speaker C: And we did. We have done a couple report outs for the Utah judiciary and each time, Emily, you've included a little sampling of the kind of free text output in addition to sampling from the full bit of questioning research.
But I will say to the credit of the judiciary, they have been very receptive to both the positive and room for opportunities available to them for change. And I think that's they're willing to see and hear from the litigants and all of the suggestions, as you mentioned, were legitimate suggestions like take a minute to let me respond feedback that could be an opportunity.
[00:33:13] Speaker D: Can I add one piece that I'd
[00:33:15] Speaker C: like to add, please?
[00:33:17] Speaker D: I think that the combination of the two, the like here's what the data shows and here's the numbers and statistics and N equals this and all that stuff. People like that. And I think it being combined with some sort of story. I think everybody at this point is familiar with the concept. There's no stories without data, no data without stories. And so the quotes and the actual feedback that Emily shared, people were like, okay, there's data. And then they would like, wait, they said what? Like that. Actual feedback. I think judges are often hungry for that feedback. And if you think about a judge's perspective or a judicial officer's perspective, you're just doing your stuff. And maybe once in a while you're going to get feedback, but you don't really ever get feedback about how you're doing. Is this good? Is this helpful? You're going to find out once every. In Utah, once every six years. And the rest of the time maybe you'll get over a turn on appeal. And if you don't, then maybe you're not doing it wrong or no one's appealing. You don't know. So I think that those pieces of actual constructive feedback give me more time. Really. We're like, you could tell that while they're uncomfortable sometimes with changing or process because everything's so complicated in a court, I think there is a hunger for how can I do better? Because people, no one's working in a court system to make a bunch of money. They're not there because they care about fabulous wealth or power. They're there because they feel called to serve in that capacity. And so getting that feedback is, I think people are just sponges sucking up that kind of feedback. So it was really, you could see it when you presented this, that they were laser focused on those sorts of things which you need to watch.
[00:34:56] Speaker B: I love that point, Nathaniel. And for any of the listeners who haven't seen the couple of prior publications on this, I also wanted to name the feedback is very positive, right? And this is, yes, the court commissioners we worked with were amazing.
And it's not so, so different than when we've done this in other courts. And so I think in that vacuum of feedback, I think it's very easy for judges or really any professional to assume in the legal context that, oh, gosh, we're ruling for some people and against others.
And the people we rule for are happy and the other ones are unhappy. And so we must have about a 50% approval rating on any given day.
But that's really a myth. This, this research shows us that people formulate opinions of fairness not just on outcome, but overwhelmingly based on their interactions with those authority figures. Right. Those elements I just named. And so judges really can influence perceptions of fairness, even if they're going to be ruling for half and against the other half, or in family law matters where both parties might walk away feeling like they lost something significant. And we saw in person the response rate, response ratings to those that question did the court treat me Fairly today was 84% positive.
So that's procedural justice at work. That 8 out of 10 people were getting that kind of fair experience from their interactions, even if the court wasn't citing for them with them in that moment. And I think that's really powerful and I hope empowering to the professionals because they really do have influence over that to promote, continued, enhance and push the bounds of these fair experiences, while, yes, hopefully making outcomes more fair along the way too.
[00:36:53] Speaker C: And just to also consider the context of this within, like Nathaniel said, Utah already has a kind of robust judicial performance review process, but it happens once every six years, it sounds like, or if an issue is appealed, then there might be some interim knowledge of performance that may need modified. But it's unlikely that these types of issues will be the subject of appeal. And so it really is good on the litigant for preserving that issue for appeal, but it really is that this is the only way to get more real time and usable feedback rather than once every six years, which is really not frequent enough to feel like you need to make these small tweaks in kind of the way you present in the courtroom. Right.
So just to round out and remind everyone in the rct, we're looking at time to disposition, durability of court order, which is really thinking about the idea of how long does it take for a family to feel like they need a modification or an enforcement of a prior dispositive order.
And the hope, if a court of an order is durable, the hope, of course, is that never happens or that the length of time in between the dispositive order and that request is longer. And then we're also looking at failure to appear.
Emily, tell us more specifically about the outcomes, if there is more for you to tell us about the outcomes you're studying.
And then for both of you, I'd like to hear what you're most interested to learn as we were getting ready to release all of the results.
[00:38:41] Speaker B: Yeah. The benefits of working within an RCT is we were able to see whether the randomly assigned litigants who showed up in remote court had different perceptions of fairness than those who were randomly sorted into in person.
And we did some coding of the qualitative feedback that came through those as well. And so the punchline is that the experiences didn't seem to differ the that 80, 84, 85% that I just mentioned. That was the experience among in person but against where it was about 20 percentage points lower of remote appearing folks. And so when you trust that really is quite random. Now footnote, we did see some people hopping, hopping over to a different one or maybe being assigned by the judge, but for the most part randomly sorted folks. They did seem to be reporting a different experience in online court before the exact same court commissioners. Right. We could all imagine that different judges have different styles and these questions could be answered differently. But so there really is something in terms of those those four elements. Something is getting missed when we are trying to translate that to a remote appearance. We saw a lot of comments in response to the open ended question about it was very hard to feel respected in a context where you don't know when someone's speaking to you.
You don't know when it's your turn to talk or you speak and then someone kind of scolds you for interrupting them.
And we've all been in zoom meetings like that. Right. It doesn't have to be a court appearance where it's like oh, you go oh no, you. Right. And that's among relative equals.
Whereas here when you have an authority figure who I believe is doing their absolute best to govern this multi multiplayer meeting every day, it can be really hard. Right. And so people who may come in with significant mistrust of the system to begin with, it's pretty hard to correct that. Right. With just these when their experiences include this sort of confusion at best. So we were looking at how perceptions of fairness varied. It's interesting that it seems that despite those differences, there weren't huge differences between the compliance in terms of showing up for court.
So people are still bringing themselves. Appearance rates weren't dramatically different. But we know that if people experience court as fair, they're more likely to show up next time. Right. So if we could help to do some remediation on online courts, I think those numbers could improve and might help make the case for policymakers or court leaders who are really wanting to invest more in in remote court level that playing field. And there's a study out that I know is cited in our recent article that basically tested how to boost the impact of remote court and they in fact found that procedural fairness was at or above the experience of those in person. So all of this needs to be viewed in a broader context. But I think we sh. This at least shows that that perceptions of fairness, I think without special care can really take a hit in remote court.
[00:42:02] Speaker C: And just for our listeners knowledge, that study that Emily references is by Victor Quintanilla at the Indiana University.
Definitely worth reading and we'll put it in the notes of the podcast.
Okay, so I think we're coming up on the amount of time I ask from the two of you and I like to be respectful of that.
So Nathaniel, I want to give you an opportunity if you have anything, any follow up from what Emily just said, which I think is a great kind of encapsulation of the work and kind of the policy issues that might move forward as a result of it. And also so any follow up from that, but also leave us with some parting dreams and wisdom.
[00:42:43] Speaker D: I'll see what I can do. I think the follow up to what Emily said is just that the perspective is that we as society, as a culture, as court systems, as a legal system, have hundreds or thousands of years of experience fine tuning and perfecting with small little adjustments the in person hearing process. And so it's pretty, it's a pretty good device that we've whittled down to be exactly the way we think it should be in our society. And the online thing is like we've got about five years now at most of experience doing it in mass and really thinking about what works and what doesn't work. It might be the case that it's not as fine tuned as the thousand year experiment that we've been expecting. That doesn't mean I don't think we shouldn't keep refining it or trying it. I think it just means the intentionality around how we do it needs to continue to be at a high level for now as we fine tune those pieces. Because if not the case that you can just take it from one modality and put it into another modality. All the things Emily's talking about, who speaks when giving people time. Also the frustration judicial officers and clerical staff had with doing the work. There's stuff that I think is hinted at in some of the writing but not said explicitly. Like in a physical court, you know who's directing traffic? The bailiff. The bailiffs saying now it's your attorney here, check in over here, do all these things. And the bailiff's like, that's not a core security thing. So I'm not doing that in an online. What is my role there? I don't actually have a role there. And so who's doing that in an online setting when everybody else is distracted with all the technology pieces, those aspects were really very present when we were digging into it and talking with the clerical staff and the judicial officers about the project.
And so I think it just seems like it's still new and we're still figuring it out. It seems like it does have the power to be meaningfully helpful in terms of creating opportunities to engage with the core process. But I think that the other piece, maybe it's not wisdom, but perhaps hope is that the perhaps encouragement and a plug for the lab is that there were so many other ancillary benefits that came from the partnership. As one example, when you guys first got involved, you gave us a whole bunch of data about here's who people are. You did an overlay of zip code and help us understand where people are coming from when they're coming to the pro se calendar. And it might surprise no one that people who are coming on a pro se calendar where they can't afford lawyers are coming from more low income areas in the Salt Lake City area. And so that informed a different project to bring community court to those neighborhoods is to actually bring court services, not just legal services, but court services to those areas. All these conversations about thinking about kiosks and thinking differently about that, the perspectives around why aren't we asking people more what their feedback is? All of that was part of this long, deep engagement process. I think because it was a long study, it forced us to continue to come back to it and pay attention to it and stay thinking about it instead of just getting through the day, getting to another week. And so it was just a really rich and enriching experience, I think for the judiciary. And it's, I think everybody is very pursuit for that.
[00:46:10] Speaker C: Thank you. That's so nice. What a great endorsement. I really appreciate your partnership and friendship in this project. Now, Emily, what parting wisdom or dreams do you have?
[00:46:23] Speaker B: Yeah, I think remote court is here to stay in many parts of the country.
And whether it is or not in person, court I think too could benefit from really centering the experience of the litigants. And I think some people that resonates with philosophically for others, you can dream about the improved compliance and cooperation that we get when we center the experiences of litigants, which I think every court, you know, every, every justice or legal agency wants people to show up at appointment, do what's being asked of them.
And I think perhaps now more than ever doing what we can to boost the legitimacy of the whatever forms of government where we work is it is meaningful and it starts in individual courtrooms. We know that from other research. I don't trust the courts but I trust Judge Dancer. That's how the progression happens. And so move is not in the cards just to be I guess my also hope not sure if it's wisdom would be that for the folks listening to just take advantage of the I guess procedural justice is turning 50 this year the Yale Law School folks tell me and there's a lot of resources out there to whether it's an in person court or remote court or your probation intake office or victim witness services to apply these practices and to explore how you might actually measure those because there really is opportunity to promote trust in these both one on one and more systematically.
And I figure we might as well it's going to kind of grease the wheels on cooperation and just fundamentally we know makes our system more fair which is I think the goal of most folks who I get to work.
So thank you for to Renee and Nathaniel for this project and I also didn't want to end without just also sharing gratitude to the State Justice Institute which brought Renee and my team together on this. They've been longtime supporters of procedural fairness as a priority. So appreciate their yes and their funding and support.
[00:48:44] Speaker C: Thank you for bringing that up. And yes the State Justice Institute has supported our work and supported Emily's work and also support said this work really needs to have both components and they were very clear about wanting that the both components so that they could inform their constituency. And we're grateful to Utah for being willing to serve with us and I'm grateful to both of you for spending this time with me today.
[00:49:10] Speaker A: Proof Over Precedent is a production of the Access to Justice Lab at Harvard Law School.
Views expressed in student podcasts are not necessarily those of the a Japanese lab.
Thanks for listening. If we piqued your interest, please subscribe wherever you get your podcasts. Even better, leave us a rating or share an episode with a friend or on social media.
Here's a sneak preview of what we'll bring you next week. And today we're going to be talking about a study that the Access to Justice Lab recently completed that did not produce the scientific results that we hoped that it would. And so we're going to talk about the study process and what happened and what we hope to find and what we actually were able to find out which was scientifically very little. Perhaps we can have we can draw a practical lesson, but with a lot of caveats and a lot of hand waving.
