Episode Transcript
[00:00:00] Speaker A: Imagine a justice system built on rigorous evidence, not gut instincts or educated guesses about what works and what doesn't.
More people could access the civil justice they deserve.
The criminal justice system could be smaller, more effective and more humane.
The Access to Justice Lab here at Harvard Law School is producing that needed evidence. And this podcast is about the challenge of transforming law into an evidence based field.
I'm your host, Jim Greiner and this is Proof Over Precedent.
Welcome to another edition of Proof over the Access to Justice Labs podcast series. My name is Jim Greiner and today I'm going to serve as your host for everybody's reminder or to remind everybody I should say. I am the Access to Justice Labs faculty Director professor at Harvard Law. And my fun fact for the day is that I am a big fan of hard boiled detective mystery stories. Mystery stories written in the genre of Dashiell Hammett, Raymond Chandler, Ross McDonald. And I have just finished listening not for the first time, but I've re listened to an audio version of Walter Mosley's Devil in a Blue Dress, which is absolutely outstanding. I recommend it to everybody and has a very good audiobook reader on Audible. Forgot the person's name, but anyway I could pick that up. So there's my fun fact for today. And today we're going to be talking about a study that the Access to Justice Lab recently completed that did not produce the scientific results that we hoped that it would. And so we're going to talk about the study process and what happened and what we hope to find and what we actually were able to find out, which was scientifically very little. Perhaps we can have, we can draw a practical lesson, but with a lot of caveats and a lot of hand waving.
My stars today of the show are Renee Dancer and Patricia Gansert. So two folks familiar to Proof Over Precedent listeners, both of whom are folks at the lab. And so why don't we get some introductions. Patricia, can you remind everyone who you are and a fun fact about you.
[00:02:05] Speaker B: Hi, I'm Patricia. I didn't know we'd have to come prepared with a fun fact today. So it may not be that.
[00:02:10] Speaker A: Come on, you always have to have that.
[00:02:13] Speaker B: And I was the lead data analyst on this project.
I like making amateur video games as a hobby.
[00:02:21] Speaker A: What's the most, what's a recent amateur video game that you've made?
[00:02:25] Speaker B: My, My White whale. The project I'm still working on is My Fire Emblem. That's a popular video game series for Licious. George the Storky is being my Fire Emblem fan game. I finished the full script. I have all the graphics and portraits. Still doing programming on the level. Got it thumbed in.
It'll be my great episode.
[00:02:42] Speaker A: Fantastic. When you're rich and famous for having produced that game, please remember your friends. And Renee, why don't we get get you to remind everyone who you are.
[00:02:52] Speaker C: Hi. Thank you for having me again. My name is Renee Dancer. I'm a researcher at the Access to Justice Lab. I was the researcher on this project.
I'm excited to get a book recommendation from you, Jim.
[00:03:02] Speaker A: I have.
[00:03:03] Speaker C: I use my Libby app with reckless abandon. So we'll put all of them in there. My fun fact this time is I have very good facial recognition.
[00:03:18] Speaker A: Excellent.
[00:03:19] Speaker C: I remember almost every face that I've seen. Now I may not remember names, but it's. It can be awkward at times where like I will remember someone I briefly met or didn't meet, just saw and they will not remember me.
And I have the ability to remember people as they age. So like I can pick out actors as when as adults, when they were actors as children and say exactly who they are. It's both a blessing and a curse.
[00:03:52] Speaker A: Renee, I think you're quite memorable. So I bet most people that meet you remember who you are as well.
Super. Let's launch right in here. Renee, we'll start with you. Tell us, give us the background of the story of this study. You basically constituted it from the beginning. Who were the field partners and what was the origin story of it and what were we hoping to find out?
[00:04:17] Speaker C: Yeah, so we worked with the third judicial District in Utah, which is Salt Lake City. And we were positioned in the family court there working with the commissioners of the family court. The commissioners are family court adjudicators. But they cannot ratify a court order that needs to go to a district judge. So it tends to be something like in the middle, like a hearing officer could be sometimes called, would be like a quasi judicial officer. So you might see other quasi judicial officers like this in other settings.
They actually a staff member at that court reached out to me because they were reconstituting their operations after the pandemic and they wanted to understand whether or not they should use in person proceedings in family law cases specifically focused on self represented litigants or use remote proceedings. And they had emerged from the pandemic with a difference of opinion on the bench and they decided they wanted to have evidence to make the decision about what they should do.
And so we worked on a randomized control trial for them where we were randomly assigning cases to either participate exclusively in person or exclusively.
[00:05:41] Speaker A: When you say participate in cases, where can you remind us what the case type was and what sort of participation we're talking about case types plural, I think, if I recall correctly, I believe there were two.
[00:05:52] Speaker C: Yes, that's correct. So we were working with divorce cases and custody cases. So I said family law matter. So we isolated to specifically divorce and custody, not other types of family law cases.
We're focused on self represented litigants. And in family law, and especially in the divorce and custody context, cases can start, go dormant, reactivate, and there's a heavy motions practice in family law.
And so when we say participate, we mostly mean through motions practice and motions argument and motions for the non practitioner out there would be a request to the court to resolve some typically interim issue. Right. So some issue prior to the disposition of the case. And we talk about this in the final report and give a few examples. But my practice area prior to non practice was family law.
And I worked with a judge who liked to refer to some of the inconsequential orders as the broccoli orders. And I actually used that example to describe inconsequential requests from the court that families may have. Things like determining whether or not a child should be permitted or precluded from eating broccoli for dinner is one type. But there's also very consequential issues too, like determining custody schedules or in the divorce context, determining the asset distribution that's fair and equitable among the parties. And those things can be done through motions practice and also through like dispositive trials and hearings. And those were the things we were looking at here.
[00:07:34] Speaker A: Patricia, just to go to you very quickly for clarification, I'll come back to Renee for a little bit more. Patricia, if a case didn't have a hearing, then what did we do with it?
[00:07:44] Speaker B: Yeah, so we did have, in theory, every case in our study should have had at least one hearing because they turned up for our study because they had a matter that had to come before the court. But we did have a small number which simply did not have any cases at all. And for those some variables we could still calculate as things like the length of the case. We can calculate from the time they joined our study to the time that resolution occurred. But sometimes the variables like failure to appear and number of hearings they simply excluded from analysis because they didn't have any. And for curious listeners, the reason why we don't list them as zero is the problem of Being equal zeros. Right. Saying that they never failed to appear is not impressive. If they never had any hearings, it would equal the analysis a little bit, since that's not quite an accurate description of what they really did. So they're just, for those analyses, the sample is a little smaller.
[00:08:30] Speaker A: So there's a difference between zero failures to appear, meaning you made all your hearings, and zero failures to appear, and meaning you never had a hearing because you didn't and therefore you didn't have the opportunity to appear. And so we did. We wanted to make sure that we distinguished that for the failure to appear outcome. But actually that, that brings us to the question here, Renee. So basically what we're going to do is when a child in an appropriate child custody or divorce case comes in on a motion or on some, something that will require a hearing for a certain period of time. So it could also be, you know, request to set the case for the, for the dispositive trial, although that was relatively rare. We're going to randomize the case and say you're going to do all of your hearings in person or you're going to do all of your hearings via video conference.
And just to add, just to clarify, did the court require a particular video conferencing technology? For example, did it use zoom or teams or what? Or was it, was it flexible on that?
[00:09:24] Speaker C: No, they were using WebEx exclusively.
[00:09:27] Speaker A: Right. Okay. So exclusively Webex. And so the court then would send the notice to the parties about that. This was going to be, this was going to be how the hearings were going to be held. For those listeners who may have picked up on the fact that we focus a lot at the Access to Justice Lab on research ethics, and we're actually doing some research, some of our own research into research ethics.
Was this a consent study? Did we ask for anybody's consent? Or did we, or did or did the IRB say that you don't have to request consent in this context?
[00:10:01] Speaker C: We did not ask for consent. The IRB approved a non consent study. Again, that's something that we address in our final report and offer the approval letter as well.
[00:10:11] Speaker A: And what. We're just at a high, very high level. Why is it that the irb, I think correctly, that you don't need to request consent in this context from the litigants?
[00:10:21] Speaker C: Yeah. So pre pandemic, all hearings were held in person. There was no choice or other option for litigants. And during the pandemic, all hearings were held remotely. There was no choice or other option for litigants so there was never a time during the study or before the study where litigants were offered an option on how they were going to appear during the study. We were also dictating which mechanism, which method they would medium they would use, but both were in play, but never giving it, still not giving an option to the litigant to make that choice. And so this was actually more status quo. There was no the study itself wasn't changing the idea that the litigants never had a choice. And so it wasn't removing any quote, unquote. I would not suggest it. The choices are right to begin with, but they it wasn't removing that right. And then all of the data that we collected also was all publicly available data. So we weren't needing to collect sensitive data that litigants already didn't have on available to the public at any point. And so there was no need to ask them permission to collect their data.
[00:11:36] Speaker A: And so this is another thing that we sometimes have to educate people in the ethical world, put it that way, on the world of IRBs, et cetera. The court data in most states, almost all court data is publicly available on a public accountability argument that these are key public institutions and the public needs to know how they are doing, what they are up to. But in some states, for instance, I believe in Pennsylvania, where you came from, Renee, divorce data is not always presumptively public or at least it's not up on the Internet in some portions of Pennsylvania at least. But in Utah it was.
In Utah it was up on the data are up on the Internet, is that right?
[00:12:17] Speaker C: That is correct. In Pennsylvania, in some jurisdictions you'll find pleadings are typically redacted, but that's neither here nor there. Utah, yes, everything was up there. Most pleadings were publicly available.
Things like demographic information of the litigant was also publicly available or birth dates. So you'll we were able to acquire age for most participants, not all of them, not birth date wasn't consistently there, but it was typically there.
[00:12:44] Speaker A: Yeah, super. Okay, so that's the basic study set up. We're going to randomize whether a case has its hearings in person or has its hearings online.
And then Renee, back to you for one more question before we get some results from Patricia, or as we're going to find out, we're going to learn about the absence of results from Patricia. Renee, what sort of outcomes were we studying and why were we picking those?
[00:13:07] Speaker C: Yeah, so we studied time to resolution, durability of court order, failure to appear and Then we also looked at court burden. And so time to resolution is fairly self explanatory. But in this instance, as Patricia mentioned, our cases were coming into the study at different postures. And so I mentioned cases in the family law context can activate, go dormant, reactivate. We were the rolling cases at the time that they activated and needed some sort of court action. So that could mean it was the first filing on the case and they were ready to go to the judge right away to make some sort of interim decision.
It could have meant that they were in the interim of that first filing and a disposition on the case and they needed something. Or it could mean that there was a disposition and now they've come back to make a request to either modify or enforce that disposition. And so we had cases that were coming in at different postures. Patricia will tell you how we dealt with that. But time to resolution was the time from that thing that brought them into the study to the resolution of the thing that brought them into the study. Now we also were able to calculate to look at time on the whole for the case. And that was one measure of court burden.
Durability of court order was meant to look at that time between a disposition of some sort and a request to make a change on that disposition, a modification or enforcement. So we were hoping to see long periods of time or never see that action happen. Meaning that the study participants were pleased with the order, seemed to be working for the family, they didn't need to have, the order was durable. Right. That was our definition of durability. And then failure to appear is again, fairly self explanatory. Did parties appear to their proceedings? We, we at, we looked at this because one argument both for and against remote hearings, depending on who you ask, is that it may improve or make failures to appear worse. So we wanted to know which of those were true. And then court burden is like the idea that family law cases can have a lot of filings and can be very contentious. And the more contentious a case is, like how means the court has to be more active in that case. So how much of a. Do either of these mediums change the burden on the court at all? We were particularly interested in that because up until this point, no one was really looking at the medium in the. From the lens of anyone but the court. So if the court is not burdened, and then maybe there is some reason to look at the lens at a, at the medium from a different lens.
[00:15:57] Speaker A: Yeah. One of my favorite parts of this study is hearing from people generally in the adjudicatory community put it that way, from people are involved in courts and judges.
I'm making this up to make the point. At 10:30am I have a conversation with someone who says, man, gosh, getting. Dragging people down to the courthouse and making them appear that way, that just is a, that just is a recipe for failure to appear. I mean, they've got to take a day off work, they've got to travel, they've got to figure out where to park, they may not know where the courthouse is, they got to get through security, et cetera. You just, you just get so many failures to appear.
And then at 11am, so 30 minutes later, having a conversation with somebody saying, man, we should be going back to in person hearings. I can't understand why we're doing this online thing.
Nobody takes online seriously. And so they're not showing up. It's causing all kinds of failure to appear. And just wishing that I had recorded both conversations so that I could play one to the other. So this is a, this is again, for those who may have followed some of the access justice efforts on ethics, this is a classic case of equipoise, where there is a lack of knowledge about which of these two mediums is superior in all of these outcome variables. And even though particular members of the field may not be aware that they are collectively in equipoise because they don't talk with each other about it, if they did, they would quickly find out that there's a massive disagreement here and which we were hoping to provide some evidence on.
Patricia, just very generally, very briefly, what did the data look like in this case?
It wasn't exactly just download from the, from the court website and then slam it into a statistical package. Right. There had to be some work. And just very briefly, what did that look like?
[00:17:48] Speaker B: Yeah, this is a very interesting one because in some ways the data is very simple and in some ways it's very complicated. So in terms of simple, we didn't have that many data sources. Each case has a docket a minute, and the docket is just a table that has a list of events that happened and the date they happen. And then the minutes are just a much more detailed version of that, including some events that the docket, the docket sheet doesn't have, minutes of individual hearings, things being submitted, various filings, and we also had a little bit of personal data about just their name, gender, potentially their age, et cetera, but not very much. We have studies where we have many, many more data sources. So in theory, Very straightforward. But turning the dockets of minutes into data we could actually analyze with incredibly complicated, because they especially the minutes. This is a very rich, detailed data source. This is literally a record of everything that happens in the case. So it has a ton of beta, but it obviously is a record that wasn't designed for researchers. It was designed to just keep track of the case.
So we used a lot of different tools. The main one being these are PDFs. We have documents that showed zoom, were typed on a typewriter based on the appearance of them. So we're using optical character recognition tools, ocr, which basically means the computer can look at a document as a picture and extract text from it by calculating what the images on the screen are as letters.
But that looks very messy because the documents themselves were a little messy. Some fonts don't work very well with optical character recognition. Document quality can matter if you have sort of parts that are a little faded, if you have things that are tilted, et cetera, that can lead to some errors in what gets recognized. And the documents, the minutes aren't structured in a way that makes a lot of sense to human beings, but not necessarily as much sense to a computer, where you have two columns, a date and then an event right next to it. Right. And for a human, we naturally read across.
But for a computer, when you have a table like that, where there's no line delineating the rows and columns of the table, just white space, computers do not necessarily read them this way. So you can have. We'll read the first column and just stick that in, and then we'll read. And columns just stick that in. And sometimes bits and pieces of those columns will just be interspersed together, depending on how the spacing was at different times.
And so when you have tables where there is no text or symbol that indicates the end of an entry, for a human being looking at it, we can tell because of LightSpace, you can end up with a big mass of text that doesn't have a clear way to link different parts.
So the process of cleaning that and turning into the data was. Was specifically, it did involve some amount of hand review and hand coding. It did get quite involved, but we did manage to do it. Ultimately, it's why we were able to do some lab analysis you'll probably hear about later, because we extracted a lot of individual, specific pieces of these court minutes in order to get very detailed data down to the level of individual hearings.
So very interesting data to work with. Sure.
[00:20:40] Speaker A: And so we eventually able. Let's take for Example, failure to appear of the example of the primary outcomes that Renee, you identified.
Take failure to appear. We eventually were able to classify, most of the time whether the petitioner, meaning the part, the party that initiated the the need for there to be a hearing, you can't say initiated the case because the case may have already been pending and it might have been the defendant who was the petitioner for this particular issue. But anyway, the petitioner, meaning the party that initiated the reason for there to be some sort of need for a hearing, and then the respondent, the person on the other side. We were able to code, for example, most of the time whether the petitioner and the respondent each was present at the hearing, but not always. There were some that we couldn't characterize, for example. Is that correct, Patricia?
[00:21:35] Speaker C: Yeah.
[00:21:36] Speaker B: Talk a little about how we did so that people understand why that's the case. So the court, as far as we can tell, doesn't really keep track of failures to appear in this level. Obviously, people maybe they're getting contempt of court, whatever they're going to use at being. But there isn't actually a big table telling you the dates and times of that. So. But what we do have is in the minutes, every single hearing has an entry, and at the start of that entry there is a list of the parties present, you know, who showed up. And that can include the share of the pundit, include lawyers, et cetera, et cetera.
And so what we could do was for every single hearing, look at whether the name we know who the petition respondent are and see whether those names fit. The format of the name is obviously very important because people can have names, maybe they do or don't use their middle initial, et cetera. So what we did was we took the docket sheet, which at the top for each state listed all the parties in the case, food, petitioner, business, attorney, frontline, et cetera. Then you match that to text in the minute to see whether each name of the patient respondent appeared at each given hearing. Now, there is a caveat to that, obviously, which is that if a name is absent, the all we can conclude is that that person was not present. We don't know whether they were supposed to be present because the minutes do not tell us who was meant to show up. So if there was a case where certain parties, maybe neither party showed up, it could be the case that maybe this was a hearing where only the attorneys had to be present. It was kind of court matter. We can't be fully certain of that. I will Say that those were pretty rare at the majority of hearings.
I think this is a pretty accurate method, but because of the way that the court records this information, all we can know is presence or absence of the person. We don't know who would oppose the beat.
[00:23:17] Speaker A: And this is a frustrating aspect of many court systems of records and high volume courts put it that way.
In courts that involve a lot of volume, a lot of case activity, they frequently, or almost never put it that way, have a box or a dedicated field in the computer system that simply says, was the petitioner or their lawyer present?
One means yes, zero means no.
Was the respondent or their lawyer present, which parties were present, et cetera. They typically just don't have that box.
And that's an astonishing thing for many law students to learn.
It's a way of. When I talk about that to law students, it's a way of attempting to communicate, you know, just how difficult court data is to work with and how little the court system tracks about its own operations. Court systems typically track about their own operations.
So we didn't have that. We had to go through minutes often to get it. And then we had. We were able typically to code matter duration and durability. There were some judgment calls to make about matter duration.
So, for example, it required characterizing exactly what the matter was that would. That brought the case to the court's attention. It necessitated the hearing. And then when that was over the most part, those were problems we solved. And at some point, when we shared our preliminary findings, which, of course, we'll talk about the findings in a moment with the court, they were fairly surprised. Renee, when we went through a verification, a partial verification, a spot check, call it of the failure to appear as well. Can you tell us just briefly about that?
[00:25:02] Speaker B: Yeah.
[00:25:03] Speaker C: I think one surprising result of this is that most people showed up right. Like there wasn't a problem of failure to appear, and it didn't matter. The medium didn't make a difference. I think lots of people that we talk about this with are surprised by that. They think failure to appear is this. Like, it is a prevalent issue, and maybe it is other places, but it just wasn't here. And I think what Patricia mentioned right. Was most people showed up right. And the court did help us spot check. And we wanted to make sure we were picking the right words and phrases. We were using the right. The entries were not uniform, but there was some. We were able to categorize them into types. So there was some typology there, but we checked that with them. They thought that was all correct too. So we appreciated all of that help, of course.
[00:25:57] Speaker A: And then at some point you listened to some audio recordings, is that correct?
[00:26:01] Speaker C: Oh, yeah, we listened to some trans. Thank you for reminding me. Yes, we listened or I listened to some transcripts just to make sure. We could spot check. Just to spot check the minutes based on maybe the transcripts, because the court thought maybe the transcripts will have better information and will reveal more than the minutes. They, the minutes are not a full transcript. It's not like a stenographer transcript. But most of the relevant information is there. They read like a full transcript, but they are not a full transcript. So I listened to the transcripts and it turned like you really need context clues or to be a practitioner to understand if someone appeared. It was very rare to actually call a case and ask for appearances from the even council.
And some, for some instances, like, it was, it really was like very unique and context specific for like, things like, oh, this person, I saw them at the magnetometer.
[00:27:03] Speaker B: Right.
[00:27:04] Speaker C: Like that's how you knew someone wasn't in the courtroom. Or you could, you could infer they weren't in the courtroom, but someone thought they were in the courthouse. But you would never really.
That wouldn't be picked up, be easily picked up through any sort of machine reading process because it's so unique and context dependent.
[00:27:23] Speaker A: But I think we looked at something between, in terms of the transcripts, 20 to 30 or so. Is that about right?
[00:27:31] Speaker B: Yeah.
[00:27:31] Speaker A: And all but half of them.
Excuse me. In all but half of one of the cases and all but half of one of the cases, the data that we already had from the process that you described, Patricia, the turned out to be the same as the, as what you coded from the transcript, is that right? There was one where we could. Where it was either it was already categorized as present and it moved to uncertain, or it was uncertain, it moved to present or something like that, where it was sort of a halfway move. But almost all of the time the minutes gave ended up giving us the same thing as the transcripts. Was that right?
[00:28:03] Speaker B: Yeah, yeah, yeah. In the vast majority of the time, the transcript pretty much confirmed the coding we had done. So that gave us more confidence that we were correctly using the minute data. Yeah.
[00:28:14] Speaker A: So we ended up again, after a lot of work, we ended up with accurate data.
And so it seems like we're going to get useful findings here in the big picture. Renee, tell us why we didn't get useful findings here. And then, Patricia, you're going to go to you for a lot more detail. What was the big picture problem, Renee?
[00:28:34] Speaker C: The commissioners or the court just didn't follow the randomization protocol. Right.
[00:28:42] Speaker B: So they.
[00:28:43] Speaker C: That's the simplest terms. So cases that were supposed to proceed virtually all the time, more often than not, did not. Or too many too frequently did not.
And cases that were meant to appear in person, all each. For each hearing, again, too frequently did not they appear. There was. Were virtual appearances. And when we. And Patricia will talk about how we looked at that, because we looked at that both from the case perspective.
[00:29:18] Speaker B: The.
[00:29:18] Speaker C: And the hearing perspective as well as the.
We.
Yeah, the case in the hearing perspective.
[00:29:26] Speaker B: But
[00:29:29] Speaker C: what kind of comes out is like a little bit of a preference for in person hearings.
[00:29:35] Speaker A: There seem to be more preference among the commissioners. Right?
[00:29:38] Speaker C: Yeah.
[00:29:38] Speaker A: Among the litigants. Right. The preference.
[00:29:40] Speaker B: Right.
[00:29:40] Speaker C: Among the commissioners, there seem to be, you know, more compliance when cases were meant to participate in person and less compliance when they were meant to. To participate virtually. But it was still like even that. I mean, you can see that. But it's.
It's still a lot of non compliance. It was a lot of non compliance.
[00:30:05] Speaker A: Yeah. So, Patricia, as Renee mentioned, we ended up counting things both in terms of each case is it counts as one unit.
And then we did. Each hearing counts as a unit. So for example, if one case had five hearings, which could have happened, you know, do you count that as five different. Five different units or just five pieces of one. We did it both ways. That ended up not mattering very much at all in terms of results, if I recall correctly. But am I right, Patricia, that say again, picking failure to appear as an example, that we'll go through, the other ones actually wouldn't matter. Sorry, this is the ram. This is randomization compliance. So it doesn't matter which outcome. Looking at if for cases that were randomized. Again, Patricia, tell me if I'm wrong about this. For cases that were randomized to be virtual, to be remote appearances, it was about 50, 50 virtual versus in person.
And then for cases that were randomized to be in person, it was on the something like 70, 30 in person versus virtual. Is that correct? Right around that.
[00:31:12] Speaker B: Yes, that's correct. And we did look at it in two different ways. We looked at categorizing each case, each individual case into the nature of its hearing. So you could have every single hearing that's virtual, every single hearing that's in person. You have both in person and virtual. Some hearing. We could not strictly categorize to either. There wasn't enough information in that sort of unknown category. And then we also looked at it, looking simply at the. For any given case, what percentage of its hearing were virtual versus in person. And no matter which way you'd like this for randomized hearing, for cases that we randomized to be virtual, it's pretty much a coin cost whether there any given hearing was going to be virtual or in person. Each was just as likely when we randomized in person. They do tend more towards in person hearing.
It is panel 70 30, but there's still a lot of hearings that were not in the intended medium. One way to put it would be of. When we look at the case level, just under 50% of our cases that were randomized in person actually had all of their hearings in person.
For virtual, about 30% had all of their hearings virtually. So you can see that the level of compliance is pretty low.
[00:32:22] Speaker A: Now, again, of course, had it actually been a coin toss, we would have been okay, but we didn't think it would.
[00:32:31] Speaker B: They're evenly distributed. The first one is that we don't know the method by which the medium was chosen. We can infer that it wasn't truly random because we randomized. Again, it didn't happen, but we don't know the thought process behind it. There's some systematic difference. Like maybe one commissioner is thinking, this is such a contentious case, it really needs to be in person to make sure the parties can only be managed. Maybe another commissioner is thinking, these would have lived really far from the courthouse. It's better to do it virtually. This could mean that the different groups are different inherently before they enter the study and in ways that we can't measure. So any differences we find between them might have nothing to do with the
[00:33:09] Speaker A: hearing, which is exactly why we wanted to impose the randomization to begin with and why the court agreed to it. And then. So when the. So when the court then didn't comply with it, we couldn't figure. We can't produce any useful science, basically, is what this comes back to. And Renee, am I correct that we. That we talked to the court about this and they hypothesized a couple of mechanisms that might have explained this. This departure from the randomization protocol from the randomized assignment. And we investigated those and basically found out that they didn't seem to be explaining the patterns that we saw. Is that all accurate? And can you give us a little more detail?
[00:33:48] Speaker C: Yeah, they. They were. I'll be honest, I was nervous to have this conversation, but they were very like, they were great to work with and Even in this, in these trying times, they were great to work with.
[00:34:00] Speaker B: They.
[00:34:00] Speaker C: But they gave some. They were genuinely surprised. They really did not. They all said, oh yeah, we definitely found some cases that we thought needed to go some one way or the other. And so we did what we thought was best.
But we didn't realize it was this much. It didn't seem like it was this frequently, which lets you know that people are often like looking at one point in time and not really able to understand the full scope of anecdote. But they did offer what they said were custom or common practice to certain types of events or specific situations where they would always schedule something remotely or always schedule something in person. And so we were able to ground truth that.
And it turns out that what they thought was custom actually wasn't happening.
[00:34:51] Speaker A: It was. Didn't explain what happened.
[00:34:53] Speaker B: Yeah.
[00:34:54] Speaker A: So we were left here with kind of a. We don't know how it and how each case or how patterns of cases ended up in person assigned to be or conducted, I should say hearings, how hearings were whether conducted in person or online.
And without knowing that, we can't do any science to try to rescue the randomization, which is sometimes possible to do. If you have good information on how the departures from randomization occurred, sometimes you can try to fix it up after the fact. If again you can, you know what happened. And we to this day still don't.
And I guess the most that we thought that we could try to take away from this was okay, we did something to the assignment of the way in which cases and hearings were assigned in person versus online.
It clearly wasn't random, which is what we had hoped.
It clearly was different from what the court was doing beforehand. And so we threw a wrench into the process in some way in terms of the case assignment process.
And Renee, tell me, did anything change?
Did we see any differences that we could just at the broad level again, Patricia, with some detail, did we see any differences, starting with you, Renee, in what in person looked like, what. What remote look liked or anything like that?
[00:36:13] Speaker C: No, as a short answer, no, we did not see any differences.
[00:36:17] Speaker A: And Patricia, in any of the three primary outcomes, did we? So either failure to appear matter duration or matter durability. Any statistically significant differences across any of the three?
[00:36:29] Speaker B: No, there were no group differences.
And it's I. What we can observe just looking at the entire group not comparing is that in general the number of events is very small. There just weren't very many enforcements and modifications at all. Failure to appear was not very common. Papers were not that that long in the window that we're looking at it. Matter that came into court when it left. So although we were not able to manipulate fully the hearing medium, we could perhaps conclude that it does not appear that either medium is causing some sort of mantra problem.
Exclude the data.
[00:37:04] Speaker A: There's no real science at the end of the day here and at the end of the study, we come back to how important it is for folks to try to follow randomization protocols that we can try to produce the science that we want to produce.
In terms of practical lessons, the one that I think that we're drawing. Renee, tell me what you think. And Patricia, tell me what you think is that it looks as though from what we can tell, which is not, which is hardly scientific, that either of these two mediums work and that our thought is that court should offer litigants a choice and be willing to work either way, but we have no science to back that up. So Renee, tell me if that's what you think the takeaway message is or what you would change or add or subtract to what I just said.
[00:37:52] Speaker C: Yeah, I think that's where that's what we thought we were going to be able to show before we realized the extent of the non compliance. We thought we'd be able to show that scientifically. I do think that holds as a good hypothesis that we think that probably is what's happening. We would like someone to be able to do an RCT to show that to be the case. I think another big takeaway is that people that were appearing. Failure to appear was low. It didn't have. It didn't seem to correlate with a specific medium. Again, we would need more science, but people were appearing for, by and large to their hearings. And so maybe that's a red herring.
[00:38:29] Speaker B: Yeah.
[00:38:30] Speaker A: Patricia, your thoughts?
[00:38:33] Speaker B: Yes, I would say that although we can't draw a statistical conclusion, I see nothing in the data to suggest that there is any problem in any of the outcomes we're looking at. There doesn't appear to be a problem with watching paid computer for anybody. There's not really a problem where there's a lot of court orders failing, not durable. There's not really a problem of a lot of papers that are taking terribly long.
So although we can investigate whether those things systematically differ depending on what kind of hearings you have, we also don't have any evidence to think that there's something wrong that needs to be fixed by changing the hearing method. Either both of them be Fine.
[00:39:09] Speaker A: Yep.
So I think then the takeaway is this is a bit of a bummer. I had very high hopes for this study because this was the kind of thing that I'm getting phone calls about from people in the field saying hey, we're, we're thinking about adopting some sort of remote program or if they're already remote, they say they're thinking about some sort of in person program or we'd like to know what evidences are out there. And we had be. We had hoped to be able to produce the first RCT of this nature. One global pandemic later, the world might be looked different from what it looked like before global pandemic and we couldn't. And bit of a bummer. Most we can say is that it looks like that litigant choice would be a way to go here, but there's no real science behind it. We do our best to try to draw everything we can from RCTs that crash and burn. And we put out this one as a, as an instance to show just how fragile field operations are. And when we complain all the time that field operations are expensive, time consuming, long and fragile, we mean it.
So for example, Renee, was this a very short term endeavor for you? Is this something that only took 10 or 20 hours of your time to put together?
[00:40:25] Speaker C: No. And when Patricia talks about the data being complicated, we went through a period of time where we were down a research assistant and so I was doing all the data downloads and literally case by case downloading dockets, it took me two full work days to do all of the cases periodically.
[00:40:47] Speaker A: So not just two full workdays to do all the cases once. This is multiple times where you're.
[00:40:53] Speaker C: Yes, I say periodically because we did download them periodically because things happen right in the interim. So anyway, I know this was not a light touch endeavor for me.
[00:41:07] Speaker B: Okay.
[00:41:07] Speaker A: So I think we'll leave it there with great thanks as always to Patricia Ganser, data analyst of the Access Justice Lab. Renee Dancer, researcher at the Access to Justice Lab. I'm Jim Greiner your host for today and thank you so much to the two of y' all and we will learn as much as we can from this endeavor and hope for better luck in the future on some of our studies. Proof over precedent is a production of the Access to Justice Lab at Harvard Law School.
Views expressed in student podcasts are not necessarily those of the ADA Jeff Lab.
Thanks for listening. If we piqued your interest, please subscribe wherever you get your podcasts. Even better, leave us a rating or share an episode with a friend or on social media.
Here's a sneak preview of what we'll bring you next week.
This study is about in a very broad sense, it's about the effect of having a lawyer versus not having a lawyer in immigration proceedings.
But there are, as several podcasts that have already posted on Proof Over Precedent have clarified, there are a lot of different types of immigration proceedings. So let's start. First of all, setting the stage.
