Episode 25: Legal Labyrinths Reveal Divorce Filing Woes

[00:00:01] Speaker A: Imagine a justice system built on rigorous evidence, not gut instincts or educated guesses about what works and what doesn't. More people could access the civil justice they deserve. The criminal justice system could be smaller, more effective and more humane. The Access to Justice Lab here at Harvard Law School is producing that needed evidence. And this podcast is about the challenge of transforming law into an evidence based field. I'm your host, Jim Griner, and this is Proof Over Precedent. Welcome to another edition of Proof over the Access to Justice Lab's blog that furthers the efforts to transform law into an evidence based field. I'm your host, Jim Greiner. And with me today to discuss a paper that has been published for some time now, but has never featured on our podcast are Rosanna Summers and Tom Ferris. And so both of whom were co authors on this paper. And so let's get each of you to introduce yourselves. So, Rosanna, why don't you begin. Who are you? And tell us a little bit about what you're interested in, what you've been working on recently. And tell us a fun fact. [00:01:23] Speaker B: Hi, I'm Rosanna Summers. I'm a law professor at the University of Michigan Law School. I am also a social psychologist by training. My research is at the intersection of law and psychology. I study consent and other questions about how we decide when people are acting freely, voluntarily, making decisions on their own. Currently, I recently had a paper published with my co author and longtime friend Jillian Jordan, another social psychologist, on how people perceive victims of sexual assault and the way that victims get evaluated by others when they have or haven't seemed to consent to certain aspects of the situation that occur before the assault takes place. If you're interested in that paper, it was recently published in pnas, the Proceedings of the National Academy of Sciences. And the the title is Sexual Assault Victims Face a Penalty for Adjacent Consent. [00:02:37] Speaker A: Terrific. [00:02:38] Speaker B: I should maybe double check that, but yes. Okay. And my fun fact, my fun fact is that I'm one of those people who. For whom cilantro tastes like soap. So for a long time I just thought I didn't like green garnishes. And then sort of, I think I was like in college, I was pretty old when my aunt mentioned that cilantro tastes like soap for her. And my cousin chimed in, me too. And then I realized, yeah, that's exactly what it tastes like. [00:03:11] Speaker A: Soap. That's a thing for certain people. [00:03:13] Speaker B: Yeah. And it must be genetic because one branch of my family all feels the same. [00:03:19] Speaker A: So I don't. It doesn't taste like soap. For me. But I actually don't like it at all because I was abroad for eight months in the Dominican Republic and sick most of that time in and it's in all, almost all Dominican food. And so now I've got this sort of, you know, classical conditioning thing going on with cilantro so you and I can hang out and have and eat cilantro free meals. Rosanna. I'm looking forward to that. [00:03:44] Speaker B: I would love that. I always feel jealous of people who go on and on about how great cilantro tastes because whatever they taste, I can't perceive it. It just tastes like soap. [00:03:53] Speaker A: Terrific. Super. Tom, why don't you tell us who you are and what you're up to now and something you're interested in and then a fun fact. Sure. [00:04:04] Speaker C: My name is Tom Ferris. My educational background is that I have a law degree and a PhD in statistics. [00:04:12] Speaker A: There's only a few of us out there. [00:04:14] Speaker C: There's only a few of us out there. Thank you for convincing me to travel this path, Jim. [00:04:21] Speaker A: Terrific. [00:04:22] Speaker C: These days I work in the private sector at a very large technology company where I manage a couple of teams of data scientists. And we think a lot about optimizing the user experience on our platform. As a fun fact, this is a pretty fun side project, actually. I have a fun side project where I'm building a machine learning algorithm that can generate crossword puzzles from scratch. Valid New York Times crossword puzzles. [00:04:55] Speaker A: Excellent. [00:04:56] Speaker C: Seems to be working reasonably well so far. [00:05:00] Speaker A: Yep. And there are people who are really bad at crossword puzzles. And what I have discovered is that I am at the lower end, the lowest end of the distribution among the people who are really bad at crossword puzzles. I mean, it's just I'm so bad at them that I just don't. My wife, who likes them, doesn't even ask me for help. Well, thank you both so much for being here. And we're here to talk about, actually another PNAS paper and its counterpart, which is on the web on ssrn. But the counterpart paper that the three of us co authored along with Ellen Degnan, who couldn't be with us, is using random assignment to measure court accessibility for low income divorce seekers. So the idea from that title suggests that what we're going to do is use an RCT to see whether a court is actually accessible, just as a way to kind of measure the pro se accessibility of a court system. And that's certainly one way to interpret this research. Another way to interpret this research, which is what the SSRN version pursues is that it is simply an illustration of an access to justice problem and an attempt to figure out why there is an access to justice problem in a particular setting and then to an assessment of a particular effort to solve it with lawyers. And the area in which we're working here is marriage and divorce, more accurately, divorce. And so the name of this paper is Trapped in Marriage. And so the theme there is that the circumstances, and as we'll talk about in a bit likely, the procedures that are required, the onerous procedures that are required in order to achieve a divorce are trapping people in undesired marriages. So, Rosanna, why don't you tell us how you got involved in this project? [00:07:06] Speaker B: I was a grad student at the time. I was a PhD student in psychology. I was actually a student at Yale, but I was living in Cambridge. And as a social psychologist, most of the research I had been doing up until that point had been laboratory based experiments which have their strengths, but one major limitation is that they're not based in the real world the way that field experiments are. I was really interested in learning more about how one conducts a field experiment. The truth is they're extremely challenging to pull off. You almost always need a team of people. And I wasn't probably going to do it on my own and I wanted to learn from the best of the best. And because I also was pursuing a JD and was interested in law and psychology and was hanging around Cambridge at the time, Jim Greiner's name came up over and over. So I reached out to him explaining, you know, I'm really interested in learning more about field experiments. How do you do them? How do you set them up, how, you know, logistically, and then also from a scholarly perspective, how you publish about, you know, these sort of complicated studies. And I got plugged into this study after it had already been set up. And the stage at the research, the stage that the research was in when I got involved was we had already identified cases where there was a case on file at the courthouse. And we needed someone to actually read through the cases where someone made it to the courthouse. They had filed at least something and we needed to learn more about what their case looked like. How complicated was it? Was there a custody dispute? Was the person represented? If they were represented by an attorney, was it one of the Philly VIP attorneys that they had been assigned to, or did they go off and find their own attorney? You know, all these questions, I believe I was blinded to condition. I didn't. [00:09:16] Speaker A: You were? Yeah. You were. You did not know the treatment. [00:09:18] Speaker B: I did not know who had been assigned an effort by Philly VIP to get them a lawyer versus had gotten the more minimal help. But I was going through these case files and sort of marking in a spreadsheet. Basically we had all these different columns, all these different attributes we were looking for. Were there any contested motions? Did, was there a master's report, questions that would be proxies for how complicated legally the case was, as well as these other features that I mentioned. Did they have a lawyer? Who was the lawyer? So I was involved in coding, I guess you could say the case outcomes in a way that would be quantifiable, talleable. And then I also got involved in a separate part of the project which was to unpack the pathway that a pro se would be litigant would would follow. If they were to go to the courthouse and ask for help, what materials would they be directed toward? I sort of combed through the self help materials that were recommended to people routinely at the time. So I got a copy of the 166 page instruction manual that was supposed to help someone file for divorce on their own. And I sort of combed through that to understand all of the steps that a person would have to go through. And you know, in addition to reading the manual, just sort of understanding, okay, when it says, go photocopy your complaint, let me understand, for a person in Philadelphia, where can they get a photocopier? Oh, at the library. Where is that library? How close is it to the courthouse, which is where they would have started? What are the hours of the library? How much does it cost to make a photocopy? Just really minute details about the journey a person would have to go on to make it from start to finish. And the reason that I was doing that was to try to understand what barriers a person might face if they were to do this without an attorney. Logistically, why is it so hard to file for divorce when legally it seems that these cases are quite simple and uncomplicated, you know, paperwork wise. Logistically, it's a multi step process that involves running around to various institutions, the post office, the photocopier, the typewriter, et cetera, and filling out a bunch of forms. And I wanted to just understand the process, you know, in as much detail as I could. So that was part of the research. And then we also benefited from speaking with a few on the ground experts who helped us understand like, where did those materials come from? Who wrote them? When people worked with individuals who had been following Them what were the steps that were the most challenging? And all of that came together to form a picture of the hassle factors or ordeals, as we called it in the paper. The. Just the challenges, the very real challenges of a quote unquote simple process, actually how complicated it was. [00:12:46] Speaker A: Terrific. So you've identified sort of three areas that you. That you worked on. Let me just ask you about each of those three. So one was in the three areas, I mean, was. One was looking through, was coding the case files and familiarizing yourself with them. And then the second was trying to the pathway of a pro se litigant from start to finish, what they would have to go through. And one of the things that we talk about in the paper is that actually surprisingly few litigants did this pro se. You know, as far as we could tell, even. Even folks who were randomized to no effort by Philly VIP to find an attorney. The only way they could actually succeed in getting to the courthouse steps was, was either to hire a lawyer anyway or to have the opposing spouse sue them to initiate the proceeding. And then the third one was to talk to some on the ground experts. So let me ask you about each of those. First of all, what were your overall impressions of the case files when you walk through them? Were they massively complicated? Say 200 or 300 docket entries and reams and reams of paperwork and lots of evidence and lots of evidentiary hearings. What were your overall impressions of the case files? [00:14:01] Speaker B: Very few were beyond just a couple of pages. They were pretty simple. If I recall, the docket list. What's it called? What's the page in front that just. [00:14:13] Speaker A: Lists all the docket sheet? Yeah, the docket sheet. [00:14:15] Speaker B: If I recall, the docket sheets were usually one to two pages max, meaning not that many entries for a given case, which is consistent with this idea that this population, at least wasn't seeking terribly complicated, legally complicated divorces. There were often not assets that needed to be divided. There was often not any contestation between the parties that they wanted to be divorced. The role for the volunteer attorney, if there was one, would mostly be about helping the individual fill out the paperwork and get it filed in the correct venue in the correct order. Not, you know, standing up in front of a judge and making arguments or something more complicated. There were a few files that had more attachments, but it was not the majority. And often those additional pieces of paperwork weren't particularly complicated or necessary. You know, an extraneous letter from a lawyer explaining that the person wants a divorce or something like that. So my overall impression is that the files were rather thin. [00:15:36] Speaker A: Excellent. And it turns out, you know, we have some statistics to back this up as well. But all of the individuals in our data set who succeeded in filing for divorce did so on no fault grounds. Pennsylvania law provided for both fault and no fault divorces. The no faults were typically either what folks in popular speech call irreconcilable differences or mutual consent. An irreconcilable difference. In Pennsylvania at the time, you had to be separated for two years. So two year separation was the phrase that the lawyers used, or mutual consent. And all of them were one or the other. And so these were really simple transactions, it turns out. And the case files bear that out. Super. Let's talk about then the ways in which people went from trying, you know, pro se litigant would try to get a divorce and we won't. The paper discloses, especially the SSR in version of the paper discloses the full or sort of walks the reader through the full set of steps in detail. And so we don't need to do that. But tell me, you know, from a, from a, you're a trained psychologist, right? And, and so, you know, you can think about how maybe you know better than most people, how, you know the average person's going to react to some of these processes and you know, what are going to be, what are going to be issues and whether one knowing that, what it's going to require, whether they actually start the process at all. Because one of the interesting findings was that 80% of the people who filed who managed to get to the courthouse steps and actually succeeded. And that was true of both the, both the Philly VIP effort group and the non Philly VIP effort group. So if they could get to filing, they could get all the way there. But the problem was, were they just so deterred from filing by the process? So what, tell me, you know, what were some impressions? Again, we don't need to walk through step by step, but what were some impressions? [00:17:48] Speaker B: It seemed like the key ingredient that would have helped a lot of people is a human expert to guide them or offer them some information. And without a lawyer, much of the staff at the clerk's office, at the library, these are people who are not lawyers and are instructed not to provide legal advice. And so they are somewhat reluctant to be too prescriptive to someone who comes in saying, which form do I fill out? Where do I go? You know, anything that smacks of legal advice is something they're told to avoid. And so if you are someone who learns better from having a person you can ask questions to, as opposed to a book that you need to flip through to figure out the next steps, it seems like that's a big barrier or gap. And, you know, people are referred to this self help manual. I think the self help manual, it's 166 pages because it's really trying to break things down to a level that's, you know, detailed enough to help people. But the truth is it takes 166 pages to explain the whole process because the process has a lot of steps. And my impression was some of these steps are overly complicated. Why is the name of the form that you need to submit to actually have the judge rule? You know, the paperwork that is needed to tell the court this divorce case is ready for a final order? Why is that called the notice of intention to file a precipice to transmit the record to the prothonotary. [00:19:50] Speaker A: And you pronounce it correctly? Actually, no, no. In prothonotary or something like that. Anyway, they have. I said prothonotary. That's actually not the right. I've learned recently that's not the right pronunciation in Pennsylvania. So whatever that person is called. Right. Which makes it even worse. Right. So go on. Sorry. [00:20:09] Speaker B: So, you know, whoever came up with that title, I'm sure they had a reason, but that is just. It's just not descriptive, it's not user friendly. And there was another form that ostensibly required both to have the quartz seal on it, which I believe was red, and for your name to be typewritten, not handwritten. So you couldn't print out from your printer this form because it wouldn't have, like, the red seal. And you couldn't take their blank page that has the seal on it and write your name. You had to, like, find a typewriter to type your name onto the form that had the seal. You know, we talk about filing fees and how those are exclusionary and reduce access because they cost money, and those are a big problem. But needing a typewriter is its own kind of problem. I would say it's a barrier in and of itself. Because, yes, there was a typewriter available, I think, at one of the libraries. But, you know, who knows that? And the library isn't always open, and the library also isn't at the courthouse where you need to submit the form. So, you know, one of the lawyers we talked to had come up with a workaround for that where they were able to use, I think, a colored printer to mimic the redness of the seal. And the court was willing to accept that. So some people who had a color printer or whose lawyer had a color printer didn't need to go find a typewriter. Good for them. But the instructions still said, go find a typewriter. And you know, what I want to emphasize is psychologically that kind of barrier can be as big a deterrent as, and you need to come up with 200 bucks. [00:21:59] Speaker A: And about that printout, if I recall correctly, the PDF to try to create that form that you could print out with a color printer and have the red that was not on the court's website. It was on the, it was if, if it, if it. You know, the only way to get it on the Internet was actually to look on the legal services provider's website and use. And use it that way. So it was very difficult to find. Even if you knew that you, even if you could figure out that you knew it and that this was available and you had a color printer, otherwise you had to go find an old fashioned typewriter. [00:22:30] Speaker B: And we didn't know about the color printer until we did the interviews with the on the ground attorneys. And they said, oh, we have a workaround for that. And we said, oh, well, we've researching this for weeks and we didn't know about that workaround. So. [00:22:43] Speaker A: And the workaround is not in the. [00:22:44] Speaker B: Self help material and the workaround is not publicly, you know, easily available. So, you know, part of it is kudos to those lawyers for figuring out a workaround. But you can see how having access to the right people, being plugged into the right information could make a difference. And if you're trying really hard, as we were, to understand all the rules and make sure that you are crossing every T and dotting every I, you might not realize that this printer option exists. [00:23:14] Speaker A: Terrific. And tell me, there were two other things I wanted to ask you to comment on. One is if there seem to be multiple waiting periods involved in the procedure. Some of these came from the Pennsylvania rules of civil procedure and therefore were set statewide and actually extended beyond divorce cases. They were just cases. They were just general purpose rules. Some of them were local by rule to the Philadelphia Family Court and then some of them came from the, you know, judges practices. And so tell me, you know, there are multiple waiting periods where basically you file something and then nothing can happen for a little while right now. And tell me, what do you think? And suppose ostensibly those were justified by saying this is a time for people to think hard about whether they really want a Divorce. Of course, by this time, in order to file, they've either had to have both parties agree by consent that they don't want to be married anymore, or there has to have been a two year separation, which seems like plenty of time to me to think about it. So let's think about then. Or can you again, as a psychologist, what do you think the effects of these waiting periods might be? [00:24:27] Speaker B: Each waiting period seems like an opportunity for someone to fall off their path. Let's say their goal is I want a divorce. Each of these waiting periods is a chance for them to get diverted from that path, either because they forget or because their, you know, circumstances change. If you want to achieve your goal, the best way to get it done is quickly and to have all the steps that, for instance, require a trip to the courthouse. Do it in one trip. Don't make them come back multiple times. Waiting 90, 90 days, waiting 10 days in between each one. Maybe each one of those requires them to get childcare or arrange to take off time from work to get to the courthouse, which I think closed at 4pm and wasn't open on weekdays. [00:25:19] Speaker A: You mean weekends? [00:25:20] Speaker B: Oh, sorry, yes, it closed at 4pm and wasn't open on weekends. And so if you think about each waiting period as a chance for someone to fall off the path or a new hassle that they have to now schedule and deal with, you can see how they compound. And when we looked at some of these situations, it seemed like the purpose of the waiting period. You know, let's give everybody a chance to contest, to have their voice heard, to really think about this. The end result sometimes just seemed absurd. Like for instance, if you had a divorce based on mutual consent. This is where both parties want to end the marriage. And at this point, the opposing spouse has already signed the affidavit of consent showing, I've been notified and I agree with the plaintiff's request for divorce. At that point there is then a statutorily mandated 90 day waiting period, which is supposed to give them time to think about it some more and consider whether they truly want a divorce. And then after the precipita transmit that we were talking about before, after that is filed, then there's still another 10 day waiting period to give them yet another chance to object. And then once the divorce is, and then once the divorce decree is made, then there's an additional 30 days for them to file an appeal. So we have like multiple chances for people to pull the plug on this. And you just wonder, who is this serving? How often does this waiting period help someone who really changes their mind, you know, effectuate their objection versus how often does it dissuade someone who really wants a divorce? And there is no objection to it from getting all of the paperwork and all of the logistics handled to make it happen? [00:27:19] Speaker A: Super. And then I think this has been implicit in the conversation. But to a psychologist and someone trained in behavioralism, what is a hassle factor? Because we use the term in the paper. [00:27:34] Speaker B: Hassle factor would be any kind of resistance that makes achieving one's goal more difficult. So when we think about how to design systems to help people achieve their goals, that could be filling out a bunch of paperwork. It could be helping you stick to a new exercise routine. It could be helping you save money for retirement. It could be any. Whatever your goal is that can be hard for you to achieve. We think about, okay, how do we design the system to help people? We want to make it easy, we want to make it attractive, we want to make it social, and we want to make it timely. This is like the east framework. E A S T. So make it easy. You know, if someone can file, you know, by mail, that's easier than making them go in person to the courthouse. If they could file, like, by text message, that's easier than filing by mail because they don't have to get a stamp. You know, how do you just make it as minimally difficult as possible? How do you make the option attractive? Maybe for divorce, this is less salient. But for something like exercise, how do you make exercise something that's salient, appealing, maybe luxurious, even social? How do you market? [00:28:50] Speaker A: We could say, how do you make it less unattractive? [00:28:53] Speaker B: Right. [00:28:53] Speaker A: That would be okay, right? Yep. Go on. [00:28:55] Speaker B: Yeah. How do you make it less unattractive? So what are some of the fears or stigmas or just bad feelings people have around going through this process? Are there ways to mitigate that? The social aspect, you know, how do you marshal people's social networks or their. Their families, their friends, their peers, their neighbors to support them? So, you know, with. As. As I was saying, if you have someone who you trust to help you through this process, maybe your sister has gone through a divorce and she can, you know, talk you through it or that sort of thing, that would make a big difference to people. And that's one reason why I'm so concerned that the first thing people who walk into the courthouse are told is, we can't help you. That's giving legal advice. Go read this manual. And then timely. I mean, how do you sequence the moments when someone can actually act on their goal and that's where the waiting periods become so problematic. Ideally, what you want to do is bundle it all together so you can knock it out all at once. At the moment when I realize I want the divorce, how can I take a step right then and there? At the moment when I need to pay the filing fee, how can we make sure that like my checkbook is right there or whatever materials are required to get it done are time to be right there. It's not helpful if I'm told, you know, I need to make a photocopy at a time when there's no photocopier there and the library is closed and like, I'm going to have to put this now on like a later to do list for me to do. It'd be much better for me to get the information at the time when I can act on it. So when we talk about hassle factors, we talk about things that make it difficult for people with limited time and cognitive, you know, bandwidth to get it done. [00:30:56] Speaker A: And let me move you then on to the last question, which is the last part that you were involved in was talking to some of the experts. And what do you think that what are some takeaways from that set of conversations? The experts here in Philadelphia, divorce proceedings and some folks who had done these, you know, cases representing clients in these proceedings, etc. What were some of the takeaways from those conversations? [00:31:23] Speaker B: It was really helpful to hear from them about the workarounds they had come up with and also the human element of representing these clients. I would say the main thing that stood out to me is how pessimistic people sounded about creating a self help infrastructure that was user friendly enough to get people through the system. It sounded like the solution that they were more interested in was how do you increase the number of legal aid attorneys? How do we increase the resources given to legal aid? How do you make sure everyone can have a lawyer? Because they really saw that as the solution, make sure everyone has a lawyer, and that it was not impossible to come up with an alternative, but that the best solution would be to find a lawyer for everyone who needed one and, you know, to marshal the resources it would require to make that happen. And I, I see that perspective, but I also, you know, with the Access to Justice Lab and with a lot of people in the Access to Justice community, I think are interested in finding alternatives. Low Bono, is that what it's called? [00:32:45] Speaker A: Sometimes called Lobono? Yes, exactly. Limited service Representation, all the usual things we talk about. I mean this setting is a good reason why unfortunately that make sure everybody has a lawyer is never ever, ever, ever going to work as a long term solution. And this as we talk about in the papers, both versions in the last two years of this study, while we were randomizing cases, the Philly vip, which was the only Philadelphia area legal services provider that was taking the standard divorces, these, you know, non complication, you know, no complication divorces, just let's, I don't, you know, I don't want to be married to that bum anymore type divorces could handle. It had the capacity to handle three out of 20, three out of every 20. So 15% of the, it was able to meet about 15% of the demand. So we could, you know, quadruple its budget and its capacity and still not get there. And so we, you know, that's why in the paper we talked about, you know, addressing directly head on some of these hassle factors that, you know, the delays, the needs and saying, well, maybe we need to reform the process. Maybe we need to, to come up with a new process. And so the answer is more in the court system than it is in the, in the legal services community. Super Rosanna, any final thoughts? [00:34:11] Speaker B: Well, just on this last point, I think it is worth mentioning how much legal aid has been defunded and how like the go go days when there was a lot more support for this type of representation, that just feels like a bygone era. So I understand the legal community saying we should go back to having more funding for legal aid, But I also agree with what you're saying, Jim, that we're so far from 100% representation that rethinking the whole system I think is also important. Okay, last thought is that it was great working with you on this project. I learned a ton about field experiments. They are extremely logistically challenging to pull off and haven't done one since without Jim's help. I can't, I can't, can't imagine as. [00:35:08] Speaker A: An effort to try, to try to seed you with the desire to do field research, we failed utterly. [00:35:13] Speaker B: What I would say is it takes a special person and it takes, you know, a village and it definitely takes. [00:35:20] Speaker A: A village, no question about it. [00:35:21] Speaker B: So thank you for including me in this project and I'm proud of the final paper. [00:35:27] Speaker A: Tom, can you tell us how you got involved with this project? [00:35:32] Speaker C: Yeah, sure. So at the time I was a student, I guess when I got involved, I had graduated from law school. [00:35:41] Speaker A: I think that's correct. [00:35:44] Speaker C: In the PhD program and in statistics and was looking for, looking for like a meaty research question to work on. And you and I were chatting about this particular study and some of the analysis is straightforward, but some of the analysis there were complications in like what data could be observed for each person who had participated in the study that made some of the more difficult questions analytically challenging. And essentially I solved that analytically challenging problem for one of the chapters of my PhD thesis. [00:36:34] Speaker A: Rosanna and I are giving celebration hands here because it was a non trivial, as the statisticians say, this is a non trivial statistical challenge here, if I recall correctly. Well, actually once you say that in a standard RCT you have an intervention group and a business as usual group. Here the intervention group is going to get an effort from Philadelphia Vit to try to find a lawyer. And here the business as usual is the legal services provider is going to say, I'm sorry, we can't help you. Here are some, you know, here's some self help materials and you know, if you can find a lawyer on your own, go to it. Right? And so in a stand, but, and so in a standard rct, the scientists get to observe what the, all of the, all of the participants in the study actually did or received. They either got the treatment and took it or they didn't get the treatment and therefore did, didn't take the treatment. And so if it's a pill, then you know, you, you, they come to the doctor's office and you watch them swallow the pill and you are 100% sure that they get the pill right? Here, here. We, we, we were interested in two separate questions. One is, is, is the Philadelphia VIP operation, Is it Philly vip? Is it, is it a good, is it work? Does it work? And all we needed to know for that is just did we randomize people to get a Philly VIP effort or didn't we? And then we could compare those two groups and we didn't have to know what they actually got because if they, if part of the problem was they never did get matched up with a lawyer, well then that's part of the problem, right, that we want to study. But if we also wanted to study the effect of having a lawyer, then we needed to observe who got lawyers. And not everybody who is randomized to an effort to affiliate for a fill of VIP match got a lawyer. And not everybody who was randomized to no Philly VIP match didn't get a lawyer. Even though Philly VIP was the last free Legal services provider, the last of the. That the provider of last resort, even though there was no one else, no other legal service provider in the Philadelphia area that would take these cases, because as it turned out, with this is what we saw, some of them did get lawyers. And so now we could see whether they got lawyers if they filed. Right? We could see if they got lawyers. We know they. We know people that filed. We can tell from the records whether they filed. But what was our statistical problem? What was our. What was our problem that you had to solve, Tom? [00:39:18] Speaker C: Yeah, so. So the main. So that problem is difficult enough even when you know, who got. Who, like, didn't comply with their treatment assignment, so to speak, like, and got a lawyer, even though we didn't randomize them to have a offer of representation from Philly VIP or who didn't have a lawyer, even though they were randomized to have someone from Philly vip. Even, even that problem, when you can fully observe who did or didn't have a lawyer, it's. [00:39:52] Speaker A: It's. [00:39:53] Speaker C: It's a fairly challenging problem. And, you know, people have made their careers solving problems like that in the not too distant past. In this case, we had the additional challenge of, for maybe half of the participants. Okay. It was actually. So the, the way that we knew somebody, if somebody was in the control group, the group. [00:40:17] Speaker A: Your control means not randomized to no Philadelphia Philly VIP effort. Right. Go on. [00:40:23] Speaker C: That's right. We didn't know if they had a lawyer or not unless they actually succeeded in filing a lawsuit, because that you would read. You would read, basically, you would read the filings and see if there was a lawyer associated with the case or not. So it's logically possible that these people in that control group, somehow they obtained a lawyer, even though they weren't randomized to receive help from Philly vip. And notwithstanding the fact that they had obtained a lawyer, they still didn't manage to file a case or get a divorce. [00:41:02] Speaker A: Maybe they weren't eligible. Maybe they couldn't get one under Pennsylvania law. And under that circumstance, the lawyer told them that I'm not going to file for you because you're just going to lose. Or maybe, you know, they, they had second thoughts even after they got the lawyer. We knew that was for true for some people because we knew from the Philly VIP effort group, Philly VIP told us who got a lawyer. And then we checked who filed. And not everybody who got a lawyer filed. Right. That was, you know, for divorce. So we knew it. We knew there were some people out there who would get a lawyer and then not end up filing because they decided not to, or they just, you know, even with a lawyer, the hassle factors were too much for them, you know, or something else happened, maybe they unfortunately passed away or something like that. Right. So we knew that, we knew that had happened. So and then the problem, as you say, was in the. No Philadelphia vip, we don't know who those people are. We know there's some of them out there, but we don't know who they are. So briefly, for a legal audience, and since you're a lawyer, you can do this in addition to a statistician. Tell us, tell us briefly what was, what was the statistical solution? [00:42:12] Speaker C: Yeah, so I'll, I'll do my best to keep this non technical. [00:42:18] Speaker A: Yep. [00:42:20] Speaker C: But basically. Okay, when you want to, when you, when you want to get an estimate, even in the, let's start with what you would do. Even in the case where you could, where you knew everybody had a lawyer or not, what you would do is you would build a statistical model that has a bunch of unknowns in it, but it broadly, and let's call those unknowns parameters. And that statistical model tells you, it models how likely somebody is to file a lawsuit and then to get a divorce in the end based on some facts that we have about them, like whether, you know, what their randomization status was and whether they would get a lawyer or not. And then a lot of other things. [00:43:22] Speaker A: Like race and gender and, you know, their assets and their income things, all those other things that we had because we got them at mta. Right, yeah, go on. [00:43:32] Speaker C: And then you would use your observed data to learn or estimate the underlying parameters of this model using standard statistical techniques. So in this case there was an extra unknown. So basically, and basically we wanted to extend the existing framework to accommodate this additional unknown that we had of. We didn't know whether these people in the treatment group had, sorry, the control group had ever gotten a lawyer or not. And so you kind of bootstrap these things one at a time. You have a probability distribution for whether they got a lawyer or not. So you can sample from that probability distribution and then based on that, you can sample whether they would file in court and then whether they would get a divorce. With some sophisticated algorithms, you're guaranteed to get good estimates of what the treatment effect would be, assuming all of your modeling assumptions are reasonable in the first place, which is kind of an assumption that you, you have to make to make progress on this problem. [00:44:56] Speaker A: Terrific. Terrific. Okay. And so there were two sorts of questions that we asked in two separate for two separate types of thinking about what the treatment in this paper was. So there are two sets of questions that we asked were, you know, did somebody succeed in filing in 18 months? And did somebody succeed for those who did succeed in filing in 18 months, for those who succeeded in getting a divorce in 36 months? And again, the fact that we had to wait 36 months after intake, three years to show some of the hassle factors, you know, and the time delays that you were talking about earlier, Rosanna So we asked those questions and then we asked them for two separate sort of ways of thinking about this research. One is, again, does the Philly VIP matching system work? And then the second is, does what's the effect of having a lawyer? And those are, as you know, two different things because again, we talked about how some people in the no Philly VIP effort group got lawyers, and a very few people in the sorry. And in the Philly VIP matching group, not everybody was successfully matched. And so we didn't succeed in getting them a lawyer. And some of them did end up with a lawyer, to our knowledge. And so let's talk about the first one. First, the set for the quants out there for the stats nerds. This is called the itt, the Intention to Treat. So we're basically just analyzing what we randomized. Was there an effect of the Philly VIP treatment? In other words, the effort, was there an effect for it? So, Tom, tell us, was there an effect? And here there's additional complication. Let's talk about just just in terms of people filing inside Philadelphia county where they were supposed to filing for divorce inside Philadelphia county. [00:46:58] Speaker C: Yeah, there was there was a very strong effect. So the difference, I think the difference in success rate for filing in Philadelphia county between the people who were randomized into the group to get help from Philly VIP versus the control group, the difference was 44 percentage points. [00:47:23] Speaker A: Yeah. So it multiplied the success rate by about a factor of four. Right. Or maybe three or four, which is just astonishing. It's actually the largest effect I've ever seen in a legal services study of any kind. I've never seen an effect that big. And then in terms of who succeeded in getting a divorce after 36 months, and again, that's of people you had to be successful under this definition. You had to have filed in 18 months. It didn't matter to us whether the participant filed or whether the opposing spouse filed. We counted that as a, quote, success either way. And it turned out Some of the, especially in the no Philadelphia VIP effort group, some of them, quote, unquote, succeeded by getting sued. Right? And we just tolerated that. So even though. So then in terms of, in terms of success In Philadelphia and 36 months. [00:48:15] Speaker C: Getting divorced, the difference between the two groups was again, gigantic, a 40 percentage point difference. And like, I also have never seen treatment effects. [00:48:25] Speaker A: I've never seen anything like it. Right. [00:48:27] Speaker C: And especially in the social sciences, this is, these are like astonishingly large. [00:48:31] Speaker A: These are huge. And it turned out that we found that there were two, two counties in Pennsylvania that were essentially known as divorce mills. And they were nowhere near Philadelphia, but you could get a divorce by mail as long as no one contested it. And that put us wise to the fact that we should search those files. And we did. And we also. And that yielded, lo and behold, a lot of our participants had filed out there, especially in the no Philadelphia VIP group effort group, a lot of our participants had filed in these two counties out in the middle of Pennsylvania in central Pennsylvania. I joked when I was presenting this research that there were, that the population of the cows was higher than the population of the people in those two counties. And I'm not sure I was wrong about that. I'm not sure it was actually a joke. So we added those in. We also searched adjacent counties adjacent to Philadelphia. If I recall, that didn't yield anything geographically. That only yielded one filing out of our data set. And so if we add in those counties that were in the middle of Pennsylvania, do the effects disappear? [00:49:42] Speaker C: They don't disappear. They're slightly diluted. Like they're, they're slightly smaller. Instead of being on the order of a 40 percentage point difference between the two groups, it's more on the order of like a 30 percentage point difference. Still very, very large. [00:49:59] Speaker A: You know, it's one of, I think in the, in the, in the, in the file in 18 months, it's, you know, sort of 20 percentage points. Even at that effect size, it's still very large. So even if you take that into account and say, oh, you know, it's still a very, very large effect size, and then as you say, sort of 30 percentage point or so difference, 25 percentage points. So it turned out this Philadelphia VIP effort to match was intensely successful. It's not the most efficient and certainly not the most fast moving operation I've ever seen in terms of providing lawyers. It's probably about the what the. Generally the way, you know, in terms of speed and efficiency, pro bono matching services work, because that's how they Work you call in you, you know, put your name on a list and they try to match you. But it often took, we, we observed from a Philadelphia records Philadelphia VIP shared with us that it often took months for there to be a match. And so it wasn't a super fast process. And yet it was still spectacularly, spectacularly effective. Which again suggested there may be something wrong. I mean, I found this, I found that result not encouraging, but depressing to say that this is just as, you know, and that's exactly the theme that we picked up on in the PNAS paper, which is this is a problem, this is an indication there's something wrong. And then we said, okay, let's actually measure, let's actually use this to figure out whether there's something wrong here. Let's look at the effect of having a lawyer, not the effect of having an affiliate VIP effort. So now we're saying again, measuring in some sense what is the pro se accessibility of the court system by saying if assist by debt, but defining accessible court system as in a simple transaction you shouldn't have to have a lawyer. That's the idea. And so if we randomize cases to where they have a lawyer and whether they don't, or at least if we can tease out, you know, with, with statistical methods, the, the effect of having a lawyer, it shouldn't be that big in if the system is pro se accessible. Right? And again, this is just the effect of having a lawyer. For that we need the effect of a lawyer, not affiliate VIP effort to find one. And so when we focused on the effect of having a lawyer, Tom, using your newly created statistical techniques, that didn't exist before and that, and that you, you created here, what did you find out? [00:52:30] Speaker C: So, yeah, I guess stepping back for a second, I think the intuition should be that even though the intention to treat effects were so large, were very. [00:52:43] Speaker A: Large. [00:52:46] Speaker C: If you focus just on the effect of having a lawyer versus not, it should be even bigger. Because in the control group, like a lot of people got lawyers, even though they like quote unquote weren't supposed to, you know, they weren't randomized to get it. So like that's, that's diluting the difference between these two groups. That should be the intuition. And that's indeed what we see. If you look at people who, if you, if you use this fancy technique to try to isolate the effect of having a lawyer versus not on filing, on the success of filing in Philadelphia county, the difference between the group that is getting the lawyer versus not getting the lawyer is about 86 percentage points. [00:53:34] Speaker A: And that's, I mean, I've never seen anything that size before or since. I've never seen anything that size. [00:53:41] Speaker C: It's taking you to close to a 0% chance, to close to 100% chance. 86 percentage points basically spans the entire. You know, it's close to spanning the entire difference from 0 to 100. So it's, it's a huge effect. [00:53:57] Speaker A: And then. And was that true for both the. Well, there's sort of four separate questions here. Filed in 18 months within 18 months in Philadelphia county, filed within 18 months within all counties, successfully divorced within 36 months in Philadelphia county only, and successfully divorced within 36 months in all counties where we looked. All the seven counties where we looked. Are those just very, very large effect sizes true in all four of those? For all four of those questions? [00:54:30] Speaker C: They are. Especially in Philadelphia County. And the difference between the two groups in terms of filing and getting a divorce, the effect sizes are pretty much are very similar for those two outcomes. If you look at all counties versus. So as I was saying, in Philadelphia, the difference between the two groups is on the order of like 86%. If you do the same comparison for all counties, that's more like 55% in terms of filing, 61% difference in terms of getting a divorce. So still, you know, huge difference. [00:55:10] Speaker A: All these results were talked about are statistically significant, you know, and not just P is less than 0.05 here for folks who have heard of that. It's, it's, you know, in some instances, we, it broke our computer system, as I recall. We had to, we had to jerry rig the computers because we kept getting P values of zero. And we knew that journals would not actually accept a P value of zero. [00:55:32] Speaker C: Yeah, that's right. A lot of these results were statistical at the level of significance that would be required for like, like discovering the Higgs boson. [00:55:43] Speaker A: Yeah, five. Five sigma. Exactly. This is sort of the five sigma level of significance, right? [00:55:48] Speaker C: That's right. [00:55:48] Speaker A: Exactly. Super. So. So, Tom, let's, let's then ask you, you know, what is what, you know, what's your. For a takeaway? Right. If you, what, you know, after this experience, it can be about the experience or about the paper, about the findings, whatever you pay, you know, what do you think? What's your takeaway? [00:56:03] Speaker C: My takeaway? It's really hard to get a divorce in Philadelphia if you don't have a lawyer. You know, maybe not that surprising given all the complexity that you and Rosanna have discussed. Like, be hard for me to figure out where to get a typewriter, for example. [00:56:19] Speaker A: And I would consider you a highly functional individual with a J.D. right. [00:56:25] Speaker C: I appreciate the vote of confidence there. You know, I. I don't know what a prothonotary is, despite my legal education. [00:56:33] Speaker A: And you were able to pronounce it, which I could not. But anyway, go on. [00:56:38] Speaker C: So, yeah, I mean, I think the results are very vivid and show that there seems to be an issue here in terms of working on the project. I mean, I loved working on this project. I learned a ton. I learned how to be, you know, do original research. Yeah, it was great working on this and it was fun to actually succeed in finding a solution. Yeah, it was a great experience. [00:57:11] Speaker A: So thank you so much to both of you. Really, really appreciate y' all taking your time to do this and to appear on Proof Over Precedent. [00:57:22] Speaker C: Thank you. [00:57:22] Speaker B: Thanks for having us. [00:57:24] Speaker A: Proof Over Precedent is a production of the Access to Justice Lab at Harvard Law School. Views expressed in student podcasts are not necessarily those of the ada. J. Thanks for listening. If we piqued your interest, please subscribe wherever you get your podcasts. Even better, leave us a rating or share an episode with a friend or on social media. Here's a sneak preview of what we'll bring you next week with your grace. I thought we might talk today about the waivers of informed consent set of regulations in the common rule. And also if we have time at the, at the very end, whether we could compare those to the exceptions to informed consent regulations that are in a different regulatory regime. And we'll talk about how it could be that there could be two sets of regulations that would govern or that would that would speak to such similar issues.