Episode Transcript
[00:00:00] Speaker A: Imagine a justice system built on rigorous evidence, not gut instincts or educated guesses about what works and what doesn't.
More people could access the civil justice they deserve.
The criminal justice system could be smaller, more effective, and more humane.
The Access to Justice Lab here at Harvard Law School is producing that needed evidence. And this podcast is about the challenge of transforming law into an evidence based field.
I'm your host, Jim Greiner, and this is Proof Over Precedent.
Welcome to another edition of the Access to Justice Labs podcast, Proof Over Preston. I'm Jim Greiner and I'm the Access to Justice Labs faculty director. And I'm going to be host today in a conversation with two fantastic scholars, Aviv Caspi and Charlie Rafkin. And we're going to be talking about a paper that I believe is still in working paper form.
I couldn't find it on any publication list yet, but I'm sure it will be published soon. And Aviv and Charlie, y' all will tell me if I've had that wrong. But anyway, called Legal Assistance for Evictions. And it is in the heart of the Civil Access to Justice movement. It is exploring when and why and whether lawyers make a difference in summary eviction proceedings. And let me just start with you, Aviv. Can you just introduce yourself, tell folks who you are and what you're interested in?
[00:01:23] Speaker B: Sure, yeah. So my name is Aviv Gaspy and I'm an economist by training. I did my PhD in economics at Cornell and then I was a postdoc at Stanford for three years at the law school. And a lot of my interests have been about the interactions of low income folks in the legal system.
So starting with public defense, and then this project with Charlie brought me into eviction. And so now a lot more of my work is in civil justice. And this year I'm actually in residence at the University of Chicago Law School doing a master's in legal studies. So so I'm getting to take the doctrinal classes and study also from a legal perspective. And it's been a really fun experience for me.
[00:01:57] Speaker A: Charlie, who are you?
[00:01:58] Speaker C: Thanks, Jim. So I'm an assistant professor of economics at the University of British Columbia. My research is on public economics and behavioral economics. So I study how do we design government programs in the most effective way to take into account how behavioral biases or kind of psychological forces might affect people's decision making. And so some of my research has looked at eviction policy. So trying to think about how do you design eviction policy to account for landlord tenant relationships. But I've also Looked at other government transfer programs like the Supplemental Nutrition Assistance Program and others.
[00:02:33] Speaker A: The paper is Legal Assistance for Evictions, and it takes place. And the study site, I believe, was Memphis, if that's correct.
And why don't y' all just give me an introduction? Charlie, why don't we start with you? How did you get interested in this project and what were you hoping to accomplish with it?
[00:02:48] Speaker C: Yeah, so I can give the big picture motivation from a policy perspective and then also give our sort of personal interest in it, as Jim, more than 20 jurisdictions have recently passed right to council policies that provide free lawyers to tenants facing eviction.
And despite the kind of importance of these policies. So policies like this have been passed in New York City, San Francisco, Los Angeles.
These policies have remarkably little evidence, and it's not at all obvious that such policies would work for.
And this was all happening at.
I was doing other research on the emergency rental assistance program in Memphis, and I came across an opportunity where I was working with collaborators, implementation partners on that other project, studying what happens when you give people emergency rental assistance where they received funding to provide lawyers to tenants facing eviction, where the idea was, we want to offer people more robust opportunities to defend themselves in court. Providing cash assistance on its own may be ineffective without somebody to shepherd the money to the landlord. And so they wanted to evaluate the program.
I was able to persuade them that it would be really valuable to try to set this up in a clean, randomized, controlled trial. So to try to avoid some of the pitfalls of using an observational design where you may have the people who get assistance who raise their hand to sign up for assistance first may be different than people who raise their hand later on to get assistance in ways that can correlate it with eviction outcomes and kind of threaten naive comparisons of people who get assistance to people who don't. It would be really valuable given that there was no way that we could provide legal assistance to the thousands of tenants every month we're facing eviction in Memphis to try to use a lottery system. And that would both be equitable, but also offer the opportunity to do a very credible evaluation of this program.
[00:04:44] Speaker A: Terrific. Terrific. And, Aviv, how did your interest generate? How did you. How. Where did it come from? Yeah, sure.
[00:04:50] Speaker B: I was. When the project was starting, I was working on a paper on public defenders and public defender caseloads and understanding kind of how public defenders are able to operate under different caseloads and what the impacts are for how they triage their time. And I lucked out to to meet Charlie at a conference right when this project was taking off. And there weren't a lot of folks doing behavioral law and econ type questions in those circles. And so Charlie invited me to join the project. And I think for me, the idea of, like, how what are the roles lawyers can play for folks, what conditions under which they can really have an impact and how we should measure those impacts in careful ways, all of those are core to the policy areas I'm interested in. And so this was a really exciting extension.
[00:05:31] Speaker A: Terrific, Terrific. Okay, and so let's look at the study itself. It's in Shelby county, and we're going to use an acronym, erap, which you've mentioned, Charlie, you've mentioned already. What is the full name for it? What is the full name for erap?
[00:05:45] Speaker C: It's the Emergency Rental and Utilities Assistance Program in Memphis. So sometimes it's known as the era. So this was a pandemic ERA program that provided payments of rental arrears that tenants may owe to their landlord, as well as some months of future rent going forward and sometimes additional utilities. This is a massive pandemic ERA program. There were lots of jurisdictions that had them all over the country. We were looking at a local kind of implementation of it, where the allocated funding was on the order of $100 million.
[00:06:18] Speaker A: And a couple of key things that questions I had about, about how it worked, who applied for it. In other words, if suppose you've got a landlord suing a tenant in an eviction proceeding and there's some suggestion that ERAP money may be able to. And the basis for the lawsuit is that the landlord is alleging that the tenant has not paid rent. And there's some suggestion that maybe ERAP can swoop in and solve the problem. Who is the formal applicant? Is it the tenant or the landlord?
[00:06:45] Speaker C: So generally tenants would apply for the program. Landlords could also apply, but I would think of for the purpose of the assistance provided in our study, it's basically tenants apply and caseworkers or lawyers could provide assistance in the application.
[00:07:04] Speaker A: Got it. Okay. And then. And again, I don't know. I'll just keep asking you, Charlie, for a minute and then I'll switch to you when we get to twi, which we'll explain in just a second. Aviv, how. What was. Was the application process burdensome? Was it complicated?
And how long did it typically take? And then how much money would a typical person get from it if they applied for erap?
[00:07:25] Speaker C: Is the application burdensome? Is a little bit in the eye of the beholder so it did involve substantial requirements in terms of submitting documentation, which could be cumbersome. In addition to providing the assistance, tenants and landlords would often need to collaborate to submit leases and ledgers that documented the amounts of back rent.
[00:07:45] Speaker A: The eligibility requirement here means unpaid rent. It means rent.
[00:07:49] Speaker C: Unpaid rent.
[00:07:50] Speaker A: Yes, exactly. So go on.
[00:07:51] Speaker C: You needed to have some sort of documentation of that. And so I would think of this as like a standard length government application for assistance. So I don't know that it would be all that different from, from applying for other major transfer programs. There wasn't an interview afterwards where they would confirm eligibility. So I think requirements were reasonably low. But in my other research, I show with Evan Soltis that it seems as though landlord tenant collaboration was very important to make it through to the end of the program. For people who weren't involved in our rct, it seemed as though people with good relationships with their landlords were more likely to submit applications and ultimately get assistance because it did require that extra collaboration to get those documents.
[00:08:39] Speaker A: And just the fact that you've characterized this as a sort of standard government application program run in the middle, I would call that a freaking nightmare for most people. But I would call that there, it's an access to justice problem right there.
And so anyway, so we should just make sure to say, yeah, might be standard for government, but could be, as you pointed out as you began, for a human being, might, might pose a barrier.
[00:09:00] Speaker C: That's totally right. And it's both think very cumbersome and absolute levels. But what I want to say is it's a standard for, relative to other, other government programs. What I didn't say though is how much money people could get at the end. And this could be many thousands of dollars. So we have, I think the average payment is $4,000 in our sample. So it's quite, quite a lot of money. There are payments that are much higher even, and there could even be payments that were made going forward for a couple months of future rent. And so in addition to clearing all of the debt that you owe, this could be really helpful because it can help you pay rent going forward for a few months to help you get on your feet. And so it is hard to apply for, but the reward is really high.
[00:09:42] Speaker A: And to center that or to contextualize that, that's a better word, to contextualize that a little bit approximately doesn't have to be exact, but approximately how many months of rent would say a payment like $4,000 BE for a modal or an average Applicant in this program in the dataset that y' all worked with.
[00:10:01] Speaker C: So the 4000 would be inclusive of back rents owed. The average rent is going to be like 900 to a thousand. This, I would think of this as two months rent was standard. One to two months of future rent.
[00:10:12] Speaker A: Got it. Okay, super. And then, Aviv, we're going to. We probably are going to use another acronym, twr, if that's what y', all, because it looks like that's what y' all did in the paper, which is short for the Works Incorporated.
Tell me, who were they?
[00:10:24] Speaker B: Sure. Yeah. And Charlie, you're welcome to jump in back and forth, too. They are housing and legal aid nonprofit that have operated in Shelby county for years. And they have a few different types of initiatives. So one is providing legal services to tenants, but they also provide other sorts of trainings and organizing for tenants. They manage buildings, if I'm not mistaken here, for tenants. And they.
And they were a key collaborator with the ERA program in Memphis when the ERA program was looking for administrators.
[00:10:55] Speaker A: Era here. Erap. Erap, yeah, the same thing. Exactly. So ERA or erap. Right, Super. And so they provided legal assistance. They did that with. For the lawyer aficionados, the lsp, Legal services, civil legal services provider aficionados. Who might be listening. They did so through a combination of staff, attorney, model and Lobono. Is that right?
[00:11:16] Speaker B: That's right, yeah. So they got this grant to provide lawyers over a couple years, and so they had a few lawyers in house who were able to take a full load, and then they recruited from the community, and that could be generalist lawyers who practice all sorts of law in their day to day life and were willing to volunteer their time at a discounted rate to represent a few tenants at a time.
[00:11:35] Speaker A: And the low bono fee was somewhere around $250 to $350 for per case. Is that right?
[00:11:42] Speaker B: That's right. And sometimes it would be a contract for a set of cases with a law firm that they would accept. And some of those law firms, my understanding is, didn't come back to collect their case. So I think we should really think of this as people volunteering their time.
[00:11:53] Speaker A: Got it, Got it. Super.
[00:11:55] Speaker B: Okay.
[00:11:55] Speaker A: And so then in terms of the field operation, it's, it seemed, and y' all tell me what I'm missing here, but it seemed like it's basically a sort of standard RCT in social sciences set up. You do outreach to let people know that the program is out there.
You have an application and enrollment process where they come in and there's a verification of eligibility, and you'll have fairly minimal eligibility requirements. And then there's a notice, and then there's a lottery system where you randomize some people to get the TWI offer of an attorney and some people not to get that offer.
And then you. Some people, many people, maybe even most people who receive the offer of the TWI attorney take it.
And then some people who don't receive the TWI attorney nevertheless find lawyers.
And so that's something you'd have to deal with statistically called non compliance. And then you analyzed in two different ways the outcomes based on. Number one, were you randomized to an offer of TWI lawyer? And then number two, did you actually use a lawyer? And you contrast that in the first sense with people who were not randomized not to get an offer of a TWI lawyer. And in the second, with statistical adjustments, you compare people who did get. Who did have lawyers with people who didn't.
Big picture. Is that about right?
[00:13:15] Speaker B: That's exactly right, yeah.
[00:13:16] Speaker A: Okay, so, super. Now tell me some of the. So a few of the details here. The outreach was an effort to reach. Is it right? Everybody who was. Had an eviction notice. Is that right? Or what was the outreach? What did the outreach look like?
[00:13:30] Speaker B: Yeah, so an eligible tenant had to have a filing.
So the goal was to reach anybody who got to that stage, and that could be through tenants who TWI had already interacted with, that could be postcards mailed to anybody who had a filing, court data, as broad of outreach as possible to people who might get to the point of a filing and at that point be eligible for a lawyer.
[00:13:51] Speaker A: Were there some electronic outreach methods?
Was there any. I'm just curious, were there any emails or text messages or anything? Usually you can't do that with civil proceedings because they don't have the contact information. But I'm just curious whether y' all were able to do that here.
[00:14:03] Speaker B: I think there was, again, for people who TWI already knew about, but I don't think there was court records because, as you said, that's a difficult thing.
[00:14:11] Speaker A: Yeah, super, super. And Charlie, so that we did the outreach, and then now we're going to do the. Have the process by which folks who want to step forward and say, I'd like to enter what amounts to the lottery, how did they do that?
[00:14:27] Speaker C: You have intake here where there's a set of people who are applying for the program. There's very minimal screening and basically conditional on some basic requirements like needing to have a filing, needing to meet a relatively lenient income requirement. I believe there then is.
We would be basically tasked with randomly assigning people to the offer of an attorney or not for the eligibility.
[00:15:01] Speaker A: You couldn't have. You couldn't have a hearing within. Within a day, is that right? And maybe there was a. And there was an income. Income ceiling, is that correct?
[00:15:11] Speaker C: Yeah. So eligibility requirements involved just making sure that the lawyer, if they were assigned to have one, would be able to assist them. So the reason why we wanted to kind of, over the course of the study, make sure that people weren't being contacted when their hearing was already had already happened. So it was basically like give the lawyer a couple of days before the hearing would occur. So that was the first one. And then actually, I think I misspoke. I don't think there was an income requirement. Is that right, Aviva?
[00:15:41] Speaker B: Yeah, I think there was a 24 hour eligibility. And then that you couldn't currently be represented by a lawyer through another program, like through the year end program.
Yeah.
[00:15:49] Speaker A: Terrific.
[00:15:50] Speaker B: And you had to have the filing. And I think the filing was one of the key ways that.
[00:15:54] Speaker C: Yeah.
[00:15:54] Speaker B: That we had to filter people for this study. And it's an open question, I think, in the literature about whether a lawyer earlier in the process would have a different effect.
[00:16:02] Speaker C: Yeah, that's right. And I think the confusing part is that the way that the process works is that landlords were required to give notice before they could file.
Now, some leases waived all these details, but if a landlord notifies you that you're overdue on rent, that's not the same thing as a formal filing in court. And so the notice alone was insufficient to make you eligible.
[00:16:28] Speaker A: Yeah, it's called a notice to quit in Massachusetts. Exactly. So they're. Yeah, exactly. They're different terminologies, but it's a fairly standard.
[00:16:34] Speaker C: But from a. From the tenants perspective, there's no reason that you would know the difference between those. And that would be, I think, one important filter.
[00:16:41] Speaker A: Yeah, exactly. So when you say there's no difference, that a tenant would know the difference between those, that's the differences between receiving the initial notice and then receiving the formal court file. Many people think when they receive the initial notice, they've been sued and they haven't been yet.
[00:16:56] Speaker B: Correct.
[00:16:56] Speaker A: Yeah, exactly. Super. So, okay, so then is it after the randomization, is it at that point that you reach out to the tenants and you say, hey, it's time to come sign a retainer agreement with TWI and be represented, or did you all have them do that prior to randomization?
[00:17:13] Speaker B: No, the randomization is to receive the. That offer.
[00:17:17] Speaker A: That. That offer. And it's at that point that they were signed. Because in the studies that I was involved with previously, we would have them sign a conditional retainer at the time prior to randomization and then it was not valid until the legal services provider counter signed it.
And so just y' all had them sign after. You all had them sign the retainer agreement after randomization?
[00:17:38] Speaker B: That's right, yeah.
[00:17:40] Speaker A: Super. Okay.
[00:17:40] Speaker B: Yeah, we might be able to get more take up by getting people to sign up front, but I think we would worry about disappointing people or if they misunderstood and thought that they had a lawyer and then didn't go out and seek other options. So, yeah, curious kind of how that would play out.
[00:17:53] Speaker A: And our IRBs would. We had to go over with. And they were actually very good at this time point. We had to go over the consent script several times with our IRBs when we did this and had them signed prior to. To make sure that there was no misunderstanding. Of course, there was still one or two instances of misunderstanding anyway, but we tried to do that. Yeah, super. Okay, so then. And so now there's going to be the legal services provided. And I was just curious whether you were able, and it's not always possible to do this, whether you were able to get any information from the attorneys, either qualitative or quantitative, about what. What services did they offer in a typical case? Were you able to say, typical case involved filing some motions.
Did it involve propounding discovery upon the. The landlord? Did it involve moving for a continuance and trying to drag things out, which some legal services attorneys try to do in eviction proceedings? What was the sort of whatever you were able to find out, what the lawyers actually did. So Viv, why don't we start with you and then Charlie, we'll hear from you as well.
[00:18:53] Speaker B: Yeah, sure. Two sources of data here. One is we had the lawyers fill out surveys after the form, after the case, we can collect some information about the different tasks that they do. And the second is we can observe continuances in the court data. And so you know, exactly as you said lawyers, the first thing they do, and really the first thing they were told to do, especially the low bono ones who might be practicing eviction law, we're told the first thing, file a continuance buys more time, both because that helps for Europe and that helps in bargaining. And so for all reasons the tenants want more time here. And so we definitely see that and we can study that in the administrative data. And then for the lawyer survey, admittedly that's one place that we have done a little less. The issue here is that some of the lawyers filled out their surveys long after their case resolved or in bulk. And so trying to get exactly a sense of what happened on each case is difficult. As at a big picture level, we can see that they're filing for these continuances, they're trying to get rental assistance while that's available, and then meeting with the client to try to understand if there are any other defenses they can raise. But in Memphis, that that list of possible defenses is pretty short.
[00:19:55] Speaker A: Charlie, anything to add on that for the next question? Anything to add on that about the. All said about the lawyer services, I'll just. As an aside, I have found lawyers to be absolutely abysmal survey respondents and abysmal at remembering with any degree of accuracy what they actually did on particular cases.
So there are legal services attorneys out there listening. You're awful sources of information. I'm sorry. Just the way that it is expressing frustration from a researcher point of view.
[00:20:22] Speaker B: Super.
[00:20:22] Speaker A: So the lawyers and provide the. The advocacy, they do their. Do the lawyer thing and then at some point you're going to measure some outcomes and what are the primary outcomes that you're going to look at in this study? What are the primary. Charlie, why don't we start with you? What are the sort of primary outcomes that you're looking at?
[00:20:41] Speaker C: So I think of it as one main primary outcome and then supporting evidence, all from the court record.
So the main outcome that we're really interested in is whether or not the person receives an eviction judgment, which is the final kind of eviction that authorizes the sheriff to. It authorizes the landlord to eventually file for a writ to have the sheriff kick you out of your apartment. But that's the final outcome of an eviction that's used in the literature. It's an eviction judgment. And so we're really interested in conditional on appearing in court. Do lawyers actually reduce the number of eviction judgments? And then the two supporting pieces of court evidence are non suits, which you can think of as like the best case outcome for a tenant, which is that the filing is taken off the court record. It's not just allowed to lapse. And then the other one would be a writ, which is beyond just the eviction. Does the landlord go ahead and ask the sheriff to kick you out of your apartment? And so you can think of the judgment is the final eviction. The Writ is like the worst possible thing for the tenant. The non suit is the best possible thing for a tenant. And I just want to set up a little bit. On the one hand you might say, okay, mechanically, it should be the case that lawyers must help tenants. Of course, if you have somebody in court, you must be able to do better. So why do we even need an RCT to study this question? And what I would say is, on the one hand that's true, directionally it should help. But on the other hand, it's not at all obvious that in the typical eviction case, a lawyer would be able to do anything. And the reason is just that a lot of these cases are open and shut from a legal perspective where, as I've said, the tenants owe months of overdue rent. And in Tennessee, the law happens to be landlord friendly. From the sense from the perspective of if you owe all this money on your, all this rent, it's not obvious that even if you have Clarence Darrow representing you, that they can really do much. Okay. So even if you have an amazing lawyer, it's not sure exactly clear what legal defenses they can raise. And so some defenses that are available in other jurisdictions, like New York City or California, where other studies like this have been done, are just not available in Tennessee. And so you can think of this as being a setting where we really do have uncertainty about what the impacts of lawyers would be. And the second thing is that the lawyers that we're providing are relatively cheap from the perspective of how much lawyers usually cost, which is like the usually cost way more than 250 to $350. But it's still not obvious. If they really only have a minor muted impact, then you think, okay, is there a better way that we could spend that money if the goal is to assist tenants? And I think we really did have uncertainty about what the impacts of lawyers would be on eviction. And that's why that's the primary Senate outcomes that we're looking at.
[00:23:24] Speaker A: Got it.
Super. And Aviv, y' all also did some.
Once you found the sort of as you were finding the primary outcome results, you also did some other analyses to try to try to both explain these results and then contextualize these results within the literature. Let's try to get the primary results out on the table first. So what did you find, and in particular, what did you find about the effect of either the lawyer or the offer of a lawyer, either of those two in relation to the ERAP program? Because I think what you're going to tell me is that the ERAP program actually became unavailable in the middle of the study. And from a scientific point of view that was serendipitous. That was actually helpful from a scientific point of view. So impacted the sort of big picture elevator pitch results for me.
[00:24:14] Speaker B: Sure. Yeah. Yeah, that's exactly right. And a nice nicely primed. So that what we find is if you cut the study in half and you naively looked at the first year of the study until December 31st of 2020 22. Right.
And. And looked at the effect of an offer or the effect of a lawyer, you would find really large effects on eviction judgments. So something depending on whether you're looking at the intent to treat or the effect of the offer or the effect of the lawyer, you're looking at 30 to 40% reductions in eviction judgments from getting a lawyer.
[00:24:47] Speaker A: And that's all right, 30 to 40% reductions or percentage point reductions. You want to make sure we are they percentage points or are they reductions? Because I think for most listeners it's what's the fraction that are going to get evicted without the either the offer of a lawyer and what's the fraction that are going to be evicted with the lawyer of the offer either. If we could just get a.
Yeah. Instead to talk about percentage reduction for most. For non quants, I found that to be less successful.
[00:25:13] Speaker B: Sure. Yeah. So for the first. So if you just focus on that first half of the study from March to December 2022.
[00:25:19] Speaker A: Yep.
[00:25:20] Speaker B: If you look 180 days after filing, among the people who didn't get the offer for a lawyer, 63% of them will have lost in court and have gotten a judgment. And if you look at the people who did get a lawyer, 20% of them. So that's a 40 percentage point reduction.
[00:25:36] Speaker A: Yep.
[00:25:37] Speaker B: That would be like the 60% reduction. And the the effect of the offer is a 24 percentage point reduction. And so that would be closer to the 40% reduction. So that's a lot of numbers, but really huge effects in this first half of the study.
[00:25:49] Speaker C: Yep.
[00:25:50] Speaker B: And then if you looked at the second half of the study, you would find basically no differences. If you looked. So there would be a delay in the eviction judgments at around 30 days. So those who got a lawyer would have had lower eviction rates at 30 days. But if you look 180 days out from the filing, you wouldn't be able to tell the two groups apart. And that is really concerning at a kind of first pass because if you just did the study in the one half or the other half, you wouldn't know about that counterfactual group. And it was really lucky that kind of. We spanned this time when ERAP was in existence and then expired. And then we could still do the study without the presence of ERAP and show that really it was the combination of access to rental assistance and lawyers that were able to achieve these huge effects in the first half of the study. And then without the rental assistance, lawyers were able to delay but not prevent evictions.
[00:26:39] Speaker A: And just for the fastidious and. How should I say this? OCD among us. OCD suffering among us. When you say ERAP expired, you mean what did the application stop or did the payment stop or which. What was it?
[00:26:52] Speaker B: Yeah, good question. The new applications, September of 2022, and they stopped. Or maybe they announced it first, Charlie.
[00:27:01] Speaker A: August, if I have it right.
[00:27:02] Speaker B: That's right. Okay.
September 1st. And stop making payments at the end of the calendar year.
[00:27:08] Speaker A: Yeah, so. So when you say first half and second half, you're picking as your date, your division date, they stop making payments.
[00:27:15] Speaker B: That's right. Yeah.
[00:27:16] Speaker A: Got it. Super. Okay. And so right away, you're thinking, it looks as though. And of course, this is always a dangerous thing to infer without a lot of more checks, which y' all did.
Right. It looks as though that. That something is going on. Lawyers are affected when ERAP's making payments. Lawyers are not affected when ERAP is not making payments. And so then you did a bunch of checks, and what did you find?
Charlie, want to get back to you, Charlie. Well, let me go back to you. What did you, in fact, find? That's what was going on.
[00:27:45] Speaker C: Yeah, the main zoom out, what we're finding is lawyers have a big effect only when emergency rental assistance is available. And so the concern that you might have is that how can we prove that it's really emergency rental assistance and not something else? And for the main part of the rct, in some sense, we have this very clean evidence because we're randomly assigning tenants to lawyers. And so if we're looking on average about what is the effect of lawyers pooling the two periods, you can be very confident in that evidence because it's like a very simple RCT where, you know, all of the I's and T's are dotted. And so you can rely on the evidence when you're then doing these kinds of heterogeneity cuts, meaning you look at before and after the expiration of Iraq, there you have to use more of the applied Microeconomics toolkit where you're naturally going to be moving from using these kinds of clean RCT methods to methods that have, that are more inferential and more scrutinizing the evidence and doing the best that we can. But we naturally should have more caution. And so we're trying to do our best to nail down the natural concern, which is, okay, maybe other things are changing around this time.
[00:28:54] Speaker A: Maybe it's just noise, had a really bad year. Right. And all of a sudden, yeah, maybe
[00:29:00] Speaker C: in, in winter lawyers are less effective than they are in summer or this sort of thing. How do you prove it? And we're going to use three main methods. So the first is if you just look at the period before and after, the differences are really striking. And so it doesn't. Yes, there may be some kind of seasonality and the impacts of lawyers, but it's hard to explain seasonality. That would be like a 40 percentage point difference in lawyers effects before versus after. So that seems a little bit implausible on its face. And we're trying to make a qualitative case that this is a reasonable hypothesis because this is a major rental assistance program.
Lawyers were able to negotiate with landlords and use this program as a bargaining chip.
The second thing that we do, which is a little bit more in the weeds, is known as mediation analysis where we're going to basically use the period when emergency rental assistance is available and try to control for the presence of emergency rental assistance. So look at those people who get emergency rental assistance with and without lawyers and see if lawyers effects in the period when emergency rental assistance is available. Once you account for the presence of emergency rental assistance statistically looks like the impacts of lawyers in the period when emergency rental assistance is not available and relative to a kind of vanilla mediation and analysis which I think can be in some cases persuasive. But oftentimes I think applied microeconomists are hold it often hold it at an arm's length. I think we have this advantage where we have a kind of out of sample test where you can look at in the period when it's not available, do the treatment effects line up to the period when it's available? But you account for emergency rental assistance and we find that it can explain about 50% of the gap. How do you explain for the other 50%? You can get into that, but I would say it's clearing at least some preliminary hurdles there. And then the last one is we have variation within the ERAP period in whether or not people were able to get ERAP based on idiosyncratic factors like whether or not they applied at a time when ERAP had all these kinds of applications going and it was just less likely to get assistance. So if you apply, they were too full. Say they were too full. Yeah, there's kind of congestion. And so you can use that as like a natural experiment to try to account for the role of ERAP in that period. And we find similar results to the mediation. There's actually those are the three main tests and in the paper there are a number of other tests where I won't bore the audience with it. I think we come away thinking there's a very clear picture about the role of erap. I think it's also reasonable to say relative to create a kind of hierarchy of the quality of the evidence here, where the evidence is highest quality when you're looking at the RCT on its own. And then as with any other applied microstudy, you need to believe assumptions more and look at the kind of collage of all the evidence more when you're looking at the emergency rental assistance impacts.
[00:31:59] Speaker A: And so then if we believe this analysis, then. And Aviv will go back to you for this part. If we believe this analysis, then the story that emerges from the paper is that lawyers were effective at getting tenants ERAP assistance and that's what avoided the eviction.
And when ERAP was unavailable or became unavailable, they didn't have any bullets to fire.
And so there was, there, there wasn't much that they could do to avoid the conviction. Is that a fair summary from a top level point of view?
[00:32:28] Speaker B: Yeah, that's right. And I think the way to, the way I understand this at least is lawyers needs tool, need tools in their toolbox in order to be effective. And in our case, Martin, rental assistance was this really powerful tool that they could use mainly to negotiate with landlords and get these evictions dismissed. One minor tool that they have is continuances, which were. Those were an important part of slowing down the process long enough for rental assistance to arrive. And that provides one nice test of our lawyers still doing everything they can without rental assistance. And we find that they are. They're still filing for these continuances and that's why we're still seeing the delays. But continuances can only take you so far. And some mix of procedural tools or these kind of out of court resources would be necessary, kind of in our view, for lawyers to have the kind of large effects that we see in the first Half of the study.
[00:33:14] Speaker A: And then so again, just restating this in the language of a legal services provider. Basically the eviction attorneys were effective because they turned themselves into public benefits attorneys. They turn themselves into benefits attorneys. Is that in the sense of avoiding the evictions?
[00:33:30] Speaker B: Yeah, I mean, I think we want to caution a little on the analogy because one, one natural question is, okay, great, if lawyers are just helping people fill out these forms, then you don't need a lawyer for that. You can have somebody else help a tenant fill out the form, get the check from the county and pay off the back rents. And so we have a kind of a small study in our paper where we provided counselors and we can talk about some of the hesitations with over interpreting from those results. But what we found is that's not sufficient and that lawyers play these kind of core legal roles in addition to helping people actually apply for the, for the rental assistance, like advocating in court continuances, helping to credibly signal to the landlord that this money actually is coming. And they're functioning as public benefits attorneys in the sense that their focus is on helping to get these resources and using them. But they're not functioning just as public benefits administrators while they still have these core legal roles that they're playing that it's not clear a tenant can't.
[00:34:25] Speaker A: And I guess for another conversation, not this one, you could ask the question about whether you actually needed at least three years of law school and a JD to formulate those roles. Typically, we think lawyers could, when they advocate or when they appear in court, they're doing lawyerly work. But if, for example, you could credibly signal without the jd, we're actually going to go get this ERAP assistance landlord, give us a couple of weeks to do it right, maybe you wouldn't need it. Totally.
[00:34:51] Speaker B: That's exactly right. Yeah. And if the continuance is in fact a necessary ingredient, unbundled legal assistance that can help tenants file for continuance without a lawyer. And that might be a more efficient way. So yeah, I'm not claiming that the takeaways you need for full representation by any stretch, but that in our study there were legal roles going on in addition to these kind of form filling out administrator benefits.
[00:35:11] Speaker A: Terrific.
And we won't have time to go through in as much detail about, about the broader analyses where you contextualize this study and the, in the larger set of studies that have been done, some randomized, some not on, on the effect of legal assistance and summary eviction. But this study did lead you to investigate and say, wait a minute, if ERAP or if rental assistance, paying arrearages, paying back rent made such a big difference in this context, mightn't it explain some of the disparate results from the literature? And just to preview this, the literature is disparate. It comes to different conclusions about the effect of representation, both in, specifically in the area of summary eviction and then more generally in the area of the effect of an employer in different legal contexts, civil and criminal. You all focus this one effort on summary eviction and again found disparate results on the effect of a lawyer. And so what did you find when you take a look? Charlie, let's go back to you. What did you find when you took a look at the other studies to see if the availability of emergency rental assistance was a good variable to look at in terms of interacting and explaining when a lawyer is effective and when not?
Thanks.
[00:36:29] Speaker C: Yeah, so there are a handful of other studies. We think that against the backdrop of this literature, our study has some qualitative advantages over those other studies in terms of just it allows us to say something about contemporary rental markets in a way that, you know, other studies are just now a few decades old or took place in other settings. And I think that said, there does exist this literature with existing randomized trials, clean evidence, including, of course, from your work, Jim, and where we think it's important to step back and aggregate what have we learned from the literature as a whole? And I think this literature has this puzzle at its heart, which is that there are all these studies that have taken place that don't seem statistically consistent with each other. That is, people are finding impacts of lawyers on eviction that don't seem like they can be reconciled. Some find positive impacts, some find null effects. Even your studies done in the same state during the same time period are finding different effects. One finds a very large impact of lawyers on eviction. The other one finds no impact. And if you were to make a recommendation to a policymaker, should you adopt a program that provides free legal assistance to tenants facing eviction? I think we have some uncertainty about A, what the average effect of that program would be and B, what are the conditions that could make it effective? And so what we wanted to do is to try to understand could emergency rental assistance play some role in explaining variation across studies? And I think some of the most convincing evidence for our for this analysis really comes from New York City, where there have been a couple of studies done of Right to Counsel in New York city. And there's this sort of celebrated paper by Sarah Nadal where it turns out that in this paper they provided lawyers to tenants facing eviction, but those lawyers were similarly able to negotiate on behalf of the tenants to receive emergency rental assistance. Now, it wasn't the ERAP program, it was long before the pandemic. But what this kind of suggested to us is that these kinds of one shot deals or rental assistance for people who are really in need fixing eviction, it's not just a pandemic ERA feature of these legal assistance programs. These pools of money do exist in other contexts. And at least in this kind of in the cross section across studies, it does seem it was available in some of the more successful ones. Now, I can't explain all of the differences, and when we do a quantitative meta analysis, our best guess is that it explains maybe a little over 50% of the variation is the presence of emergency rental assistance. And so we think of this as providing a parsimonious through line. And potentially one takeaway message for people who are interested in adopting Reiki Council is that not just in Memphis, but in other places, these programs that had positive impacts often had the availability of emergency rental assistance. And if you adopt these programs, you may want to think about pairing them with rental assistance in some cases, which
[00:39:29] Speaker A: then back to you, Aviv, for the second to the last question, just to give you an idea where we are second to last question brings us to the question of cost and efficiency of this as a way of doing business. And this is one of the qu one of the things y' all raise in the paper, which is okay, it seems like the lawyers, the lawyers actually weren't that expensive in this context, especially the Low Bono lawyers, 250 to $300, frankly, it's just dirt cheap in terms of any kind of social socioeconomic intervention. 250, 300. 350 to $300 of a pop is pretty low, right? But if you have to compare that to add on to that emergency rental assistance of something like 2000 or $3000 per person or per unit or whatever it is now, you're in a different ballgame. You are literally in an order of magnitude different ballgame. And thinking in terms for the policymakers out there, who of course all should be listening to proof over precedent, what is the take? To the extent there is a sort of clean takeaway message or a set of considerations, what would you say to them? Aviv?
[00:40:30] Speaker B: Yeah, so I would say a few things. So first is if what we're primarily interested in is preventing eviction judgments, then we need to think about what is the bundle of goods that will actually do that and the true cost of that bundle of goods. In this case, with the legal defenses available in Memphis, that bundle of goods is pretty expensive because it includes the rental assistance. But the prevention of the eviction judgment is not the only rationale for providing an attorney to tenants facing eviction. And so in the kind of last part of this of the paper, we report results on incentivized surveys that we did. And I know we're short on time, so I'll keep a very high level, but basically, we offered tenants the choice between cash and a lawyer, and we implemented some of those choices to make these kinds of real stakes decisions where a tenant had to wrestle with what would be more helpful for them in this stage of life.
And, and we focused on the period where rental assistance, where ERAP wasn't available.
And then we did all of these other elicitations to understand did tenants have miscalibrated beliefs on how effective the lawyer would be, or what role did they expect the lawyer to play? And when we focus on the tenants who had the accurate beliefs that during this period, a lawyer wasn't going to help them prevent the eviction, but their main role was going to be to help them manage stress and to move through the process and potentially have some delays, those people still valued a lawyer at more than $600, meaning they'd prefer to get the lawyer than receiving $600 in cash, which was more than twice our cost of providing them. And so I think that is one piece of data for this pool of tenants. But what it points to is the kind of, the broader policy goal is really important to specify, because the goal might be to support tenants during this time, and you can still do that with a lawyer, even if you don't see the outcomes in eviction judgments.
But being precise upfront is what's necessary in order to know whether this is a good investment of resources, or whether it be better to just give people cash, or whether it be better to give people delays automatically so that they have time to move, but they might not have support during the legal process.
So, yeah, so I think that the takeaway from us is somewhat nuanced, which is that the eviction judgment itself might be really expensive to prevent, but that lawyers might be a really efficient use of resources because they provide all these other benefits to tenants.
[00:42:45] Speaker A: And just to highlight one thing with one, one little follow up before we go to the last question you mentioned at the Outset of your answer. In a setting like Memphis or Shelby county, under Tennessee law, where there are very few defenses available, you need to give the lawyers something to do. You said tool in the toolkit. I use bullets in the gun. I like yours much better. It's much more peaceful, less violent. Another thing you could give them to do is you could actually give them defenses. So you could change the substantive summary eviction law and allow the lawyers to lawyer in the sense that they could assert defenses. And that might also potentially in some sense that in a bizarre way that becomes somewhat interchangeable with the ERAP program in the sense that it gives the lawyers a tool in the toolkit, a bullet in the gut.
[00:43:31] Speaker B: I think that's exactly right. And that's what I mean is that, you know, these tools could look very different in different jurisdictions.
[00:43:36] Speaker A: Yeah. Terrific. Okay, so final question we'll ask each of you. So Charlie, at this point, if you haven't, most people have and proof over precedent interviews, they say, man, this person interviewed me. Interviewing me is a complete idiot. He hasn't asked me the key question that I really want to answer in the podcast and make sure that everybody understands the answer too. So pretend like I answer I asked that question. What's the answer? What's the takeaway that we haven't gotten to? What's your final thought for us that we haven't discussed?
[00:44:07] Speaker C: I think for listeners of the podcast, in addition to the specifics about eviction policy, this really changed my perspective about how should we think about access to justice policy in general. And I think this paper is suggesting that access to justice policy doesn't just hinge on lawyering in court, but rather relates to the kind of bundle of services both in court and out of court and spillovers across programs. It seems like what lawyers were doing.
I had thought that what lawyers would do would be taking out their dusty tomes and, uh, making amazing arguments. And then lawyers would be. Judges would be flabbergasted and let the tenant go. But what actually happened is lawyers were connecting tenants to resources that existed far beyond the corporate.
And I think to suggest that access to policy, to justice policy might really think about lawyers as providing this kind of out of court assistance rather than exclusively serving as vehicles for representation within the court.
[00:45:09] Speaker A: Terrific. Aviv, same question. Which question would you wish we had I had asked or that leave a takeaway thought for everybody?
[00:45:16] Speaker B: Sure, yeah, I'll stay big picture, which is. I think it's their cliche criticisms or concerns about RCTs or kind of these types of analyses that one, context matters and two, that it matters what you choose to measure. And those are cliches for a reason. I think those are broadly accepted as true. But this study really highlights both of them. One, in that it's we had this gold standard RCT that randomized and that showed that if you had this RCT in either one of the pre or post erap expiration periods, you would have, you would have misunderstood the phenomenon. And, and so that really drives home this need for many studies in different jurisdictions for being really carefully attentive to the policy context that these studies operate in. And then the second is this choice of measurement where, you know, even in the case where eviction judgments aren't reduced, but the value is so high and it's really hard to pick up the right measures in administrative data that would allow you to infer that this value that's produced to tenants is really high, even if you don't see the effects on eviction judgments. And so being really clear about those welfare impacts and how close can you get to them? What are the, what are really the things that, that a study should capture if it's going to reach a conclusion about whether this is a good use of tax dollars?
[00:46:30] Speaker A: Terrific.
So, Aviv Caspi, Charlie Rafkin, thank you so much for, for being with us. We'll include a link to the what is now, I think the latest version of the paper. Look forward to the final version whenever it, whenever it comes around.
And and to remind everybody that listening to the podcast is no substitute for reading the excellent paper because there's a lot that we didn't get to. So again, Aviv, thank you so much for the conversation.
[00:46:56] Speaker B: Thank you, Jim. This is really fun.
[00:46:57] Speaker A: Charlie, thank you so much for the conversation. Really appreciate you taking the time.
[00:47:01] Speaker C: Thank you.
[00:47:02] Speaker A: Proof Over Precedent is a production of the Access to Justice Lab at Harvard Law School.
Views expressed in student podcasts are not necessarily those of the A2J lab.
Thanks for listening. If we piqued your interest, please subscribe wherever you get your podcasts. Even better, leave us a rating or share an episode with a friend or on social media.
Here's a sneak preview of what we'll bring you next week.
[00:47:27] Speaker D: We wanted to understand the effect of expungement or record clearing on housing security and employment stability.
But to do that, we couldn't really randomly assign whether or not someone received a record clearing remedy.
So we needed to identify a way that we thought individuals would be more likely to receive a record clearing remedy and randomly assign that. And so what we did was work with a legal services provider, Kansas Legal Services, and we randomly assigned either full scope representation, which we thought would be the condition in which people would be most likely to receive record clearing, or materials, which was the condition that we thought folks would be less likely to be successful in their record clearing attempts.
