Episode Transcript
[00:00:00] Speaker A: Imagine a justice system built on rigorous evidence, not gut instincts or educated guesses about what works and what doesn't.
More people could access the civil justice they deserve.
The criminal justice system could be smaller, more effective and more humane.
The Access to Justice Lab here at Harvard Law School is producing that needed evidence. And this podcast is about the challenge of transforming law into an evidence based field.
I'm your host, Jim Greiner and this is Proof Over Precedent.
This week we're bringing you a student voice.
[00:00:38] Speaker B: Hi everyone, my name is Andrew Garcia. I'm a 3L at Harvard Law School. I am particularly interested in wage theft, its rising prevalence among the immigrant and worker community nationally, but also in cities that I have a connection to like New York City and Boston.
I'm discussing this topic today with Joshua Medina, the legal director at Justicia Lab, who have developed over the course of four or five years this tool called Reclamo AI.
Reclamo AI is a chatbot that an organization that I worked with prior to law school helped pilot where workers can go through a sort of intake process or just chat with this chatbot directly about a potential wage theft case that they may have get more information about their rights as workers, if they do have a claim, how they might go about filing that claim, and then if they were to need more support, being able to connect them with worker center, legal aid service provider or some other related organization in their area.
And it is my absolute pleasure to introduce Joshua.
[00:01:58] Speaker C: Thank you so much, Andrew. It's great to talk about it a little bit. So I think in 2022, 2023, you saw a sneak peek into early version of Reclamo, the previous version, which was really a tool built in collaboration and partnership with organizations like make the Road New York. They're doing a lot of advocacy work for workers in the city.
And for us, it was a way to respond to their need. It's, it's a product that came out of a co design Sprint, which is something that we do when we come together in a room with attorneys, with advocate and with workers so that we have people from all different kind of levels of stakeholders in the, from the community to the directors of programs and things like that.
And that we asked them, what are the problems that you're seeing? What do you want to fix? What would help your work be better and what would help people in your community?
And then how might we address that? Right. So as a, as a product of that came this idea of advocates wanted a more streamlined, efficient and consistent way to get information that they needed. From workers that would then translate directly into complaints that could be filed with the wage thefts complaint, you know, office and form with the Department of Labor. So that was kind of the mandate for the first build out. And it was just a very simple backend forum that advocates could log into, and then it would give them kind of a script of questions to ask. They put in the answers.
And the smart interview would do some math for numbers and talk about, okay, this is what you're owed. These are the penalties that should be paid to you in addition to what you're owed based on the wages that you've not been paid so far. And then from there, we can help you to generate the actual wage theft complaint on the form, we can help you to generate a demand letter to send to your employer, and we can help you to generate a script to call your employer on the phone and tell them, this is what I want to do. And that could be done for this first version in English and Spanish.
We liked that a lot.
It was something that was really great to see in practice. And we also learned, given the thoroughness of the form, that that led to long interviews, right? So we started to talk about how can we shorten it? And we kind of went through and we streamlined it a bit and we made some tweaks to maximize what we had, and it was good. But as a landscape has shifted for workers and for other folks, including immigrant communities and immigrant workers in New York City, the demand changed bit, right? What people wanted, their solution to their problems, all of it. That all changed. So what partners were looking for now was a way to inject more expertise and more knowledge into the worker communities.
And folks told us, we want people to be able to talk about what their problem is, learn about what that might be considered in a legal way, but in plain language as it relates to them, to be able to do it in their language, to be able to understand what their situation is, kind of self identify what their options might be based on what they've experienced, whether it's wage theft or beyond. You know, we're also talking about, you know, discrimination, we're talking about harassment, we're talking about, you know, several different issue areas for workplace justice. So growing Rick Claville beyond the wage theft smart form to this way to deliver this knowledge around broader workplace justice areas to the community was a good challenge for us.
And we took to working more with experts. So we worked with Nida Lang, who's a wonderful expert in New York City, again working with workers rights issues and A lot of their network partners came on board and we essentially curated in the database of trustworthy, really reliable, expert curated information on workplace justice issues, the kind of thing that you would want any attorney to be trained in, to be able to counsel or give information to someone looking for help or some guidance. And from there we started building out some technology to be able to deliver that to communities in the way that people wanted.
So we started working with some AI technologies and also some different technologies that we use in our tech stack. And we thought about our challenges with using kind of any AI tool, right? And we are advocates of not everything should be solved with AI. We want to be able to deliver services that really matter. And the way that we talk about it is if this is the field and we're asking for help to solve a problem, our technology needs to be the fulcrum. It's the right level for the, for the field to be able to lift and generate some momentum for their communities. Right.
So when we looked at this, we thought, okay, this is a moment for AI. But we're very skeptical of some uses of AI.
So what we don't want is someone going out into Google or ChatGPT and starting to put a lot of their personal information, asking for help and asking for guidance in terms of what they should be doing for their problem. Because A, that's sharing a lot of personal information in a public space that can be accessed a lot of different ways, and B, it's not always reliable because any kind of AI on the Internet is looking at the whole Internet, both the very good legal articles published and also some private attorney, their predatory website. Right in some other place. So what we did was we actually built our own database. Then we built a radbot, which essentially is a delivery system that can receive a chat and go back. And it has become an expert in all of these materials that we curated and it can deliver a response back with a link back to the original source materials. So now we don't have this unknown of good and bad information, but we did that. And then to solve for that, we don't want people giving personal information even to us.
We built in a redactor that is a filter for any kind of personal information before anything touches tech like any AI. So if, if you said, my name is Andrew Garcia and I have this issue and I work, I live at 1001 South street, right, we would get that to the actual AI bot saying, Hey, I have an issue, I'm a worker, right? And that's it. It would Remove the name, it would remove where you live, it would remove that kind of stuff.
And that's been really great and comforting for us because it means that we can keep people secure.
[00:08:41] Speaker B: Yeah.
[00:08:41] Speaker C: So after that, we did a bunch of rigorous testing with, you know, hundreds and hundreds of people kind of doing reviews and running all kinds of scenarios to get the tone right for the bot, to make sure it was responding in a natural, warm way, you know, giving some empathy, even saying, I'm sorry that you've dealt with that, I'm sorry that you've been through that, but I think I can give you some resources that could help you. Right. And then we also created a list of all the workplace justice organizations in New York City, by borough, by zip code that people can plug in if they want some help. And then we have some really more exciting things that are plugged onto the backend, where in the future we can actually have the chatbot hand someone off to an intake form directly to legal services waiting lists and queues for these providers, including NYLAG and New York City. So you can go from learning about an issue saying, I can't. I have an issue. I can get help with illegal advocacy to saying, oh, I can actually fill out a form and talk with an attorney about this pretty soon. Right? Yeah.
[00:09:44] Speaker A: Is.
[00:09:44] Speaker B: Is the form, I imagine it's stripped of personal identifying information, but how would the form be queued for the attorneys on the other side?
[00:09:56] Speaker C: Yeah, so that's a great question.
And the way that it works is the AI will actually handy waf from the AI chatbot to a different kind of technology that's on our database, so they can actually receive personal information so that the attorneys can contact you. So then that, that profile, that intake form slots in to the existing intake systems at the organizations and by that
[00:10:19] Speaker B: point is covered by privilege.
[00:10:21] Speaker C: So that's kind of the way that we set it up. And the cool thing about this is you can access this bot online at Ricklamo AI, but you can also use it via WhatsApp instead of playing with it. Right?
[00:10:36] Speaker B: Yeah, yeah, yeah.
[00:10:38] Speaker C: So it's, it's been something that we're really excited about that we really, really like. And it allowed us to kind of be on the, on the precipice of AI in terms of making sure that we're using it safely and ethically.
[00:10:50] Speaker D: Yep.
[00:10:51] Speaker C: And it was a really great chance to do a lot of evaluation and, and to build rubrics and frameworks and processes just to make sure that we have an outstanding hit rate. Right. And at this point now, like in the, once we release it to the public, we're looking at the only, the, the only kinds of issues that we're spotting are ways that we can tweak the language to be a little bit better, a little bit more comforting, a little bit more concise, a little bit more whatever.
[00:11:20] Speaker D: Right.
[00:11:21] Speaker C: And so we're not having any kinds of critical errors, any kind of major issues. What we're, what we're learning is how can we make it even responsive and even more supportive in the way that it feels to engage with it. So totally we went from this smart form a few years ago now to this whole interface that people can use to get information. And what that does is it builds expertise in the communities so then they can start helping each other out, referring each other to the tool and making sure people can find help in their areas.
[00:11:49] Speaker B: Amazing. Thank you for the breakdown. I'm curious because I was able to see this earlier version.
Something that stands out to me is when I think about the pilot in 2223, I remember that this was obviously before information was being purged and sent to immigration and other federal authorities. But I remember a lot of the outputs would include, like you had said, there's the demand letter, potentially a complaint, even a state and federal DOL submission complaint form form. And when you were speaking about the fulcrum of advocates being able to use the tech to uplift both the work and the clients that you're serving, it seems like there's also been a shift in trying to perhaps not give as much of that like regenerated information or forms and actually in addition to information, connect the client with the advocates and the potential legal providers that would be able to kind of push the claim forward. I'm wondering if in parallel with this client and community centered change in how you structure the platform, if there's also been a concern about the other issues that have risen up with legal AI tools, which has been the unauthorized practice of law things with LegalZoom. And I'm wondering if that was a part of the calculus, you know what I mean? Like if there was anything that, that kind of like you guys thought not only would it make this better because it avoids that regulatory issue, but it also happens to center the, the response in a more community centered way.
[00:13:37] Speaker C: Yeah, that's exactly it. Right. And this was a big part of our calculus. Our calculus from the beginning is, okay, we're shifting from this tool which is just for advocates. Right. So it's within the safety of one office to Be able to provide information to communities and then connect them with different kinds of advocates.
[00:13:54] Speaker B: Right.
[00:13:55] Speaker C: That's a big shift in audience. And there's always, with any kind of AI tool, a really big fear and danger of UPL happening. Right? Yeah.
So for us, it's. We make tools for people. We don't make tools just to do everything on their own. This is about connecting purposefully so that people can get advice. But we also wanted to make sure that we trained the tool from the very beginning and talking about the rubric that we set up to. To make sure that we had callous evaluations run through and to make sure that we were hitting at an over 99% hit rate, with the only remaining like 5%, 0.5% being things that we can improve. Like a word used here or a sentence could be short. So, yeah, so to start that off, like, for. One of the defining pieces of the evaluation is we created a really solid definition of what is actual actionable information and what is advice.
And we built that line into the bot from the beginning. And we had it run through exercises, we had it run through tests, we had it run through hundreds and hundreds and hundreds of evaluations of all different kinds to make sure that we were never crossing that line and that it could identify how it should speak.
That was part of it is we did that diligence from the very beginning, and then we made it more user friendly and then we can make it more powerful by giving it more resources and by playing with it in the field and identifying gaps where, oh, it said it didn't have an answer to this question. Right. Which is something we want it to do. If it doesn't have an answer, we want it to say, I don't have that information right here. So, like, these are things that we're really working at doing to make it.
To strike that balance of, okay, actionable information, always legal advice, never.
So we don't run afoul of upl, but also in a way that we're creating community members who are empowered by information with the knowledge that there are dangers that differ depending upon the worker situation. Right. And we want a game plan not just for your worker who grew up as a US citizen in this country, but for the most marginalized workers across the board and the dangers that that faces. So instead of saying, hey, I can create a demand letter for you if you want. Right.
We could say, hey, one of the things that some folks have done is create a demand letter. But there are some risks with that, especially if you if you're undocumented or if you are not a citizen. Right.
[00:16:23] Speaker A: Yeah.
[00:16:24] Speaker C: And then we can talk about from there. Like that opens up conversations with the BOT transition to is there human trafficking going on Right.
In other doorways. So this is something that we took very seriously. And with Justicia Lab being committed to providing tools for the immigrant rights movement, we also wanted to make sure that we are considering all the most marginalized populations and building a tool for all workers, because that matters very, very much.
[00:16:52] Speaker B: Yeah, thank you for that. You had said earlier about the hit rate and I'm wondering like as you've conceptualized it and integrated into this, what appears based on how you've described it to be like this closed loop information system, what makes something a hit versus that 0.5% that doesn't necessarily that that isn't a hit.
[00:17:13] Speaker C: So at the very. It starts, it gets really good by starting very strong at the beginning of the training. And by that I mean we took expert attorneys and had them write out the most common questions that they received from community members. And then we also had community members write out their most common questions for attorneys. And then we had these extra attorneys double up or triple up to give the perfect ideal response to each of those questions.
And then we refine those responses based on the parameters that we had developed to say, hey, this attorney wrote a three page novel the phone's gonna get as a WhatsApp message. Let's take it down and then figure out. It's amazing, three pages, maybe give like 150 words and then offer like a follow up question to see if they want more. Right. So like we kept doing that and then we compared every single response the BOT would make to those expert responses and evaluate it. And we would do machine evaluation and then we would do human evaluation, both us and our partners.
So it was a very long back and forth process.
And our hit rate was do we have a match on all of the elements of those responses to match in tone? Right. Do we have. Are we, are we scoring good on the UPL parameter?
So these are all the things that we had to make sure. And when I talk about a 99 point, I think it's 99.6 or 99.7% hit rate for our testing on this tool as it's gone out the, the 0.3 or 4% or 0.03 or.04% that we're seeing that is being flagged for improvement.
Are, could this response be shorter? Could someone ask two follow up questions here instead of one.
[00:19:00] Speaker B: Right.
[00:19:00] Speaker C: Like for the worker, so that they can have two different options, two different paths to follow. Right. So it's just things like that as opposed to we're having hallucinations or having content errors because of the vetting that we. The testing that we've done from the beginning to vet the system and its technology. People centered testing.
[00:19:19] Speaker B: Yeah, yeah, of course. People are an expert. Centered.
It's both sides of the aisle, I guess, in that sense. Which is interesting as a tool that has begun to create more and more classifications.
How do you categorize the AI system? Because it sounds a bit like an LLM, but because I'm not particularly knowledgeable in how the AI systems work. Like where something like ChatGPT maybe has some like these training parameters, but pulls from the larger web. What is this system?
How would you identify it? I guess.
[00:19:58] Speaker C: Well, so this is a ragbot.
[00:19:59] Speaker B: If it can be classified, I guess,
[00:20:01] Speaker C: yeah, it's a ragbot. But we use agentic AI to ensure that we can put in our systems around it to make sure that we have what we need. So, for example, we can have an agent hand off to the filter and identify the kind of personal information. Right.
So like there's things that we can do to make sure that we are providing support to the ragbot that we created. But it's really just, yeah, just a call and response kind of situation where we created a bank of resources and then we instruct the bot how to, how to search and provide responses. And it's something that, you know, especially with the advances of AI, we are going to be constantly kind of reiterating because there's opportunities to improve it.
So we use something called M8N, which is a way that we can string our tech stack pieces together. That's a way that we can make sure that the bot knows when to call out to WhatsApp or that WhatsApp can call the bot. And there can be a handoff to an intake form and there can be a handoff to our directory. Right. Or a bot can read our directory. So we can have one bot that's just an expert in the directory providers and then give that back to the chatbot. Right. So there's things that we can do like that in this closed system. And then nowadays the way that things are advancing, I mean, we have, we have the actual AI, we have, for lack of a better term, like the interface that you're actually chatting with. And then we have a harness, which is like a way that things are structured and flowing between all of these pieces. And so there's a lot of different things that are put together, but I think without having to get really deeply into tech jargon and the tech stack, it's essentially just a RAND bot with add ons that help us to usher information from and where it needs to go.
[00:21:51] Speaker B: I would imagine that your hit rate is different than what I'm about to ask, but when there's something that the bot doesn't necessarily know how to respond to, how does it process the questioner or how does it output a response?
[00:22:10] Speaker C: Yeah. So I think if you ask, first of all, we set parameters around workplace justice and for this bot, because of what it is, it's specific to New York City, Right. So it has laws for New York City, has laws for New York State, and then it has, federal laws are applicable within the city. So that is its kind of realm. If I ask it, hey, what was the score of Game 3 of the 1988 World Series when the Dodgers won? Right. Like, it can, it can go through and it would say, I received your question, but I'm here to help with workplace justice issues in New York City.
So this doesn't seem like something that I should be helping with. Do you have any questions about workplace justice? So there's something like that that could come through. Right.
But then there's also, there's also different things where if you're in the bot and you try to trick it and ask a question, right? Like if you say, what are the. One of the things we did early on was we said, okay, like it has information about immigrate immigration rights in the workplace. What if I say that, you know, ICE pulled me over outside of my house, what do I do? Right.
And then it can say very, you know, very, in a very supportive way. Like, this is really scary, I'm sorry you're experiencing that.
And it could say, I'm here to help a workplace justice. If it's an emergency, you know, I'm not the one to help you.
[00:23:36] Speaker D: Right.
[00:23:37] Speaker C: But if you have questions because you're in a work vehicle, I can help you. Right. So it kind of, we, we are really working with it hard to make sure that you can parse and identify what it, what may or not may or may not be pertinent. And if it doesn't have anything, it can default back to. I don't have any information about that. But we can put you in contact with this directory of, of organizations that do work with justice advocacy. Right.
So it will look different depending on situation. I Guess is what I am trying to say, that's what we want it to be.
[00:24:05] Speaker B: Of course, before we started recording, you were talking about how you conceptualize access to justice being like a democratization of the tools that workers in the community can use in so many words. And it feels like by connecting workers with local advocates and providers, you are creating not only a community connection, but also kind of just like putting them in a place where you're both avoiding a UPL issue, but also connecting them with the support that they need.
And then as you guys are developing the tool further, how do you feel about the way in which the tool is achieving that goal of kind of just like making wage recovery tools more accessible and connecting workers with both providers and the movement, I would imagine, at a larger scale?
[00:24:58] Speaker C: Yeah. So I think that's a really powerful question because we should be talking about it more.
There are a lot of problems with our justice system in this country. Right. And in every country.
What I would say is that we want to approach our work and our collaborations from a legal empowerment perspective, meaning that we want to make sure that we are doing our part to decolonize the legal system and its power dynamics in the way that talking about access to justice justice systems are oftentimes structured in a way that are not accessible to folks. Right. So how do we inject more resources, more expertise, more information into communities so they can not just have access, but be participants who have empowered themselves with knowledge in their whole journey?
Or a case that's a wage theft case, or to make an empowered decision for what justice looks like for them, even if their attorney doesn't agree with it. Maybe their attorney says, you have a wage theft claim here.
And they say, I don't feel comfortable sharing my information about where I live with this government organization right now, given the landscapes. Right. But this is something where we want people to feel empowered to do that.
And it's also want what our advocates are asking for. They're asking for people who are involved and engaged, and they're trying to share more of their resources that they're working to create and their information to help people identify these legal issues. So the approach here is this tool, down the road, when we talk about this in five years, 10 years, has the capacity to be immensely successful in creating a New York City community of workers that are informed, that feel competent and competent to be able to engage fully and kind of diminish these. This disparity in power dynamics with an attorney when they're working on their case. So it's about taking, taking the lawyer. Right. And making them seem a little bit less intimidating. Right. But and, and taking the law and making it seem a little bit more digestible.
[00:27:10] Speaker B: Yeah.
[00:27:11] Speaker C: And that's what we're trying to do. And in doing that, we help more people recognize that their life issue is actually a legal issue and that that legal issue actually has a path towards a remedy. And it can be a remedy that makes sense to them. It doesn't need to be the remedy that makes the most sense to the attorney. So they should be able to have that conversation.
[00:27:29] Speaker B: Yeah. Right.
And it almost like engage in that, like client directed advocacy more. Right. Like if, if they are more aware of the different remedies that are available to them, they're less beholden to the attorneys. Singular legal knowledge. Yeah.
[00:27:45] Speaker C: And I think it's more of a partnership. Right.
[00:27:47] Speaker B: Yeah.
[00:27:48] Speaker C: We should be partnering on our own cases with people as opposed to just asking for an expert to take the wheel fully.
The way that we share information with our attorneys, the way that we are identifying what remedies exist, the way that we're making those remedies personal to us and what we need. Right. For our life. What's actually going to help me right now. Right.
Is sending a demand letter to an employer really realistic for me and my family? No. Right. But if I'm a worker in a different situation, let's say I'm a US Citizen who has, who has an incredible track record and feels very safe to file a Wagstaff complaint or to send a demand letter to my employer, or to make sure that there's an investigation happening by an Department of Labor that's going to help everybody that I work with.
[00:28:37] Speaker B: Right.
[00:28:38] Speaker C: So it'll look different for everybody.
[00:28:40] Speaker B: Okay, last question. When I think about organizing and when I think about movement work, I often think about how much of the work is based on the like, imagination and like, reinvention of systems as they are towards something that is yet unmaterialized.
Where would you like to see this tool go? Like, where.
What is beyond for you at this point? I would say, what, four years into this particular initiative, not usticlab generally, but for diamond, where do you see it going? And then as far as these tech and other AI tools for movement work, where do you see it going?
[00:29:23] Speaker A: Yeah.
[00:29:23] Speaker C: So, I mean, definitely I will start off by sending that up to us to dictate or decide where it should be going. But where we've heard people get really excited about this is by saying, oh, what if we can have a database of Offenders. Right. For wage theft issues and for workplace justice issues. What if we can take a look at that? Maybe there's already a letter of investigation out there that could help somebody, you know, with. With a labor issue who's also an immigrant.
Maybe there's a way to figure out list human trafficking.
And the simple employer who's taking, who's participating or facilitating human trafficking maybe has more victims out there than we knew. Right. Because now we see them popping up on this sheet over and over and over again. So there's things like that that could be really exciting. There's ways that the Department of Labor itself could say, oh, this tool is helping so many people. We would be willing to create an API with you all to connect it directly to our weight shift form. So somebody can just, in talking with this bot, give the information to our system if they want to file a complaint. Right then and there. Right? Yeah. Or that's something that we could also give to advocates if that's the best direction. So, like, there's different things that could happen where for us it's about transparency in the system, it's about accountability and what actual enforcement is happening. And it's about giving the communities the tools to decide where they want to go with the movement.
[00:30:51] Speaker B: Joshua, thank you so much for taking the time. I had a really informative experience and I'm thankful to get to hear more about the tool that we helped pilot many years ago and has come a long way in figuring out how to connect workers with the mechanisms and the tools necessary to get wages, to get benefits, learn how to prevent wage theft, and also maybe create some worker cohesion along the way.
[00:31:20] Speaker C: Thank you.
[00:31:21] Speaker A: Proof over precedent is a production of the Access to Justice Lab at Harvard Law School.
Views expressed in student podcasts are not necessarily those of the A J Lab.
Thanks for listening. If we piqued your interest, please subscribe wherever you get your podcasts. Even better, leave us a rating or share an episode with a friend or on social media.
Here's a sneak preview of what we'll bring you next week.
[00:31:47] Speaker D: So the central really question that implicates access to justice is. Well, first, as a foundational issue, the conversation is usually about a public citizen's ability to access the justice system if they have a proceeding against them. The classic example is a civil right to counsel.
Fitting this aspect of police accountability into this conversation is, I think it's very interesting. I think there is a lot of weight to the fact that you should have in the access to justice conversation. The public's ability to hold police officers to account. Right. That the public can't have access to justice unless they have some way to hold law enforcement officials accountable for alleged misconduct. And when you have procedural protections that are given to a select group of people and not to the public, that's when you start raising an eyebrow.
