Episode 57: Libraries and AI—Partners for Accessible Justice?

[00:00:00] Speaker A: Imagine a justice system built on rigorous evidence, not gut instincts or educated guesses about what works and what doesn't. More people could access the civil justice they deserve. The criminal justice system could be smaller, more effective and more humane. The Access to Justice Lab here at Harvard Law School is producing that needed evidence. And this podcast is about the challenge of transforming law into an evidence based field. I'm your host Jim Greiner, and this is proof Over Precedent. This week we're bringing you a student voice. [00:00:38] Speaker B: Hello, my name is Grace Gao. I'm a went out at Hubbard Law School. I'm here today with Aristana Skordes, Product and Research Manager at the Harvard Law School Library Innovation Lab. Ari, thank you so much for joining me today. [00:00:51] Speaker C: Hi Grace, thanks for having me. [00:00:54] Speaker B: Of course. Can you tell us a little bit about yourself, your backgrounds and experiences, especially your current role at the HLS Innovation Lab and the kinds of project you work on? [00:01:03] Speaker C: Sure. I've had a very varied background. I would say the main through line throughout my career has always been a commitment to research and access to information and also education. So I used to be a software engineer. I specifically did a lot of applied AI and machine learning in, in environmental sciences and biomedical sciences. I was, I did a lot of large data infrastructure work for national labs. I was a sailing instructor. I used to be a community, a big community organizer. I was a teacher. So all these kinds of things that all have really come together for me and just being about helping people get access to the information they need and thinking very carefully about systems and how we can care about the people in those systems. [00:01:48] Speaker B: Wait, that's fascinating here because I talked about how the legal AI tools can be not only available to people to better access information, but also to be usable and accessible in different distribution channels. And something I talked about is libraries, community centers, and the fact that you were working in a library and communities really will add a lot of insights into, in the context that we're focusing on today. So I was wondering, in your view, what does it mean for a tech tool to be accessible for people who are facing the digital barriers? [00:02:26] Speaker C: So I think accessibility comes to a lot of different things and there are a lot of different frameworks that you can think about it. Like for example, when I was at the University of Chicago in Argonne National Lab, a lot of the work was focused on making machine learning data sets and models accessible to people. And there's an acronym, it's called fair, stands for Making things Findable. Can you actually find the thing in the first place, is it accessible in the sense of can you actually then open that thing or use it? And then also there's the accessibility in terms of disability perspective of can anybody use this thing, Is it interoperable? Meaning can you use it across different systems and is it reusable, is it going to continue to be usable? And I think all of those things come into accessibility. [00:03:11] Speaker B: Great. I think that's a lot of interesting insights about accessibility. And on the other hand, when you think about AI legal help for people without lawyers, what are some of the opportunities or risks that come to your mind first? [00:03:26] Speaker C: So I think legal AI tools to me are very complicated because in the absence of any other resources, is a legal AI tool better if people can't actually get access to counsel, if they can't actually get the services that they're looking for? And I think for me it is complicated because inherently, particularly when we're talking about large language models, ChatGPT or Claude or any of those things, nothing a large language model gives you is inherently true. Right? Because that's not actually the objective function. That's not actually. The model is not built to give you true things. It's built to give you the average, most likely next word and a sequence of words. And so I think that the question of access to justice is really interesting to me here because does access to legal AI tools, and again, this very much depends on what the tools are. Also, does that actually increase access to justice if someone's potentially getting incorrect information? And so I guess I don't really have an answer there as a part. So I think it's probably a question of risk. Like how is someone using these tools? Are they using them to get definitive answers where they're like, oh great, I had this car accident. The AI tool told me I don't have to. I can't file an appeal. I guess I won't on whatever the moving violation or whatever it is, or is it more that someone is using it as a jumping off point for future information? What even are the words that I need to use that I can then go talk to, for example, state run legal services, or I can talk to a law library. And Massachusetts has law librarians that you can just reach out to. I think if it's the jumping off point, that seems super helpful. And I think that's where the community centers that you mentioned, law librarians or public services librarians in general, can really help out. But to me, I get a little concerned when it's the final answer. Like if people are starting and ending their entire journey with the AI tool. I would be a little concerned about if they're getting the right information. [00:05:20] Speaker B: Yeah, that's definitely right because I think something we talked about in class is the unauthorized practice of law, the upl. So that's definitely one of the concern. And one thing about AI I think we want to focus here is how we want to design it in a way that people can access it. So in this podcast specifically, I think we're not that concerned about giving legal advice, but we are more concerned about making it more accessible and more usable to people. What if we start by talking about the libraries? I know that you have a lot of work in the libraries and how do you think that public libraries can. If you think it can be a place where people could access the AI enabled help in any ways. [00:06:06] Speaker C: So really good question. I want to first start by saying that I have always worked in academic libraries, which is a very different cup of tea than public libraries. Our publics do so much for our communities that are like beyond their originally stated scope. So my background in libraries, I was a library assistant for about five or so years working at the circulation desk. Currently now I am at the law library at Harvard Law School. And for publics, I think for both public and academic libraries, there's a lot that we can do to assist patrons with. I want to almost just say navigating these tools. I think for a lot of people it's very confusing about what are these things Even there's a lot of language that is used by business people online, L. AI and make it seem like it's a sentient thing rather than just a statistical model, which is what it actually is. And that can be really confusing for people. And it's very personified. All of the language we have is personified. ChatGPT and Claude. The interfaces are personified. And I think just that basic kind of literacy around, hey, this thing is not sentient. It doesn't think. It's just a word generator that hopefully will give you right answers, but doesn't inherently give you right answers. I think is a really good starting place. Libraries have always been at the forefront of changing technologies. Since the beginning of libraries. It has always been librarians that help people navigate. For example, like when the first codices were bound into books, like all the way back when databases first hit the scene in the 20th century. It has always been librarians who've kind of been on the front lines of that. And so I think that whether folks have the capacity to or not these things are going to come up in our public libraries. And I think that the first step is truly just that literacy element of here's what these tools actually are and what they're actually helpful for. [00:08:00] Speaker B: Yeah, that's interesting. How do you think. What features of the library that. That make us possible? [00:08:06] Speaker C: Great question. There's truly nothing like libraries. [00:08:09] Speaker B: Honestly. [00:08:09] Speaker C: If you tried to propose even in the last 80 years the concept of a library, it would never make it in the U.S. you know, and I. Oh, you give published resources for free to people. Yes. And in a library, you have professionals who have committed their entire craft just to helping you find what you're looking for. There's truly nothing better than that. Right. And so I do think that libraries are a very unique distribution channel because you have access to a whole. Yeah. Group of people who are completely dedicated to transparency, to access, to care for you as a human being who's trying to find something. And that's unparalleled. I can't really think of no Google search is going to give you the same results that a reference librarian is going to give you. [00:08:57] Speaker B: Yeah. So the fact that there are librarians to actually help people with their questions and everything. [00:09:04] Speaker C: Yeah. And I think I would add to that. It's not just that they're there, but it's also that they have created a craft. There's this thing, for example, when you're talking to a reference librarian called the reference interview, or the librarian will, without you even really realizing it, ask you a bunch of very thoughtful, gentle questions to figure out what exactly it is that you need. And it's that skill, that very highly skilled information finding that I think is really key. [00:09:31] Speaker B: Yeah. And building onto that, the community center that I discussed in my blog post, because I feel like maybe that's also something similar to a library where people can come and ask questions. I don't know if there are. Like, in your past experiences, do you think community centers have a very similar feature? [00:09:50] Speaker C: Yeah, yeah. Yes and no. It's again, librarians are trained in a very specific way. Right. And there's a lot of honing of craft. Perhaps it's comparable to somebody who's doing an art versus a trade. Do you know what I mean? Or perhaps someone who is trained in certain ways to help you versus someone who comes up with the ways to help you. So I think community center is definitely like all the same values. Right. Transparency, access and care. Helping people find stuff. The core value that everybody should have access to information. I think the community center. I Guess a question for me would be like, what kind of training do the staff have? Are there full time there? All those things make it a little bit different, but overall, I would say very core in terms of values and purpose. Sounds very aligned. [00:10:38] Speaker B: You were working as software engineer before when you were trying to design AI tools. What are some of the important considerations? For example, the privacy issues? [00:10:49] Speaker C: That's such a great question. There's so many parts there. I'm going to start with the interface part. Right. AI has been around for a very long time, like in some form or another. We've been researching and working with AI since the 1950s. The statistical AI is more recent, and specifically large language models are even more recent. Right. The interface that you use is so important. I talked earlier about how there's a lot of business talk that kind of personifies things. I think that the interface that we typically have of a chat interface, I don't know if that's actually the best way for people to, for example, get. Try to figure out a complex legal problem. Not only because it lends an air of authority, but, but. And it lends an air of truthfulness, but it also lends that human error that kind of makes you trust an answer more than you might otherwise. Ruha Benjamin, who's a professor here, is really phenomenal. She has some really great writing about this kind of stuff and about the perceived neutrality of technology. I think actually it's really important in design. You always hear the old adage of in design, that you want everything to be seamless. It should feel nothing for the user. It should be a seamless, seamless. But there's actually been some writing recently which I really love about having seen Full Design. You actually want the user, particularly with AI tools, to know that this is a robot that does not have ground truths. You know what I mean? So that you as a human can better make decisions when you see that answer and you as a human have to decide, do I believe this or not? Does this feel accurate? And I think anything that we can do in the design to help remind our primate brains that we're like not talking to a real person, it is not objectively true. I think that's helpful. So maybe something that's not a chat interface, maybe something that has more wordings, maybe something that doesn't just write things in rows the way that ChatGPT does, but instead generates, just automatically pulls up, for example, all the forms that you might have to look for if you're going through the divorce or something like that. Right. And then it has links where you can then very easily click out to other authoritative resources that are produced by the state, produced by law firms, whatever it is. I think that interface element is actually super key in whether or not these things will be helpful. [00:13:06] Speaker B: Yes, I think that's definitely true. Because I think something that I emphasize in blog posts is that we don't want people to over trust the AIs that they're using. Maybe placing some disclaimer, for example, or maybe making it not a chat interface. I think that's a great example to use. And also if they want to access a certain website and a phone number, even for a hotline, for example, that they can do that very simply by just clicking on a certain button. [00:13:36] Speaker C: I was going to throw one more note in there about privacy actually, if that's all right. So another really interesting thing, especially with legal, is what is happening to your chats, what is happening to your queries. There have already been instances where people didn't realize that their chats with ChatGPT were admissible in court. So I think that is also a really huge thing to think about if you're doing something that requires a trial or anything like that. There's a lot of data privacy stuff. All of our interactions are going off to these third party servers. And I think there's something to be said for people having access to local models or to private, private services, but that's just not really where we're at the moment. Such is a huge consideration for anybody putting any sort of personal information into these kinds of tools. [00:14:18] Speaker B: Yeah, that's right. Privacy is definitely one of the most important consideration. And you talked about accessibility at the very beginning about how it's also important to make, for example, the people with disability and how they access the tools and how people with different language needs can understand what's going on. So what are some of the design principles here to improve accessibility? [00:14:42] Speaker C: Yeah, that's a great question. I think always you want to meet people where they're at. The good news is that we have lots and lots of guidance on this right to make a website ADA compliant is very established, which is great. Right. Some basic things for people who are using screen readers are hyperlinks formatted correctly so that the screen reader doesn't read out every single letter and number in your URL. Do you have proper use of headache for something like a legal AI tool? I think it would really depend on what the interface is, maybe for something like a legal AI tool. Actually, like a chat interface is a little Easier for folks. That's really something that would need to be studied and you'd have to actually test out with users, I think always with accessibility stuff, there's okay, what are the requirements that the ADA has set out, for example, for websites and things, things like that. And then also there's a what do my users actually need and what do my users being like for the specific product or tool that you're building? And how can I run studies or when I say studies, it doesn't have to be like something formal that you publish, but just how can I do basic user research to make sure that for the populations that I'm trying to serve, I'm serving them properly? For example, let's say that you have a prototype for a legal AI tool. Let's say that you wanted to target older folks who need assistance. Right. I would then want to have a cohort of older folks with varying disabilities, have them step through the tool, talk them through things, figure out what their experiences are, notice what they have trouble with. Just basic, what's called usability testing. How are they able to use this after I've already done all the ADA requirements stuff? Because then I can see, okay, not only have I met the general requirements, but now I can meet the specific requirements or the specific needs of the people that I'm trying to deliver this to. [00:16:27] Speaker B: Yeah, that's actually a very good idea to have people. I think it's very important in terms of product design when you are actually observing people using this tool and see if there are any obstacles they're facing while they're using. You talked a lot about the different design principles and how we can make the interface more accessible, more usable. How do you think about more reproducible so that it's consistent throughout different channels like libraries and community center. It's hard to manage all these at once. And how to. What, what is the concept of AI reproducibility and what role does it play here? [00:17:05] Speaker C: Great question. There's kind of two things at play here in general. AI reproducibility, or machine learning. Reproducibility refers to the ability to replicate the results somebody else has with whatever models they're using. And this has been a huge problem in the field for actually a very long time. The machine learning reproducibility crisis is well established to the point where it's really more of an art than a science. The field, even long before this current generative AI boom, this has been a core topic of discussion in the field because you'll have a landmark paper People will be like, wow, we received, we [00:17:42] Speaker B: got this kind of optimization. [00:17:44] Speaker C: We have so many parameters, we had data results. This classification model did all these things and then nobody can reproduce it because so much of. So much of machine learning and I is just big ol linear algebra. And that the results of these statistical processes will vary Depending on your GPUs, on the inherent randomness that are in those models. Right. Because it's statistical, so it's not deterministic. You're not going to get the exact same answer every time for a lot of these algorithms. Not all of them, but for a lot of them. So that right there is a huge problem. Right. And then now you have LLMs where you have these chat interfaces, where if I ask the same question but with slightly different words, I'll get a different answer. If I just add filler text to the beginning or end, I'll get a different answer because the prompt is longer. And factor in how these pipelines process. The prompt is. And that's a huge problem. For example, if you're seeking legal support of any kind of. And there's that part of the reproducibility question and then there's the interoperable part of, yeah, does it work across these different distribution channels in the same way? And that is actually quite a complicated thing. If you have something in the web browser to get it to work the exact same way in every single person's computer is not trivial, actually. And then likewise, if you had local first software, like something you download, same deal, operating system systems are different, versions are different. And I don't know if I fully answered your question, Grace, but basically those are two core issues. So there are all kinds of way, technically, with the technologies that we have, that we can try to preserve state and preserve the environment of something across different systems. That technology has actually also improved a lot in the last five to 10 years, which is awesome. But creating containerized environments where you can get the same results every time is a problem generally. And then you put machine learning or AI into it, and then it's doubly a problem just because of the inherent statistical nature of machine learning. And again, our GPU technology has increased a bajillion fold and our computers have increased in their operating power much more too. I can run a lot of these things on my laptop that I couldn't run on my laptop six or seven years ago. I don't know if you're familiar, this is getting pretty technical with Moore's Law, but. But basically the idea is that computing power increases significantly over time in like an exponential way, year over year, we have greater and greater capacity for compute and that dramatically changes what we can then do with technology. Like a lot of things that we're doing today with LLMs and with ChatGPT and all that kind of stuff couldn't have even happened five to 10 years ago just because the compute was not there or was so prohibitively expensive. [00:20:30] Speaker B: Yeah, I think it really adds important insights into how also technology have been developing very fast in a way that's unexpected. There are a lot of issues that might be solved using AIs or other technologies. I think it's very important to keep in mind. Maybe just like one of the takeaway is that when you're trying to design those tools, what are some of the things that can make it more accessible and more usable by users? Especially in the context of the access to justice that people without legal representation and people who are seeking legal help that can use this tools. [00:21:04] Speaker C: So on the accessibility part, actually, the compute required to do something is a huge part of accessibility, I would say. So, like in terms of, yes, our technologies are better, but not everyone's going to have a 2024 laptop, you know, and there can be this like, oh, we have infinite resources. We'll just make the software super big. Actually been like a huge problem. This is a huge problem in web development where like people in rural areas can't open basic websites because they're so heav and they require so much of your CPU to run. And so I just wanted to add that anything that I have a colleague here who calls it frugal computing, which I really appreciate. Anything that kind of like tries to conserve how much compute power you need that will increase the accessibility of any sort of tool that you build. [00:21:49] Speaker B: Oh, that's really good to know. I've never heard about it before. [00:21:52] Speaker C: It's. [00:21:52] Speaker B: Yeah. [00:21:53] Speaker C: And it's called like software bloat. Software is a huge problem right now in the industry. [00:21:58] Speaker B: Right. Yeah. It's really interesting to learn about all this. Different type of considerations. I feel like most of them I've never thought about before. So really appreciate that you're adding a lot of insights into the context. But yeah, I think we talked a lot about different things about access to justice, of being accessible and making the tools usable. And thank you so much for taking the time to speak with me today. I feel like you add a lot of the perspectives across your work at the library community and product design and really give a clear way to think about different distribution channel and also the different practical choices that make the tools succeed or fail. So thank you again for your time and for the work that you do. Thanks Grace. [00:22:44] Speaker C: This was really fun. It's not often I get to talk about all these things together, so this was really fun for me. [00:22:49] Speaker B: That is nice. I'm glad that you found it also interesting. So yeah, thank you. [00:22:55] Speaker C: Thank you too. Good luck and take care of Proof [00:22:58] Speaker A: Over Precedent is a production of the Access to Justice Lab at Harvard Law School. Views expressed in student podcasts are not necessarily those of the A J Lab. Thanks for listening. If we piqued your interest, please subscribe wherever you get your podcasts. Even better, leave us a rating or share an episode with a friend or on social media. Here's a sneak preview of what we'll bring you next week. [00:23:24] Speaker D: Does set certain limitations on ways that unions can operate. And so sometimes we like note the irony of the fact that like laws that were supposed to or thought of as empowering the purposes to empower workers that this is haven't been quite as you contested more recently, but certainly during the Obama administration we had a lot of concern about worker centers being labeled as unions forces of the law. And so the irony of these laws that were passed for the purpose of empowering workers, these worker organizations want to stay as far away from being covered by them as possible. So there's that irony.