By Olivia Janis
To help reinforce our understanding of good journalism, Mr. Newman likes to share articles in class with us from sources like The New York Times. Recently, he shared a compelling story about schools using AI to monitor their students’ devices in order to intercept self-harm before it happens.
The article features a story about a successful intervention with the help of AI. Neosho, Missouri student Madi Cholka texted her friend she was going to overdose on anxiety pills and was then intercepted by local enforcement.
According to the article Spying on Student Devices, Schools Aim to Intercept Self-Harm Before It Happens, many schools in America now have educational technologies such as GoGuardian, Gaggle, Lightspeed, Bark and Securely that offer tools that scan what students type and look for things like intention to self-harm. With the help of AI, keywords or phrases are flagged using algorithms or human review to determine which ones are a real threat. Then students can be called out of class and screened or even visited by local enforcement in their own homes.
While a life was saved in Madi’s case, the system does require law enforcement to essentially spy on students and interfere with their lives outside of school.
The article also illustrates the issues of false alarms and AI invading one’s privacy. For example, an entire health class was flagged because they were researching suicide, and a girl was visited by the local police in the middle of the night because language from a poem she wrote years ago was flagged by the system.
After reading this article, our newspaper class wondered if Lovett has any tech system that flags “self-harm” words or phrases.
I began my investigation at IT, where I spoke to Roberto Aguilar, Lovett’s senior support analyst.
“The websites are based on our firewall system, which is Palo Alto, and they pretty much put certain websites under a category of games, malware, or different types like that,” he said after I asked about Lovett’s censorship.
As we continued to talk he revealed that Lovett has no way of flagging certain searches regarding mental health. “What we search for is VPNs and things like that, but not specific text messages or anything or chats in Google,” he said. He is unsure if in the future these types of AI monitors will be implemented at Lovett.
Greg Hamrick, Head of the IT department, explained that in the middle school they once used a product that was tied to Google that would search for references to self-harm and would flag the students. But there were issues with monitoring and figuring out which flags were accurate or simply just false alarms. For middle school, IT currently uses the software company Securely in order to see what students are doing on their laptops, but this software does not directly flag phrases or words that have to do with self-harm.
Mr. Hamrick brought up a very important question: who would be monitoring the searches that got flagged? It is very essential that there are trusted adults who are able to differentiate the false alarms from the real alerts, and be able to follow up on that situation.
So, how do students feel about the prospect of being monitored? “If a student were to search up something that’s really dangerous, I feel like it would make sense for them to try and do something that could help that student. But also a lot of students may feel like it’s a breach of privacy as well, so it’s really hard to say,” said sophomore Madison Robinson.
Senior Ryan Cain had a different perspective on the topic. He felt as though if students consented to the system it would not be a breach of privacy, but also that IT would not be able to correctly identify when someone was in need of real help or a false alarm. “I feel like they’d want to be safe and then they’d investigate every false alarm. It’d be a false alarm 95% of the time,” he said. He also was concerned with the program veering away from the original purpose–mental health– and it working towards overall surveillance of students’ laptops.
Senior Davida Ohayon explained that because her mother works at Lovett, having her mental health information shared with the school would not be something she preferred. But in general, she does believe that the program could provide a sense of safety for Lovett students.
The overall student consensus was that there were some concerns about potential false alarms and overstepping on students’ private information, but most students did think the idea of trying to keep an eye on students struggling was actually beneficial for the student body.
I wondered what Dr. Wall, who knows a lot about student struggles given her role as Director of Counseling, would think about monitoring., so I made my way down to Dr. Wall’s office for a quick chat.
When I explained the new AI system in the Times article, Dr. Wall’s initial reaction was concern for students’ privacy and confidentiality.
But she could see the potential for good as well. “The more information we have about a student’s psychological space, the more information we have about where students are, the better we’re able to help them,” she said. “Now, the advantage that Lovett has is that there’s so many eyes on students. You’ve got your faculty, you’ve got the support staff, you’ve got the counselors, you have fellow students. So I think we do a pretty good job of identifying students who are hurting and are in pain.”
Dr. Wall believes that technology can have a positive impact, and while she does not know a lot about AI programs she wouldn’t be opposed to looking into it. But she still believes that “we really need to rely on each other.”
