Clarkson University Professors Publish Study on AI as a Writing Coach in First-Year Seminar
A team of Clarkson University professors recently published a study examining student confidence in their writing abilities after using ChatGPT as a writing coach. The study explores how artificial intelligence (AI) tools can assist students in the revision process while encouraging critical engagement rather than passive reliance.
“The big-picture hope is that this serves as a model for how instructors can find ways to coexist with AI writing tools instead of fearing them and banning them outright,” said Associate Professor of Sociology Matthew Manierre, a co-author of the study. “Similarly, students need to use AI tools critically and work on their prompting and evaluating abilities. Not everything these tools provide is useful, and they often benefit from scaffolding to show how to get the most value out.”
The research team is made up of Clarkson Arts, Culture, and Technology faculty, including Manierre, Associate Professor of Literature Lisa Propst, Associate Professor of Political Science and Library Director Alexander Cohen, and Associate Professor of Sociology JoAnn Rogers, who designed an exercise for the university’s first-year writing seminar, UNIV190. Students used ChatGPT to receive feedback on early drafts of their writing, alongside instructor feedback. The goal was to see if students felt more confident in their writing abilities after the exercise, as well as how useful they thought the exercise was.
Manierre said the group hoped that the project would help improve students' writing efficacy over time. While students found ChatGPT helpful, they did not see changes in writing efficacy.
“Students’ confidence in their writing abilities didn’t change at all, nor did their comfort in completing a number of tasks,” he explained. “Yet, students did report that all the tasks were at least somewhat helpful, and that the experience was largely beneficial to their writing process.”
In part, Manierre believes that students did not feel more confident in their writing because the first-year writing seminar is a difficult course, and student confidence starts at an inflated level.
“They start high, and then get knocked down a peg as the semester progresses,” he explained. “Even if they are becoming better writers, their efficacy might actually have dipped as they recalibrated.”
The study also revealed that students were critical of AI writing, something that came as a surprise to researchers. Students reported frustration with the AI rewriting some of their work, even when expressly forbidden to do so in the students’ instructions.
In the end, the vast majority of student participants in the study felt that their professor’s feedback was more valuable than the feedback they received from AI.
While he believes there is much more research to be done on the topic, Manierre believes there are some helpful takeaways for professors looking to coexist with AI moving forward.
“This project is part of a broader process of toying with ways to integrate AI into other classes for me,” Manierre said. “Introducing AI to students needs to include demonstrating its pitfalls. These things are constantly trying to get one past you on everything, so be on your guard.”
To read the full study, click here.
