Lessons on incentives and scaling from two years as a teaching assistant
22 Mar 2018
Imagine that you’re hired to work at your local public library. As an eagle-eyed
checkout clerk, you soon realize that half the patrons leave without actually
checking out their books! This leaves everyone else scratching their heads when
the catalog doesn’t match the shelves. But conveniently, the library has an
unused anti-theft alarm sitting in the back room.
There’s just one problem: your supervisor, though sympathetic to your cause,
doesn’t want you using the alarm. You see, people take books for many reasons,
and not all of them are malicious. Besides, who cares if the shelves are in
disarray? They’ve been like that for years, but everyone who works at the
library still gets their paychecks the all the same. If you want to set up the
alarm, you’ll have to do so on your own time, in addition to your other
In many undergraduate computer science programs, this is the absurd reality we
face when trying to combat plagiarism. Everyone agrees plagiarism is wrong.
Everyone wishes they could stop it. Everyone has access to the tools that find
it. But no one seems willing to take any action.
The most important goal is to keep the course fair for students who do honest
work. Instructors must assign grades that accurately reflect performance. A
student who grapples with a problem — becoming a stronger programmer in the
process — should never receive a lower grade than one who copies and pastes.
Finally, as educators, we also hope that the accused student can learn difficult
lessons about ethical behavior in the classroom rather than the workplace.
Understanding the scope of plagiarism
I’ve been a teaching assistant in a lower-division computer science course for
the past two years. Each semester, in a class of 200 to 300 students, we
typically discover 20 to 40 blatant cases of plagiarism on homework. And because
of the nature of our process, even more cases go undetected. Here’s how it
We begin by uploading our students’ code to an online plagiarism detection
tool. The tool compares the submissions with each other and with
our massive back catalog of previous solutions, flagging pairs of similar
programs. We’ll manually review nearly 1,000 of these pairs over the course of
the semester, throwing out all but the most suspicious cases. These cases end up
representing about 100 students.
Then, we apply another filter, keeping only the cases that contain
indisputable evidence — for example, hundreds of lines copied right down to
the last whitespace error. We have virtually eliminated false positives at this
point. (In the course’s entire history, only one such example exists.)
That means we aren’t catching cases where there’s plausible deniability. And we
certainly can’t find students who transform someone else’s homework beyond
recognition. (They’ve demonstrated a better grasp of programming than your
average copy-paste-rename job, but still haven’t learned what the homework was
trying to teach.)
Because our process heavily favors precision over recall,
the 20 to 40 cases at the end represent a lower bound on plagiarism in the
course. This is corroborated by the results from Lisa Yan et al, the authors of
a new plagiarism tool named TMOSS. In a Stanford course, 43 percent of true
positives detected by TMOSS wouldn’t have been discovered by a traditional tool
similar to ours.1
The large number of students suspected of plagiarism isn’t unique to our course.
In fact, if a university or course doesn’t have comparable numbers, it most
likely reflects their (lack of) plagiarism detection, rather than the actual
rate of plagiarism.
According to a 2017 New York Times report,2 a course at UC Berkeley
found about 1 in 7 students in violation of policies on copying code. At
Stanford, one course suspected up to 20 percent of its students. And in spring
2017, Harvard’s CS50 reported nearly 60 of 600 students for cheating.
What happens after an instructor confronts a student about plagiarism?
Some admit their mistake right away. Others deny it for awhile before coming
clean. And every semester, there will be a few conversations that go something
“The TAs think you copied a few homeworks. You need to tell me what’s going
“I didn’t cheat. I don’t know what you’re talking about.”
“Look, Bob, you have three screens of code that are exactly the same as
Alice’s, except she called the parameter searchQuery and you renamed it to
info_strings in your final commit.”
“Those similarities are a coincidence that happens fairly often in
programming. Stop harassing me!” *files complaint with department*
Trying to gaslight the computer science department about computer science makes
no sense, but let’s move on.
The long tail of case resolution times. After confronting students,
conversations like the above take up the vast majority of the teaching staff’s
time. A small but vocal minority of students hope to avoid consequences by
dragging out their cases.
They send endless emails to the teaching staff, sometimes even getting parents
involved. They appeal to the relevant offices in the university, claiming that
in computer programming, independently writing hundreds of identical lines of
code is a common occurrence. To refute these claims, the teaching staff might
then be asked to provide a nontechnical explanation of the similarities,
sometimes months after the semester has ended.
Combined with the time it takes to find cases in the first place, the time sink
alone is enough to discourage many instructors from looking. But don’t worry,
because there’s more:
Lack of support from the university. When there’s enough noise,
administrators in high places start getting spooked. No one wants to end up on
the front page of the New York Times. So rather than offer their
support and influence, they do nothing. Teaching staffs are left to deal with
the issue using their own time and resources.
Becoming the bad guy. At the end of it all, students take out their anger in
course evaluations. Instructors become “terrible human beings” and worse for
daring to find out why so many programs written by different people have the
exact same bugs.
These reviews matter: the hiring process for faculty positions often requires
candidates to submit their past student evaluations. And like any other form of
anonymous online abuse, repeatedly reading personal attacks and other vitriol
about yourself as part of your job carries a large mental health cost.
(Imagine if your boss outsourced a portion of your quarterly performance review
to a panel of YouTube commenters and Twitter trolls.)
On a more personal level, it hurts to know which students are plagiarizing.
We teach because we want to help people learn. It might be a little
disappointing when, for instance, a student comes to office hours at the end of
a semester not knowing how to compile programs. But it’s far worse to find out
that they’d asked for compilation help because they were trying to submit a
Competing against inaction
It’s clear that instructors who choose to pursue plagiarism cases encounter a
wide variety of disincentives, ranging from slightly unpleasant to truly
dreadful. But on an individual level, nothing bad really happens to those who
don’t bother confronting plagiarism directly. Research-track faculty can still
publish their papers. And with fewer negative reviews, teaching-track lecturers
might even improve their chances of being hired in the future.
So it should come as no surprise that most professors avoid doing anything about
Many creative solutions
I attended a discussion on plagiarism at this year’s
SIGCSE, a computer science education conference. It opened my
eyes to all the ideas faculty have to avoid facing plagiarism directly.
Some of them simply don’t generalize, like creating new homework assignments
from scratch each semester. People who bring this up have likely never asked an
instructor of an introductory course whether they’d like to spend large chunks
of time rushing out assignments in exchange for also reducing their quality. And
it still wouldn’t address plagiarism among students in the same semester.
But what really bothered me were all the moralistic solutions.
Making students sign an honor pledge. Offering leniency to those who turn
themselves in during the 72-hour “regret period” after the deadline. Or even
inviting students to email the instructor at any time if they are about to
cheat, so that the instructor can talk them out of it one-on-one.
These are great ideas — for supporting students in courses with large
enrollments. But they cannot be the only solution to combating plagiarism,
because they do nothing to address the underlying incentive structure:
Benefit: Copying and pasting someone else’s code saves a ton of
development time, and ensures a higher grade. Good grades might be tied to
desirable outcomes like graduation, financial aid, and job prospects.
Cost: Zero! If the teaching staff doesn’t look for plagiarism directly,
students know they’ll never be caught.
Empirical evidence shows that this is true. Our students sign an honor pledge at
the beginning of the semester, but we did not see any change in the rate of
plagiarism. Even for Harvard’s CS50, the birthplace of the “regret period,”
David Malan reports that it “has not materially impacted CS50’s number of
Finding our way out
Student incentives come from faculty
Student incentives are set by the faculty: for example, if the instructor
assigns a larger weight to some assignment, students usually spend more time
working on it. This means instructors have a lot of influence here! They can
combat plagiarism by making it costly.
If students knew that there would be a consistent, non-zero cost to
plagiarizing, many of them wouldn’t do it. Copying homework would no longer be
in their best interest. To have an effect, the policy would have to be
implemented consistently across all courses, beginning with the introductory
courses, which play the role of communicating the department’s standards to new
Achieving this level of uniformity means we need to provide a strong incentive
every instructor — not just the ones who care the most — to enforce the
Fixing faculty incentives
University administrators should communicate their support. Instructors
should know that, not only will they suffer no retaliation, but that the
university encourages them to enforce university policies. This might require
administrators to acknowledge the inconvenient truth of widespread plagiarism.
Next, we need to reduce the cost of looking for plagiarism.
Efficiently deploying plagiarism detection software. Most instructors I’ve
spoken with use MOSS (if they use automated detection at all).
However, each teaching staff must learn to use the software on their own,
sometimes writing additional code to format the inputs or aggregate the results.
We’ve somehow managed to turn plagiarism software — which should be written
once and deployed widely at zero marginal cost — into something that is very
expensive to use!
Instead of deploying the software on a per-class basis, the university should
pay to integrate it into their learning management systems. The software could
even automatically run on all programming assignments by default.
Spreading the workload among more TAs. Universities can further decrease the
burden by increasing the TA headcounts of courses that enforce plagiarism rules.
This ensures that time spent combing through the results of the software doesn’t
come at the cost of teaching.
Improving the quality of results from plagiarism detection software. The
software returns more false positives as class sizes increase, since the number
of pairs of students grows quadratically. But because MOSS is proprietary, the
community’s open-source development revolves around creating tools that wrap
MOSS, rather than improving the core algorithm. Anyone who has an idea for
improving the algorithm must first reimplement all existing functionality
starting from the pseudocode in the paper.4 There is an opportunity
to have a big impact by creating a free and open source alternative.
We won’t be able to fix everything. For example, personal attacks in course
evaluations may be interleaved with genuine criticisms, making them difficult
for moderators to separate.
I’m not smart enough to have all the answers. These are just some of my ideas
for moving incentives in the right direction. It’s not even clear which ideas
will work. But one thing’s for certain: students will learn to care when their
teachers start caring.
Thanks to Nelson Gomez, Joshua Zweig, John Hui, Vivian Shen,
Edward Wang, and two anonymous readers for their feedback and discussion.