SIGCSE 2018 notes

Over the weekend, I attended SIGCSE — the ACM’s conference on computer science education — with the teaching staff of Columbia’s Advanced Programming course. We learned about everything from rubric design to creating community in large classes to catching plagiarism at scale. These are the notes from the more interesting sessions I attended!

Contents

• Thursday, February 22
  • TMOSS: Using Intermediate Assignment Work to Understand Excessive Collaboration in Large Classes
  • Lightweight Techniques to Support Students in Large Classes
  • PVC: Visualizing Memory Space on Web Browsers for C Novices
  • Special Session: Watch them Teach
    • Abstraction
  • We Should Give Messy Problems and Make Students Reflect on What They Learn
  • Teaching Students a Systematic Approach to Debugging
  • Providing Meaningful Feedback for Autograding of Programming Assignments
  • An Explicit Strategy to Scaffold Novice Program Tracing
  • Discussion: Combating the Wide Web of Plagiarism
• Friday, February 23
  • How am I Going to Grade All These Assignments? Thinking About Rubrics in the Large
  • BlueBook: A Computerized Replacement for Paper Tests in Computer Science
  • Using a computer-based testing facility to improve student learning in a programming languages and compilers course

Thursday, February 22

TMOSS: Using Intermediate Assignment Work to Understand Excessive Collaboration in Large Classes

Lisa Yan, Nick McKeown, Mehran Sahami, Chris Piech

• “Excessive collaboration” is cheating, such as copying too much from online solutions or classmates
• Cheating detection is hard because of different patterns
  • Submitting an online solution
  • Viewing online solution, then modifying it
  • Viewing online solution, then working independently
• Intermediate code snapshots captured through Eclipse plugin at every compilation
• TMOSS comparison algorithm
  • Moss does \( O(N^2) \) pairwise comparison of tokenized programs
  • At average \( M = 250 \) snapshots per student, \( O(N^2 M^2) \) is too much
  • For each student, compare each snapshot against all final submissions: \( O(N^2 M) \)
  • Plot similarity score over “time” (aka snapshot index)
  • In TMOSS, the distribution of scores overlaps much less than in normal MOSS, showing it’s better at distinguishing cheaters
• Applications of TMOSS data
  • Students who cheat also start homework later, do worse on exams
  • Online matches start even later than classmate matches
  • TMOSS-only students (those who modified the code to fool MOSS) are 43 percent of students!
• TMOSS on GitHub

Lightweight Techniques to Support Students in Large Classes

Mia Minnes, Christine Alvarado, Leo Porter

• Huge class sizes
  • Enrollment is growing but we can’t hire faculty to match
  • Students feel anonymous, perhaps more for URMs
• Their previous work
  • Enhance discussion sections as “micro-classes” with the 1 TA, 2-3 tutors, who built a relationship over the entire semester
  • Students felt more included, but academic performance didn’t change!
  • Very expensive so couldn’t justify poor ROI
• This work: lightweight (cheap or free) first order approximations
• From “micro-classes,” students liked having a smaller community
  • Students liked familiar faces in assigned zone seating, about 50 per zone
  • Assigned specific seating led to backlash
    • Zones give some choice
    • Can also say “next time, your seat choice is your zone for the rest of the term”
  • Create physical space between zones, using walkways or barriers
  • Took some convincing, for example asking students to move away from the empty area, reminding people about zones
• Personal tutors (not TAs)
  • In class: tutor stays in the same part of the classroom
    • Listen to students whispering in class to ask questions or provoke a discussion if students are confused
    • Continuity from tutors coming to class helps build connection
    • “Remember when we talked about XYZ in class?”
  • Out of class: personalized congrats emails throughout the term
  • Students reported a more welcoming environment
  • Some tutors offered 15-minute 1:1 sessions and cut back on other responsibilities
  • Tutors have “their own” students
• TA said it wasn’t much more work and helped get to know students and class dynamics better

PVC: Visualizing Memory Space on Web Browsers for C Novices

Ryosuke Ishizue, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa

• Why visualization?
  • Hard for newcomers to use debugging tools like gdb because they’re text-only
  • Existing tools: Python Tutor, SeeC
  • When a program changes, SeeC requires a compile and execute cycle to update the visualization
• PlayVisualizerC (PVC)
  • Web-based interpreter that doesn’t support full C syntax
    • Interpreter is an interesting trade-off — they don’t need to modify C compiler, but also don’t get all potentially helpful diagnostics for free
    • I wonder if this would be easier in clang
  • Interpreter runs on server and sends program state to client

Special Session: Watch them Teach

Abstraction

Mehran Sahami

• Context: CS1, written small programs using Karel the Robot, but only written methods with no parameters or return value
• Abstraction: “Civilization advances by extending the number of operations we can perform without thinking about them.” —Alfred Whitehead
  • Use the sqrt() function without remembering how it works
  • Toaster analogy: put in bread, get out toast
  • One toaster works by heating coils, but if it worked by magic, we wouldn’t use it any differently
• Reuse
  • CD player does not require us to change the electronics to play a new song
  • If you buy an iPod, can reuse the same headphones rather than replacing whole system
  • Then show the code
• Teaching abstraction in a concrete way

We Should Give Messy Problems and Make Students Reflect on What They Learn

Paul Dickson

• We give students large projects because we want them to learn by figuring it out
• People learn a lot from reflection, but everyone hates it
  • Only done once
  • Not a big part of the grade

Teaching Students a Systematic Approach to Debugging

Roman Lysecky, Frank Vahid

• Students encounter problems and try to fix them by changing random stuff
  • Instructors try to teach how to use a debugger, but the problem isn’t debugging
  • Students should learn a systematic approach to troubleshooting
• Teach troubleshooting with everyday things — no code
  • “My lights do not turn on”
  • Create a hypothesis about the cause
  • Design a test to validate or reject the hypothesis
• Then move to programming
  • Experiments include reading the code, executing the code, etc
  • Web environment for writing and testing
• Book: Troubleshooting Basics

Providing Meaningful Feedback for Autograding of Programming Assignments

Georgiana Haldeman, Andrew Tjang, Monica Babes-Vroman, Stephen Bartos, Jay Shah, Danielle Yucht, Thu Nguyen

• Instructors use automatic grading to deal with CS enrollment growth
  • Students code for a test harness
  • Instructors write test cases, which the student perceives as feedback
• Impact of feedback: past research
  • “Binary instant feedback,” like unit tests, leads to trial-and-error programming and more cheating
  • Students and instructors both benefit from “directed feedback”
• Concepts and Skills Based Feedback Generation Framework (CSF^2)
  • Cluster student submissions based on unit test pass/fail vectors
  • Manually inspect the programs in each cluster and write a hint
    • Mapping of test vector to hint is known as knowledge map
  • During the next term, classify the pass/fail vector of the test cases to decide which hint to display to the student
  • Classification accuracy around 90%

An Explicit Strategy to Scaffold Novice Program Tracing

Benjamin Xie, Greg L. Nelson, Andrew J. Ko

• Programming instruction lacks strategy to apply learned knowledge
  • Their approach: teach people to trace code line by line
• Why do people struggle to trace?
  • Inaccurate knowledge of programming language semantics
  • Don’t use external representations of program state (hold all the variables in your head)
  • Weak problem solving strategies
  • Not familiar enough to put all the above pieces together
• Previous work
  • Visualization tools: requires learning another piece of software
  • “Sketching,” or writing down state of variables: students don’t use this on their own even after seeing it during lecture
• Their explicit instruction approach
  • Give explicit steps of how to solve the problem using line-by-line tracing
    • Don’t depend on students coming up with these strategies ad-hoc
    • Student: having a strategy “forces you not to skip around”
  • Use memory table, which is sketching with a printed template (columns for variable name and value history)
    • Not going to magically fix poor knowledge of programming semantics
  • Those who learned the strategy did better on practice exam questions, but didn’t better on midterm by a significant amount
• Possible extensions
  • Grade the memory table on an exam? (To provide partial credit?)
  • Online tool to fill in the memory table?

Discussion: Combating the Wide Web of Plagiarism

• Solutions may appear on websites outside the purview of US copyright law
• Tutors who give away the answers, maybe because they’re trying to save time on the student
• Is cheating a problem to be solved, or a fact of life?
• Exams in a computer lab with real compilers and documentation (Google cache), which makes the homework exactly like the exam
• Grading system where you cannot pass the course without having a passing grade in the exam category
• Intro students believe there is only one solution to programming problem, so have them do a peer review after turning in the homework
• Students who design their own project seem to cheat less
  • Maybe harder to find a solution after writing a unique problem

Friday, February 23

How am I Going to Grade All These Assignments? Thinking About Rubrics in the Large

• College Board has experience grading massive Advanced Placement CS exams consistently
  • Can apply this reading experience in the classroom
• Question development
  • Development meeting is 3 days long!
  • AP exam questions give away a lot of information because students can’t ask questions of the proctors
  • This seems like the easy part
• Rubric development
  • Decide what skills you are testing and want to reward
  • Write a canonical solution
  • Give points for each part of the algorithm that the student writes
  • Write general scoring guidelines to discourage shotgunning and cover cases outside the rubric
    • Extraneous code with side effects
    • Not declaring local variables
  • Don’t start grading — test the rubric on a sample of student work, focusing on solutions that test your edge cases
    • Simple oversights: forgetting a semicolon, incorrect syntax, misspelling
    • Incorrect execution
    • Correct solution but different approach, which may or may not violate the assignment spec
    • Student wrote almost nothing but demonstrated some knowledge
  • Write down how to handle those edge cases in the rubric
• Applying the rubric
  • Training to ensure consistency among graders, and over time (first/last exam graded)
  • Grade each individual point on the rubric, not looking at the whole thing and making up a number
  • Once a student loses points for something, continue evaluating the rest of the code as if they’d gotten that part
  • Review samples with graders first
  • Grade in pairs so people can ask questions
  • Start with grading each solution twice, then only spot check
    • Positive feedback after double checking is important but often overlooked

BlueBook: A Computerized Replacement for Paper Tests in Computer Science

Chris Piech, Chris Gregg

• Qualitatively, there are students who understand the homework but do poorly on handwritten exams
  • Writing solutions from top to bottom, rather than interatively
  • “Strange transfer task” to perform well on exams
• BlueBook app user experience
  • Takes over screen to prevent looking at notes
    • Detects when you switch to another window
    • No defense against VMs — relies on trusting that students won’t be “truly adversarial”
  • Enter password to decrypt exam
  • Editor with syntax highlighting
    • Periodic snapshots submitted to server in case of hardware failure
    • Doesn’t compile code: found that students would spend too much time trying to get it perfect
  • Problem spec in HTML, CSS, JS
    • Can include interactive examples and even a demo solution
  • Submit electronically and no way to go back
• Potential extensions
  • Automatic grading
    • 98% of student code doesn’t compile: TAs fix the syntax errors
    • Could let students compile but not run
    • Automatic syntax correction
  • Better security
    • Nothing prevents students from sending exam to a friend offsite
    • Laptops make it easier to look at screens in front of you
  • Equation editor, drawing diagrams

Using a computer-based testing facility to improve student learning in a programming languages and compilers course

Terence Nip, Elsa Gunter, Geoffrey Herman, Jason Morphew, Matthew West

• Prairie Learn: similar platform as BlueBook
• Testing model
  • Students schedule exam appointment at computer lab over multi-day exam period
    • Take the exam at any time
    • Generic proctor, since students from multiple classes can take the exam at the same time
    • Allows preventing internet access more effectively, unlike BYO hardware
  • Numbers in exam questions are randomized
  • Grades go down over the exam period, showing that questions may not be leaking
    • Selection bias issue: students who need to cheat won’t come on the first few days
• Results
  • Instructor give more exams (quizzes) because they’re cheaper now
    • Achieves scale because the exams/automatic graders becomes a fixed cost for the instructor
  • Grades have gone up at the low end