The CITRIS principal investigator and professor of computer science made a proctored, computer–based testing facility, so students gain flexibility and teachers don’t need to plan massive exams.
Scheduling exams can be a major feat for large universities. With thousands of students and few testing locations that can accommodate sizable intro–level classes, the registrar’s office often assigns examinations to unfamiliar parts of campus, at inconvenient times. (Picture a first–year English major wandering a science building at 7 p.m. on a cool December night, searching for the only auditorium large enough to hold the exam for a required course.)
Students aren’t the only ones affected: instructors must create new and unique test material for each semester or quarter to reduce opportunities for cheating, and once exams are completed, they’re required to submit grades within a matter of days. These pressures can lead them to rely less on exams altogether, making it difficult for students to prove true mastery of the course.
At the same time, faculty concerns about students using AI to do homework–without really learning the material–increases the relevance of exams as a way to determine what students actually know.
That’s where computer–based testing facilities, or CBTFs, come in. Armando Fox, a principal investigator at the Center for Information Technology Research in the Interest of Society (CITRIS) and a professor of computer science at the University of California, Berkeley, is working alongside a host of collaborators to refine and expand a CBTF on the UC Berkeley campus.
At the UC Berkeley CBTF, students can self–schedule computer–based exams to be administered during the proctored facility’s open hours, providing flexibility for students juggling multiple commitments. And, thanks to software that pulls from a pre–written bank of problems, each student’s test is unique, saving faculty members from the need to organize exams and rewrite questions.
“If you don’t get it perfect the first time, you have the opportunity to retake [with CBTFs], because exams are no longer painful to administer,” Fox said. “At the end of the day, what we care about is that you learn it, not that you learn it by a certain date.”
One of the pilot locations for the UC Berkeley CBTF was nestled within Sutardja Dai Hall, CITRIS’s headquarters and home base.
“We’re very lucky to work with CITRIS,” Fox said. “They’ve been very collaborative in working with us to use that space.”
Fox credits CITRIS Director Alexandre Bayen with helping make the pilot rollout possible by providing space for one of the facility’s first locations in Sutardja Dai Hall, the institute’s headquarters. Bayen, who also served as UC Berkeley’s founding associate provost for the Berkeley Space Center and the Liao–Cho Professor of Engineering, said the project exemplifies CITRIS’s mission of bringing together researchers and campus partners to address practical challenges through innovation.
“The CBTF is a great example of what happens when researchers, instructors and campus partners come together to solve a real–world challenge,” Bayen said. “By making assessments more flexible, scalable and effective, we’re improving the student experience while giving faculty new tools to support learning. We’re proud that CITRIS could provide a home for this effort and excited to see it expand across the UC system.”
In under four years, the CBTF has served around 6,000 students in about 20 courses at UC Berkeley. Thanks to its success, the University of California has created a new task force to find ways to implement the facilities at other UCs.
Real–world tech solutions
Fox, an award–winning teacher, mentor and researcher at the intersection of digital learning, programming systems and software engineering, began his career at Berkeley over two decades ago.
Fox got his bachelor’s degree from the Massachusetts Institute of Technology (MIT) and received his master’s degree from the University of Illinois Urbana–Champaign, both in electrical engineering, before coming to Berkeley for his Ph.D. in computer science. He admired Berkeley’s commitment to making information accessible, referencing Berkeley Unix, an open–source operating system released in 1978 that includes among its descendants Apple’s macOS and iOS, some of Windows NT’s networking utilities, and the PlayStation 5’s system software.
Fox says that sharing that information contributes to the betterment of the world, a sentiment that is “in Berkeley’s institutional DNA.”
“There’s plenty of real problems out there that need solving,” Fox said. “Software and technology are really, really powerful tools that we can use to solve those problems.”
CBTFs were pioneered by one of Fox’s alma maters, the University of Illinois at Urbana–Champaign, in 2014. Five years later, the university reported a reduced number of failing grades on exams, an increased number of As, and less grading and planning effort for instructors.
By allowing students to retake tests, they can better achieve mastery, says Oliver O’Reilly, UC Berkeley’s vice provost for undergraduate education.
“People learn by practicing and failing,” O’Reilly said. “The ability to retake exams allows students the ability to learn.”
From CITRIS admirer to collaborator
Fox first learned about CITRIS just before the institute’s 2001 launch from interim director and UC Berkeley electrical engineering professor Randy Katz.
“Randy was very excited about it. He was telling us about this new multicampus center called CITRIS,” Fox said. He became enthusiastic as well, as CITRIS’s focus on developing technology for the good of society aligns strongly with his greater goal as an academic — to collaboratively share information which promotes societal progress. “This is kind of the reason we do computer science.”
Fox and CITRIS’s paths intertwined more formally in 2016, when he received a CITRIS Seed Award to explore the use of machine learning for feedback on coding work. The processes they explored were a predecessor to the AI coding assistants that have exploded in popularity in the last three years.
During the project, Fox and seven students analyzed code using techniques borrowed from programming systems and programming languages to find program errors. They used that information to develop techniques to help students learn good coding style. Fox’s experiments demonstrated the success of the approach.
“Developers who don’t know how to use AI productively are going to be replaced by developers who do,” Fox said. “This is a new tool in the arsenal. It’s amazingly powerful. We need to teach students how to use it to make themselves even better at what they do.”
Similar AI and machine learning techniques may also help Fox supercharge the CBTF. Although AI is still in early stages, he and other instructors are experimenting with using it to create question variants in the CBTF’s question generation system, PrairieLearn, so no two tests are the same. Some are also using AI to help grade open–ended questions.
The team will continue to expand the CBTF’s capacity and capabilities as it moves from Sutardja Dai Hall to its new location in Moffitt Library, which will be triple the size of the last, allowing up to 100 students to test at a time.
“CITRIS has been wonderful in allowing us to use Sutardja Dai Hall for two years for the testing facility,” O’Reilly said. “It’s a phenomenal location. Because of the success of that location, the campus is supporting the creation of an even bigger facility.”
Translating passion to teaching
Outside the classroom, Fox finds refuge in the arts. Since toddlerhood, he’s played the piano. He even has a photo of him sitting on his mom’s lap, laying down a melody.
“I’m sort of like slamming the piano keys, and I have this big grin on my baby face,” Fox said.
In high school, Fox pursued a preparatory program certificate from Mannes College of Music. He never intended for music to be a career path and was fueled solely by passion.
“I’ve always liked having the arts as a refuge. Some people love reading, some people love cooking, some people love the outdoors. I love music and the performing arts,” Fox said. “It was pretty intense. But when I love something, I really love it.”
Years later, Fox still draws on an unexpected lesson from his musical training: the value of pursuing knowledge for its own sake. He encourages his students to develop a deep understanding of the subjects that genuinely fascinate them, trusting that expertise will prove valuable in ways they can’t yet predict.
“You will realize that you’re in a position to bring something more to a project or contribute more than you thought because you have this deep knowledge,” Fox said. “Because you went out of your way just because you’re interested in it. Just because you wanted to know it.”
For Fox, one of the greatest rewards of teaching is watching his students surpass him. That philosophy inspired A’s for All (“As Time and Interest Allow”), an initiative he developed with longtime collaborator Dan Garcia, a professor in UC Berkeley’s Department of Electrical Engineering and Computer Sciences (EECS). Like the CBTF, the effort is designed to give students more opportunities to learn through practice, demonstrate mastery and retake assessments.
Fox and Garcia first met as students in an MIT dormitory nearly 40 years ago. Their careers have remained closely intertwined ever since, collaborating on projects aimed at improving how computer science is taught.
“He’s one of the crown jewels of Berkeley,” said Garcia. “He’s remarkable.”
Garcia collaborates with Fox and O’Reilly on CBTF advocacy. Garcia is a proponent of CBTF policies and Fox advocates for the facilities and infrastructure. O’Reilly views the project as very important to undergraduate education and his team completed a 2023 survey asking faculty about their priorities, high on the list was proctoring.
“We work very well together. I need him, he needs me,” Garcia said about working with Fox. One of his classes, CS10, was in Sutardja Dai Hall — the same classroom to be converted into the CBTF.
“I’ve just been so delighted that CITRIS is in our backyard,” Garcia said. “I’m a fan of CITRIS and super happy that they’re part of our family.”
CBTFs in action
Garcia’s class has used the facility before, and he says he is “dipping his toe in.” He likes the facility because it makes exams secure. CBTF exams cannot be accessed from outside the lab. The question generator draws from a pool of random instructor–written questions, and students have to speak with a teacher’s assistant to see their results. Garcia notes it’s particularly helpful for students who require accommodations from the Disabled Students Program (DSP), such as reduced distractions and extra exam time.
“I care deeply about DSP students,” Garcia said. “So in that way, I’m supporting the students who have different accommodations because these software and hardware facilities support their success.”
Fox also highlights the benefits of CBTFs for instructors.
“The only reason we have three–hour midterms is because scheduling exams is such a challenge that we don’t do it very often,” Fox said.
By removing the scheduling barrier, instructors are more keen on allowing students to retake quizzes and tests. Additionally, teachers who assign take–home essays have demonstrated an interest in the facility. With generative AI, many are worried about legitimacy in writing, and the CBTF can help ensure that a student’s work is original.
“The convenience for students is really just extraordinary,” O’Reilly said. “And the support it provides for instructors is also something that’s very noteworthy.”
O’Reilly hopes the new facility will be the first of many across the UC Berkeley campus, and even beyond. Throughout the UC system, the demand for CBTFs is growing.
“Just imagine if we could offer it to every student in every course at Cal. That’s our moon shot,” Fox said.