Just in case, I am the author of the blog post. For our "AI" class, it felt like a good class to experiment with something novel.
No, we do not want to eliminate the pen and paper exam. It works well. We use it.
The oral exam is yet another tool. Not a solution for everything.
In our case, we wanted to ensure that the students who worked on the team project: (a) contributed enough to understand the project, (b) actually understood their own project and did not rely solely on an LLM. (We do allow them to use LLMs, it would be stupid not to.)
The students who did badly in the oral exam were exactly the students who we expected to do badly in the exam, even though they aced their (team) project presentations.
Could we do it in person? Sure, we could schedule personalized interviews for all the 36 students. With two instructors, it would have taken us a couple of days to go through. Not a huge deal. At 100 students and one instructor, we would have a problem doing that.
But the key reason was the following: research has shown that human interviewers are actually worse when they get tired, and that AI is actually better for conducting more standardized and more fair interviews. That result was a major reason for us to trust a final exam on a voice agent.
>We do allow them to use LLMs, it would be stupid not to.
I'm not sure why you're saying this so confidently. Using LLMs on school work is like using a forklift at the gym. You'll technically finish the task you set out to do, and it will be much easier. So why not use a forklift at the gym?
>But the key reason was the following: research has shown that human interviewers are actually worse when they get tired, and that AI is actually better for conducting more standardized and more fair interviews. That result was a major reason for us to trust a final exam on a voice agent.
I think that in an "AI class" for MBA students, the material is probably not complex enough to require much more than a Zork interpreter, but if you tried this on something in which nuance is required, that comparison would change dramatically. For something like this, which is likely going to be little more than knowledge spot checks to catch the most blatant cheaters, why not just have students do multiple choice questions at a kiosk?
I agree that I am not yet confident to use this approach for my technical classes. I am still very unhappy with any option for assessment for technical classes, but I would not trust an LLM to come up with good questions. NotebooksLM does come up with decent quizzes, but nothing super hard.
For the use of LLM in classes: I understand the reasoning, but I found LLMs to be extremely educational for parsing through dense material (eg parsing an NTSB report for an Uber self-driving crash). Prohibiting students from using LLMs would be counterproductive.
But I still want students to use LLMs responsibly, hence the oral exam.
No, we do not want to eliminate the pen and paper exam. It works well. We use it.
The oral exam is yet another tool. Not a solution for everything.
In our case, we wanted to ensure that the students who worked on the team project: (a) contributed enough to understand the project, (b) actually understood their own project and did not rely solely on an LLM. (We do allow them to use LLMs, it would be stupid not to.)
The students who did badly in the oral exam were exactly the students who we expected to do badly in the exam, even though they aced their (team) project presentations.
Could we do it in person? Sure, we could schedule personalized interviews for all the 36 students. With two instructors, it would have taken us a couple of days to go through. Not a huge deal. At 100 students and one instructor, we would have a problem doing that.
But the key reason was the following: research has shown that human interviewers are actually worse when they get tired, and that AI is actually better for conducting more standardized and more fair interviews. That result was a major reason for us to trust a final exam on a voice agent.