Changing Lecture Culture

Daniel Greene David Selassie Bertrand Schneider

USER GROUP
Freshmen taking science classes (CS, math, physics...) in college.

EXISTING WORK
There is a general trend of ACTIVE / PARTICIPATORY learning for cognitive engagement in the classroom: Clicker / Personal Response systems widely used Also some clickers that communicate student understanding to teacher, but not necessarily to other students Nothing addressing the stigma of asking a strange question in class... nothing helping students to understand each others collective knowledge or to engage the teacher in cooperation existing systems are generally expensive; no low-cost solution to this need

Current work aimed at increasing teacher feedback is primarily focused on increasing student cognitive engagement. The intent of these systems is to keep students engaged by answering questions, with a secondary benefit of giving the teacher feedback on inthe-moment student understanding. However, these systems fail to address two needs: the needs of students to feel safe and not judged for actually asking a question by raising their hand during class, and the need of the teacher for post-lecture data about student understanding.

EMPATHY OBSERVATIONS
Most of our observations come from the CS109 class (probability for computer scientists). One of our team member (Bertrand) is currently taking that class. We observed TAs office hours / lectures / interviews with students. Large lecture hall with hundreds of students. In situ, totally appropriate location and user groups. We were

not intrusive. Students in lecture were watching and taking notes or surfing the web; sometimes students ask overly-advanced questions that arent helpful for the class. Students in office hours were working, getting help from the TA, or waiting.

SYNTHESIS
STUDENT 1 There is a crucial moment when you learn a new

formula, and you have to stop listening to the professor in order to mentally compute each steps. Lack of big picture; youre just flooded with information and you need to recall as much stuff and procedures as possible. Video tapping is helpful because you can stop and do some consult other resources when needed. I love when the teacher says a joke after a complex equation, because it gives me time to conceptualize it. A huge cognitive load comes from the fact that you have to mentally replace the indices and variables in addition to knowing what they represent; writing their name in terms of the problem (concretely) helps a lot.

TA
them; at least for a moment. The insight often doesnt last. I try to make problems more familiar to students by using concrete examples, or by breaking it down in smaller problems. Its difficult for students to generalize from one example; you change a small thing and they lose it.

Students usually get it when I explain a concept to

STUDENT 2 The class is very useful but also painful. I feel like Im
really spending a lot of time on it. I need to look at the slides 3-4 times before class and again 3-4 times after class to feel like I have understood the lesson. Students have to learn to ask conceptual questions to the TAs; office hours have become sections because there is too much content in a lecture. Its easier for people who are always doing maths; for others, there is a huge learning curve.

OTHER STUDENTS
I never raise my hand. Im slow, so its difficult to quickly come up with a good question. Im okay asking questions 1 on 1, but I dont do it even in small groups. Often I have a basic conceptual question and I dont want to ask it because I should know the answer. It would be helpful to know what the rest of the class thinks. Repeated by all students!

INSIGHTS

Students generally arent comfortable giving direct performance feedback to the teacher, especially during class; they want to do it privately and anonymously to save face for themselves and the teacher. Students are more comfortable asking questions if they know that others have the same question. Students are more comfortable asking questions if they know the other students personally. Its hard to speak up during a lecture for two reasons - classroom culture, and size / anonymity.

POVs

POV 1

INSIGHT: People joke about not having time to write stuff down, but they dont ask the teacher to slow down! NEED: CS109 student needs a way to feel better about asking the teacher to slow down without pissing off other people or looking dumb. POV: I dont want to be that guy in class who is holding everyone back - I want to do something that helps me but is also good for everyone else.
POV 2

INSIGHT: Teacher doesnt realize that theyre going too fast.

NEED: Feedback from student. A way to create a classroom culture in which its okay to ask the teacher for things. POV: Im doing the best I can for my students, and I dont want any assumptions to hold us all back from creating a great learning experience.

BRAINSTORMING SOLUTION
We want to come up with a way to have students loose their inhibition towards asking the professor to slow down, and this will probably happen if they realize theyre not alone in that desire. Lets give students a way to discreetly let the rest of the class know that they feel rushed and want the lecture to slow down; clicker systems have been used for similar feedback before, so lets make one with that goal.

PROTOTYPE 1
NEED: CS109 student needs a way to feel better about asking the teacher to slow down without pissing off other people or looking dumb. IDEA: Give students clickers so they can show they want the professor to slow down; then show the dynamic results of this to all the students during the lecture, to show theyre not alone in that desire. VARIABLE: Will students ask the professor to slow down if prompted by a passive indicator?

We asked students what they thought of this feedback mechanism and concept, but did not test it in a real lecture setting. INSIGHTS Students liked and wanted to use the feedback mechanism; it was like a game. Students said they would be encouraged to give feedback, but if no one does, the professor still does not know about class confusion. If students have their phones out, will be tempted to goof off? Prevent students who leave the app from giving feedback? Is a generic indicator of confusion enough information to convey to students about their peers? What about students who dont have smart phones? Students might down-vote difficult sections that are still necessary. They are not the best judges of learning content. Feedback isnt constructive. This puts a lot of demand on the teacher to be a good performer; will this frustrate teachers who get bad feedback?

PROTOTYPE 2
NEED: CS109 teachers need to refer to previous feedback to identify complex concepts or slides that are confusing, since students are not giving the lecturer the feedback inclass. IDEA: Build an histogram of students confusion (updated over time as class goes on). VARIABLE: Will the lecturer use the immediate feedback to help with explanation? Based on insights from previous user testing, we choose to improve our previous prototype clicker system to include a way to explicitly give the professor immediate feedback. Students still are able to press a help button on their clicker or phone and a program records when during the class this button was pressed. We show the professor, in lecture, a graphic that shows student confusion as a function of time. They can see how well understood their comments just were, and it

allows them to go back after class and look at what parts of their presentation need to be clarified and improved, by looking at what timestamps had the most confusion. We tested this system in a small lecture given by one of Davids labmates. The lecturer could see a plot of when during his talk students were having the most confusion; the audience had no feedback, other than knowing when they themselves pressed help on the clicker page. Histogram of students confusion during lecture:

Line chart directly next to the slides:

INSIGHTS Audience felt guilty about using clickers: I should just raise my hand, not use this thing. Even after one testing session, we might be changing classroom culture! Attendees did not use clickers very much. A class-facing component seems to be necessary to keep faith and engagement in the system. Presenter did see chart clearly, and tried to act upon the new information. I didnt know what to re-explain. Presenter though that this would be more useful after lecture, when he would have time to re-assess the specific trouble spots. Prototypes need to be tested in environment that mimics the intimidating environment of a large intro class; we were in a smaller, optional lecture, that was very low stress. Presenter might need training as to how to deal with immediate feedback usefully. The system needs both prototypes simultaneously to be functional; lecture environment can be changed more if there is an audience-facing screen, lecture confusion can be reduced more if theres a professor-facing screen.

PROTOTYPE 3
NEED: Teachers needs to be able to identify what was confusing during their lecture (a histogram may not be enough); name of confusing concepts. IDEA: Allow students to directly type a keyword on their phone / laptop VARIABLE: Will students be willing to document confusing concepts? Can the lecturer incorporate current confusing topics into their talk in real-time?

We had previously ignored how the lecturer would react to this feedback, but now that our last prototype gave us specific feedback on that, we decided to incorporate more informative feedback into the system. We still wanted to keep the mechanism by which

the students asked for help very straightforward, so we show a list of most frequent topics on peoples phones and allow students to easily type in new topics at the bottom. The current most requested topics are shown on the presenter / class-facing screen along with the previous histogram data. We were unable to find an appropriate lecture that was willing to let us test out our clicker system before the end of the quarter; based on our previous testing, we knew that we needed to find a class that was as close as possible to the original lecture situation we performed needfinding on. Unfortunately, the larger lecture classes were unwilling to introduce things that might delay class this late in the term. We instead asked people and presenters for feedback on this updated system. INSIGHTS Presenter still was able to interpret data, and said he could incorporate it into his responses easier. Presenter said that this would be especially helpful in reviewing how a presentation went after the fact, and knowing what topics to focus on / review in later lectures. Similar to how people use hashtags during conference talks on Twitter; this shows the principle could be well received. Now that its possible to directly and anonymously communicate with the professor, are we preventing classroom culture from changing? Will people be less likely to ask questions now that students are aware the professor already knows their question?

FINAL THOUGHTS
Testing that any device truly changes classroom culture will be difficult and require a longer term study than we were able to complete. We are still hopeful that further testing and answering some of our outstanding questions on the interplay between social norms between student and teacher, and between students will let that goal be achieved. The fact that after one testing session one student already said, I should just raise my hand, not use this thing, leads us to think its possible. In a sense, wed like our project to be self-destructing; we think that any system that encourages students to ask questions and succeeds will positively reinforce the act and, hopefully, will make itself unnecessary. Thanks so much to Maryanna, Noah, and Mike for their teaching, support, insights, and snacks; you provided a great framework to take a stab at learning design thinking.