Skip to main content

Live E-Assessment via ARS

How to use voting systems in face-to-face courses?

With the help of so-called Audience Response Systems (ARS, also Classroom Response System - CRS), knowledge questions can be asked, information can be collected, lecture hall experiments can be carried out and feedback, e.g. for the evaluation of the event, can be obtained in a very simple way during a course. The results of these live assessments can be presented immediately via beamer and thus influence the further teaching process.

The goals and the application spectrum of audience response systems are manifold:

 

  • Interaction in face-to-face events, even with very large groups of participants
  • Maintaining and increasing attention
  • Activation of students to reflect on the subject matter they have just heard (e.g. via peer instruction)
  • Adjusting the lecture in response to participant feedback:
    • Identifying gaps in knowledge with the opportunity to immediately recap the content in question
    • Collection and agregation of comprehension questions, which are then answered in bundles
    • Feedback on lecture speed or blackboard image
  • Collection of anonymized participant data (e.g. opinion polls, lecture hall experiments)
  • Live peer review, e.g. criteria-guided, anonymous evaluation of student seminar contributions
  • Combination of questionnaire-based and dialogical course evaluation

Example of a peer instruction scenario (Source: Universität Paderborn)

 

  1. Presentation of learning content and task
  2. Recording and solving the task
  3. Evaluation of the answers and initiation of the next phase
  4. Procedure adapted to the result
    1. Answers predominantly correct → Explanation of remaining ambiguities and continuation of the event
    2. Answers predominantly incorrect  → Adjusted repetition of presentation and retuning
    3. Answers diffuse → Initiation of peer discussion: time-limited discussion of own answer with someone who chose a different answer ("Convince your neighbor of your answer") and re-vote

Alternative 1: Clicker

Clickers are small handheld transmitters that are handed out to students. Compared to online systems, they have the advantage of working in a closed environment (local database on the instructor's computer) with guaranteed performance and very reliable. The ZMML provides a set of 40 clickers (eInstruction CPS Pulse), a netbook with pre-installed voting software (eInstruction Response including PowerPoint integration) and USB receiver. Questions of different types (single choice, multiple choice, yes/no questions, number entry, short text) can be created in advance in the PowerPoint presentation and activated by calling up the slide. The voting time can be predefined, but also extended, shortened or stopped at any time. Spontaneous voting (impromptu questions) is possible at any time. If desired, the results can be displayed on the PowerPoint slide directly after the end of the voting.

Alternative 2: online CRS-Systeme

Compared to the Clicker solution, online solutions such as PINGO (Peer Instruction for very large groups) or the ClickR plugin in Stud.IP allow higher numbers of participants and a purely web-based preparation and evaluation. The aforementioned, freely available online systems lack PowerPoint integration; however, the results shown in a pop-up window of the browser or a special app can simply be placed in front of the current screen view (slide, PDF, etc.). The circle of participants is limited to owners of mobile devices (smartphones, tablets, notebooks, etc.) with WLAN connection or Internet flat rate.
 

  • Advice on didactic design and technical implementation
  • Loan of 40 clickers (CPS Pulse) including netbook with preinstalled voting software (eInstruction Response) and USB receiver

Examples from the summer semester 2015 (results of guideline-based interviews in September 2015):

 

Prof. Dr. Manfred Fahle (Department of Biology/Chemistry, Biology major):
 Professor Fahle has been using the hardware clickers awarded by the ZMML in two of his 90-minute lectures in the bachelor's program in biology since 2010: "Animal Physiology and Human Biology 2 (NHZ2)" with about 120 students and "Basic Principles of Neurophysiology and Neuroanatomy (PM1-6)" with about 30 students. Since only 40 units are available, small groups of 2 to 3 students each are formed in NHZ2 to answer. During the course of the lecture, approximately 15-minute teaching sessions are followed by one question each (2 to 3 minutes per clicker session). The total time required for ARS use is about 15 minutes It is predominantly single-choice questions of the type "Which of the following 5 statements is false?" The questions represent the primary learning objectives of the course and are similar to the questions on the e-exam. After time elapses, the voting result appears automatically. If the answers are predominantly correct, a brief explanation is given as to why the statement in question was incorrect and then the lecture continues. If the answers are only 50% or less correct, the material is repeated in approx. 2 to 3 minutes in an adapted, concentrated form. Technically, there have been no problems so far and no devices have been stolen. The time required to integrate the clicker questions into the PowerPoint presentation is low. The unrest in the lecture hall, which arises during the discussion phases at NHZ2, ends immediately after the clicker session is over and the result is presented. On average, despite non-trivial questions, over 70% of the answers were correct. The attention in the lecture could be kept throughout not only thanks to the clicker use, but also many illustrative case studies. Thought experiments such as "What would it mean to you if your excretory system did not reabsorb 99% of your primary urine?" help maintain attention. There was no detectable attrition effect at the end of the semester, and participation in voting was consistently high at about 90%. The need for group voting in NHZ2 proved to be beneficial, as reflection on the content was reinforced through discussion among students. In the teaching evaluation, the use of clickers was rated on a scale from 1 (very positive) to 5 (very negative) with an average of 1.4. One student commented: "With the help of the clicker questions, it was easy to assess whether you had understood the subject matter. In addition, you were "woken up" if your thoughts wandered. Otherwise, the lecture material was well absorbed through nice anecdotes and a relaxed style." A good prerequisite for the use of ARS was the reduction of the amount of material by approx. 25% in favor of a lecture atmosphere conducive to learning, which was achieved in the course of many years of teaching.

 

Prof. Dr. Martin Möhrle (Department 7: Economics):
 The 90-minute lecture "Marketing" is offered in the summer semester for students of business administration in the 2nd semester and all other business-related courses of study both in attendance and as a mobile lecture (mlecture.uni-bremen.de). The number of participants was 900 in 2014 and over 1000 in 2015. 700 students were present at the beginning and 200 at the end of each semester. In both semesters, PINGO was used to ask a few knowledge or assessment questions after about 45 minutes, which related to the content of the first half of the lecture. This timing was chosen because, as expected, student attention drops then and restlessness increases in the lecture hall. The multiple-choice questions were designed in such a way that the correct answer was not obvious, but rather the solution triggered an "aha" effect in the students. Example: "The relevant market for gummy bears includes ..." with the answer options fruit gums, chocolate products, play shapes for the sandbox and pacifiers (multiple choice possible). As expected, almost 70% of the students chose fruit gums, but all four answers were correct, depending on the underlying concept. Despite the large audience, there were neither technical nor organizational problems; even with more than 200 participants, the WLAN capacity was sufficient. The time required for the ARS deployment was only 3-5min. The goal of recapturing the attention of the students in this way was successful at the beginning, the participation rate was about 30 to 50%. However, the desired effect decreased significantly over the course of the semester, so that in the summer semester 2015 not every lecture was enriched with ARS questions. Prof. Möhrle comments: "In large events, ARS are like spices: They are useful if you use them sparingly."
 
 Tobias Tkaczick (Department of Social Sciences, Geography):
 Tobias Tkaczick is a lecturer in Department 8 (Geography program) and has regularly used PINGO for the 90-minute Introduction to Cartography lecture since winter semester 2012/2013. The 100 to 160 participants are full majors for the B.Sc. and the B.A. in the 1st semester, but also student teachers and complementary majors in the 3rd semester. At the beginning of the lecture, two review questions serve to recall the material of the last lecture and to find an introduction to the new topic. In the further course, each 20-minute teaching unit ends with a PINGO session in which the students can check whether they have processed the previously heard material by means of application and comprehension questions. If, for example, cartographic information systems were discussed separately in the teaching unit, they are to be identified in the question round in a complex map via a multiple choice task. Discussions in the approx. 1-2 minute voting phase are explicitly desired. After presentation of the result, silence returns immediately and the solution is, depending on the answer behavior, only briefly explained or discussed with the plenum. A mix of simple and more complex questions creates a balance between a regular sense of achievement and a challenge. The depth of the ARS questions is limited compared to those of the written exam, as they have to be oriented to the short voting time. While initially about 70% of students participate in the polls, by the end of the semester the percentage is about 40%. The reasons for this decline are unclear. In addition to providing feedback on learning success and interaction with students, ARS use also serves to break up the otherwise long presentation periods. The lecturer also receives immediate feedback on whether the content has been successfully conveyed. The reduction of the lecture material to the essentials, which is necessary due to the time required for the PINGO application (approx. 15 minutes), is perceived as a positive side effect. In the course evaluation, the students were enthusiastic about the use of PINGO, which will be continued in the coming semesters.