• Neil C. Rowe

    • ncrowe@nps.edu

    • The prevention of plagiarism has been the subject of much attention, but insufficient attention has been given to other problems of dishonesty in online assessment. We survey the types of problems that can occur and what can be done about them. We believe many educators are unaware of these problems, and most countermeasures proposed are insufficient.

    • When a student scores well for an online assessment, does that mean that they know the material?

    • Traditional assessment also requires costs: the time of human proctors, care in control of the assessment materials before and after administration, and grading effort, all of which are simplified in online assessment. But can we trust the results?

    • Unfortunately, often we cannot. Everybody lies at one time or another (Ford, 1996), and cheating is common in education (Cizek, 1999; Lathrop and Foss, 2000; Dick et al, 2003). (Bushweller, 1999) cites disturbing statistics such as that 70% of American high school seniors admit to cheating on at least one test, and 95% of the students who said they cheated were never caught. (Dick et al, 2003) reports 12 studies of cheating, mostly with college students, in which an average of 75% of students reported cheating sometime during their college career. Cizek (1999) also reports that cheating increased significantly in the second half of the twentieth century, and that cheating increases with the age of the student at least through age 25, which could have serious implications for distance learning with its often-older students. Cheating also has been observed to increase as the bandwidth (information per second) of the communications channel between assessor and assessee decreases; that is, people who feel more "distant" cheat more (George and Carlson, 1999; Burgoon et al, 2003). Online assessment has a narrower bandwidth than classroom assessment (instructors cannot watch students work, for instance) and the previous results suggest this might make cheating easier.

    • In addition, students often have less commitment to the integrity of distance-learning programs than traditional programs because distance-learning programs often lack tradition, are often taken by people with pressures from other jobs, and many programs are new and not fully debugged. In general, cheaters often point to factors like these that facilitate cheating (Bell and Whaley, 1991)

    • Some attention has been paid to the increased ease of text plagiarism using the Internet (McMurtry, 2001; Heberling, 2002) but little to the problems of focused assessments using instruments such as multiple-choice and calculation questions, necessary in most science and engineering courses.

    • Group projects can reduce cheating if students monitor one another, but group projects are not appropriate for many subjects and learning skills. Others have argued that assessment should be continuous so it is less cost-effective for students to cheat (Bork, 2001). This does require considerable work in setting up a course. It also gives students less opportunity to study and digest the material at their own paces, a key feature of self-education. It creates more of a climate of distrust, suggesting that students cannot be trusted to learn without constant testing. It is also logically impossible to simultaneously satisfy three important criteria for continuous assessment: That the assessments are of equal size, that the assessments test all material of the course to the same degree, and that each assessment tests some material covered before the previous assessment (see Appendix).

    • While an explicit cheating policy, as well as the very act of testing, could make testees distrustful because suspicion shown to someone decreases their trust (Sztompka, 1999), assessment is central to education because the main purpose of an educational institution is to validate student knowledge

    • Some anecdotal evidence (Kaczmarczyk, 2001) suggests students today cheat less in distance learning than with traditional instruction. This may be because new technologies typically first attract smarter and more motivated users with less reason to cheat.

    • Problem 1: Getting assessment answers in advance

    • it is hard to ensure all students take them simultaneously (Olt, 2002)

    • An interesting idea is to reward by a grading factor those answers that are the most atypical, but that will not work when there is only one correct or good answer.

    • If all-at-once assessment with a single test is not possible, assessment questions can be drawn from a large pool and each student given a random selection

    • it is hard to grade students fairly when they get different questions since some students will get harder questions than others. A way to reduce unfairness is to ask many questions, but then assessments become long and tedious.

    • A more serious problem with pools is that instructors systematically underestimate how large the pool must be to make negligible the overlap of questions between tests.

    • If M is the number of questions on a test and N is the number of questions in the pool, the expected number of questions in common between two randomly chosen test sets is approximately M*(M/N).

    • Even when students cannot guess the instructor’s password, they can use "social engineering" methods that have been successfully used to scam even smart people into revealing their passwords, like "emergency" calls from alleged programming staff or "please change your password temporarily for system testing" requests (Mitnick, 2002). Since few instructors are security experts, they can fall for many of these scams.

    • Even if students take an assessment simultaneously and the instructor’s password is adequately protected, students can use "spyware" to electronically sneak a look at how other students are answering questions during an assessment or what the instructor is typing on their computer. Spyware is software that secretly sends messages about you to other people. It has become a problem on the Internet where some free utilities secretly install spyware to send information to advertisers about what sites you are visiting (Mintz, 2002). The software technology of spyware is not difficult, and students who steal test answers could sell them to other students. Students could also use software called "sniffers" (McClure et al, 2001) to decipher the message packets of a local-area network used by fellow students or the instructor and thereby read their answers or passwords. Students could also use a variety of hacker attack methods to gain server-administrator privileges on the course-server machine ("privilege escalation"), just as good as obtaining an instructor password, unless the machine is kept "patched" regularly with operating-system fixes. Students don’t need to be software experts to do these things, just to download software from a Web site and follow a few simple installation instructions, just as how most hackers attacking computer systems don’t understand their attack software because they obtained it from someone else. Installed spyware and sniffers can be recognized by careful computer forensics (Prosise and Mandia, 2003), but it requires some work.

    • When a user logs off a computer, they leave in memory and on disk many records of what they have been working on, and it is not difficult for this information to be retrieved with built-in tools and free software. For instance, a student or instructor working on an assessment over the Web may leave the final version of the pages they downloaded, with their answers, in the cache of their Web browser. Even if the power is turned off, the cache will still remain on disk, and even if files are requested for deletion, operating systems often send them to a "recycle bin" before actually deleting them.

    • Problem 2: Unfair retaking of assessments

    • Another serious problem with online assessment is that it may be possible for students to retake an assessment multiple times until they are satisfied with their performance, even if that was not the intention of the instructor.

    • If the server software is not properly designed, students can break their connection to the server during an assessment, then claim they lost power and test answers and need to start over, giving them extra time to consult collaborators or unauthorized reference materials. Students could also crash (stop) the server after the grading is done but before the grades have been recorded; crashing is not difficult with the many hacker tools currently available. Another trick is to change the system clock so the grading server thinks that a new test assessment is actually prior to an earlier assessment; many operating systems do not adequately control access to their system clock.

    • Password theft of the instructor’s password as discussed above also permits a student to change previous grades, since instructional software must allow instructors to correct grading mistakes. Blackboard doesn’t even bother to tell the instructor when they last logged in, a key clue to this kind of manipulation. Again, computer forensics can detect these unauthorized activities, but this is often not easy.

    • Problem 3: Unauthorized help during the assessment

    • the most serious problem with online assessment is confirming that the student is in fact who they say they are

    • This issue of "authentication" has been subject of much research in computer security, but usually the problem is that of ensuring that a given person is present, not that they are alone, which requires different methods. Note that "high-tech" solutions of infrared or electromagnetic monitoring of test-takers are not adequate for preventing unauthorized collaboration because communication can take many forms including aural, optical, and olfactory

    • One approach is to include some traditional tests in any distance-learning program, as with the Open University, with proctors and the usual test security.

    • It is also possible for one student to impersonate another, so each student will need an identification card and it will need to be checked at the assessment site. Such tests are an imposition on the students and will need to be minimized in number because of their logistics. Hence much is riding on the outcome of these traditional assessments (since a bad score should surely override great scores on online assessments in which we are not sure who is taking the test); students will be under pressure, some students perform unfairly poorly under pressure, and this is a good incentive to cheat.

    • Countermeasures

    • (Cizek, 1999) provides a good overview of methods for recognizing, responding to, and preventing cheating in traditional paper-and-pencil assessments, and many of his insights apply to online cheating.

    • Most distance-learning assessments use multiple-choice, true/false, and matching questions since they are much easier to grade automatically than short-answer and essay questions. Then the number of identical incorrect answers between two students is a good clue, and can be given confidence intervals for a hypothesis of cheating.

    • Define cheating and encourage honesty.

    • Know the assessment takers

    • Understand what students face.

    • Maintain assessment security.

    • Assessment documents should not be stored as files on instructor machines, but only on a server machine, and the server software must be kept up-to-date to minimize hacker attacks to obtain instructor or administrator privileges. The server should have intrusion-detection software to catch attacks before they happen and should do auditing to reveal attacks. The server should also have effective physical security to prevent events like theft of disks. To keep track of all this, the server site must have a designated Security Manager.

    • Proctor the assessment. Proctors not personally related to the student are important when students use computers to do the assessment. Proctors can ensure that students take the assessment at a designated time, without collaborators, and without unauthorized materials.

    • Control the assessment situation. Prohibit all handheld devices (calculators, personal organizers, pagers, cell phones, headphones, etc.) since all can store and transmit information from outside the assessment room (Lathrop and Foss, 2000). If computers are used for the assessment, communication should be made as difficult as possible between them and the rest of the Internet.

    • Make the assessment a learning experience.

    • Use constructed-response test formats.

    • Use varied test formats.

    • Avoid situations that encourage cheating.

    • Plan for the unexpected.

    • Entrapment.

    • a useful way to catch the stealing of tests and answers is to plant fake tests in possibly accessible places, like on the grading server, while keeping the true test offline until test time. Then if a student uses answers from the fake test, we know our security precautions have been faulty and can take measures.

    • we make the following recommendations for online assessment:

    • Human-proctored traditional paper-and-pencil tests with traditional security procedures should be used for major assessments in distance learning.
      • If manual grading is too burdensome, human-proctored tests taken at a computer are a second-best choice provided that the computer’s software and networking capabilities are tightly restricted as described above.
      • If students take the same assessment at different times, it is critical to draw questions randomly from a large pool and reorder them (and answers to multiple-choice questions) randomly.
      • We should automatically and routinely compare answers given by students on assessments. When similarities beyond those due to chance are observed, especially for incorrect answers, it is usually best to just ask students to take a different assessment covering the same material since it is hard to prove guilt. Retaking should be done in a more secure manner than the original test, as for instance with essay questions instead of multiple-choice.
      • Countermeasures for cheating should be a consideration in purchase of distance-learning management software.

    • While there are countermeasures, most are unsatisfactory in some way. For these reasons, online assessment in distance-learning programs should be done with caution until more progress is made on the technical development of countermeasures. Certainly, practice quizzes can continue to be done online, and tests with essay and short-answer questions can be done online if plagiarism safeguards are used and instructors have the time to grade them, but traditional one-location one-time face-to-face testing for much of the student’s grade will need to be the assessment norm for distance learning in the foreseeable future.

Posted from Diigo. The rest of my favorite links are here.

Advertisements