Metacognitive Concerns in Designs and Evaluations of Computer Aided Education and Training:
Are We Misleading Ourselves About Measures of Success?Bob Jensen
Trinity University
715 Stadium Drive
San Antonio, TX 78212
Phone: 210-736-7347 Fax: 210-736-8134
email: rjensen@trinity.edu
Request for a Favor: This document is the first of a sequence of research papers that are related to my Working Paper 255 and Working Paper 260. All are extensions of the Jensen and Sandlin online book. The document that follows is a very rough draft. I would appreciate any feedback that you can provide by email, phone, fax, or letter. I want to improve this paper for my scheduled future workshops.
Metamemory and Metacognition: The Metalevel Activities of the Brain
The BAM Pedagogy at the University of Virginia
- Is BAM's Success Due to Metacognition and Metamemory Phenomena?
- The Need to Introduce Difficulties for the Learner
- Distributing Practice on a Given Task
- Varying the Conditions of Practice
- Providing Contextual Interference
- Reducing Feedback to the Learner
- Using Tests as Learning Events
- Impact on Teaching Evaluations
- Lessons From the Dead Poets Society
- Find the Answers Where You May
- Importance of Context
Differences Between Traditional Group Learning and
Cooperative Learning
Warnings: Suggestions for Future Research and Designs for Asynchronous Learning Networks
Acknowledgement of Paula Hertel
Appendix 1
Why Aren't Stories Good Food?
Email Messages About Evaluation Criteria and Processes
Appendix 2
The Differences Between Traditional Group Learning and Cooperative Learning
Appendix 3
The Emperor's Naked as He Can Be
Appendix 4
Update Message from Robert Bjork
MIT Encyclopedia of the Cognitive Sciences (shown in a new window)
Metacognitive Concerns
in Designs and Evaluations
of Computer Aided Education and Training: Are We
Misleading Ourselves About Measures of Success?
I've gotten to the point where I really don't lecture anymore, so if I look awkward up here talking to you, it's because I haven't lectured in four years. And the second point is that I've lost total faith in lecturing. Try telling your students something important in class, and then finding out how many heard it to where they can remember it the next day.
David Croll, University of Virginia
Transcribed from a videotape recording
Annual Meetings of the American Accounting Association
Dallas, Texas, August 20, 1997
Caveat: . I am grateful to Professors Croll and Catanach for allowing me to videotape their inspiring presentation. The quotations from Professors Croll and Catanach that appear at various points in this document have never been edited by those professors or modified from a transcript of a presentation that I videotaped at a conference. My videotape was transcribed by my secretary, Debbie Bowling. The transcription was modified by me only when Debbie failed to understand certain terminology. I prefer to minimize changes in the transcription so that what is read remains as close as possible to what the audience listened to at the conference. None of us speak with the formalized vocabulary and grammar used in our writing. Also we cannot edit what we said in the same manner that we can edit what we wrote. Hence, transcriptions should not be judged as writing. In August 1998, Tony Catanach moved to Villanova University in Villanova, Pennsylvania.
R. Jensen (1998a) documents the exponential increase in asynchronous learning where students take on greater responsibility for teaching themselves and set their own learning paces. Evidence is mounting that student performance thereby improves. Some of our finest and oldest universities in the world are experimentally replacing lectures with asynchronous learning networks where students no longer attend scheduled classes. The early success of the Sloan Foundation funded experiments, where traditional lectures are replaced by asynchronous learning networks (ALNs), are making all institutions take notice even for full-time resident students. Examples of these successes and concerns about these successes are documented in R. Jensen (1998a). In addition, traditional (synchronous) lectures and cases are taking place in electronic classrooms where live Internet connections and presentation multimedia are replacing chalk and flip charts. Students in electronic classrooms or in front of their own laptops at home can be taken into virtual learning worlds and electronic chat rooms that make learning easier, faster, more collaborative, and more fun. Virtual worlds make learning more contextual. Biology students work with virtual organs and organisms, chemistry and physic students enter virtual laboratories, medical students diagnose and heal virtual patients, sociology students colonize virtual societies, finance students choose portfolios in virtual markets, etc.
Across many years and nations, educators have long known that interactive learning tends ceteris paribus to be better than passive learning. Computer and networking technologies make interactive learning more effective and efficient. Students having keyboard, mouse, joystick, microphone, web camera, and other input controls have greater powers of interaction with instructors, other students, libraries, experts, and objects around the world. Students not only can listen to Beethoven and Chopin, they can be transported back in time to the virtual lives and times of master composers. The great libraries of online documents, major university library holdings, and government archives are at their fingertips.
Jensen and Sandlin (1998, Chapter 2) list many advantages of newer learning technologies. R. Jensen (1998a) outlines various worries and concerns. The main purposes of this paper are as follows:
Metamemory and Metacognition: The Metalevel Activities of the Brain
The brain stores long-term memory in the outer cerebral cortex with complex recollection systems among memory patterns of billions of neurons. Most things stored therein cannot be instantly recalled, but ways in which knowledge becomes embedded and patterns of use over time affect if and when recall takes place. Much depends upon associations between stored knowledge. For example, I probably would never recall the name of Don Quixote’s horse had this not been a memorable question (for me) on a 1961 GRE examination. That trivia question "sticks" in my mind all these years because of the humor of being asked about the only thing I remembered from that book (last read while I was a high school farm boy who liked horses).
There are also important distinctions between visual versus verbal memory, and these distinctions are crucial in designs of asynchronous learning materials. In fact, virtually all of Howard Gardner's theories on seven types of intelligence are important to take into account in the design of learning materials. These are described as follows by The Gardner School at http://www.swopnet.com/ed/TAG/7_Intelligences.html
:
1. Linguistic
Children with this kind of intelligence enjoy writing, reading, telling stories or doing crossword puzzles.
2. Logical-Mathematical
Children with lots of logical intelligence are interested in patterns, categories and relationships. They are drawn to arithmetic problems, strategy games and experiments.
3. Bodily-kinesthetic
These kids process knowledge through bodily sensations. They are often athletic, dancers or good at crafts such as sewing or woodworking.
4. Spatial
These children think in images and pictures. They may be fascinated with mazes or jigsaw puzzles, or spend free time drawing, building with Legos or daydreaming.
5. Musical
Musical children are always singing or drumming to themselves. They are usually quite aware of sounds others may miss. These kids are often discriminating listeners.
6. Interpersonal
Children who are leaders among their peers, who are good at communicating and who seem to understand others’ feelings and motives possess interpersonal intelligence.
7. Intrapersonal
These children may be shy. They are very aware of their own feelings and are self-motivated.
However, designing multimedia with the above types of intelligence in mind does not in and of itself aid to metacognition and metamemory. The mysterious process of "deciding" (it’s an unconscious process) what gets stored versus what gets lost or discarded takes place in a deeply embedded part of the brain known as the hippocampus. That organ is analogous to a computer’s file manager when it stimulates neurons to form an episodic network of neurons. Two important criteria affecting this process are emotional significance and association significance (associations with episodes already embedded in neuron structures). Information overload of the brain is not so much an issue of the number of neurons as it is with the input rate and complexity of processing incoming perception vis-à-vis emotions and episodic associations. Short-term memory "junk" such as a seldom-used phone number never clutters up our long-term memory episodic networks. Rote memorization of the phone number may last for a day but is not likely to last for a decade or even a week after it is used. When the hippocampus is damaged, a person might never store incoming knowledge and live only with recordings made when the hippocampus functioned properly. Alzheimer’s disease attacks the hippocampus region of the brain. Aaron, Benjamin, and Bjork (1998, p. 55) define metamemory as follows:
There has been a surge of interest in metamemory --- the study of what people know and understand about their own memory and memorial processes. From a theoretical standpoint, there has been a particular effort to explain why certain metamnemonic measures, such as the feelings of knowing . . . are accurate or inaccurate under various conditions.
The term "cognition" refers to the entire process of knowing --- a process comprised of perception, concentration, memory, judgment, and awareness. A subset of cognition known as metacognition is set apart from most cognitive activities and takes place in the frontal lobes. Metacognition is little understood even though we do know that it is impacted by episodic association formations in the hippocampus and that our well-developed metacognition is what sets human beings apart from other species. Psychologists refer metacognition as the monitoring process superimposed upon the control process of cognition. This monitoring can alter the control processes. Metacognition gives rise to what is commonly referred to as "Feeling of Knowing" or FOK judgments in cognition literature. It also gives rise to what is known as "novelty" versus "familiarity" awareness of perceptions.
Thiamine deficiency in the frontal lobes often accompanies severe alcohol abuse and is known in medicine as the Korsakoff Syndrome. Korsakoff amnesia thereby differs from amnesia that does not so dramatically affect the frontal lobes. Shimamura and Squire (1986) report that patients diagnosed with Korsakoff Syndrome have impaired FOK judgments. The theory is that frontal lobes processes modulate the hippocampus processes of forming episodic associations. Cognitive scientists make a distinction between basic memory and metamemory. Metamemory helps to make distinctions between stored memories. Weingradt, Leonesio, and Loftus (1994) note that a witness to an automobile accident may store perceptions at the scene of the accident and knowledge from what he or she read about or heard subsequent to the event. On Page 183 they elaborate as follows:
Consider a typical situation in which an eyewitness is asked to report what he or she has seen. Whether the witness is asked to identify a perpetrator, describe the events leading up to an automobile accident, or discriminate between information actually witnessed in an event or read about in a newspaper article, he or she is required to make judgments about information in memory. In metacognitive terms then, eyewitnesses are often required to make metamemory monitoring judgments. As should be evident from our discussion of the misinformation effect as a metacognitive error, the literature on metamemory monitoring can provide valuable insights into the reasons for poor eyewitness memory performance.
These metalevel processes enable humans to reflect upon thoughts and behaviors. Nelson and Narens (1994) refer to "metalevel" processes that are set apart from the lower-order "object-level" memory and cognition processes. Metalevel processes monitor perceptions to assess their novelty/familiarity and to direct attentiveness and feelings. Janet Metcalfe (1996, pp. 381-382) quotes a passage from Plato in which Socrates refers to an "inner voice." She then goes on to write the following passage:
Socrates' inner voice is the earliest well-known example of a self-reflective consciousness --- an internal monitor and control system. A self-monitoring consciousness figures in the musings of Descartes, emerging as "the thinker" whose existence cannot be doubted, even though all the content of the thought is open to question."
Later on Page 382 she continues as follows:
An accumulating body of evidence points to the prefrontal cortex as a brain region of critical importance for these [metalevel] functions. This region is distinct from those areas responsible for the knowledge and operations upon that knowledge that are the object of reflection, though, of course, they are interconnected. As Nelson and Narens (1994) have pointed out, object-level functions of cognition, such as memory storage, retrieval, and applying the operations of problem solving, may be separable from metalevel functions that monitor and control the object-level functions. In the normal person these two levels, cognition and metacognition, interact in a complex manner. This interaction is critical not only for problem solving, planning, and for memory, but arguably for sensitive assessments of one’s own appropriate social behavior.
Metacognition is essential to what earlier writers have called introspection. Nelson and Lorens (1994, p. 17) quote William James in 1890 as follows:
Introspective Observation is what we have to rely on first and foremost and always. . . . I regard this belief as the most fundamental of all postulates of Psychology.
Object-level versus metalevel distinctions become important to educators for self-evident reasons. The implications are immense. For example, it may suffice to make it as easy as possible (e.g., with visual aids) for students to memorize multiplication tables, but we are not so certain that it is best to make it as easy as possible to understand number theory theorems. Quite possibly students lose introspective powers if they do not re-discover some aspects of number theory on their own rather than start at the top of the heap of known theory. Some aids to making it easier to understand the theorems may be detrimental to long-term memory and performance.
Making Learning More Easy, Fun and Collaborative: Are We
Taking Things Too Far in Virtual Learning Worlds?
I have been making hypermedia CD-ROMs for my students since 1992. Until recently, my goals have been to make learning more easy, fun, inspirational, realistic, collaborative, and efficient. I take my students into worlds of virtual financial contracting and reporting. When seeking answers, my students can watch and listen to world experts explain complex transactions (e.g., financial instruments derivatives contracts and mathematical models for analyzing investment risks) and issues and theories about accounting for those transactions. I have videotaped experts at numerous conferences and workshops. Then I digitized the experts’ audio/video excerpts onto CD-ROMs and organized the material for pedagogic ease and efficiency. I also transcribed hundreds of hours of audio into text that can be searched using key words and is presented in a variety of formats (including Jeopardy-type games that make learning more fun). Readers can view some of my course materials from links at my web site.
I also made things easier by obtaining rights to share commercial literature databases and CPA Examination review courses on CD-ROMs. For example, by using the Price-Waterhouse Researcher CD-ROM, the accounting and auditing standards of the major developed nations are literally at the tips of student fingers on a keyboard. I can make my assignments exceedingly tough and realistic. Students can find the answers using my CD-ROMs, network server files, the Internet, textbooks, and photocopy handouts. I created a learning world in which my students can quickly find answers and do not have to conduct those slow, frustrating, and serendipitous library searches. When computer coding became involved for students (e.g., to create Microsoft Access databases or to code JavaScript web documents), I wrote tutorials that made learning much easier.
My goal was to take students to heights that were not possible in my courses before computing and networking hypermedia technologies. By making the hunt for answers easier and fun and collaborative, I could cover more material and more difficult material. My courses were not necessarily easier since I added more material. However, my courses were more efficient since students learned faster and spend less time searching for answers and completing projects. Learning became easier and in some respects more fun since student could see and hear experts whose names appear in the literature.
My own research, however, is far more serendipitous than research than I require from my students. Once in a while I pull myself away from the Internet and wander amongst the shelved books in the library. I peruse tables of contents and take a glance at sections of books and journal contents that capture my eye (I like to think of it as metacognitive success in identifying something novel that is not familiar to me). An article that I actually stumbled upon in the manner described above has had a profound impact upon my teaching and research. That article is "Memory and Metamemory Considerations in the Training of Human Beings" by Bjork (1994). Dr. Bjork convinced me that I was setting some wrong goals when creating asynchronous learning materials for my courses. In addition, he convinced me that many of the things that I had written about education technology are wrong.
I do not go so far as to say that the goals of using new technologies for making learning more easy, fun, inspirational, collaborative, and efficient are wrong per se. I do, however, suspect that there are dangers in making these goals too sacrosanct in preparing learning materials for lectures and asynchronous learning outside the classroom. By way of illustration, let me focus on a passage from Bjork (1994, p. 197-198).
The process of accessing stored information given certain cues also does not correspond to the "playback" of a typical recording device. The retrieval of stored information is a fallible, probabilistic process that is more inferential and reconstructive than literal. . . . The information in our long-term memories that is, and is not, accessible at a given point in time is heavily dependent on the cues available to us, not only on cues that explicitly guide the search for the information in question, but also on environmental interpersonal, mood-state, and body-state cues.
In years past, I created cases on the computer that outlined financial contracting situations. For example, a typical case would entail an interest rate and/or foreign currency risk exposure that a company hedged with circus swaps (combinations of interest rate and currency swap financial instruments derivatives contracts). In this virtual world of contracting, I then gave my students links to my own learning materials and Internet links that provided the information needed to assess and account for risks. They had to find the solutions and then be prepared to explain those solutions.
After pondering Bjork (1994) and related references on metacognition, I decided that my students were too dependent upon virtual worlds (cases) that I created and solutions that I provided in the course materials. Now I make my students write their own cases and propose their own solutions. Extra credit is given for attacking issues for with there are no known solutions. Credit is given to finding sources that I do not provide and, in many instances, have never encountered. Extra credit is given for searches outside the fields of accounting and finance. My goal is to multiply to encode the learning and retrieval process. Bjork (1994, p. 189) writes the following:
On the encoding side, we would like the learner to achieve, for lack of a better word, an understanding of the knowledge in question, defined as an encoding that is part of a broader framework of interrelated concepts and ideas. Critical information needs to be multiply encoded, not bound to single sets of semantic or situational cues. . . . Similar to the argument for multiple encoding, it is also desirable to induce successful access to knowledge and procedures in a variety of situations that differ in cues they do and do not provide.
Whether learning contexts are defined as "cases" or "virtual simulations" or "virtual games," the important point to consider in their designs is that they may be too context specific and/or lack variety to the detriment of long-term memory and performance. Our students may spend too much time mastering solutions that we make available to them albeit in some form of treasure hunts. Our cues may be too singular and too situational in the computer worlds that we present to our students.
The BAM Pedagogy at the University of
Virginia: Is BAM's
Success Due to Metacognition and Metamemory Phenomena?
On August 20, 1997, at the Annual Meetings of the American Accounting Association in Dallas, I videotaped an extraordinary presentation by Professors David Croll and Anthony Catanach from the McIntire School of Commerce at the University of Virginia. The topic was the Business Activity Model (BAM) revolutionary and award-winning revision of the Intermediate Accounting sequence in which all lectures and textbook assignments were replaced by a two-semester focus on accounting for a business from its inception through seven years of operation. Development of the BAM case was initially funded with $50,000 from the Accounting Education Change Commission (AECC). The AECC was set up to inspire innovation and allocate funds given by the largest eight accounting firms to foster change in accounting education pedagogy. In 1997, the BAM experiment won the prestigious $5,000 Innovation in Accounting Education Award administered by the American Accounting Association. The main innovation of the BAM pedagogy is that students teach themselves in a discovery learning pedagogy. The main purpose of BAM is make students constantly confront ambiguity. They are assigned to teams, but teams are not graded. Although spreadsheets and some other learning materials are on line, this is not a high technology ALN application with chat rooms, etc.
In my viewpoint this paper also has relevance to the findings of another AECC-funded experiment called the Project Discovery (PD) project at the University of Illinois at Urbana-Champaign. That study took a slightly different approach across multiple accounting courses to study the effects of the importance of having students learn from "complex, ill-structured, ambiguous problems and cases similar to those found in practice" as reported by Stone and Shelly (1997, p. 26). Stone and Shelly repeatedly stress that their research notes the impact of such factors on learning but make no attempt to attribute causality. I speculate that some of the causal factors are metacognitive. Even though my paper focuses on the BAM Program at the University of Virginia, I think my conclusions extend to the PD Program at the University of Illinois.
The BAM pedagogy is a "success" in terms of a number of important criteria. Students on internships and recent graduates in full-time jobs report that the BAM learning context is much more like "what really happens" on the job. In particular, they feel better prepared for dealing with ambiguities of real life. Secondly, long-term memory seems to be enhanced by the BAM pedagogy. More than a year passes between completion of Intermediate Accounting and sitting for the national CPA examination. Performance on the CPA examination improved markedly using the BAM pedagogy. Without having the traditional lecture/drill pedagogy students remember better with the BAM learn-it-yourself pedagogy. Instructors using the BAM pedagogy claim students do better. They also claim that weaker students that tend to have more trouble with examinations are given more opportunities to show that they know more than is measured on examinations. In the BAM pedagogy, students make presentations in front of fellow team members and in front of the entire class. Students have more realistic and memorable interactions in the BAM pedagogy. David Croll relates the following:
. . . afterwards she felt better about it, and participated a lot more and seemed to be much stronger at it. In student groups we have finance majors as well as accounting majors in this course, and some of the finance majors are great at speaking out. And so I get one young lady who's going off to work for a Big-Six CPA firm, and she had three finance majors, three real talkers, in her group. And on this accounting issue, the three men were adamant that they had this correct answer and she was wrong. It turns out that she really had the right answer --- and I felt great! She stuck with it. She argued them down.
And it went on for about ten minutes. We stood back and let her fight it out with them. She took them on --- all three. And she's gonna have to do that in her professional career. Go out there and say, this sounds good. This is the correct accounting. And afterwards, I think she had a lot more confidence and interest in going out and doing her job.
David Croll, University of Virginia
Transcribed from a videotape recording
Annual Meetings of the American Accounting Association
Dallas, Texas, August 20, 1997
The BAM pedagogy across two semesters is revolutionary accounting education. Intermediate Accounting in virtually all colleges is a textbook driven set of two or three courses aimed at mastering rules for accounting for financial transactions. Since business contracting and transactions have become so complex in the past three decades, the complexity and volume of the rules and practices have burgeoned. Intermediate Accounting classes are traditionally synchronous in terms of lectures and/or case discussions. Learning under the BAM pedagogy is asynchronous both inside and outside the classroom. More importantly, the BAM approach makes students search for answers on their own or in teams such that "they teach themselves."
I made a CD-ROM of the BAM presentation and feature the BAM pedagogy and outcomes assessments in my education technology workshops that I conduct on college campuses around the world. After stumbling belatedly upon the Bjork (1994) paper (subsequent to giving several workshops featuring the BAM pedagogy), it dawned on me that reasons why faculty at the University of Virginia feel that the BAM pedagogy is a resounding success relative to traditional pedagogy lie in metacognitive and metamemory phenomena. In this paper I will try to make the case that some reasons for the success of the BAM pedagogy lie in metalevel processes in the brains of students.
Please keep in mind that the success of the BAM experiment may be heavily due to the skills and dedication of the faculty and the high quality of the students participating in the experiments. It is not at all certain that other faculty can carry this off at other places and times. It is not clear that these same faculty (Croll and Catanach) can maintain the intensity and motivation year after year that were present in the first four years of experimentation. The BAM pedagogy takes students and faculty into many issues for which there are no known solutions and requires that faculty and students tolerate far more ambiguity throughout the entire year of learning. Ambiguity creates stress for students and faculty.
According to David Croll, ambiguity is the reason Robert Grinaker (original BAM case author who is now emeritus), David Croll, and Anthony Catanach proposed making such a dramatic change in pedagogy for Intermediate Accounting. Graduates prior to the BAM pedagogy graduated with an air of confidence that they had mastered accounting rules and pat solutions for the real world. In the first year on the job they discovered that these pat answers seldom applied in practice. They were not prepared for the ambiguities and complexities of real world careers. Many of them changed careers or returned to law schools according to Professor Croll. The problem is pervasive and is in no way unique to accounting graduates from the University of Virginia. The accounting firms are not happy because of the turnover in new employees, and the new employees are not happy because they were not prepared to deal with unresolved problems and ambiguities of their work assignments. Furthermore, newly minted graduates prior to the BAM pedagogy were not trained and educated to deal with ambiguities and introspective talents needed for complex business transactions. These are the major reasons the largest firms attempted to foster accounting education change with AECC grants to colleges and universities.
Hence, one goal of the BAM pedagogy is to deal with more ambiguous issues prior to graduation. These are invariably viewed as harder hurdles for students. Without knowing it, the BAM developers were in fact conforming to what Robert Bjork claims is most important for long-term performance. Bjork (1994, pp. 189-193) called this point "The Need to Introduce Difficulties for the Learner." In particular, Bjork stresses the need to introduce "variation and/or unpredictability," "contextual interference," distributing the learning of topics over extended intervals of time, reducing feedback, "using tests as learning events," and being willing take risks of lowered evaluations of instructors.
UVA010
The Business Activity Model at the University of Virginia:
The Need to Introduce Difficulties for the Learner
Whatever the exact mixture of manipulations that might turn out to be optimal, however, one general characteristic of that mixture seems clear: It would introduce many more difficulties and challenges for the learner . . . . Recent surveys of the relevant research literature . . . leave no doubt that many of the most effective manipulations of training --- in terms of post-training retention and transfer --- share the property that they introduce difficulties for the learner.
Bjork (1994, p. 189)
Metacognition: Knowing about knowing
Now one of the problems is Bob wrote the case without thinking about the accounting. If I ever wrote a case, I'd write a case about what I knew. He wrote a case about a business he knew. And so there were times when he got us into places that we didn't know how to get out of. And Bob would say, "well Dave, what do you think about how we ought to book this?" And I'd say, "gee I don't know. What do you think?" And so we were where the two of us really didn't know how to book it. And it was coming down the line the students were going to get up and gonna ask for this. And my goodness, we were gonna to be in trouble. So Bob said, "here's how I think," and he did it that way. And then he said, " the FASB has a hot line. We'll call them up."
So he called them up. And he told them the problem, and then there was a pause. And then the voice said, how are you guys gonna book it? And Bob told them, and they said "sounds good to us." So that's how we book it.
So what we're talking about is not easy It is extremely difficult; Clearly it's difficult for the students. They're in there, but when we talk about ambiguity we really mean it. I mean, they're out there no matter how bright they are.
David Croll, University of Virginia
Transcribed from a videotape recording
Annual Meetings of the American Accounting Association
Dallas, Texas, August 20, 1997
The Business Activity Model at the University of Virginia:
Distributing Practice on a Given Task
In general, compared to distributing practice sessions on a given task over time, massing practice or study sessions on the to-be-learned procedures or information produces better short-term performance or recall of that procedure or information, but markedly inferior long-term performance or recall.
Bjork (1994, p. 190)
Metacognition: Knowing about knowingWith blocked scheduling, each task is learned separately, and learning one task is completed before the trainee moves to another task. With random scheduling, the tasks are intermixed during acquisition . . . .Blocked scheduling always produced a faster rate of learning during acquisition, but, regardless of the type of text scheduling, random scheduling during acquisition resulted in the best performance at retention testing.
Healy and Sinclair (1996, p. 531)
Memory: Handbook of perception and cognition
You've got to understand, this was very innovative. We don't try to cover a topic in one sitting, or three days in a row. We've got seven years of this case. We're going to cover taxes for seven years. We don't have to teach all the tax chapter in the first year . . . we have to teach enough to get them through with how to book a loss. And then the company does turn it around in about the third year --- it's just a little early. But now they are going to have to do deal with those tax issues. So it's like peeling an onion. We go deeper and deeper into these taxes. And I had the pleasure one of the first years I taught it of having one of our very bright students come up to me in about the fifth year and say, you guys are going to make us do taxes all seven years aren't you?
And I got to smile and tell them, you're going to do taxes all of your life. Yes we did, all seven years.
David Croll, University of Virginia
Transcribed from a videotape recording
Annual Meetings of the American Accounting Association
Dallas, Texas, August 20, 1997
At this point let me say that a lot of our students don't listen to us about documenting their work and organizing their work. And so when they come back . . . this is great, I love this part! I go back in September when they return from internships when we introduce year 4. We don't go back and redo everything for them, and they need all the stuff from the first three years of the case. And they didn't organize it, and they're dead, ok? They learn very quickly by year 5 and 6 how to organize a set of work papers to support the financial statements.
Anthony Catanach, University of Virginia
Transcribed from a videotape recording
Annual Meetings of the American Accounting Association
Dallas, Texas, August 20, 1997
Whereas traditional Intermediate Accounting courses take up one text book topic at a time and then move on to the next topic over the entire two or three semester course coverage, the BAM model takes up nearly all topics repeatedly across two semesters. In each of the seven years of accounting for the business, the same problems arise each year in adjusting for accruals, estimating bad debts, valuing inventories, filing corporate tax returns, etc.
The Business Activity Model at the University of Virginia:
Varying the Conditions of Practice
.It has now been demonstrated in a variety of ways, and with a variety of motor, verbal, and problem-solving tasks, that introducing variation and/or unpredictability in the training environment causes difficulty for the learner but enhances long-term performance --- particularly the ability to transfer training to novel but related task environments.
Bjork (1994, p. 189)
Metacognition: Knowing about knowing
How's this thing delivered? Ok, and I forgot to answer your other question. I'll answer the other one . . . how the students react? First semester, remember, you're destroying their model, their learning model, with a sledge hammer. Ok? And so for the first two thirds of the semester, cuz I'm Satan! I'm a little bit worse than Dave --- my students sit there like this . . . great body language . . . for two thirds of the semester. Year 1, it's like pulling teeth.
Year 2: they finally realize that it's not going away. Ok? And that you mean it, and that they have to play, and they start playing. Now where the real, that real, hook comes in is when they go away for internships --- that's the summer between the Intermediate 1 and Intermediate 2 semesters --- cuz that's where we place internships at Virginia. After their internships, students come back and say "this is what really happens!" Afterwards, in Intermediate 2, they've been revitalized. And they've bought into BAM. And after that it's no problem
Anthony Catanach, University of
Virginia
Transcribed from a videotape recording
Annual Meetings of the American Accounting Association
Dallas, Texas, August 20, 1997
The Business Activity Model at the University of Virginia:
Providing Contextual Interference
.In Mannes and Kintsch’s (1987) experiment, for example, subjects had to learn the content of a technical article (on industrial uses of microbes) after having first studied an outline that was either consistent with the organization of the article or inconsistent with that organization (but provided the same information in either case). The inconsistent condition impaired subjects’ verbatim recall and recognition of the article’s content (compared to the consistent condition), but facilitated performance on tests that required subjects to infer answers or solve problems based on their general understanding of the article’s content.
Bjork (1994, p. 190)
Metacognition: Knowing about knowing
And so they're seeing old and new, and I think that's a pretty powerful teaching tool as well. A student discovered a problem in our case. And what had happened is this, we had been taking a tax deduction for write-offs to the bad debt. One of our students noted this --- it happened in my class. It happened in my class naturally, the new guy on the faculty. And he said, Dr. Catanach, we just got out of Ms. Jones class and she said that you couldn't take a deduction for that until it was actually written off, you couldn't take it, you know, because you are using the allowance method. I'm like, yeah, --- that sounds good to me. And then I said: "Well . . . we'll get back to this later on."
And so I ran into my colleagues --- David and I and Bob --- we had this debriefing and they went, yeah. Yes, so how are we going to do this, how are we going to fix this? And so we went back to the class the next day; we said, don't worry about the bad debt tax write-off. That's our position. We took a really weak defensive position, and this is like year 2 of the case when this student pointed this out. And so we let the mistake lie until year 6 of the case in the second semester. And in year 6 what we did is we injected an IRS audit which would allow the IRS to come in and find the tax error in year 2, and then students had to deal with a correction of an error issue. We didn't have that in the original case. So then we put correction of an error assignment in there. And if that wasn't enough, we had a stock split as well
Anthony
Catanach, University of Virginia
Transcribed from a videotape recording
Annual Meetings of the American Accounting Association
Dallas, Texas, August 20, 1997
The Business Activity Model at the University of Virginia:
Reducing Feedback to the Learner
. . . reducing the frequency of feedback makes life more difficult for the learner during training, but can enhance post-training performance.
Bjork (1994, p. 191-192)
Metacognition: Knowing about knowing. . . performance assessment during the acquisition phase of training is not a reliable indicator of training efficiency and skill learning. Only post-training retention and transfer testing, after an appropriate retention interval, can provide a true measure of learning and performance and the effectiveness of any experimental manipulation designed to enhance training efficiency.
Healy and Sinclair (1996, p. 530)
Memory: Handbook of perception and cognition. . . . described a study by Schooler and Anderson (1990) who showed that reducing the number of feedback trials during the learning of a programming language decreased acquisition performance but facilitated retention performance . . .
Healy and Sinclair (1996, p. 530)
Memory: Handbook of perception and cognition
Professors Croll and Catanach made no mention of reduced feedback in Intermediate Accounting before verus after adopting the BAM self-learning pedagogy. In both instances, examinations were given at relatively frequent intervals and students receive feedback about grade patterns to date in the course. Examinations were used to assess grades and student study teams are not graded as teams. Also examination grading was not very subjective on the types of examinations given in the courses. Frequent feedback and relatively objective grading are probably major reasons why students did not give the instructors low evaluations in the courses.
When attempts are made to improve long-term memory and performance by reducing feedback frequency and more subjective grading near the end of each course, it is highly likely that instructors will pay a heavy price in course evaluations by students. Also, keep in mind that when students are taking multiple courses, they tend to concentrate on courses more when tests are imminent. Frequency of examinations tends to increase attention given to a course even though it may impair long-term performance where students must be graded based upon final performance at the end of the course. It should be noted that if every course based 100% of the course grade on the final examination and/or the end-of-term project, it might unduly stress students at the end of the semester. There are of course other alternatives. Examinations and project deadlines can be scheduled throughout the course with the holding back of grading feedback until the entire set of things to be graded are evaluated by course instructors. Students, however, despise delayed feedback for a variety of reasons. Much remains to be researched on the tradeoff between feedback in courses and long-term memory and performance. Another problem is that impacts of feedback vary a great deal between students. Eric Jensen writes the following:
Does all this mean that external rewards are also good for the brain? The answer is no. That’s because the brain’s internal reward system varies from one student to the next. You’d never be able to have a fair system. How students respond depends on genetics, their particular brain chemistry, and life experiences that have wired their brains in a unique way. Rewards work as a complex system of neurotransmitters binding to receptor sites on neurons. . . . Most teachers have found that the same external reward is received differently by two different students.
E. Jensen (1998, p. 65)
Teaching with the brain in mind
The Business Activity Model at the University of Virginia:
Using Tests as Learning Events
And Landauer and Bjork (1978; see also Rea & Modigliani, 1995) found that "expanding retrieval practice," in which successive recall test are made progressively more difficult by increasing the time and intervening events prior to each next test of some target information, facilitates long-term recall substantially --- compared to the same number of tests administered at constant (and easier) delays.
Bjork (1994, p. 192)
Metacognition: Knowing about knowing
Now, let me cover a couple things quickly here and then I'll go back; one thing is this, I'm not really grading these students in terms of their class participation and the group work. How I'm going to grade them is I'm going to grade them the same way as I graded them before. I'm going to give them exams using old CPA problems, but I changed some things slightly so that I agreed with the answers. But for learning to learn, over two semesters, I'm going to give them six take home exams. I'm fortunate at UVA --- we have an honor system, and it works. So I thought well, maybe some schools won't have an honor system it may not work for them.
So I've tried take home exams in the summer school, where in effect, they don't take it home they simply work on it in a three hour block while they are in class. And I can watch them. And it works. The take home exams do not cover material I've covered. They cover material we're going to cover. But it's fresh and new. We give them the exam, they've got to go out, in some fashion, either with, well with both, with the official pronouncements and with the intermediate text and find out the answer to this. And sometimes to be nasty, we give them a problem having to do with a forthcoming statement that hasn't been issued yet, so of course there's nothing in the intermediate text so they have to read the pronouncement to figure out how to deal with it.
David Croll, University of Virginia
Transcribed from a videotape recording
Annual Meetings of the American Accounting Association
Dallas, Texas, August 20, 1997
The Business Activity Model at the University of Virginia:
Impact on Teaching Evaluations
The tendency for instructors to be pushed toward training programs that maximize the performance or evaluative reaction of their trainees during is exacerbated by certain institutional characteristics that are common in real-world organizations. First, those responsible for training are often themselves evaluated in terms of the performance and satisfaction of their trainees during the training, or at the end of the training. Second, individuals with the day-to-day responsibility for training often do not get a chance to observe the post-training performance of the people they have trained; a trainee’s later successes and failures tend to occur in settings that are far removed from the original training environment, and from the trainer himself or herself. It is also rarely the case that systematic measurements of the post-training on-the-job performance are even collected, let alone provided to a trainer as a guide . . .
Bjork (1994, p. 193)
Metacognition: Knowing about knowing
. . . performance assessment during the acquisition phase of training is not a reliable indicator of training efficiency and skill learning. Only post-training retention and transfer testing, after an appropriate retention interval, can provide a true measure of learning and performance and the effectiveness of any experimental manipulation designed to enhance training efficiency.
Healy and Sinclair (1996, p. 530)
Memory: Handbook of perception and cognition
A pedagogy that forces students to teach themselves or each other while relying less on instructors may lead to lower teaching evaluations than a pedagogy where the teacher lives up to student expectations of what a teacher should be doing inside and outside the classroom. The BAM experiment is an exception. Professors Croll and Catanach had high evaluations under traditional and under BAM experiment where they eliminated lectures and forced students to learn on their own. It is not clear just why their evaluations did not plummet. A major factor is that students had frequent feedback about grades and seemed to be pleased that they were actually learning. A second factor is that students were allowed to seek help in learning from most any source and were assigned to teams that seemed to aid in the learning process. During class periods, the instructors "hovered" close enough to team deliberations and most likely did more teaching than they care to admit (e.g., when students really got hung up and appeared to run into a blank wall or proceeded down wrong pathways). Outside the classroom, the instructors were available for help. Classes were relatively small (around 35 students) such that instructors could provide help to individuals and groups of students. Croll and Catanach attribute part of the success of their teaching evaluations to "working toward solutions together."
So what we're talking about is not easy. It is extremely difficult --- clearly it's difficult for the students. They're in there, but when we talked about ambiguity we really meant it. I mean, they're in deep water no matter how bright they are. They're out there swimming. Now that may be why we got good reviews --- we were swimming with them, and we all got through this thing together.
David Croll, University of Virginia
Transcribed from a videotape recording
Annual Meetings of the American Accounting Association
Dallas, Texas, August 20, 1997
A recent message from a biology professor at Trinity University reads as follows:
The July 24, 1998 Chronicle of Higher Education has a very interesting Point of View by Paul Trout of Montana State. To quote from the article:
Administrators can rebut that calumny by using evaluation forms that are more than student-satisfaction surveys. Instead of asking students to rate the professor’s "stimulation of interest," "concern for students," and impartiality in grading" - categories that allow disgruntled students to make pinatas of their professors - evaluation forms should ask whether the course was demanding, whether performance standards were high,, whether the workload was challenging, whether the grading was tough, whether the student learned a lot. Those kinds of questions make it a little harder for students who resent a heavy workload or low grades to give spiteful responses.
It would seem that the Trinity Course Evaluation form as a little of each sentiment: satisfaction and "tough." When the current form was being designed, these two "sentiments" where not on my mind. It might be curious to rewrite the evaluation form from these two perspectives and see what we get.
The Trout article was about incivility in the classroom and what he called "education lite." I found it an interesting piece.
Blystone in Texas
Robert V. Blystone, Ph.D. <RBLYSTON@Trinity.edu>
Professor of Biology
Trinity University
San Antonio, Texas 78212
210.736-7243 210.736-7229 FAX
The Business Activity Model at the University of Virginia:
Lessons From the Dead Poets Society
The movie Dead Poets Society showed examples of why students recalled so much of their learning. There were changes in location, circumstances, use of emotions, movement, and novel classroom positions. We know that learners remember much more when the learning is connected to a field trip, music, a disaster, a guest speaker, or a novel learning location. Follow up with a discussion, journal writing, a project, or peer teaching.
E. Jensen (1998, p. 110)
Teaching with the brain in mind
Just like in regular business, Jerry Loose (the error-prone Controller of the company in the BAM case) doesn't write notes. You write his notes for him; it's his notes. But you end up writing his notes for him to do his {function--not sure of this word}. And so, after the third year is completed, what's done is we've got correct balance sheets, correct income statements, correct cash flow statements, correct notes. Now how I teach that on the third day is, the students come in, in their groups and sit down. I'll take a group, I'll take the spokesperson for the group, send them to the board and say; write up the balance sheet, you are there. Write up the income statement, we're getting them to share, you get cash flow and you just sit there and we'll talk about notes when I get there. And off they go.
Now again, chances are the person I picked wasn't the one that wrote this. There is so much work in this. I mean, we are not talking about small case. When Bob got into it, he got into it. And this is the case. It can't be done by any single individual --- you have to all share in doing this thing. And then you have to help each other work your way through it. And so the person that's up doing a presentation is getting coached from the sidelines by possibly the person that wrote it. Put this here, put that here, here's my sheet. Do this. And once they're all up, I have whomever I called on explain it all. Again with normally, with short coaching, they can explain the important parts and with help from their team they can get all the parts of it.
David Croll, University of Virginia
Transcribed from a videotape recording
Annual Meetings of the American Accounting Association
Dallas, Texas, August 20, 1997
The Business Activity Model at the University of Virginia:
Find the Answers Where You May
Stated more broadly, the conditions of training need to be constructed to reveal to the subject what knowledge and procedures are, and are not, truly accessible under the types of conditions that can be expected to prevail in the post-training environment. Some of the best ways to achieve that goal involve making life seem more difficult for the learner.
Bjork (1994, p. 201)
Metacognition: Knowing about knowing
That's basically what we're doing. We're asking people questions, they give us answers. Now, since it's a case, since we've been using it for years now, one of the constant questions I get has to do with the fact that they say, well, gee, my students would get the answers to this case and from the class before and it would be useless. Well, you don't understand quite what we're doing here. I don't care whether they get the answers from the class before them. In fact, we're telling these people, you're working as a team, you're partners. Get the answers from your partner. All the teams I have in class work for the same firm. Your group doesn't know the answer, ask another group. See, as an academic I can't get over this. I won't ask questions of anybody, it's cheating.
As an academic, for years I thought, gee, you're supposed to do it all yourself. I got out into business, the more successful people asked others; that knew this information. They picked up their phone, they didn't just make luncheon appointments. They actually asked people about things. Why don't we have our students do that? What am I doing then in class when we come to these questions?
David Croll, University of Virginia
Transcribed from a videotape recording
Annual Meetings of the American Accounting Association
Dallas, Texas, August 20, 1997
The Business Activity Model at the University of Virginia:
Importance of Context
It is in such special circumstances that the risk of having contextualized training may be greatest. If we want people to respond optimally to unanticipated novel conditions, such as emergencies and/or unique conditions of some other type, the evidence summarized in this chapter suggests that we do not want to have trained those people under fixed conditions.
Bjork (1994, p. 204)
Metacognition: Knowing about knowingI want student groups to go away for a while and then come back. What I want from you are questions. I want professional questions. I want a good professional question from you, from each group, and I give them some hint as to how may questions there are --- somewhere around ten questions. I also tell them an on-going hint that Jerry has more problems as does normal business with errors of omission rather that errors of commission. And so, take a look at it. You've had a review, you have had a beginning accounting group. By then students are protesting that they don't know anything.
David Croll, University of Virginia
Transcribed from a videotape recording
Annual Meetings of the American Accounting Association
Dallas, Texas, August 20, 1997
Differences Between
Traditional Group
Learning and Cooperative Learning
In Appendix 2, I have reproduced an excerpt that lists criteria for transforming group learning into cooperative learning. It would appear that the BAM group learning is not technically cooperative learning in terms of some of the Appendix 2 criteria.
Consider the Appendix 2 criteria listed below:
- Cooperative learning groups are based on positive interdependence among group members, where goals are structured so that students need to be concerned about performance of all group members as well as their own.
- In cooperative learning groups, there is a clear individual accountability where every student's mastery of the assigned material is assessed, each student is given feedback on his or her progress, and the group is given feedback on how each member is progressing so that the other groups' members know who to help and encourage. In traditional learning groups, individual students are not often held individually accountable for providing their share of the group's work and, occasionally, students will "hitchhike" on the work of others.
Meeting the above two criteria is clearly difficult if this implies revealing each others' examination performances and interim grades. Also it is not clear that most groups work that way in the real world. At the University of Virginia, the BAM instructors wander about the classroom observing groups in action. However, grading criteria in the course apply only to individual performance and not upon group performance or performances of other members of a group.
It is not at all clear how metacognitive processes are affected along a spectrum of cooperation versus competition in learning. Clearly, colleges and universities tend to be competitive in terms of grades. Virtually all courses have some type of "curved" grading outcomes. Any course that is not somewhat competitive may lose student motivation. For example, if everyone gets a A in Course A and 50% are destined to get Cs in Course C, students most likely will strive harder in Course C. The point here is that it is very difficult to disentangle cooperative/group learning from reward systems.
From a metacognitive standpoint, long-term recall and feelings of knowing are more likely to be greatly impacted by degree of effort and motivation, including competitively inspired motivation. Also top students feel better about being distinguished as being near the top of a class where not everyone in the class is at the top. However, competition can also be destructive to aspirations and self confidence. Broad generalizations about such matters are clearly hazardous.
Warnings: Suggestions for Future Research and
Designs for Asynchronous Learning Networks
Fish Out of Water Risks
Years ago, R. Jensen (1970) warned that relying on experiments from the behavioral sciences is analogous to living with "fish out of water." These warnings apply to the experimental conclusions of virtually all metalevel studies. In nearly every instance, the experiments have dealt with memory and learning tasks far less complicated than coverage of an entire year of a Intermediate Accounting. Many of the experiments focus on remembering names and places rather than complex processes such as accounting for pensions, bad debt allowances, depreciation, inventories, etc. Given the suspected success of BAM-type pedagogy, the time is ripe for metacognitive and metamemory research in more complicated learning settings.
Contextualization Risks in Virtual Learning Worlds
One of the real problems of the BAM experiment is that it is confined mostly to Intermediate Accounting topics. In the real world, advanced topics and complex contracting issues surface that are well beyond the scope of intermediate accounting students. Although the entire basis of the BAM pedagogy is to contextualize the learning environment and force students to raise questions and find answers on their own, it is difficult to prepare them for issues that are beyond the scope of the course or, for than matter, beyond the scope of the entire college curriculum. The BAM pedagogy does, however, add some topics not found in traditional Intermediate Accounting courses. Without ever having a corporate tax course, students are required to fill out corporate tax forms across all seven years of the business case. Secondly, they encounter some auditing issues prior to having had an auditing course. By making students encounter and deal with issues for which they have no prior background, the BAM pedagogy takes a giant stride away from excess contextualization of traditional pedagogy where assignments typically are drills in applying previously-learned concepts and procedures. However, only a small portion of the plethora of novel real-world issues can be built into any case.
Misleading Program, Course, and Learning Module Evaluation Risks
I recently attended the Consortium of Liberal Arts College (CLAC) 1998 annual meetings. This organization is comprised mainly of directors of information technology on liberal arts campuses. These are the professionals in charge of assisting faculty in the design and evaluation of newer learning technologies for synchronous and asynchronous programs, courses, and learning modules. One of the most valuable sessions was entitled "How to Evaluate Instructional Technology and How Technology Can be Used in Evaluating Instruction." What struck me is the virtual consensus (among panelists and the audience) that surveys and measurements of performance are untrustworthy except for tabulations that are relatively easy to verify --- such as how many students own their own computers, how many students use the Windows operating system, etc. Other statistics such as the frequency of student visits to a web site or the number of messages in an email listserv or chat room are viewed suspiciously since these statistics can be artificially inflated in many ways.
In their opinions, anecdotal evidence and "stories" from faculty and students were far more influential in administrative evaluations of programs, courses, and learning modules than were formalized evaluations and measurement instruments. Obviously, such presenters as Eleanor Lonske (Wellesley College), Diane Balestri (Vassar College), Phil Harriman (The College of Wooster), Charles Christison (Beloit College), and Michael Westfort (Connecticut College) admit that anecdotal evidence can be one-sided and misleading, but far more dangerous in their eyes were quantitative studies that are subject to too many intervening variables such as "times of day classes meet," "wordings of questions," "times and conditions of administering the evaluation forms," and the "personalities of combinations of students in a given course." In double blind studies (e.g., having the same instructor teach one class using traditional materials and another class using newer technology aids) there is seldom, if ever, a "clean experiment." Too many other uncontrolled variables enter into such experiments. Also technologies change so fast that it is not clear that extrapolations apply even one semester into the future. Meetings at the CLAC conference did not mention Hawthorne effects. Hawthorne effects refer to distortions and possibly non-sustaining effects of a treatment just because its newness captures more of an individual's attentiveness. In double blind studies of the impact of technologies upon learning, Hawthorne effects are particularly troublesome. Students are more apt to be more attentive to newer technologies simply because they are "new" curiosities. Positive results on learning impacts may not be sustaining, however, after the novelty and curiosity factors decline with repeated use of the technology over time.
For more discussion of this topic, see Appendix 1.
Warnings040
Metalevel Oversights in Learning Material Designs and Newer Pedagogy Risks
In addition to the above concerns, I would add some metacognitive and metamemory
concerns for learning material designs. Students and faculty may have misleading feeling
of knowing (FOK) and satisfaction. Most formalized evaluations have a short-term
performance evaluations (e.g., examinations given during and at the end of a course).
Students tend to give higher evaluations to courses that did the following in their eyes:
- Made learning of complex material easier;
- Made it easier to find reference material for projects;
- Made learning more fun;
- Had new pedagogy (e.g., technologies) that captured attention because of its newness (Hawthorne Effects);
- Had high frequency feedback so that students could adjust their study intensities accordingly;
- Allowed greater topic coverage due to increased efficiencies of learning;
Metalevel researchers in cognitive science view each of the above criteria with high levels of suspicion. The problem is that the above criteria may lead to the following:
- Misleading metacognitive feeling of knowing (FOK) that does not carry over into real world contextual variations and contextual complexity;
- Impaired long-term memory that would have been enhanced by more sweat, ambiguity, frustration, and anxieties in the learning process;
- Impaired ability to adapt to changing contexts and creatively deal with variations in challenges;
- Impaired abilities to face ambiguities;
In this paper, it is proposed that we may be designing our computer aided training and education modules with the wrong criteria in mind. Perhaps it is wrong to always seek to make learning easier, less frustrating, more collaborative, and more fun. In computer aided learning, it is common to author in "rites of passage" that will not allow students to move on to the next modules without demonstrating mastery of prior modules. Perhaps rights of passage are not always ideal since reduced feedback sometimes leads to improved metacognitive performance and long-term memory. Perhaps certain types of frustrations aid learning. I see little value in the frustrations of computer failure and slow Internet performance, but having to physically search among the stacks for hard copy journals and books and having to locate references without the aid of keyword searches on a computer may improve metacognitive performance.
Metalevel research findings have grave implications for our using emerging technologies to create virtual learning worlds. The trend is toward making learning contextual. The goal is to immerse students into simulations of reality so that learning is more meaningful in the context to which it applies. However, any virtual world is a simplified copy of a more complex world. Students who with strong feeling of knowing (FOK) in a virtual world may be impaired by misleading feelings of knowing. Educators may be setting them up for failure by enhancing FOK to a point where such feelings are dysfunctional later on in life.
As mentioned already, the learner may be fooled by his or her own successes during training. Manipulations such as blocking practice by subtask, providing continuous feedback during training, and fixing the conditions of practice act like crutches that artificially support performance during training. When those crutches are absent in the post-training environment, performance collapses. The learner, however, will typically lack the perspective and experience to realize that he or she has not yet achieved the level of learning demanded by the post-training environment
Bjork (1994, p. 196)
Metacognition: Knowing about knowing
Excessive Collaboration Risks
Newer technologies such as email, electronic chat rooms, webcams, web documents, telephony on the Internet, etc. make it increasingly easy, efficient, collaborative, and effective to collaborate in training, education, and research. However, in increased collaboration there may be too much of a good thing. To the extent that collaborations reduce learning sweat and frustration there may be metacognitive impairments. There may also be misleading feelings of knowing arising from such collaborations. Which parties in the collaborations really had the "knowing" part of it? Collaborations may be taking some important subjective experience out of the learning process.
Individuals who have illusions of comprehension or competence pose a greater hazard to themselves and others than do individuals who correctly assess that they lack some requisite information or skill. The reading we take of our own state of knowledge determines whether we seek further study or practice, whether we volunteer for certain jobs, whether we instill confidence in others, and so forth. In general, then, as argued by Jacoby, Bjork, and Kelly (1993), it is as important to educate subjective experience as it is to educate objective experience.
Bjork (1994, p. 194)
Metacognition: Knowing about knowing
My colleagues and students will be surprised that I wrote this document. I am viewed as a long-time advocate of computer aided learning technologies. However, this document is entirely consistent with my emerging "gut feel" that many of the things I have tried are not the best for my students. Across the years of experimenting with newer technologies, I discovered the hard way that my masses of electronic transparencies, slide shows, and similar lecture aids were likely impairing rather than aiding my lectures. As I drifted away from the lecture pedagogy toward greater reliance upon an asynchronous online pedagogy, I began to suspect that I was making asynchronous learning too easy by giving my students instant access to experts (via audio, video, and searchable transcriptions) and digitized literature databases on CD-ROMs and in server files. My one feeling of satisfaction in the past year is that I no longer seek to make my courses easier for students. I assign tougher term projects that reduce rather than enhance feedback frequency by making students be more introspective and creative. To their consternation, my grading is very subjective. In my latest technology experimentation, my students express greater frustrations with course difficulty. Perhaps I wrote this metacognition document to justify my gut feel that I am doing some things better when my course evaluations decline.
My "discovery" of metacognitive and metamemory research strengthens my worries and concerns about how we are designing our computer aided training and education materials. It is terribly frustrating since these research findings destroy some of the comparative advantages of emerging educational technologies. Computers can aid in virtually all of Gardner's types of learning. For example, there are real comparative advantages in creating contextualized virtual learning worlds, showing animated and sequences of graphical images, increasing feedback frequency, making learning easier, making learning more fun, making learning more collaborative, networking students with experts, and making literature searches easier and more efficient. I am not promoting reduced experimenting and implementing paces in computer and networking technologies in education. What I am advocating is that the metacognitive principles be programmed into these technologies even though doing so may weaken short-term learner satisfaction with the program, course, and/or learning modules.
My "discovery" of the metacognitive and metamemory research strengthens my worries and concerns about how we evaluate our programs, courses, and learning modules. The frustrating part of metacognitive and metamemory criteria, when applied in higher education, is that learning performance cannot be evaluated prior to graduation or even in the earliest years of post-graduate work and study. Evaluating too soon may serve to put too much weight on the wrong criteria (e.g., satisfaction) that are short sighted from a metacognition standpoint. Our students may, thereby, be graduating with that misleading FOK that impairs metacognitive performance in their longer-term futures. But then how many of us in retrospect, years after graduation, have increased our regards for our toughest and most frustrating professors? Most likely these are the cussed professors buried deepest in metamemory.
In my viewpoint the above findings also have relevance to the findings of another AECC-funded experiment called the Project Discovery (PD) project at the University of Illinois at Urbana-Champaign. That study took a slightly different approach across multiple accounting courses to study the effects of the importance of having students learn from "complex, ill-structured, ambiguous problems and cases similar to those found in practice" as reported by Stone and Shelly (1997, p. 26). Stone and Shelly repeatedly stress that their research notes the impact of such factors on learning but make no attempt to attribute causality. I speculate that some of the causal factors are metacognitive. Even though my paper focused on the BAM Program at the University of Virginia, I think my conclusions extend to the PD Program at the University of Illinois.
Acknowledgement:
I want to thank Paula
Hertel, Professor and Chair of the Department of Psychology,
Trinity University. Dr. Hertel's specialty is memory and cognition research. She
remembered to give me some good leads for this document. She also reviewed the first
draft. All errors and shortcomings that remain are mine.
Appendix 1
Why Aren't Stories Good Food?
Email Messages About Evaluation Criteria and Processes
Email message from Bob Jensen to the AECM on June 28, 1998
My question is why aren't education researchers, journal editors, and referees listening to the experts about evaluating programs, courses, and learning modules? Why do we put up such hurdles for publication of "stories?"
This weekend I attended the Consortium of Liberal Arts College (CLAC) 1998 annual meetings held on the Trinity University Campus. This organization is comprised mainly of directors of information technology on liberal arts campuses. These are the professionals in charge of assisting faculty in the design and evaluation of newer learning technologies for synchronous and asynchronous programs, courses, and learning modules. One of the most valuable sessions was entitled "How to Evaluate Instructional Technology and How Technology can be Used in Evaluating Instruction." What struck me is the virtual consensus (among panelists and the audience) that surveys and measurements of performance are untrustworthy except for tabulations that are relatively easy to verify such as how many students own their own computers, how many students use the Windows operating system, etc. Other statistics such as the frequency of student visits to a web site or the number of messages in an email listserv or chat room are viewed suspiciously since these statistics can be artificially inflated in so many ways and say nothing about quality and content.
In their opinions, anecdotal evidence and "stories" from faculty and students were far more influential in administrative evaluations of programs, courses, and learning modules than were formalized evaluations and measurement instruments. Obviously, such presenters as Eleanor Lonske (Wellesley College), Diane Balestri (Vassar College), Phil Harriman (The College of Wooster), Charles Christison (Beloit College), and Michael Westfort (Connecticut College) admit that anecdotal evidence can be one-sided and misleading, but far more dangerous in their eyes were quantitative studies that are subject to too many intervening variables such as "times of day classes meet," "wordings of questions," "times and conditions of administering the evaluations forms," and the "personalities of combinations of students in a given course."
In double blind studies (e.g., having the same instructor teach one class using traditional materials and another class using newer technology aids) there is seldom, if ever, a "clean experiment." Too many other uncontrolled variables enter into such experiments. Also technologies change so fast that it is not clear that extrapolations apply even one semester into the future. Meetings at the CLAC conference did not mention Hawthorne effects, but these also tend to distort evaluations of programs, courses, and learning modules. Hawthorne effects are discussed in http://www.trinity.edu/~rjensen/245glosf.htm#Hawthorne Hawthorne effects tend to bias student evaluations upward in favor of newer technologies.
Maybe I am venting my weariness of having to referee my 23rd paper focused upon quantitative comparisons of distance education versus on-site courses. I'd rather compare the stories from the instructors and students. Are we imposing the statistical measurement criteria upon authors merely to satisfy some questionable biases of our refereeing process? Are our referees listening to the experts?
The other message from the CLAC experts is that evaluations should always be focused upon programs and not technologies per se.
Bob
Professor Robert E. Jensen (Bob) http://www.trinity.edu/~rjensen
Jesse H. Jones Distinguished Professor of Business Administration
Trinity University, San Antonio, TX 78212-7200
Voice: 210-736-7347 Fax: 210-736-8134
Email rensen@trinity.edu
Reply from Charles Walton [ cwalton@usa.net ]
Bob,
Yesterday, Saturday, I finished a year long engagement with a Board of Trustees of a college developing and implementing a new Board Governance system.
The major time drain in developing new budgeting and fiscal control system, was not restructuring Board budgeting, but was related dealing with their problems in instructional and non-instruction IS functions. The Board has invested a great deal of the taxpayers and students’ money in information technology with little data to support that investment. There was no significant difference in student satisfaction after implementation of technology and no significant difference in learning outcomes between sections with and without the technology. The amounts requested were staggering and those underrepresented or didn’t include maintenance.
On the administrative side, the systems are so poor that the IS vendor holds the school hostage. The costs for a Datatel or PeopleSoft extrication are totally out of proportion with the rest of the budget. The costs of salaries in phasing out the old vendor, the new vendor transition project manager, instructional computing head, and new CIO will be more than twice the costs of the President, VP of Instruction, VP for Student Affairs and VP for Business Affairs for at least the next two years.
I really feel for the Trustee who has to deal with these IS requests. I shouldn’t have to vent, it made for an interesting sabbatical with a lot of billable hours.
As I prepare to return from sabbatical, I must admit that I avoid IS issues at my college. My need to manage IS was satisfied years ago as an IS head.
Bob, Trustees are desperate for good food. Mostly, what college administrators feed them gives them heartburn. A word of praise for non paid Trustees who really attempt to do what is best for the college and the fanatic Trustees who work very hard with little support to make it happen.
Charles Walton, Ph.D., CPA [ cwalton@usa.net ]
Gettysburg College
If you aren’t familiar with CMM and ISO 9000 you are an IS dinosaur.
Reply from
Ailsa H. S. Nicholson [ ctiafm@uea.ac.uk ]I would like to endorse Bob’s sentiments as regards telling the story.
Here in the UK the CTI (Computers in Teaching Initiative) have been encouraging academics to ‘tell the story’ for the last nine years. Here at the CTI Centre for Accounting Finance and Management (CTI-AFM) we have been encouraging lecturers in accounting & business education to write ‘accounts’ of their use of technology with classes for publlication on our journal ACCOUNT.
Those who do write for our journal and who contribute papers for our conference are often very frank and reflective about the experience and share with others the downs as well as the ups. Such reports are of course invaluable as it can prevent new adopters from reinventing wheels or making avoidable errors.
‘Real evaluation’ is very long term and as you point out full of pitfalls and unsurmountable problems. Also technology and software packages change or are updated so often that any result of a long term study can be unhelpful in that they may refer to products which have in the meantime been improved or changed. Telling the story is more valuable to the practicioner .
Unfortunately as you point out it is difficult to get such reports published in mainstream journals and certainly here in the UK and probably with you in the states academics are strongly discouraged from any activity which does not lead to publication in a mainstream journal. So it is an uphill struggle to ensure a good supply of reflectve accounts - or stories - on the use of technology in teaching.
But CTI-AFM continues to seek these out and to persuade academics to write for our journal or to contribute conference papers for our annual conference in April. Details of both are on our www site - address below and we will shortly be putting ACCOUNT our journal o’ Why Aren’t Stories Good Food? n-line so that colleagues in the US will be able to view the experience of their UK colleagues.
Ailsa Nicholson
Ailsa H. S. Nicholson
CTI Accounting Finance and Management
School of Management
University of East Anglia
Norwich NR4 7TJ
Tel: 01603 592312
Fax: 01603 593343
E-mail: ctiafm@uea.ac.uk
Reply from Bob Dumouchel [ owl@callamerica.net ]
Hi Bob,
I just love this debate and as a vendor of self-study products I am anything but impartial. These studies all seem to boil down to the same result.
"Format Does Not Make a Difference". Without conducting a study I would contend that the difference when it is reported can be traced back to an improved student perception of the course.
If schools want to increase the effectiveness of their courses they should market their courses better and improve the student perception of the course after all "Perception is more important that reality". If a student starts a course believing in it, they will learn more. The strange thing about this is that at the university level the Profs credentials are usually impressive to say the least. Yet if you read the average college catalog you would never know it. When we have a product produced by someone with impressive credentials you can believe that we make a big deal out of it.
Student perception needs to be managed, courses need to be marketed. If students believe they need the course, if you create the want rather then stressing the need, the results will follow. Professional quality instruction is of course an assumption in all of this.
Good luck on your next study.
Bob Dumouchel
On With Learning Inc.
Phone (800) 272-0887 or (805) 481-0118
Fax (800) 508-0487 or (805) 481-0252
The Emperor's Naked as He Can Be
Prior to one of my technology in education workshops, Bob Anthony sent me the following questions that he expects me to answer in the workshop. Perhaps, more than me, you can help him with some of these answers. Bob does not subscribe the aecm listserv, but you can contact him at Robert N. Anthony at RNAnthony@valley.net
He's deadly serious about these questions. Bob is one of our most successful book authors and has been retired from Harvard for many years. Nevertheless, he still has a keen mind and his honest bluntness trademark. Years ago in Denmark, Bob was the little boy on the streets who yelled out" "The Emperor's not wearing any clothes."
These questions are fundamental and we only have a limited amount of time in my workshops to debate such issues. If those of you attending the workshops will prepare in advance, we will try to take up these questions as time permits.
Some of the issues he raises taken up in the above document. Other issues are taken up in its much larger companion piece at http://WWW.Trinity.edu/~rjensen/255wp.htm
Bob's message is repeated below below:
**************************************************************************************************************************************************
Bob,
I read with great interest the material you sent on July 31 and look forward to your session on August 16. The following comment relates to the "self directed learning" version of "distance learning," as distinguished from "asynchronous learning" within a school. The issues relating to each of these topics are somewhat different.
First, there is in implication in much of the material you quote that there is something new about "self directed learning". Actually, the descriptions I have seen aren’t much different from those of correspondence courses that have been offered for many years. Some correspondence schools prosper, which indicates that they are doing something that customers want. The inclusion of sound is new, but what else?
It seems clear that interactive courses will not succeed. The effort involved in responding to student queries is too great, especially in view of the difficulty of obtaining money for these responses. Do you agree? (Of course, the publisher must be prepared to correct errors.)
Many of the people you quote do not appreciate the difficulty of identifying worthwhile; programs. This is an important function provided by the successful textbook companies. I was consulting accounting editor for Richard D. Irwin for many years, and read all the accounting manuscripts submitted to them. Only a small fraction were worth publishing. There is a difference between the great teacher and the great author. Bill Paton was not a leading textbook author. Although the Irwin people were pessimistic, they published a text by [XXXXX} as a favor to me; it didn’t succeed.
Therefore, the self-directed learning programs published by universities are unlikely to do well. The university editors are amateurs. Moreover, the value of a university "brand name" is not high in accounting textbooks, and I don’t see why it should be higher on an Internet product. Many mediocre courses will nevertheless appear on the Internet because the cost is so low. (There is a company today that will scan a manuscript and publish it on the Internet, without even commenting on its quality, for $500.) .
You mention corporate programs. My impression is that corporations develop successful training programs, such as how to process information in a new purchase / accounts-payable system; but training is quite different from education. Colleges and universities focus on education.
I do believe that a few companies will publish good programs and therefore they will before long dominate the market. Probably they will be conventional publishers because of their editorial and marketing expertise. Maybe there will be a few new companies focusing exclusively on the Internet.. You mention Knowledge Universe, a company started by Milliken. I am familiar with UOL publishing. It’s too early to tell who will emerge as the leaders, corresponding to the publishers of conventional texts.
Arriving at sound conclusions to issues like the above is important to me!
I am trying to make a version of my Essentials of Accounting succeed on the Internet, on CD-ROM, or both. I hope some of the above topics will be discussed in the seminar.
. Bob Anthony.
Update Message from Robert Bjork
I read your online article with great interest. You do an excellent job of making concrete some very important points in your article, including, of course, some of the points I tried to emphasize in my 1994 article.
The other important thing that comes through in your article is that computer-based individualized learning techologies, for all of their potential, are not something magical: They are tools that can be used in counterproductive and ill-advised ways as well as in innovative and productive ways. Starting last year, for example, there was an initiative at UCLA to have an internet site for every course at UCLA. That initiative—well meaning, in my own opinion—has raised cries of alarm from certain professors about infringement of intellectual property. Those concerns are legitimate in a few cases, I think, but I (and I alone, I sometimes think) have been concerned about the kind of issues you address in your article—namely, the potential for such "hi-tech" resources to be used in ways that impede, rather than promote, learning. Instructors now get day-to-day pressures from students to put most everything on the web—overheads, outlines, lecture notes, etc. The web site then becomes a kind of remedial device. Students decide that they can skip the lecture, or, when they do attend, that they don’t need to take good notes, understand the lecture, or ask questions when things are confusing, because they can (hopefully) get a repetition on the course site.
There are, of course, creative ways to use a course site—ways that make the learner an active participant in the learning process, but such exercises/materials to enrich a course demand the professor’s time and energies and will not necessarily be used or appreciated by students, who are prone to view such enriching exercises as an additional course burden.
When I teach the graduate course on learning and memory this fall term, I would like to consider using your article as one of the packet of course readings. I printed a copy from the web site, but that copy is not one that would lend itself to being reproduced for the course. Would you be willing to either send me a hard word-processing copy or attach one to an email message?
A couple related articles of my own that appeared after the Bjork "Memory and Metamemory Considerations" article are:
Bjork, R.A. Institutional impediments to effective training. (1994). In D. Druckman and R.A.Bjork (Eds.), Learning, remembering, believing:
Enhancing human performance (pp.295-306). Washington, DC: National Academy Press.
Bjork, R. A. (in press). Assessing our own competence: Heuristics and illusions. In D. Gopher and A. Koriat (Eds.), Attention and Peformance XVII. Cognitive Regulation of Performance: Interaction of Theory and Application. Cambridge, MA: MIT Press. (41 pages)
I’ll have my assistant send copies of those articles to you.
Best regards,
RAB
Robert A. Bjork, Editor
PSYCHOLOGICAL REVIEW
Department of Psychology
UCLA
Los Angeles, CA 90095-1563
(310-825-7028; fax 310-206-5895)
Click here to return to the top of this document.