Friday, August 28, 2015
Don’t waste your reader’s time. There is a difference between reading from the printed page and reading from a screen. Reading from paper makes use of reflected light, whereas the screen is lit from behind. This makes it more difficult to read on screen, particularly when large amounts of text are involved, and where the user is required to read for long period of time. It is almost always a mistake to simply transfer text from the printed page or workbook to the screen, without taking into account this fundamental difference.
In all reading, we do not simply scan letter-by-letter, word by word, sentence by sentence. We scan, skip and look for anchors. When we read text on screens, these behaviours become more extreme. More than this, in online learning, we need to be aware of chunking, meaning, order, cues and retention. The art of writing text for online learning is a skill, especially when that text will be read across a whole range of devices and different screen sizes.
Ten quick suggestions:
1. Cut until it bleeds
Several pieces of research point towards overlong text as a destructive force in learning. Mayer, Bove, Bryman, Mars and Tapangco (1986) showed that too much text can reduce learning by up to 50%. Edit out unnecessary words, use shorter words, break down long sentences. Give yourself a maximum word length and rewrite to within that limit.
2. Chunk text
When we read to learn, text must be chunked to increase retention. This is a well researched area. Chunking text means more paragraphs. Break down the text into more paragraphs than you would on the written page. From Ebbinghaus through to Miller and many other psychologists of learning, chunking is a proven technique to increase retention.
3. More structure and meaningful headings
Bruner researched how structure (content must be structured so that it can be grasped by the learner), sequence (material must be presented in the most effective sequences) and generation (encourage extrapolation, manipulation and a filling in the gaps, just beyond the learners existing knowledge) all aid learning. Write to these rules. Like this blog post, screen text benefits from more headings. These headings act as structural cues for meaning, understanding and encoding, all of which aid retention. Don’t use headings like ‘Introduction’ ‘Section 3’ and so on. Make headings short, meaningful cues. Start with a question, capture main learning point in the heading. Use headings as they are used in this blog post.
4. Highlight key words
Highlighting ‘cue’ words within text in bold or italics. These act as cues for retention but don’t overuse, one or two at most per paragraph. Tulving explored cues in learning showing that they are vital in the process of encoding, storage and recall, and therefore retention.
You often see this in good magazine articles, quotes and main points pulled out in a separate or much larger font, even placed separately at the side of the text. This is a useful learning device that can be used to reinforce the main learning point. Again, these act as cues, which Tulving thought were essential in memory and retention.
6. Use stab points
Long paragraphs, that are in fact lists, work better on screens as stab points. These also increase encoding and retention. In one study, Mayer, et al. (1996) presented 600 pieces of scientific learning and found that briefer versions, which were concise, coherent and co-ordinated, resulted in far more effective learning. They are precise in their recommendations, ‘There is a clear pattern in which the more words added to the core verbal explanation, the more poorly the student does in producing the core explanative idea units. These results are consistent with the idea that the additional words overload verbal working memory, drawing limited attentional and comprehension resources away from the core verbal explanation.’
Formal and conversational styles (using terms such as I, me, my, you, your and we) of text have been compared by Moreno and Mayer (2000) showing that the conversational style resulted in 20-46% better learning. Nass & Reeves also produced 35 studies showing that learners see technology as a dialogue on a personal level.
8. Keep text close to image.
Mayer (1989), Mayer Steinhoff Bower (1995) and Moreno and Mayer (1999) in five separate studies compared graphics with text close to the graphics, and graphics with text below the graphics, at the foot of the screen. In all five studies, learners who used the co-located text and graphics improved their problem solving by between 43-89%. Similar results have been found by Chandler and Sweller (1991), Pass and Van Merrienboer, (1994). Making the learner’s eye jump from one part of the screen to another is disruptive and reduces the effectiveness of the learning.
9. Don’t do audio and text on screen
Words, in both text and accompanying audio narration, can hurt learning. This is interesting, as it is often assumed that one needs both to cover accessibility issues. Clark and Mayer (2003) argue that ‘audio or text on their own’ are better than ‘text and audio together’. This is confirmed by another study by Kalyuga, Chandler and Sweller (1999) where the group with audio scored 64% better than the group with both text and audio. They claim that one or other is redundant and will overload the visual and aural channels.
10. Don’t use unnecessary graphics
A review of studies around this concept, known as the redundancy effect, by Sweller et al. (1998) cites a list of research studies that all point to the damage done to learning when redundant material interferes with the efficacy of the learning. For example; they illustrate a point about leaving out extraneous or distracting graphics in media with an experiment, conducted by Harp and Mayer (1997), in which students were given a text to read on lightning strikes. Students who read the passage accompanied by elaborate colour photos with additional captions - as opposed to the text with simple graphics - showed 73% less retention of knowledge and 52% fewer solutions on a transfer test.
The problem, in many online learning courses and resources, is not that they are in text but that there is too much text and that it is badly written and presented. A good editor can often take subject matter text, book text or initial documents and use these techniques to make the text screen-ready. When I hired interactive designers, I gave them all literacy skill tests and was keen to see evidence of their ability to write well and edit. In many ways this is a primary skill with screen friendly writing something that had to be acquired. A good editor will reduce text like a good sauce and the learner invariably benefits in terms of understanding, encoding, retention and perseverance.
Wednesday, August 26, 2015
Top 10 stupid mistakes in design of Multiple Choice questions
Some online learning designers write their assessment items first, as they can match the assessment to the objectives or competences before being distracted by the detail in content. This avoids the trap of writing test items that simply test atomic facts and words from the presented text.
Many test items are quite simply not fit for purpose, as they don’t really assess, test the wrong thing or are so badly designed that they mislead or annoy learners. Bad test item writers tend to simply pluck out key words from the text and test the recognition or the meaning of those terms. Good test item writers, and this is a skill that few subject matter experts possess, understand good design and the need to test deeper understanding.
Here’s a few rules to follow if you’re new to this game:
1. Test understanding, not facts
What’s easy is to simply extract all the nouns, objects and quantities, then testing for recall. The trick is to push beyond this to test understanding. It’s not the ‘what’ but the ‘why’ that often matters but remains untested. What’s hard is to write questions that really do test the learner’s actual knowledge and ability to apply that knowledge. Professor Dylan Wiliam, calls great questions ‘hinge questions’, as they literally diagnose poor understanding. He explains how one can use these questions as powerful verbal test items in a classroom, where it is difficult to diagnose 30 kids quickly but the principle applies just as well to online learning. Here’s an example:
The ball sitting on a table is not moving. It’s not moving because:
A. No forces are pushing or pulling on the ball.
B. Gravity is pulling down, but the table is in the way.
C. The table pushes up with the same force that gravity pulls down.
D. Gravity is holding it on to the table.
E. There’s a force inside the ball keeping it from rolling off the table.
This question not only catches common misconceptions, it diagnoses between those who have understood the ‘physics’ and those who have not. There are forces at play, two in fact, so C is correct. The other options introduce common misconceptions about the absence of forces in stationary objects (A), single forces (B & D) or inner forces (E).
There are four components that make great questions:
Make them THINK
Give helpful FEEDBACK
Keep it CONVERSATIONAL
A really bad question will allow the learner to guess the right answer or simply test recognition from a list. A good question will make the learner think, a great question will make them look away, even close their eyes to recall what they know, do something in their head and move towards the answer. It will push the learner.
Let’s try this:
A bat and ball cost £1.10.
The bat costs one pound more than the ball.
How much does the ball cost?
This question really is a test of a learner’s grasp of ‘ratios’ in maths, it does a great job, as it really requires you to know how to apply a principle in mathematical thinking. The answer is not 10p as many choose. Think about it. If the ball was 10p, the bat, if it were a pound more, would be £1.10p, making a total of £1.20. The right answer is 5p. If the ball was 5p, the bat, if it were a pound more, would be £1.05p, making a total of £1.10. That’s a great question.
Remember also, that a great question also needs great feedback. In fact, it is better to call these, not questions, but ‘test items’ as you need, not only the question but the feedback. So let’s think about explanatory feedback for this question, that allows the learner to learn from the question.
The bat costs one pound more than the ball.
How much does the ball cost?
20p -> Try again.
15p -> Try again.
10p -> Try again.
5p -> Correct.
This doesn’t really give any help to the learner who has jumped to the wrong conclusion or hasn’t thought deeply enough about the question. We need to provide HELPFUL feedback.
The bat costs one pound more than the ball.
How much does the ball cost?
20p -> 20p+120p=140p Try again.
15p -> 15p+115p=130p Try again.
10p -> 10p+110p=120p Try again.
5p -> Correct. 5p+105p=110p
The feedback here tries to unpack the nature of the problem for the learner and gives an explanation as to why they are wrong.
Now let’s add another touch, that human element. Make it CONVERSATIONAL. In conversation, you wouldn’t ask a question like this and simply feedback the statement ‘10p+110p=120p Try again.’ You’d make it a bit more conversational and friendly.
20p -> If the ball was 20p, and the bat is £1 more at 120p. The ball (20p) plus bat (120p) gives a total of 140p. Try again.
15p -> If the ball was 15p, and the bat is £1 more at 115p. The ball (15p) plus bat (115p) gives a total of 130p. Try again.
10p -> If the ball was 10p, and the bat is £1 more at 110p, the ball (10p) plus bat (110p) gives a total of 120p. Try again.
5p -> Correct. If the ball was 5p, and the bat is £1 more at 105p, the ball (5p) plus bat (105p) gives a total of 110p
Now be a little more POSITIVE:
20p -> Sorry. Most people get this wrong on first attempt. Let’s see why this is wrong. If the ball was 20p, and the bat is £1 more at 120p. The ball (20p) plus bat (120p) gives a total of 140p. Think of the ratio of the cost between the ball and bat within the 110p cost. Try again.
15p -> Sorry. Most people get this wrong on first attempt. If the ball was 15p, and the bat is £1 more at 115p. The ball (15p) plus bat (115p) gives a total of 130p. Think of the ratio of the cost between the ball and bat within the 110p cost. Try again.
10p -> Most people get this wrong first time around and choose this answer. If the ball was 10p, and the bat is £1 more at 110p, the ball (10p) plus bat (110p) gives a total of 120p. Think of the ratio of the cost between the ball and bat within the 110p cost. Try again.
5p -> Well done. If the ball was 5p, and the bat is £1 more at 105p, the ball (5p) plus bat (105p) gives a total of 110p.
You get the idea. Make test items meaningful, not mundane, and be helpful, fulsome and conversational in the feedback. This type of approach to test items will make the learner feel good and propel them forward, rather than act as barriers or make them feel like failures.
2. Binary Choice language
We don’t ask each other questions in normal language using the terms True and false, so consider more natural terms such as Yes/No, Agree/Disagree
Margaret Thatcher was the only British woman Prime Minister.
Yes No or Agree Disagree
3. Binary choice mistakes
Binary choice questions are often too long or turn into complex tests of logic by having two or more ideas expressed. Using NOT in a binary choice simply leads to a logic test, so best avoided. Finally, and I have seen this, asking the user to try again is stupid.
4. Edit out unnecessary words in options
How many earths would fit into the sun?
This is a good question, a real test of your knowledge of the relative sizes of the earth and sun. However, take out the repeated word of each of the options, as there is no need to make the learner go to that extra effort in reading the same word five times.
Approximately how many earths would fit into the sun?
And with meaningful feedback….
1000 -> Sorry. It is easy to underestimate the size of the sun. The Sun has a radius 100 times that of the Earth. Try again.
10,000 -> Sorry. It is easy to underestimate the size of the sun. The Sun has a radius 100 times that of the Earth. Try again.
100,000 -> Sorry. It is easy to underestimate the size of the sun. The Sun has a radius 100 times that of the Earth. Try again.
1,000,000 -> That’s right. In fact, you could fit 1.3 million Earths inside the Sun! It’s huge.
5. Make question grammatically agree with all answers
In golf, a one under par is a:
The word ‘a’ agrees with two of the options but not the other two. It’s grammatically wrong, as well as almost giving away the right answer!
What do you call one under par in golf?
6. Avoid negative questions
Avoid having NOT in the question, (Which of the following is not…..) unless they really are testing a competence than involves a common error or misconception. The ‘not’ can be easily missed by the learner, so capitalize if used. These questions often simply reinforce irrelevant negative information, rather than testing much that is useful.
7. Avoid making the longest option the answer
Go through the assessment items and click on just the longest option. You’d be surprised to see how often these are weighted towards the correct answer, as designers often write this first, then come up with easier, shorter distractors. Poundstone showed that this was common, even in high stakes tests. Options must be roughly the same length, of similar grammar and random in order of presentation.
8. Avoid ‘All of the above’ and ‘None of the above’
These are often imprecise tests and you don’t know what actually misled the learner if they get it wrong. They are also easy to write, so there is a tendency for poor test writers to have these as their ‘right’ answer, a fact shown by Poundstone’s analysis of thousands of test items.
9. All options should be believable.
Answers that look obviously wrong are pointless or even worse, patronizing and condescending. These are two questions from BBC Bitesize:
What is magma?
A chocolate ice cream
Bubbles of gas
What are the tiny air sacs in the lungs called?
I rest my case. A lesser but similar error is to make one or more of the options too obviously right or wrong.
10. Test your test items
First test your test items with an expert, namely an experienced test-item writer. It is a skill that takes some time to master. Just because you know your subject, does not mean that you know how to assess and write good test items. Second, try the questions out with real target learners and ask them to ‘voice’ what they’re thinking. They invariable say things like ‘I don’t understand this question… What do you mean by…”. Don’t argue, just change it.
Questions encourage, not just the acquisition of learning but also critical thinking. Questions stimulate curiosity. Questions can intrigue and pull the learner towards exploring a subject in more detail. Good questions can be learning experiences in themselves not just test items. They can be used to stimulate thought but also, through good feedback, explain to the learner why they have a misconception, even act as a way of getting a point across. Questions drive learners forward. Good questions diagnose strengths and weaknesses. You don't know what you don't know and questions uncover the often uncomfortable truth that you know less than you thought you know. Finally, questions can assess and determine whether a learner is competent against certain criteria.
Questions really do matter in learning, that’s why it is important to design them so that they are fit for purpose. Professional examination bodies all too frequently write questions that are poorly designed and in some cases, unbelievably, impossible to answer. Experts in their subjects, often write poor test items, as it is difficult to put yourself in the shoes of a novice and assessment skills are different from subject matter expertise. In online learning, even professional vendors often produce badly designed test items, due to a lack of interactive design expertise. In many ways the test items are more important than the content.
Friday, August 14, 2015
7 ways women are ignored in education from Jane Roland Martin
Jane Roland Martin is a remarkable woman as she widened up educational theory in line with those such as Dewey and Illich. yet her real contribution is in the many issues around women in education. Her views are inclusive but above all expansive.
1. Women ignored
1. Women ignored
She saw the role of women as a largely unexamined issue in education. Women had for centuries been excluded from access to education but even recently, ignored in teaching, schooling, higher education.
2, New sources
As women have been denied access to education, excluded from education, educational institutions and the vehicles of educational theorising, she recommends that we must look to wider sources for the views, in magazines, pamphlets, more general literature and the wider media landscape.
3. The educational ideal ignores women
The definition of education itself, Martin claims, has neglected women. Work by women and work about women are often ignored, with the role of women in education hugely under researched and understood. This, she thinks, distorts and misrepresents reality. The very idea of an educated person or educational ideal must take women into account.
4. Spectator knowledge
Martin excavated, in an anthropological manner, hidden assumptions and problems within education, especially the curriculum of fixed cultural subjects and knowledge, as well as male cognitive perspectives, at the expense of other disciplines and real-world knowledge and skills. She saw curricula and institutions, that focus largely on ‘spectator’ knowledge, as promoting a one-sided view of education. In particular she came to focus on gender through feminism.
5. Education to include domestic context
For Martin, the school as home should be far more relevant to life outside of school, especially the domestic environment of the home. It needs to draw back from a curriculum that focuses largely on ‘spectator’ knowledge. The role of mothering, issues around the ghettoisation of women on professions such as teaching and nursing, as well as the insensitivity to women’s often difficult role at work and at home, led her to a radical stance on taking action across a broad front to rebalance the educational system, in response to this analysis.
6. Multiple Educational Agency
Yet the education debate so often focuses solely on schools and schooling. Dewey and many others have downplayed the role of school in the development of young people. For Martin this narrow and obsessive emphasis on schools and schooling may result in cultural miseducation. Parents, family, community, church, youth groups, sports clubs, the media, internet and other institutions, even the education of animals, have played roles in education long before schools and schooling were common. Martin has an expansive view of learning as a process of Multiple Educational Agency. To focus just on schools, which are notoriously difficult to change, is to ignore the very many other opportunities to educate
7. Learning happens everywhere
Above all she reconceptualises education as something that is all-pervasive, often informal, through many agents in many different contexts. The idea that education is a fundamental aspect of life that lasts for the whole of one's life underpins her entire re-evaluation.
Critics have claimed that her work is so general that it is difficult to either object or test. Others claim that her widening education to include, for example animals, is taking the inclusivity idea too far. Others, while agreeing with Martin’s descriptive work, have argued that Martin’s prescriptive recommendations around the reconstitution of education are either impractical, simply reinforce inequalities or replace them with new ones. Her work certainly aroused debate and passions.
Martin may not be regarded as the only or even leading voice of feminism in education but her reputation as a dogged researcher, determined to re-establish historical wrongs and implement future rights is unarguable. The re-evaluation of Plato, Rousseau, James, Wollstonecraft, Montessori, Beecher and Gilman is one contribution. Stimulating feminist and gender debate and research is another. Widening the view of education beyond schooling is another. Above all she reconceptualises education as something that is all-pervasive, often informal, through many agents in many different contexts. This was Martin’s great achievement, simply opening the world of education up to a much wider set of perspectives around gender, women and what it is to educate and be an educated person.
Martin, J. R. (1985). Reclaiming a Conversation: The Ideal of the Educated Woman. New Haven: Yale University Press.
Martin, J. R. (1992). The Schoolhome: Rethinking Schools for Changing Families. Cambridge, MA: Harvard University Press, 1992.
Martin, J. R. (1994). Changing the Educational Landscape: Philosophy, Women, and Curriculum. New York: Routledge, 1994.
Martin, J. R. (2000). Coming of Age in Academe: Rekindling Women's Hopes and Reforming the Academy. New York: Routledge.
Martin, J. R. (2002). Cultural Miseducation: In Search of a Democratic Solution. New York: Teachers College Press.
Martin, J. R. (2011). Education Reconfigured: Routledge.