06 / 10Notes

Notes for problems

Some subjects do not come as paragraphs. A statistics class, a physics problem set, a tax form, a spreadsheet formula that will not work, a page of code. The lecture shows a problem solved on the board, and the notes copy every line. Days later the homework has a problem that looks almost the same. The copied lines do not say why the third step was a division and not a multiplication. Notes for this kind of material keep the steps and the reason for each step. They also keep a record of every problem that went wrong: what was done, where it went wrong, and a date to try it again from a blank page. Prose notes ask what an author claimed. Problem notes ask what to do next, and why.

In five minutesTake one problem you got wrong this week. On a card, write the problem, the step where it went wrong, and the rule you missed. Add a date three days out, and redo it then without looking.

Method · Worked examples and an error log  /  Skill · Recall  /  Teaching · Research psychology

↺From last time · Notes from a video

Answer these from memory before you start. Tap to check.

Why pause the video instead of writing while it plays?

While it plays, the speaker keeps going and attention splits. A pause gives time to recall the chunk and put it in your own words, and lessons taken in segments are learned better.

What does writing a question for each chunk add?

It turns the notes into a test for later. In one study, short quizzes between parts of a lecture reduced mind wandering and improved learning of the parts that followed.

When do captions help, and when do they get in the way?

They help catch terms and fast speech, and help most in a second language or a new topic. Copying them word for word skips deciding what mattered.

01Where it comes from

In 1985 John Sweller and G. A. Cooper, at the University of New South Wales, ran five experiments on algebra [1]. Students who studied worked examples, problems already solved step by step, needed considerably less time than students who solved the same problems. They then solved similar problems faster and with fewer errors. The gain held only for problems with the same structure. In 1989 Michelene Chi and colleagues listened to students think aloud while studying worked examples in mechanics. The stronger students explained each step to themselves and tied it to a principle in the text. The weaker ones explained less and leaned on the examples later [2]. Alexander Renkl found the same pattern in probability, with study time held equal [3]. A 2000 review by Robert Atkinson and colleagues set out how to use examples well: several per problem type, each paired with a practice problem, studied with explanations [4]. Labeling the parts of a solution is covered in the Systems course. The error log is a teacher's tool more than a research finding. We found no controlled study of a student's error log as such.

Also called Studying worked examples, self-explanation, mistake log, error notebook, error journal.

02The method

Worked examples and an error log

  1. Write one step of a solution per line, with room to its right.
  2. Beside every step, write why it is taken: the rule, formula or reason. "Divide by 0.8 to undo the discount."
  3. Before reading the next step of a worked example, cover it and say what you expect it to be.
  4. After studying an example, solve a similar problem with the example covered.
  5. Log every problem you miss: the problem, what you did, where it went wrong, the rule or step you missed, and a retry date a few days out.
  6. On the retry date, solve it again from a blank page. If it goes right, retire it. If not, log it again with a new date.

03An example

What it looks like done

Worked on A percent problem from a business math course: one worked example studied with reasons, then one missed homework problem in the error log

Worked example, with the why

  • A price rises 25% to $150. What was it before?
  • Step 1: new = old × 1.25. Why: a 25% rise keeps all of the old price and adds a quarter.
  • Step 2: old = 150 ÷ 1.25. Why: undo a multiplication by dividing.
  • Step 3: old = $120. Check: 120 × 1.25 = 150.

Problem

  • A jacket costs $80 after a 20% discount. What was the original price?

What I did

  • 80 × 1.20 = $96.

Where it went wrong

  • I added 20% of the sale price. The discount was 20% of the original, not of $80. Check: $96 less 20% is $76.80, not $80.

The rule or step I missed

  • After a 20% discount, the sale price is 80% of the original. So original = 80 ÷ 0.80 = $100. Check: 100 × 0.80 = 80.
  • Same move as the worked example: write the change as a multiplication, then divide to undo it.

Retry on

  • Thursday, from a blank page, with new numbers: a coat costs $63 after a 30% discount.
  • Thursday: 63 ÷ 0.70 = $90. Check: 90 × 0.70 = 63. Right the first time. Retired.

04Try it · 30 minutes

For a class

Take the next problem set for a math, science, accounting or programming course. Pick two worked examples from the lecture or textbook and write the reason beside every step, covering each step before you read it. Then do the set. For each problem you miss, fill one row of an error log with a retry date three days out. Retry on that date from a blank page. About 30 minutes, plus the retries.

For work and life

Use it on a skill that runs in steps: a spreadsheet formula, a tax form, a recipe scaled up, a bug you fixed. Write the steps of one solution with the reason beside each. Keep an error log for a month: each mistake, what you did, where it went wrong, and the rule you missed. Retry each one a few days later without notes.

The blank page for this is in the Notes workbook.

05Where it goes wrong

  • Copying the board line by line. Steps without reasons look complete and say nothing about the next problem. Write why beside each step, even in a few words.
  • Reading examples like prose. Reading every step straight through skips the prediction. Cover the next step and say what it should be first.
  • Logging only the right answer. Writing the correct solution over the wrong one hides what happened. Keep what you did, and name the exact step.
  • Retrying the same day. The fix is still in view, so the retry tests nothing. Leave a few days and start from a blank page.
  • Examples forever. Worked examples help most early in learning a skill. Once the steps come easily, switch to solving problems and keep the log going.

06Research

What research has measured

Studying worked examples

Shown that studying worked examples improves mathematics performance by a medium amount, from elementary school through college.

The numbers and sources
Effect
A meta-analysis of 55 studies and 181 effect sizes found an average effect of g = 0.48 on mathematics performance. Correct examples alone did better than incorrect examples alone or a mix. The effect was similar whether examples were used for practice or for first learning a skill [5]. In the 1985 algebra experiments, studying examples took considerably less time than solving problems, and later similar problems were solved faster and with fewer errors [1].
Population
Students from elementary grades to postsecondary, in 43 articles screened from 8,033 abstracts [5]. Year 9, Year 11 and university mathematics students in [1].
Caveats
In [1] the gains held only for problems with the same structure as the examples. The review in [4] ties worked examples to the early stages of a skill.

Explaining the steps to yourself

Shown that prompts to explain material to yourself improve learning across many tasks. In math worked-example studies, adding such prompts did not add to the benefit.

The numbers and sources
Effect
A meta-analysis of 69 effect sizes from 64 reports found an overall effect of g = 0.55 for self-explanation prompts, across solving problems, studying worked problems and studying text [6]. A meta-analysis of worked examples in mathematics found that conditions adding self-explanation prompts did worse than worked-example conditions without them [5]. In one study of 36 university students learning probability, the quality of their own explanations predicted learning, with time on task controlled [3].
Population
Learners at several levels of education [6]. Elementary to postsecondary math students [5]. 36 first-year education students [3].
Caveats
The prompts in [6] were written by instructors. The math result in [5] compares across studies and is not a direct test. The explanations that went with learning in [2] and [3] came from the students themselves, unprompted.

Learning from errors

Shown that training built on making and learning from errors improves transfer to new tasks. Errors followed by corrective feedback can help. No study tested an error log itself.

The numbers and sources
Effect
A meta-analysis of 24 studies with 2,183 trainees found error management training beat the comparison trainings, d = 0.44 overall. The effect was larger after training (d = 0.56) and on tasks unlike the ones trained (d = 0.80) [7]. A review of laboratory work found that errors followed by corrective feedback help learning, especially errors made with high confidence. The review calls corrective feedback crucial, including an analysis of the reasoning that led to the mistake [8]. In algebra, explaining incorrect examples helped conceptual understanding [9].
Population
2,183 trainees in 24 studies [7]. Laboratory studies reviewed in [8]. Algebra I students using a computer tutor in two experiments [9].
Caveats
An error log combines feedback, an analysis of the mistake and a delayed retry. Each part has support. The log as a whole has thin direct evidence. The math meta-analysis found incorrect examples weaker than correct ones for performance [5].

07Check yourself

Answer from memory first. The next lesson asks these again.

Why write a reason beside every step?

Steps alone fit only the problem they came from. The reason, the rule each step applies, is what carries over to the next problem that looks different.

Why cover the next step of a worked example before reading it?

Predicting the step makes you explain the example to yourself. Students who learn most from examples tend to anticipate the next step and name the principle behind it.

What does an error log add to simply redoing the homework?

It names the exact step that went wrong and the rule missed, then sets a retry from a blank page days later. The mistake becomes feedback and a delayed test.

Sources
  1. Sweller J, Cooper GA. The use of worked examples as a substitute for problem solving in learning algebra. Cognition and Instruction. 1985;2(1):59-89. https://doi.org/10.1207/s1532690xci0201_3
  2. Chi MTH, Bassok M, Lewis MW, Reimann P, Glaser R. Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science. 1989;13(2):145-182. https://doi.org/10.1207/s15516709cog1302_1
  3. Renkl A. Learning from worked-out examples: A study on individual differences. Cognitive Science. 1997;21(1):1-29. https://doi.org/10.1207/s15516709cog2101_1
  4. Atkinson RK, Derry SJ, Renkl A, Wortham D. Learning from examples: Instructional principles from the worked examples research. Review of Educational Research. 2000;70(2):181-214. https://doi.org/10.3102/00346543070002181
  5. Barbieri CA, Miller-Cotto D, Clerjuste SN, Chawla K. A meta-analysis of the worked examples effect on mathematics performance. Educational Psychology Review. 2023;35(1):11. https://doi.org/10.1007/s10648-023-09745-1
  6. Bisra K, Liu Q, Nesbit JC, Salimi F, Winne PH. Inducing self-explanation: A meta-analysis. Educational Psychology Review. 2018;30(3):703-725. https://doi.org/10.1007/s10648-018-9434-x
  7. Keith N, Frese M. Effectiveness of error management training: A meta-analysis. Journal of Applied Psychology. 2008;93(1):59-69. https://doi.org/10.1037/0021-9010.93.1.59
  8. Metcalfe J. Learning from errors. Annual Review of Psychology. 2017;68:465-489. https://doi.org/10.1146/annurev-psych-010416-044022
  9. Booth JL, Lange KE, Koedinger KR, Newton KJ. Using example problems to improve student learning in algebra: Differentiating between correct and incorrect examples. Learning and Instruction. 2013;25:24-34. https://doi.org/10.1016/j.learninstruc.2012.11.002

MVIII puts the day in front of you each morning. Get MVIII