03 / 09Study

Turn notes into questions

A lecture ends with three pages of notes. They look complete, and on the train home they get read again. A week later, in a quiz, one question asks for a thing those pages covered, and the answer is somewhere near the tip of the tongue. At work the same thing happens after a training day, or before a meeting with a client whose details matter. The notes were there the whole time. What was missing was ever having to pull the answer out without them. This lesson turns notes into questions, so that each review is a small test. The test is the part that does the work. Reading the notes again feels like work and does much less.

In five minutesAfter your next class or meeting, write three questions about it on a scrap of paper. Tomorrow morning, answer them without looking, then check your notes.

Method · Active recall  /  Skill · Recall  /  Medicine · Research psychology

↺From last time · Why rereading feels like learning

Answer these from memory before you start. Tap to check.

Why does a second read feel like learning when it may not be?

The page is familiar, so it reads smoothly, and smooth reading feels like knowing. Familiar and remembered are different things. Only recall with the book shut shows which one you have.

Why recite before checking, instead of checking as you go?

Recall with the page out of sight is the practice that makes material last. A glance turns it back into reading. The check afterward catches gaps and errors so a wrong version doesn't stick.

When can rereading still help?

After a gap of days, and on the parts you missed. A second read straight after the first helped on a test that day, and the advantage was gone two days later.

01Where it comes from

The observation is old. Francis Bacon wrote in 1620 that reading something ten times while trying to repeat it in between beats reading it twenty times, and William James said much the same in 1890 (see Voices). Psychologists call it the testing effect, or retrieval practice. Henry Roediger and Jeffrey Karpicke at Washington University in St. Louis brought it into study research with prose passages in 2006 [1]. Medicine is where students took it up on their own and made it routine. Medical students preparing for the US licensing exam make or share flashcard decks, often in the program Anki, and work through practice question banks. In one survey of 72 students at one school, every student used practice questions, about 3,870 each on average, and 31% used Anki [2].

Also called Retrieval practice, practice testing, the testing effect, self-testing, flashcards.

Ten readings with recital

Thus, if you read anything over twenty times, you will not learn it by heart so easily as if you were to read it only ten, trying to repeat it between whiles, and when memory failed looking at the book.

Francis Bacon, Novum Organum, Book II Aphorism XXVI · tr. James Spedding

Bacon uses memory as a worked example of his method for sorting cases into kinds. He finds six aids to memory. This one belongs to things "waited for" that "raise the attention." The reader who tries to repeat the passage has to wait for it to come, and checks the book only when memory fails. Four centuries later the same comparison, fewer readings with recall against more readings alone, is the basic design of testing-effect studies.

Read it in full

Active over passive

A curious peculiarity of our memory is that things are impressed better by active than by passive repetition.

William James, The Principles of Psychology, Volume 1 Chapter XVI, Memory

James goes on to describe learning a piece by heart. When it is almost known, he writes, it pays to wait and recall it "by an effort from within" before looking at the book again. Words recovered that way will probably come the next time. Words looked up will likely need the book again. His explanation was about paths in the brain. The behavior he describes is the one this lesson builds.

Read it in full

02The method

Active recall

  1. After a lecture, chapter or meeting, read through your notes once and turn each main point into a question. Question on the front, answer on the back.
  2. Keep one idea per question. Ask why and how, as well as what.
  3. Cover the answers. Answer each question from memory, out loud or on paper, before you look.
  4. Check every answer. Mark the ones you missed or only half got.
  5. Go back over the marked ones until you have answered each correctly once.
  6. Come back to the whole set on a later day. The next lesson, Space it out, covers when.

03An example

What it looks like done

Worked on Chapter 1 of the Enchiridion, about 400 words, as six question cards written on a Sunday and answered on Monday

Card 1

Q What does Epictetus say is in our power?

A Opinion, movement toward a thing, desire, aversion. In a word, "whatever are our acts."

Card 2

Q What does he say is not in our power?

A The body, property, reputation and offices. Whatever are not our own acts.

Card 3

Q Why does he call the things in our power free?

A By nature no one can restrain or hinder them. The others are in the power of others.

Card 4

Q What happens to someone who treats what belongs to others as their own?

A They will be hindered, lament, be disturbed, and blame both gods and men.

Card 5

Q Why can't someone aim at the great things and at office and wealth too?

A Chasing both, they may miss even office and wealth, and will fail at the things that bring happiness and freedom.

Card 6

Q What is the first test for a harsh appearance?

A Ask whether it relates to what is in our power. If not, be ready to say it does not concern you.

Answered the next day

  • Right: 1, 2, 4, 6.
  • Half: 3. I had "free" but not why.
  • Missed: 5. I remembered "great things" and nothing after.
  • Cards 3 and 5 go in the misses pile for Saturday.
The text: The Enchiridion, Chapter 1

04Try it · 25 minutes

For a class

Within a day of your next lecture, spend ten minutes writing eight to ten questions from your notes, one idea each. The next day, answer them on a blank page without looking. Check, and keep the misses in a separate pile for the weekend. To practice first on something short, read chapter 1 of Epictetus's Enchiridion, write five questions, and answer them an hour later.

Open it in the reader: The Enchiridion, Chapter 1

For work and life

After a training session, a meeting, or a chapter of a book for work, write five questions a colleague might ask you about it. The next morning, answer them from memory before you open your notes. Check each answer. The ones you could not answer show exactly where to look again. About 15 minutes across the two days.

The blank page for this is in the Study workbook.

05Where it goes wrong

  • Questions that copy the sentence. A card that blanks one word from a copied sentence tests whether you recognize the wording. Ask for the idea in your own words.
  • Turning the card too soon. Looking before you have committed to an answer turns recall back into reading. Say or write the answer first.
  • Answers a paragraph long. A big answer gets half-recalled and marked right. Split the card until each has one answer you can check.
  • No checking. Recall without feedback can keep a wrong answer alive. Check each one against your notes.
  • Only definitions. Questions that ask what a word means miss how ideas connect. Mix in why, how, and what happens next.

06Research

What research has measured

Testing compared with restudying

Shown across hundreds of studies that answering from memory leads to better long-term retention than studying the same material again.

The numbers and sources
Effect
Across 159 effect sizes, testing beat restudying by g = 0.50, and recall tests gave larger benefits than recognition tests [3]. A second meta-analysis found g = 0.61 for practice tests against all comparison conditions, and g = 0.51 against restudying [4]. A week after studying a passage, students who had recalled it three times remembered 61% of it, against 40% for students who had read it four times [1].
Population
61 studies [3]. 118 articles with 272 effect sizes and 15,427 participants [4]. 180 undergraduates [1].
Caveats
Much of this is laboratory work with college students and short materials, and effects varied widely [3][4]. On a test given minutes after study, rereading can come out ahead [1].

Quizzes in real classes

Shown that quizzing in real classrooms raises achievement by a medium amount, with the size depending on how the quizzes are run.

The numbers and sources
Effect
Across 222 classroom studies, quizzing raised achievement by g = 0.499. The effect depended on what the comparison group did, whether quiz and exam formats matched, whether corrective feedback was given, and how many times students were tested [5].
Population
48,478 students in 222 independent studies [5].
Caveats
One meta-analysis pooling many designs and subjects. The many moderators mean the average hides a wide spread [5].

In medical training

One trial found doctors in training remembered more of a teaching session six months later after repeated tests on it than after repeated study of a review sheet.

The numbers and sources
Effect
Repeated testing with feedback gave final scores of 39%, against 26% for repeated study of identical information, d = 0.91, more than six months after teaching [6]. In a survey, the number of practice questions completed and the number of unique Anki cards seen each predicted licensing exam scores [2].
Population
40 pediatric and emergency medicine residents [6]. 72 medical students at one school [2].
Caveats
Small samples at single sites. The survey is correlational, so students who use more questions may differ in other ways [2].

07Check yourself

Answer from memory first. The next lesson asks these again.

Why does answering from memory help more than reading the answer again?

Pulling an answer out without the notes is what makes it last. In studies, recalling a passage beat rereading it on tests a week later, even though rereading looked better minutes after.

Why keep one idea per card?

A long answer gets half-recalled and marked right. With one idea per card, each answer is clearly right or wrong, so the misses show exactly what to study again.

Why check every answer?

Recall without feedback can keep a wrong answer alive. Checking against the notes corrects it before the next review.

Sources
  1. Roediger HL III, Karpicke JD. Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science. 2006;17(3):249-255. https://doi.org/10.1111/j.1467-9280.2006.01693.x
  2. Deng F, Gluckstein JA, Larsen DP. Student-directed retrieval practice is a predictor of medical licensing examination performance. Perspectives on Medical Education. 2015;4(6):308-313. https://doi.org/10.1007/s40037-015-0220-x
  3. Rowland CA. The effect of testing versus restudy on retention: A meta-analytic review of the testing effect. Psychological Bulletin. 2014;140(6):1432-1463. https://doi.org/10.1037/a0037559
  4. Adesope OO, Trevisan DA, Sundararajan N. Rethinking the use of tests: A meta-analysis of practice testing. Review of Educational Research. 2017;87(3):659-701. https://doi.org/10.3102/0034654316689306
  5. Yang C, Luo L, Vadillo MA, Yu R, Shanks DR. Testing (quizzing) boosts classroom learning: A systematic and meta-analytic review. Psychological Bulletin. 2021;147(4):399-435. https://doi.org/10.1037/bul0000309
  6. Larsen DP, Butler AC, Roediger HL III. Repeated testing improves long-term retention relative to repeated study: A randomised controlled trial. Medical Education. 2009;43(12):1174-1181. https://doi.org/10.1111/j.1365-2923.2009.03518.x

MVIII puts the day in front of you each morning. Get MVIII