08 / 08Systems

Review and adjust

At the end of a project, a trip or a semester, the usual question is "how did it go?" The usual answer is "fine" or "rough", and neither one changes the next time. A review asks narrower questions. What was supposed to happen? What actually happened? Why the difference? What will we keep, and what will we change? Fifteen minutes on those after a group project, a big presentation or a family move turns one experience into a better plan for the next. The same questions work on a whole week. A week has parts, flows, loops and bottlenecks, like any system in this course. The weekly review is where it gets adjusted. This lesson closes the course there.

In five minutesTonight, write four lines about today: what you meant to do, what happened, why the two differed, and one thing to keep or change tomorrow.

Method · After-action review  /  The military · Research psychology

↺From last time · Track it

Answer these from memory before you start. Tap to check.

Why mark slips each evening instead of once at the end of the week?

By Sunday the week has blurred. An evening mark records the day while it is still clear, and a row of days shows which ones slip and what they share.

Why give one row the main attention each week?

Franklin found that guarding one fault left him open to another. Focusing on one row while the rest are only recorded keeps the effort small enough to keep up.

In the research, what made monitoring progress help more?

Writing the information down, and reporting it to others or making it public. How often people monitored also explained much of the benefit.

01Where it comes from

The after-action review comes from the US Army, which has used it to improve performance in training for decades [1]. The Army's Training Circular 25-20, A Leader's Guide to After-Action Reviews, was issued on September 30, 1993 [2]. A guide adapted from it for US schools says the method grew up at the Army's National Training Center, with the unit that played the opposing force in training exercises, and that companies such as Shell Oil, Colgate-Palmolive and Harley-Davidson later took it up [2]. Medicine and aviation use the same practice under the name debrief [1]. Two meta-analyses have since measured what debriefs do for performance [1][3]. The habit is older than the name. In chapter 9 of his Autobiography, Benjamin Franklin set aside the evening for an "Examination of the day," following the advice of Pythagoras's Golden Verses [5].

Also called AAR, debrief, after-event review, weekly review, retrospective, lessons learned.

The evening question

Question. What good have I done to-day? 6, 7, 8, 9. Put things in their places. Supper. Music or diversion, or conversation. Examination of the day.

Benjamin Franklin, Autobiography Chapter 9, "Plan for Attaining Moral Perfection," his scheme for the twenty-four hours of a day

Franklin's plan for a day opens with a morning question, "What good shall I do this day?", and closes with this one. Between them the day's business gets contrived in the morning and examined at night. The review sits beside supper, music and conversation, as one ordinary part of an evening. The morning question sets the plan and the evening one checks it, the same pairing as an after-action review.

Read it in full: Autobiography of Benjamin Franklin, Chapter 9. Plan for Attaining Moral Perfection

02The method

After-action review

  1. Set a time within a few days of the event, or at the end of each week. Bring everyone who took part, or just yourself.
  2. Write what was supposed to happen: the goal, the plan, the standard.
  3. Write what actually happened, as facts in order, before any judging.
  4. Ask why the two differed. Ask it of what went well, as well as what went wrong.
  5. Pick one thing to keep and one to change. Write each as a step for next time, with a date.
  6. Put the steps where the next plan gets made: the calendar, the checklist, the weekly chart.

03An example

What it looks like done

Worked on One week for someone working full time and taking an evening statistics class

The week of

  • November 9 to 15.

What I learned

  • Statistics: a 95% confidence interval comes from a method that captures the true value in about 95% of samples. It is not a 95% chance for this one interval.
  • Work: the client reads only page one, so the summary goes there.

What I'll test myself on

  • Write the confidence interval formula from memory, and explain what 95% means in one sentence.

Notes to connect or rewrite

  • Link the confidence interval note to last week's note on sampling error (is the next step after).

What slipped, and what I'll change

  • Planned: three study sessions. Happened: one, on Sunday.
  • Why: Tuesday and Thursday sessions sat after a 6 p.m. meeting, and the reading pile kept growing.
  • Bottleneck: Thursday evening. Change: move one session to Saturday morning.
  • Keep: the 15-minute review on the train after class worked both nights.

Next week

  • Saturday 9 a.m.: statistics problem set at the library.
  • Add "summary on page one" to the client report checklist.

04Try it · 20 minutes

For a class

When your next exam or graded assignment comes back, take 15 minutes. Write what you planned to do to prepare, what you actually did, and what the grade and comments show. For each gap, write why. Then write one study step to keep and one to change for the next exam, each with a date on your calendar.

For work and life

At the end of this week, take 20 minutes with the review page. What did the week plan to do? What happened, day by day? Where did it slip, and why? Use the course: one loop that ran, one bottleneck, one checklist line that would have caught a miss. Pick one change and put it on next week's calendar.

The blank page for this is in the Systems workbook.

05Where it goes wrong

  • Turning into blame. A review that hunts for who failed gets guarded answers. Ask what happened and why, about the plan and the conditions.
  • Only the failures. Reviewing only what went wrong leaves the reasons for success unknown. Ask why the good days went well too.
  • Lessons with no step. "Plan better" changes nothing. Write one step, with a date, where next week's plan gets made.
  • Too late. A month after an event, memory has already rewritten it. Review within a few days, or at the end of each week.

06Research

What research has measured

Debriefs and performance

Shown that structured debriefs after training or work improve later performance, by about a quarter on average in one review.

The numbers and sources
Effect
Debriefs improved effectiveness over control groups by about 25% (d = .67), with similar effects for teams and individuals, in simulated and real settings, and in medical and other samples [1]. A later meta-analysis found d = 0.79. Reviews aligned to the individual or the team, and reviews using objective records of performance, worked better [3].
Population
2,136 participants in 46 samples [1]. 915 teams and 3,499 individuals in 61 studies [3].
Caveats
Most studies tested debriefs in training, in the military, medicine and similar settings. One person's review of their own week is a smaller cousin of these, and was not tested itself. The later analysis corrected for sampling error only [3].

Reviewing successes as well as failures

One study found soldiers improved more when daily reviews covered their successes as well as their failures, than when they reviewed failures only.

The numbers and sources
Effect
In successive navigation exercises, performance improved significantly more in the group debriefed on failures and successes after each training day. Before the reviews, both groups' mental models of failed events were richer than their models of successful ones. The gap closed over later measurements [4].
Population
Soldiers in navigation training. The abstract does not give the number [4].
Caveats
One quasi-field experiment, in one military course.

07Check yourself

Answer from memory first. The next lesson asks these again.

Why write what happened before asking why?

Judging first bends the facts toward a story. Laying out the plan and the events in order makes the gap visible, so the why can be about the gap.

Why review what went well, not only what went wrong?

In one study, soldiers who reviewed successes as well as failures improved more than those who reviewed only failures. What worked is also something to keep on purpose.

What makes a review change the next week?

A step with a date, put where the next plan is made: the calendar, the checklist, the chart. A lesson that stays in the review notes rarely gets used.

Sources
  1. Tannenbaum SI, Cerasoli CP. Do team and individual debriefs enhance performance? A meta-analysis. Human Factors. 2013;55(1):231-245. https://doi.org/10.1177/0018720812448394
  2. Layland A, Koehler P, Simpson J. After Action Review Guide. National Comprehensive Center at Westat; 2020. https://www.adi.org/downloads/After-Action-Review-Guide.pdf
  3. Keiser NL, Arthur W. A meta-analysis of the effectiveness of the after-action review (or debrief) and factors that influence its effectiveness. Journal of Applied Psychology. 2021;106(7):1007-1032. https://doi.org/10.1037/apl0000821
  4. Ellis S, Davidi I. After-event reviews: Drawing lessons from successful and failed experience. Journal of Applied Psychology. 2005;90(5):857-871. https://doi.org/10.1037/0021-9010.90.5.857
  5. Franklin B. Autobiography of Benjamin Franklin. Pine FW, ed. Henry Holt; 1916. https://www.gutenberg.org/ebooks/20203

In MVIII: Today. The day's workout, food and tasks on one list. Get MVIII