Transforming Physician Education With Adaptive Learning

Speaker: Matthew O'Rourke

Amount of medical knowledge is increasing exponentially. While physicians don’t need to know everything, part of the problem is that they don’t know what they need to know. They started as a medical journal, it evolved and became more multi-media and interactive. However, to maintain relevance they needed to go beyond that format completely. Their needs assessment showed:

What MDs want:

  •  Progressive learning

  • Make the most of my time

  • Make it rewarding for me

  • Learn it my own way

What MDs need:

  •  Need to demonstrate clinical knowledge in training

  • Need to demonstrate clinical knowledge inexams

  • Need to maintain and grow clinical knowledge in practice

Current learning science:

  • Spaced repetition

  • Metacognition (do you know what you don’t know?)

  • Bite-sized learning

  • ILPs (individualized learning plans – a trend now in medical education)

  • Adaptive learning

  • Physicians are frustrated by the piecemeal approach of occasional required courses, the need for both continual and just-in-time learning was great.

Adaptive learning (as opposed to traditional)

  •  Adapts to individual

  • Assess first, then offer targeted content (for them they are pretty much combined, for traditional assessment last)

  • Take into consideration what learner already knows and focuses on knowledge gaps.

  • Self-directed, with follow-up on what the learner has trouble with.

They created NEJM Knowledge Plus, a tool that either provides questions or let’s admins create them. It is used mostly for CMEs and also preparation in taking the Board exams.

There is a self-assessment dashboard for the learner to see. The majority of their users not only access on phone, but don’t even use a desktop. They use the term “probes” instead of questions, since traditional multiple-choice questions are one way to do this, but not the only way.

When they answer the question they have to also answer their confidence level in their answer - I know it, I think so, not sure, or no clue. So if they get it right, but without confidence they will see this kind of question again to make sure it wasn’t a guess. But spaced out to help with mastery. It also “primes the pump” for the learner to think more about their response.

Ebbinghaus Forgetting Curve shows how forgetting time shortens the more one is exposed to the knowledge.

For the content creators, the hardest part of the process is to provide the feedback on the questions – why the correct answer is correct, and why the other answers are incorrect. They are finding this also leads to a good click through rate on the additional resources.

They also have a “challenge us” with each question – a way for learner to give feedback to them to help them keep on top of information and keep it current. This also helps learner to more deeply think about the question to articulate why they think it might be inaccurate or unclear.


They show the learner how their confidence level aligned with the correct/incorrect answer. Doctors who have a high degree of confidence and are often wrong are the most dangerous, and need to have this self-awareness. Other than system error, over-confidence is one of the biggest causes of medical error.