Understanding Firecracker's Efficacy

Each year, we rigorously test, validate, and publish Firecracker efficacy results. The reasons for this are simple...

1. There is a lot at stake.

Being able to prove increased student outcomes is critical for all parties involved, including Firecracker, individual students, Faculty members, residency programs, hospitals, and ultimately even patients.

2. Nobody else does it.

No other company consistently validates and transparently shares the efficacy of their platform and/or content. With so much at stake, we find this to be alarming, as well as a valuable source of differentiation.

Study Methodology:

How do we do it?

  • Ongoing retrospective study since August 1, 2014.
  • Collect verified and unverified USMLE and COMLEX score reports starting June 2014 from any medical student regardless of Firecracker use.
  • Avg scores and standard deviation of non-Firecracker submitted reports are compared to official scores to determine accuracy.
  • 1,500+ official score reports and 2,000+ unofficial scores for the USMLE and COMLEX exams (over 5% of all board exam scores in past 3 years).
  • Reports are linked to Firecracker student accounts and analyzed using students full history of effort, progress, & performance.

Efficacy Results:

And the results are in!

Firecracker students score higher on licensing exams than non-Firecracker students.
The relationship between Firecracker & licensing exam performance appears causal, not correlational.
There is a significant correlation between Firecracker effort & licensing exam scores.
There is a significant correlation between Firecracker performance & licensing exam scores.
Firecracker efficacy continued to improve following the release of major product updates.

Score Report Validation

Firecracker students consistently score higher than non-Firecracker students. This increase in performance is also repeated across different licensing and standardized exams.

USMLE Step 1 Scores

245 Firecracker Average Score

229 Non-Firecracker Average Score

Not only that, the gap has increased over the last 3 years! And there is less variation in Firecracker scores than Non-Firecracker scores.

COMLEX Level 1 Scores

587 Firecracker Average Score

527 Non-Firecracker Average Score

NBME Family Medicine Shelf Exam

77 Firecracker Average Score

70 Non-Firecracker Average Score

Firecracker users aren't better students.

The relationship between Firecracker usage and licensing exam performance appears causal, not correlational. Results of our analysis show that Firecracker usage is associated with an equally significant increase in licensing exam score, regardless of incoming MCAT scores.

And not only that, our students continue to score higher with new product updates.

Q3 2015

Launch Firecracker MD/DO

Q1 2016

Launch MD/DO mobile apps

Align first course curriculum

Q2 2016

Launch Dedicated Test Prep mode

Assign first formative quiz

Q3 2016

Launch High Yield mode

Launch Institutional Dashboard

Q4 2016

Launch standalone clerkships

Align first clerkship - iPad clerkships

Q1 2017

1,200+ student clinical clerkships

Q2 2017

Launch summer remediation course

Just compare the efficacy of Firecracker Legacy (retired in late 2015) to Firecracker MD/DO.

245 Firecracker MD avg. score.

242 Firecracker Legacy avg. score.

228 Non-Firecracker avg. score

It’s all in the memory science.

In a 2007 report by the Institute for Education Sciences (IES) entitled “Organizing Instruction and Study to Improve Student Learning”, the IES isolated a collection of principles from hundreds of studies on student and instructor behavior to inform the education community of actionable techniques that improve learning outcomes. Firecracker embodies these principles through a combinatorial approach of curriculum alignment, daily concept review, and higher-order assessment. Some principles are demonstrated inherently in Firecracker’s study functionality, while other principles rely on the interaction effect between different functionality. We believe that our adoption and referral rates represent the definitive returns from these principles that Firecracker students experience throughout their medical education.

Firecracker actively employs the following wide range of proven memory science principles into its instructional design.

Strong Evidence
Moderate Evidence
Minimal Evidence
Ask deep explanatory questions.
Use quizzes to re-expose students to key content.
Interleave worked example solutions with problem solving exercises
Space learning over time.
Combine graphics with verbal descriptions.
Connect and integrate abstract and concrete representations of concepts.
Use tests and quizzes to identify content that needs to be learned.
Use pre-testing to introduce new topics.
Teach students how to use delayed judgements of learning to identify content that needs further study.

Deep questions lead to complex insight.

To advance their understanding, students should face questions that require deep explanations, which are based on causal mechanisms, planning, and logic.

How Firecracker integrates this:

  • Clinical Vignettes require memorization and higher-order thinking.
  • Complex concepts presented via the clinical scenario, question, and answer choices.
  • Authored by our content team to match the style and difficulty of the board exams.

Use quizzes to emphasize information.

“Closed-book” assessments expose students to the most important concepts in a course and lead to higher performances on final exams.

How Firecracker integrates this:

  • Weekly quizzes in institutional partnership courses for weekly content recap.
  • Multitude of full- or partial-length timed exams featuring sequestered questions.
  • Exam reports including answer explanations and comparative analytics.

Interleave example solutions and exercises.

Students who review example solutions in tandem with their own problem-solving attempts develop frameworks that they can apply to new scenarios.

How Firecracker integrates this:

  • “Solve Cases” for step-by-step practice solutions.
  • “Take an Exam” for authentic assessment experience.
  • Expert Skills Program test-taking procedure and guide for clinical vignettes.

Space learning over time using repetition.

Students learning new subjects are more likely to retain concepts and deeply integrate knowledge if the material is reviewed consistently over time.

How Firecracker integrates this:

  • Previously studied concepts reintroduced into daily review.
  • Forgetting Curves used to prioritize interval spacing.
  • “Urgent”, “Current”, and “Past” markings determine rate of concept recurrence.

Combine graphics with text descriptions.

Students learn more effectively when verbal or text-based topics are paired with relevant visuals in close proximity.

How Firecracker integrates this:

  • Firecracker Topic Summaries equipped with charts, diagrams, and pictures.
  • Images visible in answer explanations.
  • Captioned and curated by content team for accuracy and relevance.

Use abstract and concrete concept models.

Students learning with a mixed-diet of concrete and abstract representations build both foundational knowledge and lateral application skills.

What type of hypersensitivity is in allergic contact dermatitus?

A patient comes into the emergency department. She had a headache the previous day which has since resolved. She is worried that a brain tumor is the cause of her headache. Her physical exam is completely normal, and there is nothing in her history to suggest she might have a brain tumor. She is insistent that she have a head MRI to check for a brain tumor.

Which core ethics principle dictates that this procedure not be done so that limited resources can be reserved for those who truly need them?

Assessments can highlight knowledge gaps.

After the introduction of new material, assessments can help isolate concepts that require additional coverage and lead to more efficient use of study time.

How Firecracker integrates this:

  • Mastery Scorecard presents performance ranges based on user history.
  • Assessments prompt remediation steps for incorrect answer choices.
  • Faculty get remediation recommendations for “at-risk” population based on user data.

Introduce new topics with pre-questions.

Answering pre-questions before formally engaging with the lesson helps students focus on key information and summon relevant past knowledge.

How Firecracker integrates this:

  • Pre-course diagnostic exams with institutional partnership courses.
  • Dedicated Test Prep mode diagnostic exam for final-sprint study plan creation.
  • Weekly follow-up diagnostic exams to gauge student progress.

Delay judgment to identify further review.

Students unclear about their competency should delay assessment for a period after they study and should address the question cue before answering.

How Firecracker integrates this:

  • Case questions separated from daily recall questions to separate assessment and study.
  • Students must view and address the question cue before clicking “show answer”.
  • Students self-report confidence ratings to consciously judge their aptitude.

Ready to get started with Firecracker?

Whether you're a student preparing for your boards, or a Faculty member looking for a better way to assess your students, Firecracker has the solution for you.