AI Tutors May Boost Test Scores Without Improving Real Learning

There is a particular kind of educational promise that keeps returning, decade after decade, with the same basic shape: master this system, and your mind will be sharper everywhere. In the early 2000s, companies like Lumosity and BrainHQ made exactly this pitch. Spend a few minutes daily on memory games and logic puzzles, they argued, and cognition itself would strengthen—better performance in the games would translate into better performance at school, at work, in life. By 2010, the brain-training industry had become a multi-billion-dollar enterprise. Millions subscribed. Schools bought licenses. Investors saw the future. Today, the movement has nearly vanished from public consciousness.

What happened is instructive, because the same pattern is now repeating with artificial intelligence tutors—and the mechanism behind both failures is the same: a misunderstanding of how learning actually travels from one context to another.

Transfer—the ability to apply knowledge learned in one situation to a new one—is not binary. It exists on a spectrum. Near transfer happens when you apply learning to a situation that closely resembles the original. Learning to drive a Toyota and then switching to a Honda is near transfer. So is learning Beethoven on a piano and then playing a keyboard. The underlying structure, environment, and goals remain largely the same. Far transfer is harder: it happens when you successfully apply learning to a situation substantially different from the original. Learning to drive a car and then operating an agricultural tractor. Learning piano and then playing a pipe organ. Learning chess and then playing Shogi. In the classroom, a student who solves twenty textbook problems about Newton's laws has demonstrated near transfer. Weeks later, if she correctly explains to her parents why passengers lurched forward when the bus braked suddenly, she has demonstrated far transfer—applying familiar concepts in a novel, untrained context.

Educators have long known that even near transfer is difficult. Students routinely perform well in one setting yet struggle to apply the same knowledge elsewhere. Effective curricula deliberately vary how students encounter ideas: different modes (writing, speaking, acting), different media (textbooks, whiteboards, flashcards), different environments (classroom, laboratory, field trips), different instructional approaches (explicit instruction, hands-on work, group discussion). The surface changes; the underlying principle does not. The goal is to help students recognize larger patterns so knowledge becomes portable.

Brain training violated this principle. The central claim—that repeatedly exercising memory would make cognition generally stronger—ran counter to decades of transfer research. A 2016 meta-analysis pooling results from 87 working-memory training studies found a clear pattern: the closer an assessment resembled the training exercises, the larger the apparent benefit. But as assessments moved further from the training environment, benefits shrank until they vanished. A second-order meta-analysis years later confirmed the same pattern across different populations. Brain training produced modest improvements on tasks closely resembling the training exercises, but those gains failed to generalize. The billion-dollar promise—that people would become better thinkers overall—never materialized.

Now consider digital tutors. Their overall effect size sits at +0.52, comfortably above the +0.40 threshold often used to indicate meaningful educational impact. But how much of this benefit reflects improvements within the tutoring ecosystem versus beyond it? International assessments provide an answer. In 2012, 2015, and 2018, PISA asked students how frequently they used digital technology at school for practice and drilling. Across all years and subjects, greater use was associated with lower performance on broader measures. TIMSS in 2019 and 2023 found the same pattern: greater reliance on digital quizzes and learning games correlated with lower performance on assessments removed from the tutoring environment. Most tellingly, NAEP asked 4th- and 8th-graders in 2017, 2019, 2022, and 2024 how frequently they used digital tutoring games for reading instruction. Students using these games as their primary mode achieved the lowest scores. Those using them to supplement traditional instruction performed better. Those never using them achieved the highest scores. This pattern replicated across all four testing cycles.

The implication is stark: while performance on assessments closely aligned with digital tutoring format may be impressive, those advantages largely disappear on broader standardized measures. This is not an obscure methodological detail. It changes how to interpret virtually every claim about digital tutoring effectiveness. The closer an assessment resembles the learning environment, the more likely it is to overestimate how well learning will generalize beyond that environment.

Some developers have exploited this insight deliberately. They design assessments that mirror the structure and cognitive demands of the tutor itself. A typical adaptive digital tutor presents a continuous stream of questions, evaluates each response immediately, estimates mastery, and adjusts the next question accordingly. MAP Growth, a common assessment, uses the same adaptive cycle delivered within essentially the same digital interface. If students spend hundreds of hours learning through an adaptive tutor, then an adaptive assessment is about as close to the learning environment as a standardized test can get. Alternatively, developers can change the assessment to more closely mirror the tutoring environment. Through 2023, the SAT looked essentially the same for decades: a paper test booklet with the same questions for all students, completed with pencil and bubble sheet under strict time limits. Structurally and cognitively, this differed substantially from adaptive digital tutors. In 2024, however, the SAT underwent fundamental redesign. It moved online, adopted a multistage adaptive format, and many students now complete it on the same school-issued device they use for digital learning throughout the year. The redesign made the SAT substantially more similar to modern adaptive tutoring systems. As the SAT becomes more similar to the tutoring environment, the transfer distance between the two shrinks. If near transfer is driving much of digital tutors' reported success, students trained on those systems should perform relatively better on the digital SAT than on its paper predecessor. But such improvements need not reflect deeper or more durable learning. They may simply reflect that the assessment itself has become more similar to the environment in which students practiced. Sometimes the easiest way to improve measured learning is not to change the teaching—it is to change the test.

Over more than a century, the pattern repeats: from Pressey's teaching machine to brain training to AI tutors, the promise remains the same. Master the program, and that mastery will generalize. But evidence tells a different story. Students often become highly proficient within the training environment yet struggle to carry that knowledge into novel situations. The learning produced is often more contextualized, less durable, less transferable than knowledge developed through traditional pedagogies. The effectiveness of any digital tutor cannot be separated from the transfer demands of the assessment used to evaluate it. The true test of learning is not what students can do with the tutor—but what they can still do without it.