Research

Every study here is run through one rigorous bar before it earns a verdict. Ordered by a blend of how well it holds up, how influential it is, and how recent. Click any row for the full scoring.

The bar every study clears

Each paper is scored 0 to 100 across nine dimensions of research quality (causal design and decision-relevance carry the most weight), then a devil's advocate tries to refute it. Any flaw rated critical caps the score and blocks a passing verdict outright. A study is REAL only at 75 or higher with no critical flaw, WATCH if the direction is real but the evidence is not there yet, and anything weaker is held back. The order blends three normalized signals: the Gate score, citation velocity (citations per year), and recency. A rigorous study surfaces even with fewer citations, an influential one even if older, and a strong new study is not buried.

Real · 75+, no critical flaw Watch · promising, evidence not there yet
Scored studies · ordered by influence + recency
YearVerdictScoreCitedStudyWhat it means for leaders
2026 Watch 58/100 n/a
Vanhoyweghen, Holst, Mobini, Van de Voorde, Vanleke, Algaba, Verboven, Guerry, Van Droogenbroeck & Ginis · arXiv 2610.27881 (preprint)
AI feedback quality & learning outcomes
One of the few studies to ask both halves of the question: how good is the AI feedback, and did it change the exam score. The quality answer splits sharply by criterion, with generated practice exercises rated strong far more often than diagnostic specificity or clarity and tone, and feedback on the lowest-scoring answers rated less actionable. The outcome answer is a null, and a weak one: the authors report no statistically significant differences after adjustment across four comparisons, while stating they cannot separate feedback from attendance and practice. University mathematics, two cohorts, no school-age students. ›
Causal design7/15
Decision-relevance11/15
Sample & power7/10
Effect size5/10
Stat validity7/10
Peer-review3/10
Replication7/10
Conflicts / funding7/10
Generalizability4/10
The readGPT-5.1 generated feedback and practice exercises from handwritten university mathematics answers and their grades, across six weekly bonus tests in an 'English (n=343)' and a 'Dutch (n=236)' cohort, with a crossover feedback design on tests 1 and 3. Six reviewers assessed '157 messages from three English-cohort tasks' against 'six criteria scored from 0 to 3'. The abstract reports that 'the practice exercises received the highest proportion of strong ratings (95.5%), while diagnostic specificity (41.1%) and clarity and tone (33.4%) received fewer', and that feedback for the lowest-scoring answers was rated as less actionable. On outcomes, comparisons of related exam questions showed mixed results, with 'no statistically significant differences after adjustment across four comparisons'. Greater attendance and feedback exposure were associated with higher exam scores, but the authors state the contribution of feedback could not be separated from attendance and practice.
Strongest counter-argumentThe null exam result is the headline a reader will take away, and it is the weakest part of the paper. The authors themselves say the design cannot separate feedback from attendance and practice, and only tests 1 and 3 used the crossover design, so the four comparisons are closer to an underpowered open question than to evidence that the feedback does not work. The reverse reading is equally available: a real effect could be masked by the same confounding the authors flag.
  • MajorUniversity mathematics in two cohorts at one institution. Nothing measured in a K-12 classroom.
  • MajorThe null exam finding cannot be attributed to feedback, because the authors state the contribution of feedback could not be separated from attendance and practice.
  • MajorThe abstract states no peer-review status and no preregistration, and the arXiv abstract page carries no funding statement.
  • MinorThe quality ratings rest on 157 messages from three tasks in the English cohort only, so they do not describe the Dutch cohort or the tool in general.
  • MinorA single model version, GPT-5.1, which will date quickly.
Confidence: high on the reported figures, read verbatim from the arXiv abstract on 2026-10-21. Lower on the exam analysis, because the full 10-page manuscript was not read and the abstract does not give the adjustment model. Citation count not shown because the paper is too recent for a stable count.
2026 Watch 51/100 n/a
Zamanifard, Goudarzi & Robb · arXiv 2610.27448 (preprint)
Learner profiling & personalization claims
Relevant to any vendor offering to profile a learner from how they chat. Two commercial models were asked to infer three learning-style dimensions from short student-AI dialogues, against questionnaire labels from 153 undergraduates. Where accuracy and F1 looked moderate, the authors report it was often explained by majority-class prediction, and the models under-identified Intuitive and Global learners. The authors conclude the labels are uncertain, bias-prone signals rather than stable profiles. Note the yardstick: the ground truth is a learning-styles instrument whose own validity is contested. ›
Causal design4/15
Decision-relevance10/15
Sample & power6/10
Effect size4/10
Stat validity7/10
Peer-review3/10
Replication6/10
Conflicts / funding7/10
Generalizability4/10
The readGPT-4o and Gemini 2.0 Flash were prompted to infer three Felder-Silverman learning-style dimensions, Active-Reflective, Sensing-Intuitive and Sequential-Global, from short task-focused student-AI problem-solving dialogues. Ground truth came from Index of Learning Styles questionnaire responses from '153 undergraduate participants', evaluated with five-fold cross-validation under zero-shot and few-shot prompting. The authors report that moderate accuracy and F1 on some dimensions were often explained by majority-class prediction, that the models under-identified Intuitive and Global learners, that few-shot prompting had mixed effects, and that Active-Reflective results were unstable across models. Their conclusion is that 'LLM-generated learner-style labels should be treated as uncertain, bias-prone signals rather than stable learner profiles', with a call for 'bias-aware evaluation and uncertainty-sensitive design'.
Strongest counter-argumentThe construct, not the model, may be the problem. The ground truth here is the Index of Learning Styles, and matching instruction to learning styles has not held up well in the learning-sciences literature. A model that fails to reproduce those labels is therefore not cleanly a failure of the model, and a model that succeeded would be confidently reproducing a category of contested value. Either way the practical claim a vendor makes about learner profiling is unsupported by this paper, which is the usable finding.
  • MajorThe ground truth is a learning-styles questionnaire whose construct validity is contested, so the paper cannot separate model error from instrument error.
  • MajorThe abstract reports no accuracy or F1 figures, so the magnitude of the effect cannot be checked from it.
  • Major153 undergraduate participants. No school-age students, so this does not transfer to a K-12 classroom as published.
  • MinorThe abstract states no peer-review status, and the arXiv abstract page carries no funding statement.
  • MinorTwo specific model versions, GPT-4o and Gemini 2.0 Flash, which will date quickly.
Confidence: high on the design and the stated conclusion, read verbatim from the arXiv abstract on 2026-10-21. Lower on effect magnitude, because the abstract publishes no accuracy figures and the full paper was not read. Citation count not shown because the paper is too recent for a stable count.
2026 Watch 61/100 n/a
Zhao, Jiao & Xu · arXiv 2610.09460 (preprint)
AI grading & feedback quality
The clearest evidence yet that AI feedback can look like a diagnosis without being one. Across 3,041 graded responses, LLM feedback named a student misconception in 5 to 7 percent of cases against 15 to 31 percent for teachers, the rubric sub-scores were so tightly correlated they carried almost no independent information, and the tone stayed uniformly positive no matter how poor the response. Read it as the question to put to a vendor, not as proof of harm: this is one computer-science dataset in higher education and no learning outcome was measured. ›
Causal design7/15
Decision-relevance13/15
Sample & power6/10
Effect size8/10
Stat validity6/10
Peer-review3/10
Replication8/10
Conflicts / funding6/10
Generalizability4/10
The readA head-to-head comparison of human-instructor and LLM grading on the JorGPT dataset of '3,041 student responses to 50 open-ended computer science questions, scored by both human instructors and three commercial LLMs'. The authors report that sub-dimension scores are 'highly correlated (r = 0.82-0.99, VIF up to 45), providing redundant rather than independent diagnostic information', that textual feedback 'rarely detects student misconceptions (5-7% vs. 15-31% for teachers), functioning as a coverage checklist rather than a diagnostic instrument', and that 'the feedback tone remains uniformly positive regardless of response quality'. Grading accuracy varied by domain, with procedural topics most reliable.
Strongest counter-argumentThe misconception gap is a comparison between what human instructors chose to write and what LLM feedback produced, so some part of the 5-7 against 15-31 could reflect differing annotation conventions rather than a real difference in diagnostic capability. Nothing in the study measures whether students who received LLM feedback learned less, because no learning outcome was collected at all.
  • MajorOne dataset, computer science only, higher education only. Nothing measured in a K-12 classroom.
  • MajorNo learning outcome measured, so the paper cannot speak to student harm or benefit.
  • MajorarXiv preprint. No peer-review status, no preregistration and no funding disclosure stated.
  • MinorThree commercial LLMs are compared but not named per-result on the abstract page.
Confidence: high on the reported figures, read verbatim from the arXiv abstract on 2026-10-14. Lower on mechanism and on the annotation procedure behind the misconception counts, because the full paper was not read. Citation count not shown because the paper is too recent for a stable count.
2026 Watch 61/100 n/a
Wang, Modi Jain, Burbano, Guzman, Keum, Kim, Kogut & Wright · arXiv 2610.10463 (preprint)
AI & classroom engagement
A preregistered field experiment with a clean within-student design that found a real behavioural effect and honestly declined to call it learning. Students who prepped for a class session with a voice-based AI discussion partner later made about 31 percent more voluntary contributions, and reported more comfort speaking up, but no gain in focus or motivation. Useful as a reason to pilot a rehearsal step, not as evidence of achievement: these are MBA students and no test, grade or knowledge measure was collected. ›
Causal design12/15
Decision-relevance8/15
Sample & power8/10
Effect size7/10
Stat validity8/10
Peer-review4/10
Replication5/10
Conflicts / funding5/10
Generalizability4/10
The readThe authors describe 'a preregistered field experiment with 759 MBA students enrolled in ten sections of a course, in which each student was randomly assigned two of ten class sessions to prepare for with a purpose-built voice-based AI discussion partner'. Randomising which sessions a given student prepared with AI makes each student their own comparison. They report that 'after two uses of the AI discussion partner, students made about 31% more voluntary contributions in each later class session', and that heavier users 'reported greater comfort speaking up and greater perceived learning, but not greater focus or motivation'. The authors characterise the outcome as 'a critical intermediate learning outcome'.
Strongest counter-argumentVoluntary contributions are a behaviour, not an achievement, and the authors say so. A student told to prepare for two specific sessions with a novel tool may contribute more because of the salience of the assignment rather than any durable change in engagement, and nothing in the design separates rehearsal from the attention the assignment itself draws. No learning outcome was measured, so the headline cannot be converted into an academic claim.
  • MajorMBA students in a single course. The population is remote from K-12 and the finding should not be assumed to transfer.
  • MajorThe outcome is class participation. No test, grade or knowledge measure was collected.
  • MajorThe AI discussion partner is 'purpose-built' and unnamed, and no funding or author stake is disclosed on the listing.
  • MinorarXiv preprint, not peer reviewed, though preregistration is stated.
Confidence: high on the design and reported figures, read verbatim from the arXiv abstract on 2026-10-14. Lower on who built the discussion partner and on the exact contribution-counting procedure, because the full paper was not read. Citation count not shown because the paper is too recent for a stable count.
2026 Watch 65/100 n/a
Domínguez Figaredo & Fernández De la Cruz · arXiv 2610.12125 (preprint)
AI tutor benchmarks & procurement
The most directly useful paper on this page for anyone reading a vendor's benchmark. The authors wrote a plausible metric of good teaching, fine-tuned a tutor to score better on it, and succeeded: the metric rose while 31 educators rating blind judged the output substantially worse, because the model learned to repeat its single highest-scoring move. Carry the habit it implies, which is to ask what a rising number is made of and to have a teacher read the output blind. It is a model experiment, and the abstract reports no learning outcome and never describes a real student. ›
Causal design10/15
Decision-relevance12/15
Sample & power6/10
Effect size9/10
Stat validity7/10
Peer-review3/10
Replication6/10
Conflicts / funding7/10
Generalizability5/10
The readThe authors built a metric of pedagogical adaptivity and tested the obvious improvement strategy against it. They 'audited a frontier tutor across 2,000 learner scenarios, corrected its weakest cases by fine-tuning an open-weights proxy, and asked 31 trained educators to rate the pedagogical alignment of the outputs blind, before and after correction'. The two measures diverged: 'the metric increased from +0.05 to +0.42 for the corrected cases, while the expert ratings decreased from 4.46 to 3.03, with the unmodified controls remaining unchanged and a base-proxy control ruling out the change of model'. The mechanism generalises, because 'any metric that scores decisions independently and averages them is maximized by repeating the single best decision', and the authors conclude that 'measurement validity does not imply optimization validity'.
Strongest counter-argumentThe collapse may be an artifact of this fine-tuning setup rather than a general property of metric optimisation. The authors' own weight analysis found the corrected model 'had memorized its training strings rather than learned to adapt', which is the classic signature of overfitting a small open-weights model on a narrow correction set. On that reading the result says more about this run than about benchmark design in general, although the averaging argument stands on its own.
  • MajorNo learning outcome reported and no real student described. 2,000 learner scenarios and a 31-educator rating panel.
  • MajorOne frontier tutor and one open-weights proxy. A narrow base for a general claim about metrics.
  • MajorarXiv preprint. No peer-review status and no preregistration stated.
  • MinorFunding not stated on the listing, though no vendor product is being promoted.
Confidence: high on the reported figures and the mechanism argument, read verbatim from the arXiv abstract on 2026-10-14. Lower on the fine-tuning details and the educator rating instrument, because the full paper was not read. Citation count not shown because the paper is too recent for a stable count.
2026 Watch 74/100 n/a
Liu, Sweet, Chen, Engelberg, Clark, Persaud, Lancaster, Hollingsworth, King Rice & Masters · EdWorkingPaper 26-1598 (Annenberg, Brown)
AI tutoring & achievement
The rare randomized trial where an AI tutor did worse than nothing. Access reduced final grades and sharply reduced participation in existing coursework, with larger estimated losses for first-generation students. Read it as a reason to keep a real comparison group in any tutor pilot and to ask what activity the tool is displacing, not as a settled verdict on AI tutoring, because the grade estimate rests on a same-course subsample and the tutor product is not named. ›
Causal design13/15
Decision-relevance15/15
Sample & power9/10
Effect size10/10
Stat validity6/10
Peer-review4/10
Replication5/10
Conflicts / funding6/10
Generalizability6/10
The readA randomized controlled trial of access to a course-integrated generative AI tutor across 2,379 undergraduates and 30 instructors at one large US public university. The authors report that 'among sections of the same course, tutor access reduced final grades by 0.37 standard deviations and learning management system participation by 0.90 standard deviations', that participation declines were similarly large in the full sample, and that estimated academic losses were greater among first-generation students.
Strongest counter-argumentThe headline grade effect is reported for the comparison among sections of the same course, and the abstract confirms full-sample robustness only for the participation decline, not for grades. That leaves open that the grade estimate rests on a narrower and possibly more favorable slice than the 2,379 figure implies, and no preregistration is stated to rule out specification choice.
  • MajorGrade effect scoped to a same-course subsample; full-sample robustness stated only for the participation outcome.
  • MajorWorking paper, not peer reviewed, and no preregistration stated.
  • MajorThe tutor product is not named and no funder is listed on the paper's landing page, so independence cannot be confirmed.
  • MinorUndergraduates at a single university. Does not transfer directly to K-12.
Confidence: high on the reported figures and their scope, all read verbatim from the paper's abstract and DOI record on 2026-09-30. Lower on mechanism and on what the comparison condition actually involved, because the full paper was not read. Citation count not shown because the paper is too recent for a stable count.
2026 Watch 54/100 n/a
Racovan, Rawat, May & Turkstra (Purdue University) · arXiv preprint
AI detection & academic integrity
Notable less for its detection rate than for what its own authors insist on: no automated detector should run without a structured human review process. The 45 percent figure is a rate of flagged behavioral patterns over six years of one programming course, not a measured cheating rate, and no false-positive rate is reported. Use it as the standard to hold a detection vendor to, not as a prevalence estimate. ›
Causal design6/15
Decision-relevance12/15
Sample & power8/10
Effect size5/10
Stat validity5/10
Peer-review2/10
Replication6/10
Conflicts / funding6/10
Generalizability4/10
The readA behavioral and stylistic detection system for LLM-assisted work in undergraduate C programming assignments, applied across six years of Spring offerings of a large-enrollment CS2 course at Purdue. The authors report that '45% of enrolled students exhibited patterns consistent with LLM-assisted code development in Spring 2026' and find a significant negative correlation between flagged use and performance on written, in-person proctored examinations. They argue that 'any automated detection system must be accompanied by a structured process for human review'.
Strongest counter-argumentThere is no causal identification and no reported false-positive rate, so the 45 percent cannot be read as a cheating prevalence. The negative correlation with proctored-exam performance is equally consistent with weaker students leaning on a model, which would make the flag a proxy for struggle rather than for misconduct, and the authors evaluate a tool they built themselves.
  • MajorNo false-positive rate reported, so the flag rate cannot be converted into a prevalence claim.
  • MajorCorrelational only; flagged use and lower proctored performance have at least one common cause in student difficulty.
  • MajorAuthors evaluate their own system. Non-commercial and university-internal, but still self-evaluation.
  • MinorOne CS2 course, one institution, C programming. Preprint, not peer reviewed.
Confidence: high on the quoted figures and the authors' own human-review caveat, read verbatim from the arXiv abstract on 2026-09-30. The full paper and its detection methodology were not read. Citation count not shown because the paper is too recent for a stable count.
2026 Real 85/100 n/a
Allcott, Baron, Dee, Duckworth, Gentzkow & Jacob · NBER Working Paper 35132
Phones, attention & achievement
The strongest causal evidence yet that phone restriction changes behaviour and climate but not test scores. Budget and justify a ban on attention and climate over a multi-year horizon, and plan for a worse first year, not on promised achievement gains. ›
Causal design13/15
Decision-relevance15/15
Sample & power10/10
Effect size8/10
Stat validity8/10
Peer-review5/10
Replication9/10
Conflicts / funding9/10
Generalizability9/10
The readA staggered difference-in-differences study of lockable phone pouches nationwide, combining large-scale surveys, GPS pings, standardized test scores, school administrative records and the largest pouch provider's sales records. Pouch adoption substantially reduces phone use. In year one disciplinary incidents rise and student well-being falls; both reverse in later years. Average effects on test scores are consistently close to zero, with modest positive high-school effects (particularly math) and small negative middle-school effects.
Strongest counter-argumentPouch adoption is not randomly assigned, and staggered difference-in-differences can be biased when treatment effects vary across adoption cohorts. The GPS compliance evidence, however, rules out the strongest rival explanation, that the bans were simply never enforced.
  • MinorNBER working paper; not yet peer reviewed.
  • MinorMeasures one mechanism, lockable pouches, so it does not settle what a bell-to-bell ban enforced another way would produce.
  • MinorMany outcomes examined, creating some multiple-comparison exposure, though nulls are reported evenhandedly.
Confidence: high. Scored from the NBER abstract and paper listing; design, data sources and directional findings are stated explicitly by the authors. Citation count not shown because the paper is too recent for a stable count.
2026 Watch 78/100 n/a
Robinson, Gormley, Trindade Ribeiro & Loeb · EdWorkingPaper 26-1451 (Annenberg Institute / Stanford SCALE)
AI tutoring & engagement
The cleanest available answer to "will students actually use the AI tutor we bought." Nearly half of the elementary students given dedicated session time never opened the platform at all, and those who did averaged two to five minutes a week against a thirty-minute target. Adding a human tutor whose only job was engagement raised usage by one to four minutes and did not move reading scores. Budget for the engagement problem before the licence. ›
Causal design12/15
Decision-relevance14/15
Sample & power6/10
Effect size7/10
Stat validity8/10
Peer-review5/10
Replication9/10
Conflicts / funding9/10
Generalizability8/10
The readTwo randomized controlled trials run by the Stanford SCALE Initiative, covering 355 students in grades 1 to 5 across two school districts, assigning students either to use an AI literacy platform independently or to use the same platform with an in-person tutor whose role was engagement and accountability rather than instruction. Both groups had dedicated session time, twice a week for thirty minutes. Nearly half the students in the independent-use group never used the platform at all, and those who did averaged two to five minutes per week. Human support raised average weekly usage by one to four minutes and raised engagement by 71 to 80 percent in relative terms, but absolute usage stayed far below the thirty-minute weekly dose the platform's own guidance associates with academic benefit, and the intervention did not improve reading achievement. The authors' conclusion is that access alone is insufficient and that implementation is the binding constraint.
Strongest counter-argumentThis is one AI literacy product, in two districts, with 355 children in grades 1 to 5. Very young students failing to self-direct through a chat-based tool may say more about developmental fit and product design than about AI tutoring as a category, and a middle or high school leader should discount accordingly. The headline "71 to 80 percent increase in engagement" is also a relative gain on a very small base, which is a figure that travels badly once it leaves the paper. And the null on reading achievement is close to uninformative, because the delivered dose never approached the level at which anyone expected gains: this is a study of a failed implementation, not a test of whether the tool works when used.
  • MajorSingle product, two districts, grades 1 to 5 only. Generalization beyond young elementary students and beyond this platform is not supported.
  • MinorAnnenberg working paper; not yet peer reviewed. Preregistration not confirmed from the accessible record.
  • MinorN of 355 across two trials is modest and underpowered to detect small reading effects, so the achievement null should be read as uninformative rather than as evidence of no effect.
  • MinorThe relative engagement increase (71 to 80 percent) is a large-sounding number on a base of two to five minutes per week and is easily over-read.
Confidence: high on the design and the reported usage figures, which are stated in the abstract and in the authors' own August 2026 write-up. The verdict is Watch rather than Real because of the single-product, two-district generalizability flag, not because the trials are weak. Converges strongly with the two-year Khanmigo trial below and with Khan Academy's own founder conceding the first Khanmigo did not move learning as hoped. Citation count not shown because the paper is too recent for a stable count.
2026 Watch 83/100 n/a
Oreopoulos & Low · NBER Working Paper 35620 / EdWorkingPaper 26-1551
AI tutoring & achievement
The longest randomized trial of a mainstream AI tutor to date, and the gains it produced resemble those from the same practice platform without AI. Before buying an AI tutor, ask what the comparison condition is, and budget for the engagement problem rather than the licence: 96 percent of students tried it once and the median student used it on a third of their practice days. ›
Causal design14/15
Decision-relevance15/15
Sample & power8/10
Effect size5/10
Stat validity9/10
Peer-review6/10
Replication9/10
Conflicts / funding8/10
Generalizability9/10
The readA two-year pre-registered cluster randomized trial (AEARCTR-0013519) across 18 Tennessee middle schools, testing access to Khan Academy's Khanmigo tutor configured to coach rather than to give answers, inside existing daily remedial math sessions. Assignment raises math achievement by 1.3 national percentile ranks per term, about 0.06 to 0.08 SD over a school year, with a full year of active participation implying 0.14 SD. The authors state these gains resemble those from Khan Academy practice without AI assistance. Engagement is the binding constraint: 96 percent of students tried Khanmigo at least once, but the median student messaged it on only a third of the days they practiced and in only 17 percent of exercise sessions in which they made a mistake, and messages were mostly bare answers or clicks on suggested prompts. Funded by Smith Richardson, SSHRC, NSERC and J-PAL; Khan Academy provided platform guidance and a data-sharing agreement but did not fund the study.
Strongest counter-argumentThe trial estimates the effect of assignment, not of use. With the median student messaging the tutor on a third of practice days and in 17 percent of mistake sessions, the finding that AI adds little over non-AI practice may describe the dosage actually delivered rather than the technology's ceiling. Separately, 53 grade-within-school clusters across 18 schools is a modest number of randomization units for a 0.06 to 0.08 SD contrast, so failing to detect a difference is not the same as demonstrating equivalence.
  • MajorIntent-to-treat by design, so the headline comparison cannot separate "AI tutoring does not add much" from "students did not use the AI tutoring." The paper is explicit that engagement is the binding constraint.
  • MajorUnderpowered to establish equivalence. Fifty-three grade-within-school clusters and a small effect mean the confidence interval around "no difference" is wide, and absence of a detected difference is not evidence of no difference.
  • MinorThe 0.14 SD full-participation figure is what the paper calls an implied effect on participants: a Wald estimate that scales the intent-to-treat result by the participation rate. It adds the assumption that assignment moved scores only through participation, so it carries less weight than the randomized intent-to-treat estimate. (Corrected 2026-09-16 after reading the full paper: an earlier version of this flag called it a non-randomized extrapolation, which misdescribed the method.)
  • MinorNBER and Annenberg working paper; not yet peer reviewed.
  • MinorSetting is existing remedial math sessions in one Tennessee county, which is a real-world strength for generalizability but a single-district limitation for breadth.
Confidence: high on the design and the reported findings, which are stated explicitly by the authors and pre-registered. The verdict is Watch rather than Real because two major flags are attached, not because the trial is weak. On design quality this is among the strongest studies on this page. Citation count not shown because the paper is too recent for a stable count.
2024 Real 76/100 513
Fan, Tang, Le, et al. · British Journal of Educational Technology
AI, writing & critical thinking
Credible randomized evidence that AI help improves the artifact more than the learner. Design assessments and tooling so AI assists the process rather than substituting for it. ›
Causal design13/15
Decision-relevance13/15
Sample & power7/10
Effect size7/10
Stat validity8/10
Peer-review10/10
Replication5/10
Conflicts / funding8/10
Generalizability5/10
The readIn a randomized four-condition experiment, ChatGPT users improved their essay scores the most but gained no more knowledge transfer than other groups, suggesting AI support can boost the output while offloading the thinking.
Strongest counter-argumentThe 'metacognitive laziness' label over-reads a null. ChatGPT users produced better essays and merely showed no advantage on knowledge transfer in one short lab task, and a missing transfer benefit in an underpowered single session is not evidence of cognitive harm.
  • MajorSmall cells: 117 students across four arms (~29 per group), so the null knowledge-transfer finding is likely underpowered.
  • MinorSingle lab essay task, one session, 2023-era ChatGPT; limited ecological validity.
  • MinorNo preregistration or direct replication.
Confidence: medium. Scored from the abstract and detailed secondary summaries confirming the four-arm design and N=117; full tables not read directly.
2026 Watch 71/100 n/a
Oreopoulos, Liut, Sungu & Low · NBER Working Paper 35621 / EdWorkingPaper 26-1552
AI tutoring & achievement
Suggests the workflow around an AI tutor matters more than the tutor. The clearest benefit came after a student made a mistake, and the mastery rule alone did not improve delayed learning. Useful for designing how a tool is used, not for deciding whether to buy one, because the whole trial is a single 50-minute session. ›
Causal design13/15
Decision-relevance12/15
Sample & power9/10
Effect size4/10
Stat validity6/10
Peer-review6/10
Replication8/10
Conflicts / funding9/10
Generalizability4/10
The readA pre-registered factorial randomized field experiment (AEARCTR-0018678) with more than 6,000 Hamilton County middle schoolers on the NUMI computer-assisted-learning platform, crossing AI versus platform-only with mastery versus non-mastery progression across two math topics, with a delayed assessment one week later. Students with AI progressed more slowly and attempted fewer questions but answered more accurately conditional on attempting. The clearest mechanism was post-mistake: AI improved next-attempt correctness and reduced the attempts needed to get back to a correct answer, while increasing time per question. Mastery progression raised three-correct-in-a-row attainment but did not by itself improve delayed learning. The best delayed-test evidence came when AI was embedded in the mastery workflow, but those gains were only marginally significant and confined to practiced material. Funded by Smith Richardson, SSHRC, NSERC and J-PAL.
Strongest counter-argumentA single 50-minute session with a one-week delayed test cannot support a claim about learning. The headline advantage was marginally significant, did not extend to harder problems, and did not transfer beyond the practiced material, which is the pattern you would expect from a short-run practice effect rather than from durable learning. The lead author declined to claim a breakthrough on exactly these grounds.
  • MajorSingle 50-minute session. Dosage is far too small to speak to a school-year decision, and the delayed test is one week out.
  • MajorThe favourable result is described as only marginally significant and confined to practiced material, in a factorial design with two topics and multiple arms, which creates real multiple-comparison exposure.
  • MinorNo headline standardized effect size reported in the abstract, which makes practical magnitude hard to judge independently.
  • MinorNBER and Annenberg working paper; not yet peer reviewed.
Confidence: high on the design and the reported direction, both pre-registered and stated by the authors. Listed as the mechanism companion to the two-year Khanmigo trial by the same lead author, posted the same day. Read the two together: the long trial says deployment did not beat the non-AI platform, and this one says the moment where AI did help was immediately after a mistake. Citation count not shown because the paper is too recent for a stable count.
2025 Watch 51/100 408
Kosmyna, Hauptmann, et al. (MIT Media Lab) · arXiv preprint
AI, writing & critical thinking
Provocative and directionally consistent with offloading concerns, but too small and unreviewed to act on. Treat it as a hypothesis to watch, not proof that AI assistance damages thinking. ›
Causal design10/15
Decision-relevance11/15
Sample & power4/10
Effect size5/10
Stat validity4/10
Peer-review2/10
Replication3/10
Conflicts / funding8/10
Generalizability4/10
The readAcross 54 participants writing essays under EEG, LLM users showed the weakest brain connectivity and the poorest recall of their own essays, which the authors frame as accumulating 'cognitive debt.'
Strongest counter-argumentThe viral 'your brain shuts off with ChatGPT' conclusion is statistically fragile: the headline crossover result rests on 18 participants in a non-randomized fourth session, with dozens of EEG comparisons, no peer review, and a single artificial essay task.
  • MajorNot peer reviewed at release (arXiv preprint), a status the authors themselves emphasized.
  • MajorHeadline 'cognitive debt' claim relies on 18 participants in session 4, non-randomized, heavy multiple-comparison EEG analysis.
  • MinorSingle SAT-style essay task and a Boston-area convenience sample; widely over-interpreted in media.
Confidence: low. Scored from the abstract and detailed secondary summaries; full statistical detail not read directly.
2025 Watch 76/100 98
Kestin, Miller, Klales, et al. · Scientific Reports
AI tutoring vs active learning
A well-engineered AI tutor can match or beat good instruction in a controlled lesson, but the evidence is too narrow to justify replacing classroom teaching. Treat it as promising for piloting supplemental practice, not a deployment mandate. ›
Causal design13/15
Decision-relevance13/15
Sample & power6/10
Effect size9/10
Stat validity8/10
Peer-review7/10
Replication6/10
Conflicts / funding9/10
Generalizability5/10
The readA crossover RCT of 194 Harvard physics students found a custom GPT-4 tutor with engineered pedagogical prompts produced post-test gains of about 0.63 SD, up to 0.73 to 1.3 SD by quantile, highly significant, in less instructional time than in-class active learning.
Strongest counter-argumentThe result comes from 194 elite Harvard physics students on just two topics over roughly 50 minutes, so novelty effects, ceiling effects, and the narrow setting make it unsafe to assume the same gains in a typical K-12 or community-college classroom.
  • MajorSingle elite institution, single subject, two short topics; weak external validity.
  • MinorNot preregistered.
  • MinorCeiling effects addressed via quantile regression but still present.
Confidence: high. Read the full published paper including methods, effect sizes, and the no-competing-interests statement.
2025 Watch 72/100 128
Letourneau, Deslandes Martineau, et al. · npj Science of Learning
Evidence base (review)
The best single map of where ITS evidence is strong versus thin in K-12. The strategic nuance for a buyer: the gain comes from structured tutoring, not necessarily the AI layer, since ITS rarely beat simpler tutoring software. ›
Causal design6/15
Decision-relevance13/15
Sample & power8/10
Effect size6/10
Stat validity6/10
Peer-review8/10
Replication8/10
Conflicts / funding9/10
Generalizability8/10
The readA peer-reviewed synthesis of 28 studies (4,597 K-12 students) finding intelligent tutoring systems generally beat human-teacher-only conditions (Hedges g roughly 0.68 to 1.30 in 7 of 8 such studies) but showed little advantage over non-intelligent tutoring software.
Strongest counter-argumentBecause it is a narrative synthesis that declines to pool effects, the headline rests on vote-counting across 28 heterogeneous quasi-experimental studies, which can mask publication bias and overweight a few high-effect comparisons.
  • MajorNarrative review, not a meta-analysis; no pooled effect estimate, so synthesis relies on vote-counting.
  • MinorUnderlying evidence base is overwhelmingly quasi-experimental.
Confidence: medium. Scored from the open-access full text including study counts, design mix, and effect-size ranges.
2026 Watch 70/100 n/a
Marcoccia, Quattrociocchi, Capraro · arXiv preprint
AI & judgment / overreliance
The cleanest experimental evidence yet that simply having AI advice on screen changes whether people admit they do not know something. Adults, not students, so treat it as a mechanism to design around (make "I don't know" a graded option), not a finding about your classrooms. ›
Causal design13/15
Decision-relevance9/15
Sample & power8/10
Effect size8/10
Stat validity9/10
Peer-review3/10
Replication7/10
Conflicts / funding9/10
Generalizability4/10
The readFive experiments (N = 3,132; four preregistered, one direct replication) had adults answer hard questions with a real option to decline. Mere access to deliberately wrong AI advice nearly eliminated willingness to say "I don't know": participants answered more, were correct about a third as often as without AI, and their confidence nearly doubled. Accuracy incentives only partially restored suspension of judgment.
Strongest counter-argumentThe advice was engineered to be wrong on deliberately difficult trivia-style questions with incentives attached, which is not how students meet AI in schoolwork, so the near-elimination effect may overstate what happens when advice is usually right and the task is familiar.
  • MajorAdult online participants and artificial question sets; no student sample, so school generalization is untested.
  • MajorNot peer reviewed (arXiv preprint, July 2026).
  • MinorDeliberately wrong advice isolates the mechanism cleanly but caps ecological validity.
Confidence: medium. Scored from the full abstract with design, N, preregistration status, and headline effects stated; full tables not read directly.
2026 Watch 70/100 n/a
Mills et al. · AugmentED (Advanced Education Research and Development Fund) · Educational Data Mining (EDM) 2026 proceedings
AI assessment & grading
The most specific answer yet to "can we let AI score argumentative writing." No critical-thinking sub-skill scored better than 78 percent accuracy even after fine-tuning, and AI failed four of the six: evaluating counterarguments, separating fact from opinion, drawing conclusions, and spotting logical fallacies. Do not buy a single headline accuracy number. Ask for per-sub-skill accuracy and the human inter-rater baseline on the same essays. ›
Causal design9/15
Decision-relevance13/15
Sample & power5/10
Effect size9/10
Stat validity6/10
Peer-review5/10
Replication8/10
Conflicts / funding8/10
Generalizability7/10
The readAugmentED, a program of the Advanced Education Research and Development Fund, tested whether AI systems can score critical thinking in argumentative essays written by students in grades 6 to 12, against a six-sub-skill rubric developed by ETS and the Carnegie Foundation. Presented by researcher Mills and AugmentED executive director Sherry Lachman. No sub-skill was measured at better than 78 percent accuracy, and that ceiling held even after fine-tuning and additional prompting. Performance split cleanly by task type: the models did best at judging synthesis across multiple sources and at evaluating the strength of evidence, and did poorly on four of the six sub-skills, namely evaluating counterarguments, distinguishing fact from opinion, drawing conclusions, and identifying logical fallacies. The failing four are the sub-skills that require holding a claim and its opposite in view simultaneously.
Strongest counter-argumentThe 78 percent ceiling is being read as a failure without the only benchmark that would make it one. Trained human raters scoring critical-thinking rubrics frequently agree with each other only in the seventies, so an AI at 78 percent may be performing at or near the human ceiling rather than below it, and the accessible record does not report a human inter-rater baseline on these essays. Sample size is also not disclosed in available reporting, so power cannot be assessed. What survives the objection is the sub-skill pattern rather than the headline number: even if the overall accuracy is defensible, the consistent failure on counterarguments and logical fallacies specifically is a structural finding that a single aggregate figure would have hidden.
  • MajorNo human inter-rater reliability baseline reported on the same essays, so the 78 percent ceiling cannot be judged as good or bad in absolute terms.
  • MajorSample size and essay count not disclosed in available reporting; power to distinguish sub-skill accuracies cannot be assessed.
  • MinorSix sub-skills scored without reported correction for multiple comparisons, and accuracies are reported as point estimates without confidence intervals.
  • MinorConference proceedings (EDM 2026) rather than a journal; preregistration not indicated in the accessible record.
Confidence: medium. Scored from detailed trade reporting of the EDM 2026 presentation rather than from the full paper, so sample, statistical-validity and peer-review dimensions were scored conservatively. The verdict is Watch rather than Real because of the missing human baseline and the undisclosed sample, not because the measurement work is weak. Converges with the existing evidence on this page that AI scoring is reliable on surface and structural features and unreliable on reasoning quality. Funder AERDF is a philanthropic education R&D organization, not a vendor selling an assessment product, and the rubric is ETS and Carnegie Foundation work, so conflict risk is low. Citation count not shown because the paper is too recent for a stable count.
2026 Watch 67/100 0
Rismanchian, Uzun, Matayoshi, et al. · arXiv preprint (under review)
AI & learning (math)
A strong counterweight to AI-tutoring optimism: unsupervised generative AI access during practice may erode the learning it appears to accelerate. Useful for framing guardrails, but it is a single preprint from a vendor team pending peer review. ›
Causal design10/15
Decision-relevance14/15
Sample & power10/10
Effect size9/10
Stat validity8/10
Peer-review2/10
Replication4/10
Conflicts / funding4/10
Generalizability6/10
The readA large quasi-experiment arguing that since ChatGPT, students spend 27 to 31 percent less time on AI-susceptible math problems and retain less, with the effect vanishing under proctoring, suggesting offloading rather than efficiency.
Strongest counter-argumentIdentification rests on AI susceptibility of problem type as a proxy for AI use rather than measured use, so unobserved curriculum or platform changes over 2015 to 2025 could mimic the post-ChatGPT ramp.
  • MajorFour of five authors are employed by McGraw Hill, owner of the ALEKS platform studied.
  • MajorarXiv preprint, not yet peer reviewed.
  • MinorNot preregistered; AI use inferred from a proxy, never directly observed.
Confidence: medium. Scored from the full text including design, sample, effect sizes, and funding.
2026 Watch 66/100 0
Sungu, Lira, Duckworth (Wharton / UPenn) · SSRN preprint
AI & teacher practice
The first randomized field evidence on teachers using AI to prepare class materials: access alone did not lift learning, students rated the classes duller, and students of the weakest teachers lost ground. Treat "give teachers AI" as an implementation problem, not a productivity upgrade. ›
Causal design12/15
Decision-relevance13/15
Sample & power7/10
Effect size5/10
Stat validity7/10
Peer-review3/10
Replication6/10
Conflicts / funding8/10
Generalizability5/10
The readA 10-week randomized field experiment (193 teachers, 2,800+ middle and high school students in a Turkish private school chain) gave treatment teachers a curriculum-customized ChatGPT assistant. Student intrinsic motivation fell 0.11 SD, classes were rated less enjoyable and less important, average achievement on externally administered exams was unchanged, and achievement and confidence declined for students of lower-performing teachers.
Strongest counter-argumentControl teachers were free to use consumer AI tools, so this estimates a customized assistant versus the status quo, not AI versus no AI; and the 'harms teaching' headline rests on a modest motivation decline plus subgroup effects, while average achievement did not change.
  • MajorSSRN preprint (June 2026), not yet peer reviewed.
  • MajorHeadline harm is concentrated in subgroup analyses (lower-performing teachers, prior heavy AI users); multiple outcomes raise forking-paths risk.
  • MinorSingle private school network in one country; a customized tool, so transfer to US districts is untested.
  • MinorMechanism unobserved: no classroom observation or analysis of the AI-generated materials.
Confidence: medium-low. Scored from the SSRN abstract and detailed independent reporting (Hechinger Proof Points, 2026-07-13); full tables not read directly, so effect and validity dimensions were scored conservatively.
2026 Watch 72/100 6
Hadra, Cambridge, Mesbah · Int'l Journal for Educational Integrity
AI detection
Do not use AI-detector scores as standalone evidence in integrity cases. The false-classification risk is high enough that any policy should require corroborating evidence and a fair appeals process. ›
Causal design9/15
Decision-relevance13/15
Sample & power5/10
Effect size7/10
Stat validity7/10
Peer-review8/10
Replication8/10
Conflicts / funding9/10
Generalizability6/10
The readTesting 192 balanced texts against Turnitin and Originality, overall accuracy was only 61 percent and 69 percent respectively, with both detectors failing badly on mixed human-AI text and degrading on longer and scientific writing.
Strongest counter-argumentThe accuracy figures rest on only 192 texts and two detectors at a single point in time, and because detector models are retrained frequently, the specific numbers may be stale soon even though the broad unreliability finding is robust.
  • MajorSmall corpus (192 texts) and only two commercial detectors tested.
  • MinorDetector versions evolve quickly, limiting the shelf life of exact accuracy numbers.
Confidence: medium. Verified title, authors, journal, design, sample, and the no-competing-interest statement.
2025 Watch 70/100 12
Zhao, Yue, Sun, Jiang, Li · Journal of Intelligence
Evidence base (meta-analysis)
Converging evidence that generative AI can meaningfully support higher-order thinking, strongest for K-12 and 8 to 16 week interventions. High unexplained heterogeneity and no preregistration mean the pooled number is a range, not a guarantee. ›
Causal design10/15
Decision-relevance12/15
Sample & power7/10
Effect size8/10
Stat validity8/10
Peer-review6/10
Replication5/10
Conflicts / funding8/10
Generalizability6/10
The readA peer-reviewed random-effects meta-analysis of 29 experiments and quasi-experiments (59 effect sizes) on generative AI and higher-order thinking. Pooled Hedges g = 0.61 (95% CI 0.49 to 0.73): moderate gains in problem-solving, critical thinking, and creativity. Heterogeneity was high.
Strongest counter-argumentAn I-squared of 77 percent means effects vary enormously, so a single pooled g masks contexts where generative AI does little or even harms; gray literature excluded and no PROSPERO registration also raise selective-reporting risk.
  • MajorHigh unexplained heterogeneity (I-squared = 77 percent) limits interpretability of the pooled effect.
  • MajorNo preregistration and gray literature excluded, raising publication and selection bias risk.
  • MinorIncludes quasi-experiments; weighted toward short Chinese and English-language studies.
Confidence: high. Full article read: effect sizes, moderators, Egger's test, heterogeneity, and funding all verified.
2025 Real 79/100 2
Heinrich, Baily, Chen, et al. · PLOS One
AI grading / workload
Solid evidence that rubric-anchored AI grading of short answers performs on par with instructors and can save grading time, with the caveat that it was one university in one discipline and did not measure student learning. ›
Causal design14/15
Decision-relevance12/15
Sample & power8/10
Effect size8/10
Stat validity9/10
Peer-review9/10
Replication5/10
Conflicts / funding9/10
Generalizability5/10
The readA preregistered RCT: 3,080 short-answer gradings across 271 students in four courses, with responses randomized to GPT-4-assisted versus human grading. AI grading was statistically indistinguishable from human grading; human feedback was rated only marginally more helpful (about 2.1 percentage points).
Strongest counter-argumentConcordance with human grades is not the same as accuracy; human grading is an imperfect gold standard, and the outcomes are proxies (regrade rates, perceived helpfulness) rather than learning. One institution, one short-answer format.
  • MinorSingle institution and one discipline limit generalizability.
  • MinorOutcomes are grading-concordance and perception proxies, not student learning.
Confidence: high. Full peer-reviewed article read: randomization, sample, preregistration, effect sizes, and funding all verified.
2023 Watch 73/100 485
Weber-Wulff, Anohina-Naumeca, et al. · Int'l Journal for Educational Integrity
AI detection
Strong, independent, widely-cited evidence not to base disciplinary decisions on AI-detector scores. Treat the specific numbers as dated but the unreliability conclusion as robust. ›
Causal design4/15
Decision-relevance14/15
Sample & power7/10
Effect size8/10
Stat validity7/10
Peer-review9/10
Replication9/10
Conflicts / funding9/10
Generalizability6/10
The readAcross roughly 14 detection tools, AI-text detectors were neither accurate nor reliable, were easily defeated by paraphrasing and translation, and skewed toward labeling text as human-written.
Strongest counter-argumentA vendor would call the findings obsolete (2023-era detectors vs GPT-3.5). The defense: the structural failure modes, paraphrasing and translation defeating detection plus a bias toward 'human,' have been independently replicated, so the qualitative conclusion is more durable than the percentages.
  • MajorRapid tool obsolescence: the detectors and models tested are now superseded.
  • MinorLargely descriptive classification scoring rather than formal diagnostic-accuracy statistics.
  • MinorModest corpus mixing human, AI, machine-translated, and obfuscated text.
Confidence: medium. Scored from the abstract, metadata, and detailed secondary summaries; the full PDF was not read end to end.
2024 Real 80/100 36
Wang, Ribeiro, Robinson, Loeb & Demszky (Stanford) · arXiv, preregistered RCT
Human-AI tutoring at scale
A low-cost AI assist for existing human tutors is a credible way to lift outcomes, and it is most worth funding where your tutor bench is weakest rather than as a replacement for strong tutors. ›
Causal design13/15
Decision-relevance14/15
Sample & power9/10
Effect size7/10
Stat validity8/10
Peer-review6/10
Replication6/10
Conflicts / funding8/10
Generalizability9/10
The readIn a tutor-randomized, preregistered RCT of 900 tutors and 1,800 K-12 students, giving tutors real-time AI guidance raised topic mastery by about 4 percentage points overall, with the largest gains (about 9 points) among initially lower-rated tutors.
Strongest counter-argumentThe research team is evaluating its own tool on a single tutoring platform, the work is not yet peer reviewed, and the 4 percentage point average effect is small with the real benefit concentrated almost entirely in lower-rated tutors.
  • MinorarXiv preprint, not yet peer reviewed (mitigated by OSF preregistration).
  • MinorCreators evaluating their own system; single tutoring platform context.
Confidence: medium. Verified design, sample, effect size, and preregistration; funding scored on visible facts.
2024 Watch 72/100 n/a
NFER, for the Education Endowment Foundation · EEF/NFER evaluation report
Teacher workload
The cleanest causal evidence to date that AI can reduce teacher prep workload without hurting resource quality. Treat the 25-minute figure as directional given self-report, and note it says nothing about student learning. ›
Causal design13/15
Decision-relevance14/15
Sample & power7/10
Effect size7/10
Stat validity7/10
Peer-review6/10
Replication4/10
Conflicts / funding8/10
Generalizability6/10
The readA school-randomized RCT of 259 science teachers across 68 English secondary schools. The ChatGPT arm cut weekly lesson-prep time from 81.5 to 56.2 minutes (about 31 percent, roughly 25 minutes) with no drop in blindly rated resource quality.
Strongest counter-argumentThe 25-minute weekly saving rests entirely on teachers' self-reported planning diaries, and teachers knew which arm they were in, so demand and recall effects could inflate the gap. It never tested whether pupils learned more.
  • MajorPrimary outcome (planning time) is self-reported by unblinded teachers, vulnerable to demand and recall bias.
  • MinorPublished as an EEF/NFER evaluation report, not a peer-reviewed journal article.
  • MinorSingle context (England KS3 science, one term); no pupil learning outcome measured.
Confidence: high. Design and results read across the EEF project page and NFER summary. Citation count: not indexed (gray-literature report).
2025 Watch 50/100 n/a
Savoldi, Attanasio, et al. · arXiv preprint
Equity & access
Population-representative evidence that AI access and use gaps track existing socioeconomic and gender lines, so an unmanaged AI rollout risks widening inequality. Weigh cautiously: it is a preprint, correlational, and about Italian adults generally. ›
Causal design4/15
Decision-relevance10/15
Sample & power8/10
Effect size5/10
Stat validity6/10
Peer-review2/10
Replication4/10
Conflicts / funding6/10
Generalizability5/10
The readA representative quota-sampled survey (n = 1,906, stratified to the Italian population) found less-educated, older, lower-income, and female respondents adopt and use generative AI less; 40 percent cite competence barriers.
Strongest counter-argumentThe design is purely cross-sectional and correlational, so it documents divides but cannot show AI is causing or will widen inequality, and its numbers should not be transported to a classroom or another country without confirmation.
  • MajorNot peer reviewed (arXiv preprint).
  • MajorPurely correlational and cross-sectional; no causal identification.
  • MinorGeneral Italian adult population rather than students; transferability limited.
Confidence: low. Abstract and partial PDF only; some statistics not verified. Citation count: not retrievable.
2026 Watch 55/100 n/a
Chambers, Kelley · AIED 2025 (Springer LNCS 15879), via arXiv
AI detection & equity
First empirical test of the claim that AI detectors disproportionately flag autistic writers: flag rates were under 2 percent overall, but significantly higher for likely-autistic authors. One more independent reason not to treat a detector score as evidence against a specific student, especially one with a disability. ›
Causal design5/15
Decision-relevance9/15
Sample & power5/10
Effect size4/10
Stat validity6/10
Peer-review7/10
Replication7/10
Conflicts / funding9/10
Generalizability3/10
The readA corpus study of roughly 60,000 Reddit posts split into likely-autistic and general-Reddit subcorpora, run through OpenAI's GPT-2 output detector. Under 2 percent of either subcorpus was flagged as AI-generated, but significantly more likely-autistic texts were flagged, and the textual features driving it did not map cleanly onto known features of AI text.
Strongest counter-argumentThe autism labels are proxies (subreddit membership, not diagnoses) and the detector tested is OpenAI's retired GPT-2 model, so the specific flag rates say little about the commercial detectors schools actually run today; what survives is the direction, which converges with the documented bias against non-native English writers.
  • MajorObsolete detector: OpenAI's GPT-2 output detector, not the commercial tools in current school use.
  • MajorProxy labeling: "likely-autistic" inferred from subreddit membership, not verified authorship.
  • MinorReddit posts, not student schoolwork; absolute flag rates were low in both groups.
Confidence: medium. Peer-reviewed venue (AIED 2025) and full abstract read; full tables not read directly.
2026 Watch 54/100 n/a
Benazet i Montobbio, Rotter & Hernández-Leo · arXiv (preprint)
AI literacy & PD design
Relevant to how you design AI training, not to whether AI helps students. Hands-on beat lecture immediately, then the gap closed by five weeks. If you are planning a one-off AI session for staff, this is the argument for building in a follow-up rather than expecting one afternoon to hold. ›
Causal design9/15
Decision-relevance9/15
Sample & power5/10
Effect size3/10
Stat validity6/10
Peer-review3/10
Replication6/10
Conflicts / funding9/10
Generalizability4/10
The readA quasi-experiment with 126 first-year engineering undergraduates compared a two-hour hands-on session on learning with generative AI against a lecture-based version of the same content, measuring metacognitive awareness before, immediately after, and again five weeks later. The hands-on group scored higher on engagement and on knowledge of cognition immediately after. By five weeks the two groups had converged on those measures, with only the hands-on group showing a continued within-group rise in regulation of cognition. No numerical effect sizes are reported in the abstract.
Strongest counter-argumentThe between-group advantage disappears at the only timepoint a school would plan around. What is left is a within-group trend on one subscale of a self-report instrument, which is a much smaller claim than "experiential instruction works better."
  • MajorBetween-group difference did not survive to the five-week follow-up; the surviving result is a within-group trend on a single subscale.
  • MinorSelf-reported metacognitive awareness, not observed behavior or learning outcomes.
  • MinorPreprint, no preregistration stated; 126 students in one first-year engineering course in one institution.
  • MinorHigher education, not K-12; transfer to school staff PD is plausible but untested.
Confidence: medium-low. Scored from the arXiv abstract and metadata; effect sizes are not reported there, so magnitude and statistical-validity dimensions were scored conservatively. Citation count: too new, none recorded.
2026 Watch 47/100 n/a
Nagashima, Siegrist, Scholz, et al. · Proc. ACM HCI (CSCW 2026), via arXiv
Student-AI control & trust
Early qualitative evidence that teachers and students want different things from classroom AI, especially on how much control students get and how much they trust it. Useful as a checklist of rollout questions to ask both groups, not as evidence for any policy. ›
Causal design6/15
Decision-relevance8/15
Sample & power2/10
Effect size3/10
Stat validity5/10
Peer-review8/10
Replication3/10
Conflicts / funding9/10
Generalizability3/10
The readA storyboard speed-dating study with 16 school students and 15 teachers in Germany found systematic misalignment between teacher and student views on student-AI decision-making control, on how much each side trusts AI, and on the social and emotional side of learning with it. Accepted at CSCW 2026.
Strongest counter-argumentThirty-one participants in one country reacting to storyboards measure stated preferences, not behavior; nothing here shows what teachers or students actually do with control once they have it, and the specific misalignments may not describe any other school system.
  • MajorVery small sample (16 students, 15 teachers) in a single national context.
  • MinorStoryboard speed-dating elicits stated preferences rather than observed classroom behavior.
  • MinorNo replication; first study of its pairing design in K-12 AI.
Confidence: medium. Scored from the abstract and venue metadata (peer-reviewed CSCW acceptance confirmed); full findings not read directly. Citation count: too new, none recorded.
2024 Watch 63/100 126
Lee, Pope, Miles, Zarate · Computers and Education: Artificial Intelligence
Academic integrity
A useful counterweight to moral-panic framing, but the flat self-report likely undercounts AI use that students do not label as cheating, so do not treat it as proof that AI cheating is contained. ›
Causal design5/15
Decision-relevance13/15
Sample & power6/10
Effect size6/10
Stat validity6/10
Peer-review9/10
Replication6/10
Conflicts / funding7/10
Generalizability5/10
The readSelf-reported high school cheating stayed roughly stable after ChatGPT's release, with changes varying by cheating type.
Strongest counter-argumentThe reassuring 'cheating did not rise' result may be a measurement artifact. If students do not categorize AI assistance as cheating, self-reported rates stay flat even as AI-assisted shortcutting grows.
  • MajorSelf-reported cheating via anonymous repeated cross-sectional surveys, vulnerable to social-desirability and shifting norms.
  • MajorOnly three Challenge Success partner schools that skew affluent and high-achieving.
  • MinorPre/post window straddles COVID-era disruption, a confound.
  • MinorStability could reflect definitional drift rather than unchanged behavior.
Confidence: low. Scored from abstract and secondary summaries; full methodology and exact sample size were paywalled.
2023 Watch 64/100 60
Thomas, Lin, Gatz, et al. · ACM Learning Analytics & Knowledge
Human-AI tutoring
Encouraging directional evidence that a human-plus-AI model can lift outcomes for struggling students at a defined cost, but the quasi-experimental design means it should inform pilots, not procurement. ›
Causal design8/15
Decision-relevance12/15
Sample & power7/10
Effect size5/10
Stat validity6/10
Peer-review7/10
Replication7/10
Conflicts / funding6/10
Generalizability6/10
The readAcross three urban low-income US middle schools (585 students), adding human tutors supported by AI tools to math software was associated with better proficiency and usage, with larger apparent gains for lower-achieving students, at roughly $700 per student per year.
Strongest counter-argumentWithout randomization across three differently structured deployments, the positive differences could reflect selection into the hybrid condition rather than the tutoring itself, and no quantified effect sizes are reported.
  • MajorQuasi-experimental with no randomization; conditions differ across schools.
  • MinorNo numerical effect sizes available in the abstract.
  • MinorFunding and conflict-of-interest disclosures not visible.
Confidence: low. Scored largely from the abstract; effect sizes could not be verified, forcing conservative scoring.
2024 Watch 74/100 n/a
Demszky, Liu, Hill, Sanghi & Chung · EdWorkingPaper 23-875
AI coaching for teachers
Pre-registered randomized evidence that automated feedback can shift one specific teaching practice in real classrooms. Useful for professional learning pilots, but it measures teacher talk, not student learning, so do not let a vendor cite it as an achievement result. ›
Causal design14/15
Decision-relevance12/15
Sample & power7/10
Effect size6/10
Stat validity9/10
Peer-review7/10
Replication8/10
Conflicts / funding5/10
Generalizability6/10
The readA pre-registered randomized controlled trial with 224 Utah mathematics and science teachers, run in partnership with TeachFX, testing automated email feedback on "focusing questions" that press students for explanation. Teachers opened the feedback emails 53 to 65 percent of the time, and their use of focusing questions rose 20 percent (p < 0.01) against control. No other teaching practices moved. Interviews with 13 teachers surfaced skepticism about accuracy, data privacy concerns and time constraints.
Strongest counter-argumentThe outcome measured is teacher talk, a process proxy, not student learning. A 20 percent increase off a low base may be small in absolute terms, and the study cannot show that any student understood more as a result.
  • MajorProcess outcome only; no student learning measured.
  • MajorConducted in partnership with TeachFX, the vendor whose product is evaluated. Disclosed, but a live conflict.
  • MinorSingle state, mathematics and science only; working paper, not peer reviewed.
Confidence: medium-high. Scored from the EdWorkingPapers abstract, which states the pre-registration, sample, effect size and p-value directly. Citation count not shown because a verified count was not available at scoring time.
2026 Watch 72/100 n/a
Mata, Russell & Page · EdWorkingPaper 26-1409
Chatbots & student support
A rare long-horizon randomized test of a support chatbot. It moved administrative task completion and nothing else. Buy this category to reduce paperwork friction, not to move learning, and say so out loud when you buy it. ›
Causal design13/15
Decision-relevance11/15
Sample & power8/10
Effect size6/10
Stat validity8/10
Peer-review5/10
Replication9/10
Conflicts / funding8/10
Generalizability4/10
The readA four-year randomized controlled trial of an AI-enabled text-messaging chatbot at a large urban public university, combined with system observation and administrator interviews. Students stayed receptive over time and impacts concentrated in improved completion of time-sensitive administrative tasks, with no detectable effects on academic performance or persistence. Centralized ownership and flexible communication emerged as conditions for sustained implementation.
Strongest counter-argumentA precise null is not proof of absence. The study may be underpowered to detect small but real effects on persistence, and a single institution with one chatbot cannot rule out that a different implementation would move learning.
  • MajorSingle institution and post-secondary only; transfer to K-12 is an inference, not a finding.
  • MinorWorking paper; not yet peer reviewed.
  • MinorNo preregistration stated in the abstract.
Confidence: medium. Scored from the EdWorkingPapers abstract and metadata; full tables and power calculations not read directly. Citation count not shown because the paper is too recent for a stable count.
2026 Watch 52/100 n/a
Harrison, Khowaja, Dobson, Uwimpuhwe & Higgins (What Worked Education; Durham University) · arXiv preprint 2609.14789
AI tutoring & achievement
A positive four-week result for an AI revision platform in English GCSE science, funded and commissioned by the company whose product it tests. Treat it as a reason to ask for an independent trial with a standardized outcome, not as evidence to buy. ›
Causal design11/15
Decision-relevance10/15
Sample & power6/10
Effect size6/10
Stat validity5/10
Peer-review2/10
Replication5/10
Conflicts / funding2/10
Generalizability5/10
The readThirty-nine practitioner-led, individually randomized micro-trials in English secondary schools already using Medly, an AI tutoring platform, on GCSE Biology, Chemistry and Physics topics over four weeks. Control students were allocated 30 minutes a week of self-directed revision on the same topic using resources other than Medly. Of 929 students at baseline, 644 completed the post-test, 30.7 percent attrition overall, 33.2 percent in control and 27.7 percent in the intervention group. Medly students scored higher on a 35-mark GCSE-aligned post-test, an adjusted difference of 2.56 marks, Hedges' g = 0.33 (95% CI 0.18 to 0.48), with no evidence that the effect differed by subject or by Pupil Premium status. The authors write that they interpret the findings as preliminary.
Strongest counter-argumentThe evaluation was funded by Medly, which commissioned the evaluators, gave access to schools already using its platform, and supplied the usage data, while the paper describes itself as an independent evaluation. The outcome is a non-standardized test, AI-marked and then checked by class teachers who knew which students were treated, and the control arm lost more students than the treatment arm. Each of those can inflate a four-week effect.
  • MajorVendor-funded and vendor-commissioned, with the vendor supplying school access and usage data. Disclosed in the declarations, so not critical, but it sits beside an "independent evaluation" framing.
  • MajorPrimary outcome is a non-standardized, curriculum-aligned test, AI-marked and checked by unblinded class teachers who could amend marks.
  • Major30.7 percent attrition, higher in control (33.2 percent) than intervention (27.7 percent), reported descriptively without a formal test.
  • MinorNo preregistration found; arXiv preprint.
  • MinorSchools already using the platform, headteacher consent, four-week window; only 6 of 39 teacher trials returned the process survey.
Confidence: high on the reported design and numbers, read from the full arXiv PDF including the funding and competing-interests declarations. Citation count not shown because the paper is too recent for a stable count.
2026 Watch 63/100 n/a
Restrepo Morales, Rodríguez Flores, Giraldo Betancur & Zea Restrepo (Universidad Autónoma del Perú; Tecnológico de Antioquia) · EdArXiv preprint
Reading AI claims
A worked method for testing a headline AI-tutoring result before believing it. When a pilot group is said to match a rich country's scores, ask for the participating schools' baseline and the sampling protocol: here, selecting the top 5.5 percent of students would reproduce the math claim with no learning at all. ›
Causal design8/15
Decision-relevance11/15
Sample & power5/10
Effect size7/10
Stat validity7/10
Peer-review3/10
Replication7/10
Conflicts / funding9/10
Generalizability6/10
The readA secondary analysis of a public claim. In September 2026 the World Bank reported that students from 171 public schools in El Salvador's education modernization pilot with AI tutors had reading, mathematics and science results comparable to the 2022 averages of Germany and Sweden; the government put the number of volunteer students assessed at 1,198, and no technical report of the assessment has been released. The authors recover the spread of Salvadoran achievement from published PISA 2025 figures and compute what share of the top of that distribution would reproduce the benchmark with zero learning: 5.5 percent in mathematics, 11.7 percent in science and 17.2 percent in reading against the German averages. The pilot assessed 7.0 students per school; the national PISA 2025 sample assessed 25.4. Taken at face value, the claim implies a gain of 1.29 standard deviations, about 6.5 years of schooling in one year.
Strongest counter-argumentThe bound rests on a distributional assumption used to recover the standard deviation from two published figures, and the authors observe neither the selection nor the participating schools' baseline. They say plainly that the paper is not evidence the gain is unreal: a genuine effect of 0.30 standard deviations combined with 15 percent selection reproduces the announced result exactly.
  • MajorThe standard deviation is inferred from a published mean and a Level 2 share under a distributional assumption. The authors report robustness to skewness and to a 20 percent error in the standard deviation.
  • MajorBounds, not estimates: selection and baseline are unobserved, so the paper shows how much selection would suffice, not how much occurred.
  • MinorPreprint and not preregistered; a secondary analysis of published aggregates, with code and an independent reimplementation deposited openly.
  • MinorThe authors disclose that a generative AI assistant drafted text under their direction.
  • MinorA single national case; what transfers to a US district is the method and the reporting standard, not the numbers.
Confidence: high on the method and the reported figures, read from the full 44-page manuscript. No funding and no competing interests declared. Citation count not shown because the paper is too recent for a stable count.
How a study is scored

Every research candidate is run through the Research Integrity Gate before it earns a verdict. Nine dimensions, scored 0 to 100. Expand any one to see the exact criterion the pipeline scores against, taken verbatim from the ranking prompt.

Causal identification / design15

Is this an RCT or a strong quasi-experimental design (diff-in-diff, RD, IV, matched comparison) with confounds controlled, or is it merely correlational? Reward clean causal identification; demote "X is associated with Y" framing dressed up as causal.

Decision-relevance to a school / education leader15

Would a principal, district leader, or department head actually change a decision because of this? Inert-but-interesting scores low here too.

Sample & power10

Adequate N, representative sample, and adequately powered to detect the claimed effect. Underpowered or convenience samples score low.

Effect size & practical magnitude10

Is the effect big enough to matter in a real school, not just statistically detectable? A significant-but-trivial effect scores low.

Statistical validity10

Watch for p-hacking, uncorrected multiple comparisons, and garden-of-forking-paths signals (many outcomes, flexible specifications, results that hinge on one cut of the data). Pre-specified, robust analyses score high.

Peer-review / preregistration status10

Preprint scores lower than peer-reviewed; WWC-vetted scores highest. Preregistration adds points; an undisclosed analysis path loses them.

Replication & convergence with prior evidence10

Does it replicate or converge with the existing body of evidence, or is it a lone surprising result? Reward convergence; treat outliers with caution.

Conflicts of interest / funding10

Who funded and authored it? Vendor-funded or self-funded studies of the funder's own product are demoted hard. Independent funding scores high.

Generalizability to real classrooms / districts10

Does the finding transfer to a working leader's context, to real classrooms, real constraints, and real populations, or only to a lab or a single atypical site?

Then a devil's advocate tries to refute the study and names the single strongest counter-argument. Any flaw rated critical caps the score and blocks a passing verdict. A study is Real only at 75 or higher with no critical flaw, Watch when the direction is real but the evidence is not there yet, and anything weaker is held back rather than published.

How the list is ordered. The Gate score decides whether a paper appears, and it also shapes the order. Order blends three normalized signals: the Gate score (50 percent), citation velocity in citations per year (30 percent), and recency (20 percent). So a rigorous study surfaces even with fewer citations, an influential one surfaces even if older, and a strong new study is not buried under older, more-cited ones. A paper with no reliable citation count contributes zero on the citation axis and is placed by quality and recency, shown as n/a.

Attribution

The Gate adapts the architecture of three open peer-review skills built for the Claude ecosystem. Full credit to their creators:

Citation counts via Semantic Scholar / OpenAlex, retrieved 2026-06-29.

The Margin · A weekly AI-in-education digest for education leaders
Home · Methodology · My Planning Partner · @myplanningpartner