[aied-2026-itextbooks] LLM feedback isn't automatically better: Static scaffolds outperform dynamic feedback in textbook-embedded practice

README.md

@@ -35,11 +35,12 @@ Directory | Paper
 [aied-2024-evallac](https://github.com/vitalsource/data/tree/main/aied-2024-evallac) | [Exploring large language models for evaluating automatically generated questions](https://drive.google.com/file/d/1vO21K60lDf18izQdr79CpJxOvfXvHQBM/view)
 [edm-2024](https://github.com/vitalsource/data/tree/main/edm-2024) | [Investigating student ratings with features of automatically generated questions: A large-scale analysis using data from natural learning contexts](https://doi.org/10.5281/zenodo.12729796)
 [jedm-2025](https://github.com/vitalsource/data/tree/main/jedm-2025) | [Intrinsic and contextual factors impacting student ratings of automatically generated questions: A large-scale data analysis](https://doi.org/10.5281/zenodo.15174917)
-[its-2025](https://github.com/vitalsource/data/tree/main/its-2025) | [Scaling effective characteristics of ITSs: A preliminary analysis of LLM-based personalized feedback](https://doi.org/10.1007/978-3-031-98281-1_13) ![NEW](https://img.shields.io/badge/status-new-brightgreen) <!-- remove badge after Oct 2, 2025 -->
+[its-2025](https://github.com/vitalsource/data/tree/main/its-2025) | [Scaling effective characteristics of ITSs: A preliminary analysis of LLM-based personalized feedback](https://doi.org/10.1007/978-3-031-98281-1_13)
 [edm-2025-causaledm](https://github.com/vitalsource/data/tree/main/edm-2025-causaledm) | [Improving automatically generated fill-in-the-blank answer selection with an LLM-based agreement filter](https://drive.google.com/file/d/1qJzkLz78t1afIJ4KKKxI7tCxrhWklOcP/view)
 [l@s-2025](https://github.com/vitalsource/data/tree/main/l@s-2025) | [Refining sentence selection for automatic cloze question generation with large language models](https://doi.org/10.1145/3698205.3733926)
 [aied-2025-evallac](https://github.com/vitalsource/data/tree/main/aied-2025-evallac) | [Open-ended questions need personalized feedback: Analyzing LLM-enabled features with student data](https://drive.google.com/file/d/15HCyN1uU6AtIT8aVpMJCS5ZQa75buA_g/view)
 [aied-2025-itextbooks](https://github.com/vitalsource/data/tree/main/aied-2025-itextbooks) | [Improving textbook accessibility through AI simplification: Readability improvements and meaning preservation](https://ceur-ws.org/Vol-4010/itb25_s2p2.pdf)
+[aied-2026-itextbooks](https://github.com/vitalsource/data/tree/main/aied-2026-itextbooks) | [LLM feedback isn’t automatically better: Static scaffolds outperform dynamic feedback in textbook-embedded practice](https://intextbooks.science.uu.nl/workshop2026/files/itb26_s1p1.pdf) ![NEW](https://img.shields.io/badge/status-new-brightgreen) <!-- remove badge after Dec 1, 2026 -->
 
 Unless otherwise noted, our datasets are available under the
 [Creative Commons Attribution 4.0 International

aied-2026-itextbooks/README.md

@@ -0,0 +1,148 @@
+# Dynamic vs. Static Feedback for Textbook-Embedded FITB Practice Dataset
+
+This directory contains the dataset and analysis code for our paper:
+
+Johnson, B. G., Dittel, J. S., Ortiz, O. J., Bistolfi, R., Clark,
+M. W., Jerome, B., Benton, R., & Van Campenhout, R. (2026). LLM
+feedback isn't automatically better: Static scaffolds outperform
+dynamic feedback in textbook-embedded practice. In *Proceedings of the
+Seventh Workshop on Intelligent Textbooks at the 27th International
+Conference on Artificial Intelligence in Education*. CEUR Workshop
+Proceedings. [https://intextbooks.science.uu.nl/workshop2026/files/itb26_s1p1.pdf](https://intextbooks.science.uu.nl/workshop2026/files/itb26_s1p1.pdf)
+
+This paper was presented at [AIED 2026](https://www.aied-conference.org/2026) as part of the
+[Seventh Workshop on Intelligent Textbooks (iTextbooks)](https://intextbooks.science.uu.nl/workshop2026/).
+
+## Description
+
+Formative practice embedded within textbooks has been shown to increase learning
+outcomes through the doer effect — the principle that doing practice while learning
+is substantially more effective than reading alone. The VitalSource Bookshelf ereader
+platform delivers formative practice through CoachMe, a free study feature that
+integrates automatically generated questions directly alongside textbook content.
+CoachMe supports several question types including fill-in-the-blank (FITB) cloze
+questions, which are the focus of this study.
+
+When a student answers an FITB question incorrectly, as shown in Figure 1, they
+receive feedback along with options to retry, reveal the correct answer, and rate the
+question.
+
+<p align="center">
+<img alt="An example FITB formative practice question in a chemistry textbook." src="./CoachMe_screenshot.png"/>
+<br><b>Figure 1.</b> An example FITB formative practice question in a chemistry textbook.
+</p>
+
+Prior to this study, CoachMe deployed three types of static feedback for FITB
+questions (common answer, context, and outcome feedback) selected according to a
+preference hierarchy. Common answer feedback presents a second sentence from nearby
+textbook content with the same target term removed, providing another retrieval cue.
+Context feedback provides an extended excerpt of the surrounding passage. Outcome
+feedback informs the student that their response is incorrect. Prior research found
+that common answer feedback performed best on key behavioral outcomes and serves as
+the benchmark for any new feedback approach.
+
+Large language models (LLMs) lower the practical barrier to generating dynamic,
+error-sensitive feedback conditioned on a student's actual incorrect answer.
+Applying this approach to FITB questions, the system uses GPT-4.1 nano to generate
+feedback intended to acknowledge the student's specific answer, explain why it does not
+fit, and redirect the student without revealing the correct answer. A no-leak
+guardrail checks generated feedback for the presence of the correct answer word; if
+the answer is detected, the system falls back to static feedback. Figure 2 illustrates
+common answer and dynamic feedback side by side.
+
+<p align="center">
+<img alt="Examples of common answer and dynamic feedback for FITB questions." src="./feedback_examples.png"/>
+<br><b>Figure 2.</b> Examples of common answer and dynamic feedback for FITB questions.
+</p>
+
+This dataset captures 33,834 student-question sessions collected during a randomized
+deployment from April 9, 2026, through May 8, 2026, using textbooks from eight
+publishers who granted permission for generative AI research. Each incorrect first
+attempt was randomly assigned with equal probability to either the dynamic feedback
+condition or the existing static feedback approach. The unit of analysis is the
+student-question session: all interactions by a given student on a given question,
+anchored at an incorrect first attempt and followed through the next recorded action.
+The final dataset covers 5,596 students, 23,148 questions, and 1,363 textbooks. The
+top subject domains by percentage of sessions were Psychology (22.7%), Social Science
+(19.1%), Political Science (15.7%), Business & Economics (12.2%), and Law (7.5%).
+
+The research goals were to:
+
+- Determine whether dynamic LLM-generated feedback improves student outcomes relative
+  to the existing static feedback approach under random assignment.
+- Identify which delivered feedback types are most associated with the observed
+  differences in outcomes.
+- Draw implications about the relationship between feedback design and target term
+  recovery in textbook-embedded formative practice.
+
+## Example Session
+
+The following example, drawn from the textbook *Biological Psychology*
+(Lyons et al., 2014), illustrates how the no-leak guardrail shapes
+feedback delivery. For the item "The ANS is essentially the collection
+of ______ that act as the manager of your internal organs," a student
+answered "neurons."
+
+The generated dynamic feedback was: *"You answered 'neurons,' which are the cells
+that make up nerves, but the question is asking about the overall collection that acts
+as the manager of your internal organs. What do we call that collection of nerve
+fibers? Would you like to try again?"* This response is coherent and engages the
+student's answer, but it approaches a definition of the target term and is rejected
+by the no-leak guardrail. The system falls back to common answer feedback: *"Some
+texts refer to a sub-set of the ANS known as the enteric nervous system, which refers
+to a fine network of ______ that are found only in the walls of the digestive tract
+and control the digestive process."* The correct answer is "nerves."
+
+In the dataset, this session would appear with `assigned_condition` = `dynamic` and
+`realized_condition` = `common_fallback`, reflecting the discrepancy between
+assignment and delivered feedback introduced by the guardrail.
+
+## Data Files and Analysis Code
+
+The provided files are:
+
+| File | Description |
+| --- | --- |
+| `sessions.parquet` | Session-level dataset of 33,834 incorrect-first-attempt sessions with feedback conditions and behavioral outcomes |
+| `Feedback Analysis.ipynb` | Jupyter notebook for replication of the primary and secondary analyses in the paper |
+| `dynamic_feedback_prompt.txt` | The full LLM prompt used to generate dynamic feedback |
+
+The dataset contains the following fields:
+
+| Field | Type | Description |
+| --- | --- | --- |
+| `timestamp` | datetime | Date and time of the incorrect first attempt |
+| `student_id` | string | Anonymized student identifier |
+| `question_id` | string | Unique question identifier |
+| `textbook_id` | string | Unique textbook identifier |
+| `subject` | string | Textbook BISAC major subject heading (e.g., "Psychology") |
+| `question_stem` | string | Question text with the target term replaced by a blank (`______`) |
+| `correct_answer` | string | Target term for the blank |
+| `student_answer` | string | Student's incorrect first attempt |
+| `assigned_condition` | categorical | Randomized assignment: `dynamic` or `static` |
+| `realized_condition` | categorical | Feedback type actually delivered: `dynamic`, `common_assigned`, `common_fallback`, `context_assigned`, `context_fallback`, `outcome_assigned`, or `outcome_fallback` |
+| `feedback_text` | string | Text of the feedback shown to the student |
+| `next_action` | categorical | Student's next recorded action: `answer_reveal`, `correct_retry`, `incorrect_retry`, `answer_suggestion`, `no_action`, `rating_thumbs_up`, or `rating_thumbs_down` |
+| `answer_reveal` | categorical | 1 if the student's next action was to reveal the correct answer, 0 otherwise |
+| `correct_retry` | categorical | 1 if the student's next action produced a correct response, 0 otherwise |
+| `edit_distance` | integer | Damerau-Levenshtein edit distance between `student_answer` and `correct_answer` |
+| `edit_distance_quartile` | categorical | Quartile of `edit_distance` within the dataset (Q1–Q4) |
+
+## Acknowledgments
+
+We thank the following publishers for granting permission to include student use of
+CoachMe formative practice questions in their textbooks as part of this open dataset:
+
+- Cambridge University Press
+- Emond Publishing
+- F. A. Davis Company
+- Human Kinetics Publishers
+- OpenStax
+- SAGE Publications, Inc. (US)
+- SAGE Publications, Ltd. (UK)
+- Taylor & Francis
+
+## Contact Us
+
+If you have questions, please feel free to email
+[benny.johnson@vitalsource.com](mailto:benny.johnson@vitalsource.com).

aied-2026-itextbooks/dynamic_feedback_prompt.txt

@@ -0,0 +1,115 @@
+Dynamic Feedback Prompt — LLM Feedback Isn't Automatically Better (iTextbooks 2026)
+======================================================================================
+
+Model: gpt-4.1-nano
+Temperature: 0.0
+
+This file documents the prompt used to generate dynamic, error-sensitive feedback for
+incorrect fill-in-the-blank (FITB) responses in the VitalSource CoachMe platform. The
+prompt is split into a system message and a user message, as sent to the OpenAI Chat
+Completions API. Placeholders shown in curly braces — {question}, {correct_answer},
+{student_answer} — are substituted with the actual values for each session at runtime.
+
+--------------------------------------------------------------------------------------
+SYSTEM MESSAGE
+--------------------------------------------------------------------------------------
+
+You are an AI tutor giving feedback on student answers to practice questions. You are
+very professional. All your responses are ethical. You never use profanity and you never
+respond to messages that violate any ethical, moral or legal standard.
+
+--------------------------------------------------------------------------------------
+USER MESSAGE
+--------------------------------------------------------------------------------------
+
+You will be given a fill-in-the-blank question, the correct answer, and a student's answer that is incorrect.
+Write feedback that helps the student learn. Follow these rules:
+
+1. Clarity & tone
+- Keep feedback clear, concise, and supportive.
+- Use everyday language, not overly formal or textbook-like.
+- Keep responses short: about 2–3 sentences max.
+
+2. Acknowledgment
+- Usually quote the student's answer directly (e.g., "You said 'boiling'…").
+- Do this when it makes the explanation clearer and ties the feedback directly to their response.
+- If the answer is nonsense or quoting it would be awkward, just restate the focus of the question instead.
+
+3. Explain the contradiction
+- Point out why the student's answer doesn't fit with what they should already know.
+- Example: "You said 'boiling,' but that's when liquid turns to gas, not when solid turns to liquid."
+
+4. Refocus on the question
+- Remind them of what the question is really asking about, without giving the correct answer word.
+- Example: "This question is asking about the temperature where a solid turns into a liquid."
+
+5. Encourage the student to retry
+- Phrase the last part of the feedback as a question to invite them to try again.
+- Example: "What do we call that point?"
+
+6. Nonsense answers
+- If the answer is irrelevant, gently redirect them back on track. No need to repeat the answer.
+- Example: "That doesn't seem related. This is about the temperature where a solid turns into a liquid."
+
+7. Close-but-not-exact answers (synonyms or near-synonyms)
+- If the student's answer is basically the same idea with different wording:
+    - Acknowledge that it makes sense.
+    - Explain that the textbook uses different terminology, and strongly encourage the student to use the textbook terminology.
+- Example: "You said "fusion" which also refers to the solid-to-liquid transition. The textbook uses different terminology. Do you know the word the textbook uses?"
+
+8. Valid-but-different answers
+- If the student's answer is plausible or correct in another context but not the specific concept the textbook/question is targeting:
+    - Acknowledge that their answer makes sense.
+    - Note that the item is focusing on a different concept in the context of their textbook.
+    - Gently steer them toward that context without giving away the word.
+- Example (science): "You said 'hydrides,' which are hydrogen-containing compounds. In this section, though, the focus is on substances known for acidic behavior. What are those called?"
+- Example (non-science): "You said 'legislature,' which is the general type of body that makes laws. Here the question is asking for the specific U.S. body made up of the House and Senate. What is it called?"
+
+---
+
+FEW-SHOT EXAMPLES
+
+Example 1
+Question: The temperature at which the solid and liquid phases of a given substance are in equilibrium is called the _______ point of the solid or the freezing point of the liquid.
+Correct answer: melting
+Student answer: boiling
+Feedback: You said **boiling**, but that's when a liquid turns to gas, not when a solid turns to liquid. This question is asking about the temperature where a solid changes into a liquid. What do we call that point?
+
+---
+
+Example 2
+Question: The process by which plants use sunlight to make their own food is called _______.
+Correct answer: photosynthesis
+Student answer: respiration
+Feedback: You answered **respiration**, which is when cells break down food to release energy. This question is about the opposite process—how plants *make* food from sunlight. What's that process called?
+
+---
+
+Example 3
+Question: The _______ War was fought between the North and South regions of the United States from 1861 to 1865.
+Correct answer: Civil
+Student answer: banana
+Feedback: "Banana" doesn't seem related to this question. Think about U.S. history—the war between the North and South in the 1860s. What war was that?
+
+---
+
+Example 4
+Question: The temperature at which a solid turns into a liquid is called the _______ point.
+Correct answer: melting
+Student answer: fusion point
+Feedback: You said **fusion point**, which does describe the solid-to-liquid transition. The textbook uses a different word here. Can you think of another way to say it?
+
+---
+
+Example 5
+Question: Some compounds containing hydrogen are members of an important class of substances known as ________.
+Correct answer: acids
+Student answer: hydrides
+Feedback: You said **hydrides**, and those are indeed compounds containing hydrogen. In this section, though, the question is about the group of substances known for their acidic behavior. What are those called?
+
+---
+
+NEW INPUT
+Question: {question}
+Correct answer: {correct_answer}
+Student answer: {student_answer}