diff --git a/README.md b/README.md index b5b1168..fcc7fac 100644 --- a/README.md +++ b/README.md @@ -35,11 +35,12 @@ Directory | Paper [aied-2024-evallac](https://github.com/vitalsource/data/tree/main/aied-2024-evallac) | [Exploring large language models for evaluating automatically generated questions](https://drive.google.com/file/d/1vO21K60lDf18izQdr79CpJxOvfXvHQBM/view) [edm-2024](https://github.com/vitalsource/data/tree/main/edm-2024) | [Investigating student ratings with features of automatically generated questions: A large-scale analysis using data from natural learning contexts](https://doi.org/10.5281/zenodo.12729796) [jedm-2025](https://github.com/vitalsource/data/tree/main/jedm-2025) | [Intrinsic and contextual factors impacting student ratings of automatically generated questions: A large-scale data analysis](https://doi.org/10.5281/zenodo.15174917) -[its-2025](https://github.com/vitalsource/data/tree/main/its-2025) | [Scaling effective characteristics of ITSs: A preliminary analysis of LLM-based personalized feedback](https://doi.org/10.1007/978-3-031-98281-1_13) ![NEW](https://img.shields.io/badge/status-new-brightgreen) +[its-2025](https://github.com/vitalsource/data/tree/main/its-2025) | [Scaling effective characteristics of ITSs: A preliminary analysis of LLM-based personalized feedback](https://doi.org/10.1007/978-3-031-98281-1_13) [edm-2025-causaledm](https://github.com/vitalsource/data/tree/main/edm-2025-causaledm) | [Improving automatically generated fill-in-the-blank answer selection with an LLM-based agreement filter](https://drive.google.com/file/d/1qJzkLz78t1afIJ4KKKxI7tCxrhWklOcP/view) [l@s-2025](https://github.com/vitalsource/data/tree/main/l@s-2025) | [Refining sentence selection for automatic cloze question generation with large language models](https://doi.org/10.1145/3698205.3733926) [aied-2025-evallac](https://github.com/vitalsource/data/tree/main/aied-2025-evallac) | [Open-ended questions need personalized feedback: Analyzing LLM-enabled features with student data](https://drive.google.com/file/d/15HCyN1uU6AtIT8aVpMJCS5ZQa75buA_g/view) [aied-2025-itextbooks](https://github.com/vitalsource/data/tree/main/aied-2025-itextbooks) | [Improving textbook accessibility through AI simplification: Readability improvements and meaning preservation](https://ceur-ws.org/Vol-4010/itb25_s2p2.pdf) +[aied-2026-itextbooks](https://github.com/vitalsource/data/tree/main/aied-2026-itextbooks) | [LLM feedback isn’t automatically better: Static scaffolds outperform dynamic feedback in textbook-embedded practice](https://intextbooks.science.uu.nl/workshop2026/files/itb26_s1p1.pdf) ![NEW](https://img.shields.io/badge/status-new-brightgreen) Unless otherwise noted, our datasets are available under the [Creative Commons Attribution 4.0 International diff --git a/aied-2026-itextbooks/CoachMe_screenshot.png b/aied-2026-itextbooks/CoachMe_screenshot.png new file mode 100644 index 0000000..ed0abf1 Binary files /dev/null and b/aied-2026-itextbooks/CoachMe_screenshot.png differ diff --git a/aied-2026-itextbooks/Feedback Analysis.ipynb b/aied-2026-itextbooks/Feedback Analysis.ipynb new file mode 100644 index 0000000..c9487d2 --- /dev/null +++ b/aied-2026-itextbooks/Feedback Analysis.ipynb @@ -0,0 +1,1579 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This notebook contains Python code for reproducing the results in our paper on comparing dynamic LLM-generated feedback to static feedback for automatically generated fill-in-the-blank practice questions delivered in a textbook ereader platform:\n", + "\n", + "Johnson, B. G., Dittel, J. S., Ortiz, O. J., Bistolfi, R., Clark, M. W., Jerome, B., Benton, R., & Van Campenhout, R. (2026). LLM feedback isn't automatically better: Static scaffolds outperform dynamic feedback in textbook-embedded practice. In *Proceedings of the Seventh Workshop on Intelligent Textbooks at the 27th International Conference on Artificial Intelligence in Education*. CEUR Workshop Proceedings. [https://intextbooks.science.uu.nl/workshop2026/files/itb26_s1p1.pdf](https://intextbooks.science.uu.nl/workshop2026/files/itb26_s1p1.pdf)\n", + "\n", + "This paper was presented at [AIED 2026](https://www.aied-conference.org/2026) as part of the [Seventh Workshop on Intelligent Textbooks (iTextbooks)](https://intextbooks.science.uu.nl/workshop2026/).\n", + "\n", + "Results are presented in the order they occur, organized by the paper's sections. For each result, an excerpt from the paper is given followed by code to compute the result from the dataset provided. Example:\n", + "\n", + ">The final dataset contains 33,834 sessions across 23,148 questions, 5,596 students, and 1,363 textbooks.\n", + "\n", + "```len( sessions ), sessions.question_id.nunique(), sessions.student_id.nunique(), sessions.textbook_id.nunique()```\n", + "\n", + "Please refer to the paper for additional context." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "ExecuteTime": { + "end_time": "2023-10-02T09:20:39.221294Z", + "start_time": "2023-10-02T09:20:37.660620Z" + } + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "import pandas as pd" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "%load_ext rpy2.ipython" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Some features are not enabled in this build of Arrow. Run `arrow_info()` for more information.\n", + "The repository you retrieved Arrow from did not include all of Arrow's features.\n", + "You can install a fully-featured version by running:\n", + "`install.packages('arrow', repos = 'https://apache.r-universe.dev')`.\n", + "\n", + "Attaching package: ‘arrow’\n", + "\n", + "The following object is masked from ‘package:utils’:\n", + "\n", + " timestamp\n", + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "%%R\n", + "library( arrow )\n", + "library( glmmTMB )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Read dataset of student-question sessions" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
timestampstudent_idquestion_idtextbook_idsubjectquestion_stemcorrect_answerstudent_answerassigned_conditionrealized_conditionfeedback_textnext_actionanswer_revealcorrect_retryedit_distanceedit_distance_quartile
02026-04-09 16:47:34RN4R7PMVJZ3HP4QCY7VQ1c1a04325be5e658547de661371e44373920d4c7bab976...9781071925287Business & EconomicsIn conjunction with the latter view, it has be...globalsetdynamiccommon_fallbackThe same answer also completes the following s...incorrect_retry006Q1: 1–6
12026-04-09 16:48:40RN4R7PMVJZ3HP4QCY7VQc7f7b15894ef8d1945dfef087d7055a6aa9a02e09937f8...9781071925287Business & EconomicsWith each successful ______ business venture (...internationalsetstaticcontext_assignedHere's a bit more to help you. Try again. <spa...answer_reveal1011Q4: 11+
22026-04-09 16:49:31RN4R7PMVJZ3HP4QCY7VQ6dc15f0bd8663f347219248a28a5e81ceea837f35b5319...9781071925287Business & EconomicsPeople from more collectivist cultures may be ...constructivesetdynamicdynamicYou said \"set,\" but that doesn't fit with the ...answer_reveal1010Q3: 9–10
32026-04-09 16:49:57SNVHNNPFVD2F7CT8ES3W880ba64e728318c6645177a3d5a98731acaa111d453597...9781544309217Social ScienceA textbook ______ gives the impression—incorre...definitionon drugsdynamicdynamicYou said \"on drugs,\" but that doesn't relate t...no_action0010Q3: 9–10
42026-04-09 16:50:20PWMWHRFF22TCDTTE25UK0a998b1f6d8bc9e2c6ae090fd85a5cd2c5f71b715eec08...9781317342465Social ScienceAs such, women's religions center on acknowled...healingembodimentdynamicdynamicYou said \"embodiment,\" but this word refers to...incorrect_retry008Q2: 7–8
\n", + "
" + ], + "text/plain": [ + " timestamp student_id \\\n", + "0 2026-04-09 16:47:34 RN4R7PMVJZ3HP4QCY7VQ \n", + "1 2026-04-09 16:48:40 RN4R7PMVJZ3HP4QCY7VQ \n", + "2 2026-04-09 16:49:31 RN4R7PMVJZ3HP4QCY7VQ \n", + "3 2026-04-09 16:49:57 SNVHNNPFVD2F7CT8ES3W \n", + "4 2026-04-09 16:50:20 PWMWHRFF22TCDTTE25UK \n", + "\n", + " question_id textbook_id \\\n", + "0 1c1a04325be5e658547de661371e44373920d4c7bab976... 9781071925287 \n", + "1 c7f7b15894ef8d1945dfef087d7055a6aa9a02e09937f8... 9781071925287 \n", + "2 6dc15f0bd8663f347219248a28a5e81ceea837f35b5319... 9781071925287 \n", + "3 880ba64e728318c6645177a3d5a98731acaa111d453597... 9781544309217 \n", + "4 0a998b1f6d8bc9e2c6ae090fd85a5cd2c5f71b715eec08... 9781317342465 \n", + "\n", + " subject question_stem \\\n", + "0 Business & Economics In conjunction with the latter view, it has be... \n", + "1 Business & Economics With each successful ______ business venture (... \n", + "2 Business & Economics People from more collectivist cultures may be ... \n", + "3 Social Science A textbook ______ gives the impression—incorre... \n", + "4 Social Science As such, women's religions center on acknowled... \n", + "\n", + " correct_answer student_answer assigned_condition realized_condition \\\n", + "0 global set dynamic common_fallback \n", + "1 international set static context_assigned \n", + "2 constructive set dynamic dynamic \n", + "3 definition on drugs dynamic dynamic \n", + "4 healing embodiment dynamic dynamic \n", + "\n", + " feedback_text next_action \\\n", + "0 The same answer also completes the following s... incorrect_retry \n", + "1 Here's a bit more to help you. Try again. The dynamic condition uses an LLM to generate feedback based on the student’s actual incorrect answer. The model used in this deployment was GPT-4.1 nano [32]. For each incorrect attempt, the model is given three inputs: the question stem, the correct answer, and the student’s incorrect answer. The feedback generation prompt instructs the model to produce brief supportive feedback that acknowledges the student’s answer, explains why it does not fit, redirects the student without revealing the correct answer, and encourages a retry. ... The generated response is constrained to two or three sentences. The full prompt, along with the dataset used in this study, will be made available in our open data repository [33]." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Dynamic Feedback Prompt — LLM Feedback Isn't Automatically Better (iTextbooks 2026)\n", + "======================================================================================\n", + "\n", + "Model: gpt-4.1-nano\n", + "Temperature: 0.0\n", + "\n", + "This file documents the prompt used to generate dynamic, error-sensitive feedback for\n", + "incorrect fill-in-the-blank (FITB) responses in the VitalSource CoachMe platform. The\n", + "prompt is split into a system message and a user message, as sent to the OpenAI Chat\n", + "Completions API. Placeholders shown in curly braces — {question}, {correct_answer},\n", + "{student_answer} — are substituted with the actual values for each session at runtime.\n", + "\n", + "--------------------------------------------------------------------------------------\n", + "SYSTEM MESSAGE\n", + "--------------------------------------------------------------------------------------\n", + "\n", + "You are an AI tutor giving feedback on student answers to practice questions. You are\n", + "very professional. All your responses are ethical. You never use profanity and you never\n", + "respond to messages that violate any ethical, moral or legal standard.\n", + "\n", + "--------------------------------------------------------------------------------------\n", + "USER MESSAGE\n", + "--------------------------------------------------------------------------------------\n", + "\n", + "You will be given a fill-in-the-blank question, the correct answer, and a student's answer that is incorrect.\n", + "Write feedback that helps the student learn. Follow these rules:\n", + "\n", + "1. Clarity & tone\n", + "- Keep feedback clear, concise, and supportive.\n", + "- Use everyday language, not overly formal or textbook-like.\n", + "- Keep responses short: about 2–3 sentences max.\n", + "\n", + "2. Acknowledgment\n", + "- Usually quote the student's answer directly (e.g., \"You said 'boiling'…\").\n", + "- Do this when it makes the explanation clearer and ties the feedback directly to their response.\n", + "- If the answer is nonsense or quoting it would be awkward, just restate the focus of the question instead.\n", + "\n", + "3. Explain the contradiction\n", + "- Point out why the student's answer doesn't fit with what they should already know.\n", + "- Example: \"You said 'boiling,' but that's when liquid turns to gas, not when solid turns to liquid.\"\n", + "\n", + "4. Refocus on the question\n", + "- Remind them of what the question is really asking about, without giving the correct answer word.\n", + "- Example: \"This question is asking about the temperature where a solid turns into a liquid.\"\n", + "\n", + "5. Encourage the student to retry\n", + "- Phrase the last part of the feedback as a question to invite them to try again.\n", + "- Example: \"What do we call that point?\"\n", + "\n", + "6. Nonsense answers\n", + "- If the answer is irrelevant, gently redirect them back on track. No need to repeat the answer.\n", + "- Example: \"That doesn't seem related. This is about the temperature where a solid turns into a liquid.\"\n", + "\n", + "7. Close-but-not-exact answers (synonyms or near-synonyms)\n", + "- If the student's answer is basically the same idea with different wording:\n", + " - Acknowledge that it makes sense.\n", + " - Explain that the textbook uses different terminology, and strongly encourage the student to use the textbook terminology.\n", + "- Example: \"You said \"fusion\" which also refers to the solid-to-liquid transition. The textbook uses different terminology. Do you know the word the textbook uses?\"\n", + "\n", + "8. Valid-but-different answers\n", + "- If the student's answer is plausible or correct in another context but not the specific concept the textbook/question is targeting:\n", + " - Acknowledge that their answer makes sense.\n", + " - Note that the item is focusing on a different concept in the context of their textbook.\n", + " - Gently steer them toward that context without giving away the word.\n", + "- Example (science): \"You said 'hydrides,' which are hydrogen-containing compounds. In this section, though, the focus is on substances known for acidic behavior. What are those called?\"\n", + "- Example (non-science): \"You said 'legislature,' which is the general type of body that makes laws. Here the question is asking for the specific U.S. body made up of the House and Senate. What is it called?\"\n", + "\n", + "---\n", + "\n", + "FEW-SHOT EXAMPLES\n", + "\n", + "Example 1\n", + "Question: The temperature at which the solid and liquid phases of a given substance are in equilibrium is called the _______ point of the solid or the freezing point of the liquid.\n", + "Correct answer: melting\n", + "Student answer: boiling\n", + "Feedback: You said **boiling**, but that's when a liquid turns to gas, not when a solid turns to liquid. This question is asking about the temperature where a solid changes into a liquid. What do we call that point?\n", + "\n", + "---\n", + "\n", + "Example 2\n", + "Question: The process by which plants use sunlight to make their own food is called _______.\n", + "Correct answer: photosynthesis\n", + "Student answer: respiration\n", + "Feedback: You answered **respiration**, which is when cells break down food to release energy. This question is about the opposite process—how plants *make* food from sunlight. What's that process called?\n", + "\n", + "---\n", + "\n", + "Example 3\n", + "Question: The _______ War was fought between the North and South regions of the United States from 1861 to 1865.\n", + "Correct answer: Civil\n", + "Student answer: banana\n", + "Feedback: \"Banana\" doesn't seem related to this question. Think about U.S. history—the war between the North and South in the 1860s. What war was that?\n", + "\n", + "---\n", + "\n", + "Example 4\n", + "Question: The temperature at which a solid turns into a liquid is called the _______ point.\n", + "Correct answer: melting\n", + "Student answer: fusion point\n", + "Feedback: You said **fusion point**, which does describe the solid-to-liquid transition. The textbook uses a different word here. Can you think of another way to say it?\n", + "\n", + "---\n", + "\n", + "Example 5\n", + "Question: Some compounds containing hydrogen are members of an important class of substances known as ________.\n", + "Correct answer: acids\n", + "Student answer: hydrides\n", + "Feedback: You said **hydrides**, and those are indeed compounds containing hydrogen. In this section, though, the question is about the group of substances known for their acidic behavior. What are those called?\n", + "\n", + "---\n", + "\n", + "NEW INPUT\n", + "Question: {question}\n", + "Correct answer: {correct_answer}\n", + "Student answer: {student_answer}\n", + "\n" + ] + } + ], + "source": [ + "with open( 'dynamic_feedback_prompt.txt', 'r' ) as f:\n", + " print( f.read() )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2.3. Observation window, outcomes, and analysis plan" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">The final dataset contains 33,834 sessions across 23,148 questions, 5,596 students, and 1,363 textbooks." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(33834, 23148, 5596, 1363)" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "len( sessions ), sessions.question_id.nunique(), sessions.student_id.nunique(), sessions.textbook_id.nunique()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">Using the Book Industry Standards and Communications major subject heading classification [36] available for most of the textbooks, the top subject domains as a percentage of session data were Psychology (22.7%), Social Science (19.1%), Political Science (15.7%), Business & Economics (12.2%), and Law (7.5%)." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "subject\n", + "Psychology 22.7%\n", + "Social Science 19.1%\n", + "Political Science 15.7%\n", + "Business & Economics 12.2%\n", + "Law 7.5%\n", + "Education 4.7%\n", + "Language Arts & Disciplines 4.4%\n", + "Medical 3.3%\n", + "MISSING 1.3%\n", + "Science 1.1%\n", + "History 1.1%\n", + "Art 1.1%\n", + "Health & Fitness 1.0%\n", + "Family & Relationships 0.8%\n", + "Computers 0.8%\n", + "Technology & Engineering 0.7%\n", + "Sports & Recreation 0.7%\n", + "Study Aids 0.4%\n", + "Nature 0.4%\n", + "Religion 0.4%\n", + "Philosophy 0.2%\n", + "Transportation 0.1%\n", + "Juvenile Nonfiction 0.1%\n", + "Music 0.0%\n", + "Performing Arts 0.0%\n", + "Architecture 0.0%\n", + "Juvenile Fiction 0.0%\n", + "Photography 0.0%\n", + "Travel 0.0%\n", + "Name: proportion, dtype: object" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "sessions.subject.value_counts( normalize=True ).apply( lambda p: f'{p:.1%}' )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3. Results and Discussion" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3.1. Randomized comparison" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">**Table 1**
Randomized ITT and CACE estimates for answer reveal and correct retry." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [], + "source": [ + "analysis = sessions[ [ 'student_id', 'assigned_condition', 'realized_condition',\n", + " 'answer_reveal', 'correct_retry' ] ].copy()\n", + "analysis[ 'Z' ] = ( analysis.assigned_condition == 'dynamic' ).astype( int )\n", + "analysis[ 'D' ] = ( analysis.realized_condition == 'dynamic' ).astype( int )\n", + "analysis[ 'Y_answer_reveal' ] = analysis.answer_reveal.astype( float )\n", + "analysis[ 'Y_correct_retry' ] = analysis.correct_retry.astype( float )" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [], + "source": [ + "def cluster_bootstrap_itt_cace(\n", + " data, outcome_col, cluster_col='student_id',\n", + " n_boot=100_000, random_state=42, progress_every=10_000,\n", + "):\n", + " rng = np.random.default_rng( random_state )\n", + " cluster_codes, cluster_labels = pd.factorize( data[ cluster_col ], sort=False )\n", + " n_clusters = len( cluster_labels )\n", + " print( n_clusters, 'clusters' )\n", + "\n", + " Z_dyn = ( data.assigned_condition == 'dynamic' ).to_numpy()\n", + " Z_sta = ( data.assigned_condition == 'static' ).to_numpy()\n", + " D = data.D.to_numpy( dtype=float )\n", + " Y = pd.to_numeric( data[ outcome_col ], errors='coerce' ).to_numpy( dtype=float )\n", + "\n", + " rows = np.empty( ( n_boot, 5 ), dtype=float )\n", + " for b in range( n_boot ):\n", + " if ( b + 1 ) % progress_every == 0 or b + 1 == n_boot:\n", + " print( f'Sample {b+1} of {n_boot}' )\n", + " sampled = rng.integers( 0, n_clusters, size=n_clusters )\n", + " cluster_counts = np.bincount( sampled, minlength=n_clusters )\n", + " w = cluster_counts[ cluster_codes ].astype( float )\n", + " w_dyn, w_sta = w[ Z_dyn ], w[ Z_sta ]\n", + " mean_dyn = np.sum( Y[ Z_dyn ] * w_dyn ) / np.sum( w_dyn )\n", + " mean_sta = np.sum( Y[ Z_sta ] * w_sta ) / np.sum( w_sta )\n", + " itt = mean_dyn - mean_sta\n", + " compliance = np.sum( D[ Z_dyn ] * w_dyn ) / np.sum( w_dyn )\n", + " cace = itt / compliance if compliance != 0 else np.nan\n", + " rows[ b ] = ( mean_dyn, mean_sta, itt, compliance, cace )\n", + "\n", + " draws = pd.DataFrame( rows, columns=[ 'mean_dynamic', 'mean_static',\n", + " 'ITT_difference', 'compliance_rate', 'CACE' ] )\n", + " summary = draws.agg( [ 'mean', 'median' ] ).T.rename(\n", + " columns={ 'mean': 'boot_mean', 'median': 'boot_median' } )\n", + " summary[ 'ci_2.5' ] = draws.quantile( 0.025 )\n", + " summary[ 'ci_97.5' ] = draws.quantile( 0.975 )\n", + " return draws, summary" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "5596 clusters\n", + "Sample 10000 of 100000\n", + "Sample 20000 of 100000\n", + "Sample 30000 of 100000\n", + "Sample 40000 of 100000\n", + "Sample 50000 of 100000\n", + "Sample 60000 of 100000\n", + "Sample 70000 of 100000\n", + "Sample 80000 of 100000\n", + "Sample 90000 of 100000\n", + "Sample 100000 of 100000\n", + "5596 clusters\n", + "Sample 10000 of 100000\n", + "Sample 20000 of 100000\n", + "Sample 30000 of 100000\n", + "Sample 40000 of 100000\n", + "Sample 50000 of 100000\n", + "Sample 60000 of 100000\n", + "Sample 70000 of 100000\n", + "Sample 80000 of 100000\n", + "Sample 90000 of 100000\n", + "Sample 100000 of 100000\n" + ] + } + ], + "source": [ + "boot_reveal_draws, boot_reveal_summary = cluster_bootstrap_itt_cace(\n", + " analysis, 'Y_answer_reveal', n_boot=100_000, progress_every=10_000 )\n", + "boot_retry_draws, boot_retry_summary = cluster_bootstrap_itt_cace(\n", + " analysis, 'Y_correct_retry', n_boot=100_000, progress_every=10_000 )" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [], + "source": [ + "def fmt( point, ci_lo, ci_hi, scale=100 ):\n", + " return f'{point*scale:.1f} [{ci_lo*scale:.1f}, {ci_hi*scale:.1f}]'" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
Dynamic (%)Static (%)ITT [95% CI]CACE [95% CI]
Outcome
Answer reveal61.064.3-3.3 [-4.4, -2.2]-5.3 [-7.2, -3.6]
Correct retry16.516.9-0.4 [-1.3, 0.4]-0.7 [-2.1, 0.7]
\n", + "
" + ], + "text/plain": [ + " Dynamic (%) Static (%) ITT [95% CI] CACE [95% CI]\n", + "Outcome \n", + "Answer reveal 61.0 64.3 -3.3 [-4.4, -2.2] -5.3 [-7.2, -3.6]\n", + "Correct retry 16.5 16.9 -0.4 [-1.3, 0.4] -0.7 [-2.1, 0.7]" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "rates = analysis.groupby( 'assigned_condition' )[ [ 'Y_answer_reveal', 'Y_correct_retry' ] ].mean()\n", + "\n", + "dyn_reveal = rates.loc[ 'dynamic', 'Y_answer_reveal' ]\n", + "sta_reveal = rates.loc[ 'static', 'Y_answer_reveal' ]\n", + "dyn_retry = rates.loc[ 'dynamic', 'Y_correct_retry' ]\n", + "sta_retry = rates.loc[ 'static', 'Y_correct_retry' ]\n", + "\n", + "itt_reveal = dyn_reveal - sta_reveal\n", + "itt_retry = dyn_retry - sta_retry\n", + "\n", + "compliance = analysis.loc[ analysis.assigned_condition == 'dynamic', 'D' ].mean()\n", + "cace_reveal = itt_reveal / compliance\n", + "cace_retry = itt_retry / compliance\n", + "\n", + "table_1 = pd.DataFrame( {\n", + " 'Outcome': [ 'Answer reveal', 'Correct retry' ],\n", + " 'Dynamic (%)': [ round( dyn_reveal * 100, 1 ), round( dyn_retry * 100, 1 ) ],\n", + " 'Static (%)': [ round( sta_reveal * 100, 1 ), round( sta_retry * 100, 1 ) ],\n", + " 'ITT [95% CI]': [ fmt( itt_reveal,\n", + " boot_reveal_summary.loc[ 'ITT_difference', 'ci_2.5' ],\n", + " boot_reveal_summary.loc[ 'ITT_difference', 'ci_97.5' ] ),\n", + " fmt( itt_retry,\n", + " boot_retry_summary.loc[ 'ITT_difference', 'ci_2.5' ],\n", + " boot_retry_summary.loc[ 'ITT_difference', 'ci_97.5' ] ) ],\n", + " 'CACE [95% CI]': [ fmt( cace_reveal,\n", + " boot_reveal_summary.loc[ 'CACE', 'ci_2.5' ],\n", + " boot_reveal_summary.loc[ 'CACE', 'ci_97.5' ] ),\n", + " fmt( cace_retry,\n", + " boot_retry_summary.loc[ 'CACE', 'ci_2.5' ],\n", + " boot_retry_summary.loc[ 'CACE', 'ci_97.5' ] ) ],\n", + "} ).set_index( 'Outcome' )\n", + "table_1" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">The reduction in answer reveal did not translate into more students recovering the correct answer on the next action. Incorrect retry rates were correspondingly higher under dynamic (18.5%) than static (14.6%), consistent with the persistence shift being absorbed primarily by failed rather than successful retries." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "assigned_condition\n", + "dynamic 18.5%\n", + "static 14.6%\n", + "Name: next_action, dtype: object" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "( sessions.next_action == 'incorrect_retry' ).groupby( sessions.assigned_condition ).mean().apply( lambda p: f'{p:.1%}' )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">Compliance with dynamic delivery was moderate rather than high (61.6%), reflecting fallback under the no-leak guardrail in a substantial minority of dynamic-assigned cases." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "61.6%\n" + ] + } + ], + "source": [ + "print( f'{compliance:.1%}' )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">**Table 2**
Mixed effects logistic regression results for the randomized ITT comparison. The assigned dynamic condition is the reference level." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [], + "source": [ + "%%R\n", + "sessions <- read_parquet( 'sessions.parquet' )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Answer reveal" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [], + "source": [ + "%%R\n", + "sessions$assigned_condition <- relevel( factor( sessions$assigned_condition ), ref = 'dynamic' )\n", + "model <- glmmTMB( answer_reveal ~ assigned_condition + (1|student_id) + (1|question_id),\n", + " family=binomial(link=logit), data=sessions )" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " Family: binomial ( logit )\n", + "Formula: answer_reveal ~ assigned_condition + (1 | student_id) + (1 | \n", + " question_id)\n", + "Data: sessions\n", + "\n", + " AIC BIC logLik deviance df.resid \n", + " 31093.9 31127.7 -15543.0 31085.9 33830 \n", + "\n", + "Random effects:\n", + "\n", + "Conditional model:\n", + " Groups Name Variance Std.Dev.\n", + " student_id (Intercept) 5.22669 2.2862 \n", + " question_id (Intercept) 0.05071 0.2252 \n", + "Number of obs: 33834, groups: student_id, 5596; question_id, 23148\n", + "\n", + "Conditional model:\n", + " Estimate Std. Error z value Pr(>|z|) \n", + "(Intercept) -0.07543 0.04460 -1.691 0.0908 . \n", + "assigned_conditionstatic 0.30515 0.03473 8.788 <2e-16 ***\n", + "---\n", + "Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1\n" + ] + } + ], + "source": [ + "%%R\n", + "summary( model )" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " OR\n", + "(Intercept) 0.93\n", + "assigned_conditionstatic 1.36\n" + ] + } + ], + "source": [ + "%%R\n", + "round( exp( cbind( OR=fixef( model )$cond ) ), 2 )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Correct retry" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [], + "source": [ + "%%R\n", + "model <- glmmTMB( correct_retry ~ assigned_condition + (1|student_id) + (1|question_id),\n", + " family=binomial(link=logit), data=sessions )" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " Family: binomial ( logit )\n", + "Formula: correct_retry ~ assigned_condition + (1 | student_id) + (1 | \n", + " question_id)\n", + "Data: sessions\n", + "\n", + " AIC BIC logLik deviance df.resid \n", + " 23565.8 23599.5 -11778.9 23557.8 33830 \n", + "\n", + "Random effects:\n", + "\n", + "Conditional model:\n", + " Groups Name Variance Std.Dev.\n", + " student_id (Intercept) 5.8417 2.4170 \n", + " question_id (Intercept) 0.3097 0.5565 \n", + "Number of obs: 33834, groups: student_id, 5596; question_id, 23148\n", + "\n", + "Conditional model:\n", + " Estimate Std. Error z value Pr(>|z|) \n", + "(Intercept) -2.66801 0.07987 -33.41 <2e-16 ***\n", + "assigned_conditionstatic 0.07274 0.04134 1.76 0.0785 . \n", + "---\n", + "Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1\n" + ] + } + ], + "source": [ + "%%R\n", + "summary( model )" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " OR\n", + "(Intercept) 0.07\n", + "assigned_conditionstatic 1.08\n" + ] + } + ], + "source": [ + "%%R\n", + "round( exp( cbind( OR=fixef( model )$cond ) ), 2 )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3.2. Realized condition decomposition" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">**Table 3**
Descriptive outcome rates by realized feedback condition." + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
answer_revealcorrect_retry
realized_condition
common_assigned63.118.4
common_fallback55.521.3
context_assigned63.216.2
context_fallback57.717.8
dynamic63.614.5
outcome_assigned70.412.9
outcome_fallback60.916.9
\n", + "
" + ], + "text/plain": [ + " answer_reveal correct_retry\n", + "realized_condition \n", + "common_assigned 63.1 18.4\n", + "common_fallback 55.5 21.3\n", + "context_assigned 63.2 16.2\n", + "context_fallback 57.7 17.8\n", + "dynamic 63.6 14.5\n", + "outcome_assigned 70.4 12.9\n", + "outcome_fallback 60.9 16.9" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "sessions.groupby( 'realized_condition' )[ [ 'answer_reveal', 'correct_retry' ] ].mean().mul( 100 ).round( 1 )" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">Cell sizes differed across realized feedback conditions, ranging from 1,098 to 10,410." + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "realized_condition\n", + "dynamic 10410\n", + "common_assigned 9557\n", + "context_assigned 4580\n", + "common_fallback 3623\n", + "outcome_assigned 2789\n", + "context_fallback 1777\n", + "outcome_fallback 1098\n", + "Name: count, dtype: int64" + ] + }, + "execution_count": 23, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "sessions.realized_condition.value_counts()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">Common answer feedback accounted for 56.3% of static-delivered cases." + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "56.3%\n" + ] + } + ], + "source": [ + "static_delivered = sessions[ sessions.realized_condition != 'dynamic' ]\n", + "is_common = static_delivered.realized_condition.str.startswith( 'common' )\n", + "print( f'{is_common.mean():.1%}' )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">**Table 4**
Mixed effects logistic regression results for the realized feedback condition decomposition. Dynamic feedback is the reference level." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Panel A. Answer reveal" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [], + "source": [ + "%%R\n", + "sessions$realized_condition <- relevel( factor( sessions$realized_condition ), ref = 'dynamic' )\n", + "model <- glmmTMB( answer_reveal ~ realized_condition + (1|student_id) + (1|question_id),\n", + " family=binomial(link=logit), data=sessions )" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " Family: binomial ( logit )\n", + "Formula: answer_reveal ~ realized_condition + (1 | student_id) + (1 | \n", + " question_id)\n", + "Data: sessions\n", + "\n", + " AIC BIC logLik deviance df.resid \n", + " 30980.7 31056.6 -15481.3 30962.7 33825 \n", + "\n", + "Random effects:\n", + "\n", + "Conditional model:\n", + " Groups Name Variance Std.Dev.\n", + " student_id (Intercept) 5.27632 2.297 \n", + " question_id (Intercept) 0.05617 0.237 \n", + "Number of obs: 33834, groups: student_id, 5596; question_id, 23148\n", + "\n", + "Conditional model:\n", + " Estimate Std. Error z value Pr(>|z|) \n", + "(Intercept) -0.09270 0.04925 -1.882 0.059808 . \n", + "realized_conditioncommon_assigned 0.16500 0.04540 3.634 0.000279 ***\n", + "realized_conditioncommon_fallback -0.11379 0.05992 -1.899 0.057557 . \n", + "realized_conditioncontext_assigned 0.30779 0.05736 5.366 8.05e-08 ***\n", + "realized_conditioncontext_fallback 0.09189 0.08082 1.137 0.255543 \n", + "realized_conditionoutcome_assigned 0.88446 0.07250 12.200 < 2e-16 ***\n", + "realized_conditionoutcome_fallback 0.40386 0.10140 3.983 6.81e-05 ***\n", + "---\n", + "Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1\n" + ] + } + ], + "source": [ + "%%R\n", + "summary( model )" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " OR\n", + "(Intercept) 0.91\n", + "realized_conditioncommon_assigned 1.18\n", + "realized_conditioncommon_fallback 0.89\n", + "realized_conditioncontext_assigned 1.36\n", + "realized_conditioncontext_fallback 1.10\n", + "realized_conditionoutcome_assigned 2.42\n", + "realized_conditionoutcome_fallback 1.50\n" + ] + } + ], + "source": [ + "%%R\n", + "round( exp( cbind( OR=fixef( model )$cond ) ), 2 )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Panel B. Correct retry" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "metadata": {}, + "outputs": [], + "source": [ + "%%R\n", + "model <- glmmTMB( correct_retry ~ realized_condition + (1|student_id) + (1|question_id),\n", + " family=binomial(link=logit), data=sessions )" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " Family: binomial ( logit )\n", + "Formula: correct_retry ~ realized_condition + (1 | student_id) + (1 | \n", + " question_id)\n", + "Data: sessions\n", + "\n", + " AIC BIC logLik deviance df.resid \n", + " 23468.1 23544.0 -11725.1 23450.1 33825 \n", + "\n", + "Random effects:\n", + "\n", + "Conditional model:\n", + " Groups Name Variance Std.Dev.\n", + " student_id (Intercept) 5.9127 2.4316 \n", + " question_id (Intercept) 0.3121 0.5587 \n", + "Number of obs: 33834, groups: student_id, 5596; question_id, 23148\n", + "\n", + "Conditional model:\n", + " Estimate Std. Error z value Pr(>|z|) \n", + "(Intercept) -2.72708 0.08408 -32.43 < 2e-16 ***\n", + "realized_conditioncommon_assigned 0.33755 0.05480 6.16 7.27e-10 ***\n", + "realized_conditioncommon_fallback 0.31069 0.07079 4.39 1.14e-05 ***\n", + "realized_conditioncontext_assigned -0.04613 0.06974 -0.66 0.508 \n", + "realized_conditioncontext_fallback -0.11231 0.09752 -1.15 0.249 \n", + "realized_conditionoutcome_assigned -0.38538 0.08781 -4.39 1.14e-05 ***\n", + "realized_conditionoutcome_fallback -0.17746 0.12203 -1.45 0.146 \n", + "---\n", + "Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1\n" + ] + } + ], + "source": [ + "%%R\n", + "summary( model )" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " OR\n", + "(Intercept) 0.07\n", + "realized_conditioncommon_assigned 1.40\n", + "realized_conditioncommon_fallback 1.36\n", + "realized_conditioncontext_assigned 0.95\n", + "realized_conditioncontext_fallback 0.89\n", + "realized_conditionoutcome_assigned 0.68\n", + "realized_conditionoutcome_fallback 0.84\n" + ] + } + ], + "source": [ + "%%R\n", + "round( exp( cbind( OR=fixef( model )$cond ) ), 2 )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">Using edit distance between the student’s answer and the correct answer as a proxy for response proximity, we found that fallback was more likely for responses closer to the correct answer, suggesting that leakage risk is greater when the student answer is already close to the target term." + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "edit_distance_quartile\n", + "Q1: 1–6 46.2%\n", + "Q2: 7–8 38.8%\n", + "Q3: 9–10 35.0%\n", + "Q4: 11+ 29.7%\n", + "Name: realized_condition, dtype: object" + ] + }, + "execution_count": 31, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dynamic_assigned = sessions[ sessions.assigned_condition == 'dynamic' ]\n", + "fallback = dynamic_assigned.realized_condition != 'dynamic'\n", + "fallback.groupby( dynamic_assigned.edit_distance_quartile ).mean().apply( lambda p: f'{p:.1%}' )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Fallback rates decrease monotonically across quartiles, from 46.2% for responses closest to the correct answer (Q1) to 29.7% for those furthest away (Q4), consistent with the paper's finding that leakage risk is greater when the student answer is already close to the target term." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + ">However, descriptive differences between conditions remained similar across edit distance strata, and adding\n", + "edit distance quartile as a covariate did not materially change the assigned condition effect in the mixed effects models." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Answer reveal" + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "metadata": {}, + "outputs": [], + "source": [ + "%%R\n", + "model <- glmmTMB( answer_reveal ~ assigned_condition + edit_distance_quartile + (1|student_id) + (1|question_id),\n", + " family=binomial(link=logit), data=sessions )" + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " Family: binomial ( logit )\n", + "Formula: \n", + "answer_reveal ~ assigned_condition + edit_distance_quartile + \n", + " (1 | student_id) + (1 | question_id)\n", + "Data: sessions\n", + "\n", + " AIC BIC logLik deviance df.resid \n", + " 31077.1 31136.1 -15531.5 31063.1 33827 \n", + "\n", + "Random effects:\n", + "\n", + "Conditional model:\n", + " Groups Name Variance Std.Dev.\n", + " student_id (Intercept) 5.20483 2.2814 \n", + " question_id (Intercept) 0.05029 0.2242 \n", + "Number of obs: 33834, groups: student_id, 5596; question_id, 23148\n", + "\n", + "Conditional model:\n", + " Estimate Std. Error z value Pr(>|z|) \n", + "(Intercept) -0.21095 0.05341 -3.950 7.81e-05 ***\n", + "assigned_conditionstatic 0.30859 0.03474 8.882 < 2e-16 ***\n", + "edit_distance_quartileQ2: 7–8 0.14940 0.04497 3.322 0.000893 ***\n", + "edit_distance_quartileQ3: 9–10 0.20999 0.04805 4.370 1.24e-05 ***\n", + "edit_distance_quartileQ4: 11+ 0.20350 0.05915 3.440 0.000581 ***\n", + "---\n", + "Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1\n" + ] + } + ], + "source": [ + "%%R\n", + "summary( model )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `assigned_condition` coefficient remains statistically significant and similar in magnitude to the primary ITT model (Table 2), confirming that the answer reveal result is not driven by response proximity to the correct answer." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Correct retry" + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "metadata": {}, + "outputs": [], + "source": [ + "%%R\n", + "model <- glmmTMB( correct_retry ~ assigned_condition + edit_distance_quartile + (1|student_id) + (1|question_id),\n", + " family=binomial(link=logit), data=sessions )" + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " Family: binomial ( logit )\n", + "Formula: \n", + "correct_retry ~ assigned_condition + edit_distance_quartile + \n", + " (1 | student_id) + (1 | question_id)\n", + "Data: sessions\n", + "\n", + " AIC BIC logLik deviance df.resid \n", + " 23504.3 23563.3 -11745.1 23490.3 33827 \n", + "\n", + "Random effects:\n", + "\n", + "Conditional model:\n", + " Groups Name Variance Std.Dev.\n", + " student_id (Intercept) 5.7213 2.392 \n", + " question_id (Intercept) 0.3102 0.557 \n", + "Number of obs: 33834, groups: student_id, 5596; question_id, 23148\n", + "\n", + "Conditional model:\n", + " Estimate Std. Error z value Pr(>|z|) \n", + "(Intercept) -2.38898 0.08456 -28.251 < 2e-16 ***\n", + "assigned_conditionstatic 0.06811 0.04137 1.647 0.0997 . \n", + "edit_distance_quartileQ2: 7–8 -0.28022 0.05310 -5.277 1.31e-07 ***\n", + "edit_distance_quartileQ3: 9–10 -0.37219 0.05747 -6.476 9.40e-11 ***\n", + "edit_distance_quartileQ4: 11+ -0.51126 0.07305 -6.999 2.59e-12 ***\n", + "---\n", + "Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1\n" + ] + } + ], + "source": [ + "%%R\n", + "summary( model )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "As in the primary ITT model (Table 2), `assigned_condition` does not reach statistical significance for correct retry. The non-significant result here is consistent with the paper's conclusion: controlling for edit distance does not change the qualitative finding that dynamic and static feedback produce no detectable difference in correct retry." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.13" + }, + "varInspector": { + "cols": { + "lenName": 16, + "lenType": 16, + "lenVar": 40 + }, + "kernels_config": { + "python": { + "delete_cmd_postfix": "", + "delete_cmd_prefix": "del ", + "library": "var_list.py", + "varRefreshCmd": "print(var_dic_list())" + }, + "r": { + "delete_cmd_postfix": ") ", + "delete_cmd_prefix": "rm(", + "library": "var_list.r", + "varRefreshCmd": "cat(var_dic_list()) " + } + }, + "types_to_exclude": [ + "module", + "function", + "builtin_function_or_method", + "instance", + "_Feature" + ], + "window_display": false + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/aied-2026-itextbooks/README.md b/aied-2026-itextbooks/README.md new file mode 100644 index 0000000..daf15c8 --- /dev/null +++ b/aied-2026-itextbooks/README.md @@ -0,0 +1,148 @@ +# Dynamic vs. Static Feedback for Textbook-Embedded FITB Practice Dataset + +This directory contains the dataset and analysis code for our paper: + +Johnson, B. G., Dittel, J. S., Ortiz, O. J., Bistolfi, R., Clark, +M. W., Jerome, B., Benton, R., & Van Campenhout, R. (2026). LLM +feedback isn't automatically better: Static scaffolds outperform +dynamic feedback in textbook-embedded practice. In *Proceedings of the +Seventh Workshop on Intelligent Textbooks at the 27th International +Conference on Artificial Intelligence in Education*. CEUR Workshop +Proceedings. [https://intextbooks.science.uu.nl/workshop2026/files/itb26_s1p1.pdf](https://intextbooks.science.uu.nl/workshop2026/files/itb26_s1p1.pdf) + +This paper was presented at [AIED 2026](https://www.aied-conference.org/2026) as part of the +[Seventh Workshop on Intelligent Textbooks (iTextbooks)](https://intextbooks.science.uu.nl/workshop2026/). + +## Description + +Formative practice embedded within textbooks has been shown to increase learning +outcomes through the doer effect — the principle that doing practice while learning +is substantially more effective than reading alone. The VitalSource Bookshelf ereader +platform delivers formative practice through CoachMe, a free study feature that +integrates automatically generated questions directly alongside textbook content. +CoachMe supports several question types including fill-in-the-blank (FITB) cloze +questions, which are the focus of this study. + +When a student answers an FITB question incorrectly, as shown in Figure 1, they +receive feedback along with options to retry, reveal the correct answer, and rate the +question. + +

+An example FITB formative practice question in a chemistry textbook. +
Figure 1. An example FITB formative practice question in a chemistry textbook. +

+ +Prior to this study, CoachMe deployed three types of static feedback for FITB +questions (common answer, context, and outcome feedback) selected according to a +preference hierarchy. Common answer feedback presents a second sentence from nearby +textbook content with the same target term removed, providing another retrieval cue. +Context feedback provides an extended excerpt of the surrounding passage. Outcome +feedback informs the student that their response is incorrect. Prior research found +that common answer feedback performed best on key behavioral outcomes and serves as +the benchmark for any new feedback approach. + +Large language models (LLMs) lower the practical barrier to generating dynamic, +error-sensitive feedback conditioned on a student's actual incorrect answer. +Applying this approach to FITB questions, the system uses GPT-4.1 nano to generate +feedback intended to acknowledge the student's specific answer, explain why it does not +fit, and redirect the student without revealing the correct answer. A no-leak +guardrail checks generated feedback for the presence of the correct answer word; if +the answer is detected, the system falls back to static feedback. Figure 2 illustrates +common answer and dynamic feedback side by side. + +

+Examples of common answer and dynamic feedback for FITB questions. +
Figure 2. Examples of common answer and dynamic feedback for FITB questions. +

+ +This dataset captures 33,834 student-question sessions collected during a randomized +deployment from April 9, 2026, through May 8, 2026, using textbooks from eight +publishers who granted permission for generative AI research. Each incorrect first +attempt was randomly assigned with equal probability to either the dynamic feedback +condition or the existing static feedback approach. The unit of analysis is the +student-question session: all interactions by a given student on a given question, +anchored at an incorrect first attempt and followed through the next recorded action. +The final dataset covers 5,596 students, 23,148 questions, and 1,363 textbooks. The +top subject domains by percentage of sessions were Psychology (22.7%), Social Science +(19.1%), Political Science (15.7%), Business & Economics (12.2%), and Law (7.5%). + +The research goals were to: + +- Determine whether dynamic LLM-generated feedback improves student outcomes relative + to the existing static feedback approach under random assignment. +- Identify which delivered feedback types are most associated with the observed + differences in outcomes. +- Draw implications about the relationship between feedback design and target term + recovery in textbook-embedded formative practice. + +## Example Session + +The following example, drawn from the textbook *Biological Psychology* +(Lyons et al., 2014), illustrates how the no-leak guardrail shapes +feedback delivery. For the item "The ANS is essentially the collection +of ______ that act as the manager of your internal organs," a student +answered "neurons." + +The generated dynamic feedback was: *"You answered 'neurons,' which are the cells +that make up nerves, but the question is asking about the overall collection that acts +as the manager of your internal organs. What do we call that collection of nerve +fibers? Would you like to try again?"* This response is coherent and engages the +student's answer, but it approaches a definition of the target term and is rejected +by the no-leak guardrail. The system falls back to common answer feedback: *"Some +texts refer to a sub-set of the ANS known as the enteric nervous system, which refers +to a fine network of ______ that are found only in the walls of the digestive tract +and control the digestive process."* The correct answer is "nerves." + +In the dataset, this session would appear with `assigned_condition` = `dynamic` and +`realized_condition` = `common_fallback`, reflecting the discrepancy between +assignment and delivered feedback introduced by the guardrail. + +## Data Files and Analysis Code + +The provided files are: + +| File | Description | +| --- | --- | +| `sessions.parquet` | Session-level dataset of 33,834 incorrect-first-attempt sessions with feedback conditions and behavioral outcomes | +| `Feedback Analysis.ipynb` | Jupyter notebook for replication of the primary and secondary analyses in the paper | +| `dynamic_feedback_prompt.txt` | The full LLM prompt used to generate dynamic feedback | + +The dataset contains the following fields: + +| Field | Type | Description | +| --- | --- | --- | +| `timestamp` | datetime | Date and time of the incorrect first attempt | +| `student_id` | string | Anonymized student identifier | +| `question_id` | string | Unique question identifier | +| `textbook_id` | string | Unique textbook identifier | +| `subject` | string | Textbook BISAC major subject heading (e.g., "Psychology") | +| `question_stem` | string | Question text with the target term replaced by a blank (`______`) | +| `correct_answer` | string | Target term for the blank | +| `student_answer` | string | Student's incorrect first attempt | +| `assigned_condition` | categorical | Randomized assignment: `dynamic` or `static` | +| `realized_condition` | categorical | Feedback type actually delivered: `dynamic`, `common_assigned`, `common_fallback`, `context_assigned`, `context_fallback`, `outcome_assigned`, or `outcome_fallback` | +| `feedback_text` | string | Text of the feedback shown to the student | +| `next_action` | categorical | Student's next recorded action: `answer_reveal`, `correct_retry`, `incorrect_retry`, `answer_suggestion`, `no_action`, `rating_thumbs_up`, or `rating_thumbs_down` | +| `answer_reveal` | categorical | 1 if the student's next action was to reveal the correct answer, 0 otherwise | +| `correct_retry` | categorical | 1 if the student's next action produced a correct response, 0 otherwise | +| `edit_distance` | integer | Damerau-Levenshtein edit distance between `student_answer` and `correct_answer` | +| `edit_distance_quartile` | categorical | Quartile of `edit_distance` within the dataset (Q1–Q4) | + +## Acknowledgments + +We thank the following publishers for granting permission to include student use of +CoachMe formative practice questions in their textbooks as part of this open dataset: + +- Cambridge University Press +- Emond Publishing +- F. A. Davis Company +- Human Kinetics Publishers +- OpenStax +- SAGE Publications, Inc. (US) +- SAGE Publications, Ltd. (UK) +- Taylor & Francis + +## Contact Us + +If you have questions, please feel free to email +[benny.johnson@vitalsource.com](mailto:benny.johnson@vitalsource.com). diff --git a/aied-2026-itextbooks/dynamic_feedback_prompt.txt b/aied-2026-itextbooks/dynamic_feedback_prompt.txt new file mode 100644 index 0000000..8f00305 --- /dev/null +++ b/aied-2026-itextbooks/dynamic_feedback_prompt.txt @@ -0,0 +1,115 @@ +Dynamic Feedback Prompt — LLM Feedback Isn't Automatically Better (iTextbooks 2026) +====================================================================================== + +Model: gpt-4.1-nano +Temperature: 0.0 + +This file documents the prompt used to generate dynamic, error-sensitive feedback for +incorrect fill-in-the-blank (FITB) responses in the VitalSource CoachMe platform. The +prompt is split into a system message and a user message, as sent to the OpenAI Chat +Completions API. Placeholders shown in curly braces — {question}, {correct_answer}, +{student_answer} — are substituted with the actual values for each session at runtime. + +-------------------------------------------------------------------------------------- +SYSTEM MESSAGE +-------------------------------------------------------------------------------------- + +You are an AI tutor giving feedback on student answers to practice questions. You are +very professional. All your responses are ethical. You never use profanity and you never +respond to messages that violate any ethical, moral or legal standard. + +-------------------------------------------------------------------------------------- +USER MESSAGE +-------------------------------------------------------------------------------------- + +You will be given a fill-in-the-blank question, the correct answer, and a student's answer that is incorrect. +Write feedback that helps the student learn. Follow these rules: + +1. Clarity & tone +- Keep feedback clear, concise, and supportive. +- Use everyday language, not overly formal or textbook-like. +- Keep responses short: about 2–3 sentences max. + +2. Acknowledgment +- Usually quote the student's answer directly (e.g., "You said 'boiling'…"). +- Do this when it makes the explanation clearer and ties the feedback directly to their response. +- If the answer is nonsense or quoting it would be awkward, just restate the focus of the question instead. + +3. Explain the contradiction +- Point out why the student's answer doesn't fit with what they should already know. +- Example: "You said 'boiling,' but that's when liquid turns to gas, not when solid turns to liquid." + +4. Refocus on the question +- Remind them of what the question is really asking about, without giving the correct answer word. +- Example: "This question is asking about the temperature where a solid turns into a liquid." + +5. Encourage the student to retry +- Phrase the last part of the feedback as a question to invite them to try again. +- Example: "What do we call that point?" + +6. Nonsense answers +- If the answer is irrelevant, gently redirect them back on track. No need to repeat the answer. +- Example: "That doesn't seem related. This is about the temperature where a solid turns into a liquid." + +7. Close-but-not-exact answers (synonyms or near-synonyms) +- If the student's answer is basically the same idea with different wording: + - Acknowledge that it makes sense. + - Explain that the textbook uses different terminology, and strongly encourage the student to use the textbook terminology. +- Example: "You said "fusion" which also refers to the solid-to-liquid transition. The textbook uses different terminology. Do you know the word the textbook uses?" + +8. Valid-but-different answers +- If the student's answer is plausible or correct in another context but not the specific concept the textbook/question is targeting: + - Acknowledge that their answer makes sense. + - Note that the item is focusing on a different concept in the context of their textbook. + - Gently steer them toward that context without giving away the word. +- Example (science): "You said 'hydrides,' which are hydrogen-containing compounds. In this section, though, the focus is on substances known for acidic behavior. What are those called?" +- Example (non-science): "You said 'legislature,' which is the general type of body that makes laws. Here the question is asking for the specific U.S. body made up of the House and Senate. What is it called?" + +--- + +FEW-SHOT EXAMPLES + +Example 1 +Question: The temperature at which the solid and liquid phases of a given substance are in equilibrium is called the _______ point of the solid or the freezing point of the liquid. +Correct answer: melting +Student answer: boiling +Feedback: You said **boiling**, but that's when a liquid turns to gas, not when a solid turns to liquid. This question is asking about the temperature where a solid changes into a liquid. What do we call that point? + +--- + +Example 2 +Question: The process by which plants use sunlight to make their own food is called _______. +Correct answer: photosynthesis +Student answer: respiration +Feedback: You answered **respiration**, which is when cells break down food to release energy. This question is about the opposite process—how plants *make* food from sunlight. What's that process called? + +--- + +Example 3 +Question: The _______ War was fought between the North and South regions of the United States from 1861 to 1865. +Correct answer: Civil +Student answer: banana +Feedback: "Banana" doesn't seem related to this question. Think about U.S. history—the war between the North and South in the 1860s. What war was that? + +--- + +Example 4 +Question: The temperature at which a solid turns into a liquid is called the _______ point. +Correct answer: melting +Student answer: fusion point +Feedback: You said **fusion point**, which does describe the solid-to-liquid transition. The textbook uses a different word here. Can you think of another way to say it? + +--- + +Example 5 +Question: Some compounds containing hydrogen are members of an important class of substances known as ________. +Correct answer: acids +Student answer: hydrides +Feedback: You said **hydrides**, and those are indeed compounds containing hydrogen. In this section, though, the question is about the group of substances known for their acidic behavior. What are those called? + +--- + +NEW INPUT +Question: {question} +Correct answer: {correct_answer} +Student answer: {student_answer} diff --git a/aied-2026-itextbooks/feedback_examples.png b/aied-2026-itextbooks/feedback_examples.png new file mode 100644 index 0000000..cb61734 Binary files /dev/null and b/aied-2026-itextbooks/feedback_examples.png differ diff --git a/aied-2026-itextbooks/sessions.parquet b/aied-2026-itextbooks/sessions.parquet new file mode 100644 index 0000000..93d5a05 --- /dev/null +++ b/aied-2026-itextbooks/sessions.parquet @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5bb0ea27af2258f9e64517e0a8872b6ace3eff00963dd4cff08eaab4dd21f3df +size 9202623