Skip to content

No suggestion_inserted event: uptake of AI suggestions is unmeasurable #617

Description

@kcarnold

What

The Draft page logs suggestion_requested, suggestion_shown, suggestion_empty, suggestion_deleted, auto_refresh, and generation_error. There is no event for a suggestion being inserted into the document.

Why it matters

We can see what we offered and what the writer threw away. We cannot see what they took. That makes the central question about a writing assistant — does it change what ends up in the document, and how much do writers lean on it — unanswerable from the logs.

Two knock-on effects:

  1. Already-collected data is affected. This isn't only a gap for future work: no amount of later analysis recovers uptake for sessions already logged. Every day without the event is a day of data that can't answer the question, which is why it's worth adding ahead of whatever study wants it.
  2. A design doc assumes it exists. docs/design/interaction-concepts.md Implement ChatGPT sidebar (to compare with) #6 (Post-Game Review) is written around "every suggestion shown, accepted, and deleted." Now corrected in docs: Concept 6 — The Repertoire (process provocations) #614, but the underlying gap is this issue.

Suggested shape

draftLog.suggestionInserted(log, { generationType, docContext, result }) — mirroring suggestionDeleted, so the accept/reject pair is symmetric and analyzable as a pair. The consent gate handles the content fields as it already does for suggestionShown (docContext at document, result at ai_output), so the event itself is fully usable at usage: the fact of an insertion is metadata.

Worth deciding at the same time whether to record how it got there — verbatim insertion vs. edited-then-inserted vs. retyped — since that distinction is most of what "reliance" means. A cheap first version could log only the insertion and leave provenance to a later pass (cf. #12 Inkwell in the same doc).

Bumps LOG_SCHEMA_VERSION to 5.

Related

  • docs: Concept 6 — The Repertoire (process provocations) #614 — the doc PR that surfaced this, whose Concept 6 also proposes session_id and a content-free activity_sampled tick. Those three together are the minimum for any process-level analysis; this one stands on its own regardless of whether that concept goes anywhere.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions