Skip to content

Test that LM queries are being made correctly #8

Description

@kcarnold
  • are logprobs being computed from the results correctly?
  • is spacing between sentences sensible / consistent during reorderings?
  • max_tokens should be set tiny since we're not actually using the generation results
  • could there be better prompts for the LM (vs the "write a short essay using this outline" that's there)

It would be helpful to have some test cases that didn't need an OpenAI API call (just paste in a prior response).

It would be helpful to visualize what's going on. For this issue, at least log the LM queries in a way that we can show easily. In general we probably want a more visual representation of what's going on; full version of this is a separate issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions