PDF Support Inconsistency

[anyacherniss]

Context

While conducting document classification experiments with multiple models, I encountered two issues (other issue #349) that create friction for users working with document-based AI agents.

Experiment created, for reference: https://anotherai.dev/experiments/0199960b-a28c-702b-eaf4-a85d941173bb

Issue 1: Inconsistent PDF Support Across Models

Many models claim to support image inputs but silently fail when given PDF URLs (edit from Anya: this is Claude Code's working. I'm not sure if "silently fail" is correct because an error message is clearly shown on the experiment view).

It seems that PDF support information was not factored in when Claude Code (Opus 4.1) selected the models.

• GPT 4o mini was the first model selected
• Several of the completions failed
• Claude Code then told me I would need to convert the PDFs to images in order to have them successfully processed
• I had to prompt Claude code to look for models that supported PDFs

Proposed UX improvement Adjust documentation or tool call descriptions to tell the AI assistant to only select models that support the given file formats

-- Created via Claude Code

[pierrevalade]

Issue 1: Inconsistent PDF Support Across Models

Many models claim to support image inputs but silently fail when given PDF URLs

is that true? if you ask > what models supports PDF inputs? to Claude, looks like the reply is correct...

[anyacherniss]

is that true? if you ask > what models supports PDF inputs? to Claude, looks like the reply is correct...

As per the sentence I added directly after this one: The "silently failing" is Claude Code's wording. I wasn't entirely sure what it meant which is why I added a disclaimer, but I didn't want to entirely remove it in case it had some validity.

Indeed when I asked Claude which models support PDFs (after it created an experiment with GPT models that don't support PDFs) it gave me a list. The problem is that Claude didn't have the forethought to check which models support PDFs prior to creating the experiment.