LLM and Agent usage

[jpmckinney]

Tips

Project selection

Choose projects:

• With high coverage. If missing, use the agent to add more tests, first.
• With soundness checks. This can be via the Rust compiler or Python type checker (like mypy, but see Look into ty and pyright, compare to mypy, and document our type checking approach #114). If type annotations are missing in Python, use the agent to add more annotations, first.
• With strict linters and formatters. Our default is to enable all rules in ruff and cargo.

Task selection

Choose tasks:

• Where you are an expert in the project and task. Do not ask the LLM to solve or build something you couldn't do on your own with your current knowledge.

Agents are also good at:

• Complex refactoring, e.g. prompting to:
  • rewrite contributed code in the style of the rest of the file or project
  • reduce duplication, single-use functions, single-use variables and single-letter variables
  • extract methods

Planning

• For complex requests, it is recommended to write a specification (and to ask the LLM to update it as it makes progress). This planning step can be assisted by a different model. Thisblog post has examples.

Steering

• Agents sometimes "anchor" to a particular direction. When steering the agent off that direction, make sure to delete any generated code in that direction.

Edit prediction

• When refactoring, this can be especially useful, e.g. after making a change, you can press tab to advance to the next part of the code that needs to be changed (e.g. when changing function signatures).
• Otherwise, it is often preferable to hide predictions by default (like in Zed's subtle mode), to have fewer distracting suggestions. (Cursor has an open feature request.)

git

• Use git worktree so that you and the agent can work on the repository at the same time, independently.
• Use git add to stage accepted changes between prompts, and git diff to see incremental changes.

Cursor

• Create a Cursor Rule whenever you correct the agent, so that you don't need to make the same correction in future. The blog post describes an example workflow.
• When using Cursor, explicitly set the model to Sonnet (from Claude) when working with Python notebooks; otherwise, the app knows that it needs Sonnet (and displays a message), but the LLM instead keeps retrying to make changes that it can't.
• Accept or reject changes before changing the code and re-prompting the agent. Otherwise, the agent tends to ignore your changes.
• Be careful about using undo/redo if the agent has made changes. If you did not accept a changeset, then "redo" might not work to re-suggest those changes.

Recommended reading

• Overall: https://simonwillison.net/2025/Jun/18/coding-agents/
• https://simonwillison.net/about/#subscribe

Things I read but that were not particularly useful:

• https://devin.ai/agents101

Untested

GitHub Copilot

• https://docs.github.com/en/copilot/using-github-copilot/using-github-copilot-to-create-issues
• https://docs.github.com/en/copilot/getting-started-with-github-copilot/getting-started-with-github-copilot-in-visual-studio-code

MCPs

In general, try without MCPs. It's easy to load up on MCPs, but: (1) the MCPs might fill up the context, adding "noise" to the "signal" (your prompt); (2) the agent's behavior might become even more unpredictable and harder to steer, under the influence of additional MCPs; (3) the agent needs to know which MCP to use when, which you need to configure; and (4) the authors of the agent have already tried to make it as good as possible, and the MCPs might duplicate or conflict with functionality/tuning of the agent. In other words, only add an MCP that solves a real problem you have experienced.

That said, these are likely useful:

• https://github.com/microsoft/playwright-mcp (used)
• https://github.com/github/github-mcp-server
• https://docs.anthropic.com/en/docs/claude-code/mcp#example%3A-monitor-errors-with-sentry
• https://github.com/googleapis/genai-toolbox or similar for PostgreSQL

Otherwise, if the agent has access to a terminal, you can instead write a script or Makefile and ask the agent to use that (e.g. to run tests and lints, read logs, etc.).

• https://github.com/oraios/serena: Language server. Works with Claude Code, Cursor, presumably Zed.
• https://github.com/upstash/context7: Access up-to-date dependency documentation. Works with Claude Code, Cursor, Zed.
• https://github.com/BeehiveInnovations/zen-mcp-server: Access other LLMs from CLI agents like Claude Code or Gemini CLI. Has orchestration tools.

Reference

• https://www.youtube.com/watch?v=nfOVgz_omlU (not yet watched)

Other learning

• Fine-tuning https://ai.google.dev/gemma/docs/core/huggingface_text_full_finetune
• Building MCPs https://gofastmcp.com/getting-started/welcome

[jpmckinney]

Moved to #136

[jpmckinney]

Moved to #136

[jpmckinney]

Cursor really restricts the free tier now (would have to pay $20/mo, which is maybe worth it, but would need to compare against others).

Gemini CLI

So, I am trying Gemini CLI, which has a more generous free tier.

• Running gemini starts a terminal user interface that breaks scroll back. Run it in a separate terminal if you want to see your previous output!
• It immediately misunderstood my first prompt. I asked it to change .egg files in a specific directory, and it wrote back that it would change the files in a different one. I stashed that directory to not get it confused.
• The second time, it started working, but it seems like it got in a loop, removing files that had already been removed, adding lines to tests.yml that had already been added, etc.

So, for now, I think Gemini is not worth the effort.

My prompt:

Decompile the .egg files at tests/fixtures and make it such that the eggs are built during testing. As part of this change:

- Update the CI workflow at .github/workflows/tests.yml
- Update the documentation at docs/contributing/index.rst

[jpmckinney]

Claude Code

Reading

• https://www.anthropic.com/engineering/claude-code-best-practices
• https://docs.anthropic.com/en/docs/claude-code/overview
• https://claudelog.com/mechanics/claude-md-supremacy/ (unofficial, decent, not a must-read)

General tips:

• https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices

Setup

If needed:

• claude mcp add playwright npx '@playwright/mcp@latest' then prompt like Use playwright mcp to open a browser to example.com (source)
• Running Claude Code in a Docker container https://github.com/anthropics/claude-code/tree/main/.devcontainer

~/.claude/CLAUDE.md

- Always use `uv` to run Python scripts, manage packages and create virtual environments (if needed)
- Use `uv run` instead of `python` or `python3`
- Run `ruff format` after generating code

For individual-project CLAUDE.md, some initial potential content:

• A description of the project, if starting from scratch.
• Links to relevant documentation (e.g. specific APIs that we'll be using).

Tips

• In Sublime Text, remember to use Claude Code with Terminus, to switch windows less.
• Run CLAUDE.md or prompts through https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prompt-improver

Tests

Switching to pathlib

Details

I now have a budget for LLM agents, so I subscribed to Claude Code Pro, and issued the prompt, for Scrapyd:

Update the project to use pathlib instead of os.path

This took a while, but I sent the prompt before starting a call with Yohanna, and it was finished by the end of the call.

It ran the tests, too. After reviewing (no issues!), I did a git add and then:

Update eggstorage.py to return a Path from FilesystemEggStorage._get_path, and make corresponding changes as needed

Run `ruff check` and fix the lints

Could Config.__init__ omit the str() around the Path objects?

Claude Code answered the question, then made the change. (Unlike Cursor, I think it always expects to make changes and does not distinguish questions from instructions.) Edit: You can toggle into "Plan Mode" by pressing Shift-Tab twice.

By running npx ccusage@latest I see that this would have cost $2.76 if using the Sonnet 4 API directly. We can compare expected monthly API costs to the actual costs of fixed rate plans, to decide what's best for us (e.g. maybe as a team, it makes more sense to use APIs directly, as long as we monitor our spending).

┌────────────┬───────────────┬───────────┬───────────┬───────────────┬─────────────┬───────────────┬─────────────┐
│ Date       │ Models        │     Input │    Output │  Cache Create │  Cache Read │  Total Tokens │  Cost (USD) │
├────────────┼───────────────┼───────────┼───────────┼───────────────┼─────────────┼───────────────┼─────────────┤
│ 2025-07-25 │ - sonnet-4    │       374 │    24,978 │       266,403 │   4,632,823 │     4,924,578 │       $2.76 │
├────────────┼───────────────┼───────────┼───────────┼───────────────┼─────────────┼───────────────┼─────────────┤
│ Total      │               │       374 │    24,978 │       266,403 │   4,632,823 │     4,924,578 │       $2.76 │
└────────────┴───────────────┴───────────┴───────────┴───────────────┴─────────────┴───────────────┴─────────────┘

https://ccusage.com/guide/daily-reports#understanding-the-columns

Re: "Cache Read" compared to "Cache Create" (i.e. Write) and "Input": see https://forum.cursor.com/t/cache-read-bug-using-90-million-tokens-120x-more-than-it-should/120544/33 and https://forum.cursor.com/t/frustrated-with-cursor-s-token-hungriness/118086/197

Getting tests passing in cove-ocds

Details

Prompt:

PROBLEM STATEMENT: Some tests in tests/test_functional.py are failing, due to a dependency upgrade. PROGRESS SO FAR: Run git diff HEAD to see the changes made to get some tests passing. DIAGNOSIS: The problem is often due to a test using simply "body" as the CSS selector. Doing so fails with: "selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: stale element not found". SOLUTION: The solution is to use a more specific selector. PROPOSED PLAN: Run pytest tests/test_functional.py to get a list of failing tests. Start a server with uv run manage.py runserver. Then for each failing test: (0) Use playwright MCP to open http://127.0.0.1:8000 to load the form. (1) Each test mentions a file. Enter the file's contents into the form's paste element. Submit the form to load the results page. (2) Each test has expectations around rendered (or not rendered) text. Find the text on the page and get its selector. (3) Update the test with the new selector. (4) Run the individual failing test to check that it passes. (5) If the test still fails, stop. If the test passes, move onto the next test. PROMPT: Are there any possible improvements to this plan?

It came up with an updated plan, which seemed fine to me, so I accepted.

╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Ready to code?                                                                                                                                        │
│                                                                                                                                                       │
│ Here is Claude's plan:                                                                                                                                │
│ ╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │
│ │ Plan to Fix Failing Functional Tests                                                                                                              │ │
│ │                                                                                                                                                   │ │
│ │ Overview                                                                                                                                          │ │
│ │                                                                                                                                                   │ │
│ │ Fix failing tests in tests/test_functional.py that are experiencing StaleElementReferenceException due to dependency upgrades. The root cause is  │ │
│ │ tests using overly generic selectors like "body".                                                                                                 │ │
│ │                                                                                                                                                   │ │
│ │ Execution Steps                                                                                                                                   │ │
│ │                                                                                                                                                   │ │
│ │ 1. Initial Assessment                                                                                                                             │ │
│ │   - Run pytest tests/test_functional.py -v to identify all currently failing tests                                                                │ │
│ │   - Start development server with uv run manage.py runserver                                                                                      │ │
│ │ 2. For Each Failing Test (systematically):                                                                                                        │ │
│ │   - Use Playwright browser automation to navigate to http://127.0.0.1:8000                                                                        │ │
│ │   - Load the test file's content into the form's paste element                                                                                    │ │
│ │   - Submit form to reach results page                                                                                                             │ │
│ │   - Locate the expected text/elements on the page                                                                                                 │ │
│ │   - Identify more specific, stable CSS selectors (prioritizing ID > class > tag combinations)                                                     │ │
│ │   - Update the test with the new selector                                                                                                         │ │
│ │   - Run individual test to verify it passes                                                                                                       │ │
│ │   - Continue to next test only if current one passes                                                                                              │ │
│ │ 3. Selector Strategy Priority                                                                                                                     │ │
│ │   - ID-based selectors (most stable)                                                                                                              │ │
│ │   - CSS class selectors                                                                                                                           │ │
│ │   - Tag + class combinations                                                                                                                      │ │
│ │   - Avoid generic tag selectors like "body"                                                                                                       │ │
│ │ 4. Validation                                                                                                                                     │ │
│ │   - Run individual tests after each fix                                                                                                           │ │
│ │   - Final run of entire test suite to ensure no regressions                                                                                       │ │
│ │   - Stop immediately if any test still fails for investigation                                                                                    │ │
│ │                                                                                                                                                   │ │
│ │ This approach builds on the significant improvements already made in recent commits and focuses specifically on the remaining stale element       │ │
│ │ reference issues.                                                                                                                                 │ │
│ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                                                                       │
│ Would you like to proceed?                                                                                                                            │
│                                                                                                                                                       │
│ ❯ 1. Yes, and auto-accept edits                                                                                                                       │
│   2. Yes, and manually approve edits                                                                                                                  │
│   3. No, keep planning                                                                                                                                │
│                                                                                                                                                       │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

I had to tell it to use uv run (a CLAUDE.md file, or mentioning uv run in our own docs, would help here).

(Playwright MCP is cool - you get to see the agent click around the browser.)

The application errored; it turns out my local DB had been migrated, so I had to fix the DB and then ask:

I had to migrate the database. Continue from the earlier step, when you first navigated to http://127.0.0.1:8000 in playwright

It ended up starting on a difficult-to-fix test, so I interrupted it and directed it to simpler ones:

Fix the tests/test_functional.py::test_headlines_class tests first (after confirming those that fail)

It got the test to pass. Sweet! I told it to continue:

 Okay, now continue until all tests in tests/test_functional.py are passing. If wait mechanisms are a common solution, you can restore similar logic. (To see the logic that was removed, use git diff --cached)

The cove-ocds tests take too long to run. I could probably configure CC's timeout, but I just did:

Work on these tests specifically: test_url_input, test_url_input_with_version, test_extension_validation_error_messages, test_validation_error_messages

It got stuck. I'm beginning to think we should just avoid using Selenium, so I'll start a new task around that.

Switching away from Selenium in cove-ocds

Details

> PROBLEM: Selenium is slow.   
                                                                
  TASK: Refactor tests, where possible, to django.test.Client and assertHTMLEqual.

  PROPOSED PLAN:                                                                                                                                    
  - Start a server with: uv run manage.py runserver
  - Use playwright MCP to open http://127.0.0.1:8000 to load the form
  - Add a method corresponding to url_input_browser to submit the form using django.test.Client              
  - Ask me for a list of tests to convert. Then for each test:
                                                                 
    - Skip the test if it performs click() or similar operations       
    - Rewrite the test to django.test.Client and assertHTMLEqual instead of Selenium
                                                                                 
  PROMPT: Are there any possible improvements to this plan?               

Claude's new plan:

     Phase 1: Setup & Infrastructure

     1. Create tests/test_functional_django.py for Django Client-based tests
     2. Create helper functions to replicate form submission workflow using Django test client
     3. Set up proper test base classes (TestCase vs TransactionTestCase)

     Phase 2: Test Analysis & Categorization

     1. Analyze all tests in test_functional.py and categorize them:
       - Easy conversions: Simple form submissions with content validation
       - Medium conversions: Tests with dropdowns/redirects but no complex JS
       - Keep as Selenium: Tests requiring JavaScript, CSS validation, or complex interactions

     Phase 3: Convert Easy Tests First

     1. Start with test_url_input parametrized test - convert simple validation scenarios
     2. Convert tests that only check response content (no form interactions)
     3. Use assertContains()/assertNotContains() instead of string searching

     Phase 4: Convert Medium Complexity Tests

     1. Handle dropdown selections using Django Client
     2. Convert tests with simple form interactions
     3. Adapt redirect following tests

     Phase 5: Optimization & Cleanup

     1. Compare test execution times between Selenium and Django Client versions
     2. Remove redundant Selenium tests where Django Client equivalent exists
     3. Keep complex interaction tests in Selenium

     Key Differences from Original Plan:
     - No manual server management needed
     - Focus on assertContains() over assertHTMLEqual
     - Systematic categorization approach
     - Preserve complex tests in Selenium rather than skipping them

I saw a change I didn't like, so I stopped it and:

Rollback your most recent changes. The original tests in test_functional.py typically compare against text (with HTML tags stripped). Use the striptags function from Django, or the lxml package (which is already installed) to strip tags, if needed.

It stopped after converting some tests. Cleanup:

Remove the redundant tests from tests/test_functional.py

Is TestUrlInputDjango redundant with test_url_input_scenarios_django?

And then asked it to continue:

Convert to the Django approach: test_footer_ocds, test_index_page_ocds, test_index_page_ocds_links, test_terms_page, test_validation_error_messages, test_extension_validation_error_messages, test_url_invalid_dataset_request, test_url_input_with_version, test_url_input_with_extensions, test_url_input_extension_headlines, test_additional_checks_section, test_additional_checks_section_not_being_displayed, test_key_field_information, test_records_table_releases_count and the tests starting with test_records_table_ and test_releases_table_

It did up to and including test_url_invalid_dataset_request . Cleanup:

 The original tests starting with test_records_table_ and test_releases_table (corresponding to the new test test_table_row_limits_django) used fixtures to modify Django's settings. Please replicate that logic, then remove the original tests.

Then re-prompted with the updated list of tests to convert.

[jpmckinney]

@yolile My tendency is for OCP to use Claude Code as it only requires a terminal and doesn't require adopting a new IDE or having the team collectively use the same IDE(s). It also gives us access to (for now) the best models for software development (Sonnet 4 and Opus 4).

That said, among the IDE options, ordered by Crunchbase rank as a rough indication of longevity:

• Windsurf has its own IDE, but it also integrates with VS Code, Visual Studio, JetBrains, Vim/Neovim, Eclipse, Xcode.
  • Recently purchased by Cognition, who have Devin, which also has an IDE based on VS Code, though Devin is more "autonomous".
• Cursor (by Anysphere) is its own IDE, that builds on VS Code.
• Augment Code has plugins for VS Code, JetBrains, Vim/Neovim.
• Zed is its own IDE.
• GitHub Copilot has extensions for VS Code, Visual Studio, JetBrains, Vim/Neovim, Eclipse, Xcode.
• Cline and Roocode (a fork of Cline) are specific to VS Code.

It depends how much we like the IDE features (edit prediction). Of these, I only tested Cursor.

For the ones that are IDEs, we can either subscribe directly, or subscribe to e.g. Anthropic, and provide API keys. Since we prefer allowing different IDE options, we would probably provide API keys. I know Zed for sure is free if only API keys are provided – would need to double check for the other IDEs/IDE extensions.

[jpmckinney]

"Skills" are moving toward a standard specification: https://agentskills.io

[jpmckinney]

Can add as seealso:

• https://claude.com/plugins/claude-md-management

~/.claude/settings.json (replacing paths to commands)

  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "/home/USER/.claude/hooks/rewrite-python-to-uv.sh"
          }
        ]
      }
    ],
    "PostToolUse": [
      {
        "matcher": "Write|Edit",
        "hooks": [
          {
            "type": "command",
            "command": "/home/USER/.claude/hooks/python-post-write.sh"
          }
        ]
      }
    ]
  }

rewrite-python-to-uv.sh

#!/bin/sh
# Rewrites python/python3/pip/pip3 commands to use uv equivalents.

cmd=$(jq -r '.tool_input.command // empty')

if echo "$cmd" | grep -qE '^(python3?|pip3?) '; then
    echo "$cmd" | sed -E 's|^python3? |uv run python |; s|^pip3? |uv pip |' | jq -R '{"hookSpecificOutput": {"hookEventName": "PreToolUse", "updatedInput": {"command": .}}}'
fi

python-post-write.sh

#!/bin/sh
# After Write/Edit on Python files, update __generated_at__ if unreviewed, then run ruff format.

file=$(jq -r '.tool_input.file_path // empty')

case "$file" in
  *.py)
    if head -5 "$file" | grep -q '__review_status__ = "unreviewed"'; then
      sed -i '' "s|^__generated_at__ = \".*\"|__generated_at__ = \"$(date +%Y-%m-%d)\"|" "$file"
    fi
    ruff format "$file" 2>/dev/null || true
    ;;
esac