Making better use of AI tooling

[alexdewar]

We're using AI in a few ways for MUSE2 at present:

• I usually tag Copilot for review on PRs
• I've used VS Code's Copilot to write tests in a few places and I know @tsmbland was/is using Cursor

These tools nowadays allow you to add a markdown file with extra instructions specific to your repo, such as coding style etc. I was thinking it could be worth writing some things about coding style in a document anyway (e.g. "error handling is done using anyhow, but we should just panic for developer-facing errors", "you can use the itertools crate to simplify operations involving iterators"). The idea isn't to be prescriptive, so much as to steer new contributors/our robot overlords in the right direction.

What do we think?

On a related note, it seems you can automatically request reviews from Copilot on PRs, which I think is probably worth doing. Some of the suggestions aren't that helpful (though perhaps we can better tune things with additional instructions), but it has found logic errors in my code a few times, so I think it's worth doing for that alone.

@tsmbland @Aurashk @dalonsoa

[tsmbland]

Funny you bring this up because I've been thinking about this today/yesterday, particularly with regards to bringing the in-code documentation (doc comments, inline comments) up to date, as I've noticed a few things are missing or wrong. A lot of these are simple things already implied by the code which should be easy for AI to pick up. For example, a comment saying that a map is keyed by commodity and year when really it's just keyed by commodity - case in point. Another example might be places where we use upwrap or [] but don't explain why this is safe - a bit harder but AI should still be able to look through the code and figure this out. I was having a go at writing a prompt that we could apply to each module to fix these kinds of things (in fact, getting the AI to write this prompt for me by looking over the coding and getting a feel for our preferences/conventions, then manually refining). Very much work in progress, but if interested I've attached. Wasn't sure whether this is something I'd just use as a one time thing or add to the repo to help guide the AI in the future. But on that note I like your idea of having a markdown file(s) with points about coding style/preferences (I do find myself frequently having to remind it to use anyhow)

[tsmbland]

But on that note I like your idea of having a markdown file(s) with points about coding style/preferences (I do find myself frequently having to remind it to use anyhow)

Alternatively, we could aim to put together a comprehensive developer guide in the documentation and include any points about coding style in this, then point the AI to the developer guide every time it's called (i.e. put a note in copilot-instructions.md instructing the AI look at the developer guide)

[alexdewar]

Alternatively, we could aim to put together a comprehensive developer guide in the documentation and include any points about coding style in this, then point the AI to the developer guide every time it's called (i.e. put a note in copilot-instructions.md instructing the AI look at the developer guide)

That's what I was leaning towards, because that sort of info might be useful to new (human) contributors too. That said, there are probably also some small details that we really only need to tell AIs about, so maybe a bit of both would be best.

Your prompt looks useful! Were you planning on using it to check the repo over for documentation problems?

[tsmbland]

That's what I was leaning towards, because that sort of info might be useful to new (human) contributors too. That said, there are probably also some small details that we really only need to tell AIs about, so maybe a bit of both would be best.

Yeah that sounds good

Your prompt looks useful! Were you planning on using it to check the repo over for documentation problems?

For now I was just going to apply the prompt one-by-one to each module and see how it gets on, probably refining the prompt a bit as I go along. Longer term, I think some of the points in the prompt could be incorporated into the developer guide / copilot markdown so we can (hopefully) keep everything up to date as we go along

[Aurashk]

I'd be in favour of trying out having automated copilot reviews on PR's as I often forget to do it. It does generate some useless output, but I've found it particularly useful for raising relevant rust-stuff which I can look into to in more detail.

[tsmbland]

Likewise

[alexdewar]

Ok cool. I'll make that change now then.

[alexdewar]

I had to change from a classic branch protection rule to a ruleset to make it work. Think it should work now, but if things start breaking, this is probably why!