How can we integrate with RubyLLM?

[stevepolitodesign]

RubyLLM is becoming the default library for interacting with LLMs. If we want to increase Top Secret adoption, we need to ensure it can seamlessly integrate with RubyLLM.

[stevepolitodesign]

The challenge is that #ask does all of the work:

# Create a chat instance (uses the configured default model)
chat = RubyLLM.chat

# Ask a question
response = chat.ask "What is Ruby on Rails?"

# The response is a RubyLLM::Message object
puts response.content
# => "Ruby on Rails, often shortened to Rails, is a server-side web application..."

We need a way to "hook" into it so that...

• We only send filtered information to the LLM
• We can restore the the response from the LLM
• We can store the original message and restored response to the database

[clarissalimab]

When using Rails + RubyLLM's acts_as, you can use a callback on the Message model to process the content before sending the message. We kept a column on the Chat model with the mapping generated by top secret to be able to restore the message in memory when sending back the response to the user (and we do the same when retrieve the history). This works well on simple setups, but if you're calling .ask in multiple contexts outside of your Chat model without storing messages, then it might not work.

Another idea is to override .ask to filter the content before doing anything with it and have an attribute accessor write to an attribute with the mapping that gets stored to the chat model. So something like:

class Chat < ApplicationRecord
  acts_as_chat

  def ask(message, **options, &block)
    filtered = TopSecret::Text.filter(message)

    # This is not as simple as jjust merging because we would have to make sure the keys are unique, but this is just an example!
    self.top_secret_mappings = top_secret_mappings.to_h.merge(filtered.mapping)
    save!

    response = super(filtered.output, **options, &block)

    restore_response(response)
  end

  private

  def restore_response(message)
    return message if message.content.blank? || top_secret_mappings.blank?

    # This will change in memory only, so the persisted mapping remains the same for future calls.
    message.content = TopSecret::FilteredText.restore(
      message.content,
      mapping: top_secret_mappings.symbolize_keys
    ).output
    message
  end
end

You could patch it to override every ask call instead of just on the model.

Another idea is to have a .with_filtered_content method that would optionally filter the content, that would be a nice interface that would suit well with RubyLLM (but I haven't thought much about how that would work).

I could see these last ideas becoming a mini gem to extend RubyLLM with TopSecret capabilities, with a generator to include the column to store the mappings.

[clarissalimab]

One thing to consider is that sometimes a request will call .ask multiple times. For example, on the project I'm working on we have a categorizer that calls a light LLM to categorize messages, another one that generates dynamic loading feedback for the user while the messages are being processed and the main LLM, so ideally a solution that wouldn't apply to ALL .ask calls would be nice.

[stevepolitodesign]

@clarissalimab I was actually looking at your work with OpenTelemetry RubyLLM Instrumentation, and I think overriding #complete, and the using prepend might be a decent path forward.

[stevepolitodesign]

This was resolved with https://github.com/thoughtbot/ruby_llm-top_secret