Full Test Discovery inside entire codebase

[Splines]

The RubyLSP Test Explorer offers the following config option:

"rubyLsp.featureFlags": {
  "fullTestDiscovery": true
}

However, when I turn it on, the rspec command is executed as following:

bundle exec bin/rspec -r /my/path/.rbenv/versions/3.3.8/lib/ruby/gems/3.3.0/gems/ruby-lsp-rspec-0.1.28/lib/ruby_lsp/ruby_lsp_rspec/rspec_formatter.rb -f RubyLsp::RSpec::RSpecFormatter /my/path/spec/models/user_cleaner_spec.rb:27

If I set fullTestDisocvery to false, tests are executed without the -r and -f arguments:

bundle exec bin/rspec /my/path/spec/models/user_cleaner_spec.rb:27

For my setup, the latter works fine and I want to avoid the -f and -r options since I'm using docker compose exec. Is there a way for ruby-lsp-rspec to still show the tests for the entire codebase, and not only for open files?

On a side-note: I've also tried to write a custom Ruby script that maps -f and -r to the respective files in the Docker container (including the rspec_formatter.rb file). This works fine when executed in the terminal, however the VSCode Ruby LSP extension doesn't detect the end of the test running, i.e. it just continues to spin in the "running test" state, even though the test has already passed or failed, which one sees as stdout output.

[Splines]

To be clear, I'm talking about the VSCode test explorer, which only shows the test for the current file when the config option fullTestDiscovery is set to false.

[Splines]

For the sake of completeness, here is also my custom bin/rspec file.

#!/usr/bin/env ruby

DOCKER_COMPOSE_FOLDER = File.expand_path("../../docker/test", __FILE__)
DOCKER_SERVICE_NAME = "mampf"
PROJECT_ROOT = File.expand_path("../..", __FILE__)
DOCKER_PROJECT_ROOT = "/usr/src/app"

def get_container_id
  container_id = `cd #{DOCKER_COMPOSE_FOLDER} && docker compose ps -q #{DOCKER_SERVICE_NAME}`.strip
  
  if container_id.empty?
    puts "\nError: The #{DOCKER_SERVICE_NAME} test container is not running."
    puts "\nPlease start it first by running:"
    puts "  cd docker/test && docker compose up -d #{DOCKER_SERVICE_NAME}"
    puts "\nOr from the project root:"
    puts "  docker compose -f docker/test/docker-compose.yml up -d #{DOCKER_SERVICE_NAME}"
    exit 1
  end
  
  container_id
end

def map_path_to_docker(path)
  return path unless path.start_with?(PROJECT_ROOT)

  path.sub(PROJECT_ROOT, DOCKER_PROJECT_ROOT)
end

def get_docker_formatter_path(container_id)
  gem_path = `docker exec #{container_id} sh -c "gem which ruby-lsp-rspec 2>/dev/null"`.strip
  return nil if gem_path.empty?
  
  gem_dir = File.dirname(gem_path)
  "#{gem_dir}/ruby_lsp/ruby_lsp_rspec/rspec_formatter.rb"
end

def map_args_to_docker(args, container_id)
  docker_formatter_path = nil
  mapped_args = []
  i = 0
  
  while i < args.length
    arg = args[i]
    
    if (arg == "-r" || arg == "--require") && i + 1 < args.length
      next_arg = args[i + 1]
      if next_arg.include?("ruby_lsp_rspec") && next_arg.include?("rspec_formatter.rb")
        docker_formatter_path ||= get_docker_formatter_path(container_id)
        if docker_formatter_path
          mapped_args << arg
          mapped_args << docker_formatter_path
          i += 2
          next
        else
          puts "Could not find ruby-lsp-rspec gem inside the container (needed to load the formatter)."
          puts "Please install the gem inside the container (run bundle install inside the container)."
          exit 1
        end
      end
    end
    
    mapped_args << map_path_to_docker(arg)
    i += 1
  end
  
  mapped_args
end

def main
  container_id = get_container_id
  docker_args = map_args_to_docker(ARGV, container_id)
  rspec_args = docker_args.join(' ')
  rspec_command = "RAILS_ENV=test bundle exec rspec #{rspec_args}"
  # puts "Running command in Docker container #{container_id}: #{rspec_command}"

  flags = STDOUT.tty? ? ["-it"] : ["-i"]
  status = system("docker", "exec", *flags, container_id, "sh", "-c", rspec_command)
  exit_code = $?.exitstatus || (status ? 0 : 1)
  exit exit_code
end

main

[Splines]

So I think this is what is happening and why it's not working:

• On my host, bin/rspec is called when I click on the CodeLens for a test. fullTestDiscovery is set to true.
• Inside bin/rspec (see the file above), I map the -r flag to the RSpecFormatter that lives within the Docker container (in the Gems in an anonymous volume).
• The RSpecFormatter within the Docker container requires "ruby_lsp/test_reporters/lsp_reporter" in order to report back the test results. However, the host knows nothing about this and is not being informed.
• That's why on the host, I see the spinning wheel and the test never ends.

My wrong assumption was that the test reporting happens by checking the stdout. But instead, the formatter class requires the Ruby LSP reporter and uses it directly to report results.

---

So I'm wondering how my scenario could work. Note that I still want to use VSCode without Dev Containers as described here. So I do a bundle install inside my local environment and a bundle install when the app starts in the Docker container. Is there any way to report back the results from within the Docker container to the host?

It's not an option to mount all my gems to the Docker container as that defeats the purpose of the Docker container to be self-contained. It'd be an option to mount just the formatter file to the Docker container. But given my first analysis, I'm not sure if this would help at all since the require would still be evaluated from within the Docker side, right?

---

@westonkd Maybe I'm not reading #27 (or rather your PR description #57) correctly, but in the end, it doesn't seem to be as trivial as just setting the rspecCommand to something like docker compose exec myapp rspec. The option useRelativePaths does not exist and therefore, I had to manually set the path of the formatter and map the paths of the spec files to paths inside the container. While it works on the command-line, the reporting mechanism back to Ruby LSP doesn't, so the test will never halt as described above.

[westonkd]

👋 Hi @Splines

Maybe I'm not reading #27 (or rather your PR description #57) correctly, but in the end, it doesn't seem to be as trivial as just setting the rspecCommand to something like docker compose exec myapp rspec...

You are correct, it's likely not as easy as that. As you noted, the useRelativePaths option was not included in the merged PR (see discussion starting here for context).

I'm unfortunately busy with unrelated work so have not had time to follow recent developments here, but I'd be happy to reproduce your setup and collaborate on ideas when I get time if it's helpful.

To make sure I understand correctly, do I have the gist of your setup right?

• Containerized Ruby app. The container has required non-development gems installed, but not development dependencies (ruby-lsp / ruby-lsp-rspec)
• Development dependencies (ruby-lsp / ruby-lsp-rspec) are installed on the host machine. Are these managed via Bunder or installed as "system gems" (gem install ...)?
• VS Code is running on the host machine without the dev container extension used
• A bundle install is run on the host machine using the containerized app's Gemfile/Gemfile.lock

Is that correct?

[westonkd]

One more note that might help guide the direction of this issue:

I mentioned in my previous comment I removed useRelativePaths before merging. What led me to that decision was this section of the ruby-lsp README.

My read of this section left me with the impression that the ruby-lsp maintainers' position is to use Dev Containers when using the gem with containerized development, with ruby-lsp running in the container. This made useRelativePaths not needed for the use-cases I considered, since path construction and resolution would occur within the container.

This sentence from that section might be relevant as we explore solutions:

To provide its functionality, the Ruby LSP must be running in the same place where your project files exist and dependencies are installed.

[Splines]

My read of this section left me with the impression that the ruby-lsp maintainers' position is to use Dev Containers when using the gem with containerized development

Yes indeed, see also the current discussion on Ruby-LSP: Understanding how to improve container support. This image would be my dream, but I understand that the maintainers will probably not want to go into that direction since the world has dev containers.

To make sure I understand correctly, do I have the gist of your setup right?

You got everything right. Just a remark: we do also install the dev dependencies even in the Docker container, although they are not used there. That could be optimized. But conceptually yes, they are not needed there, at least for the majority. Since something like the debug gem is still needed inside the Docker container to work.

In this current issue, I also installed ruby-lsp-rspec inside the Docker container, just in order to have a -r path that points to the formatter in the Docker container. But apparently that did not work as the formatter really has to communicate (not only via stdout) with the Ruby LSP extension that is running on my host machine. I haven't tried to volume-map these files yet.

Are these managed via Bunder or installed as "system gems" (gem install ...)?

These are managed via Bundler. We have a Gemfile on the host where we do bundle install. The same Gemfile (and lock-file) is copied over to Docker, where we do another bundle install. These dependencies are cached in an anonymous volume.

---

In summary, we currently code without dev containers in VSCode (connecting to WSL). Docker is only used to get our runtime up, e.g. connecting to a database inside another container, having a mailcatcher running etc. So I modify code in the editor, then go to localhost served by Docker to see the result. Debugging works fine in this setup via the debug gem, just testing is a bit of a pain. We currently use my solution in this gist via the Ruby Test Explorer extension that is going to be deprecated, see #92. And admittedly, my script is also hacky (and yes, I could have written it in Ruby instead of Python 😅).

I'm searching for a better solution right now. I once gave Dev Containers a try and started migrating our setup, but refrained from it after a while since it quickly became a major task and we as a small team wanted to rather focus on feature development and bug fixes. Maybe I should give it another try, but given our working setup (locally, in CI/CD and in production using Docker containers), I don't see the immediate benefits.

Should you like to look into our setup, it is here on MaMpf. Certainly, there are many things to improve on and we do this step by step ;)

I'm unfortunately busy with unrelated work so have not had time to follow recent developments here, but I'd be happy to reproduce your setup and collaborate on ideas when I get time if it's helpful.

No worries, it's not pressing and any ideas are appreciated.

[st0012]

Ruby LSP leans on Dev Container is because that's VS Code's recommendation. Communicating between container & host can be tricky depending on the setup (which can probably vary a lot), so Ruby LSP relies on VS Code's server & client to handle that automatically.
Given that we don't use containers ourselves, it's not likely that we can design a better communication approach than what VS Code offers. And as a small team, it also doesn't make sense for us to prioritize this work.

Still, I appreciate your discussions here because it means that Ruby LSP should still keep the "fullTestDiscovery": false implementation for the time being so some users can still use the (now we call) legacy test explorer features.

[Splines]

Thanks for your response. I'm now trying to migrate our setup to dev containers. We previously used a dockerized test setup with an empty database to run our unit tests. Via the database cleaner, the database is emptied before each new test run. This is to ensure test reproducibility.

However, the main service that we use for the dev containers is the app in development mode, not in test mode (i.e. with different environment variables). Based on the Container Development section and the Ruby LSP Test docs, I didn't find any information for this scenario.

How do other people solve this? Is this really exotic? Maybe there's a simple way that I don't see? I thought of setting the rspecCommand to something that executes the test in the other docker container, but that would mean that the two app containers (built from different environments) would have to talk to each other, which sounds problematic.