Apparent bug with paths when deploying dependencies

[jmcdonagh]

Hi, using latest Ceedling gem downloaded from your GH release artifacts.

I have a big monorepo managed by rake that I am trying to integrate with ceedling for all the C stuff and C dependencies. One of the dependencies is bash. Here is the relevant configuration from my root project.yaml:

:dependencies:
  :deps:
    - :name: bash
      :paths:
        :fetch: zarf/vendor/bash
        :source: zarf/vendor/bash
        :build: zarf/vendor/bash
        :artifact: zarf/vendor/bash
      :fetch:
        :method: :none
      :build:
        - "./configure --prefix /opt/slithernix --bindir /opt/slithernix/bin"
        - make -j8
      :artifacts:
        :dynamic_libraries:
          - bash
          - bashbug

first off, as a suggestion, artifacts should support an 'executables' field because sometimes dependencies are just a standalone executable and don't need to be linked. i know by default this example breaks linking, but that's not the issue I am here for (aware that i can alter the linking config elsewhere in project.yaml).

The issue is when trying to deploy, the code's FileUtils.cp call is simply looking at the value of the current element in dynamic libraries, but it does not prefix that value with the artifacts path. so, what happens with this config is when you do a

ceedling -v debug dependencies:deploy

you get

🧨 EXCEPTION: No such file or directory @ rb_sysopen - bash

Debug Backtrace ==>
/Users/jmcdonagh/.rvm/rubies/ruby-3.2.8/lib/ruby/3.2.0/fileutils.rb:2290:in `initialize': (Errno::ENOENT)
	/Users/jmcdonagh/.rvm/rubies/ruby-3.2.8/lib/ruby/3.2.0/fileutils.rb:2290:in `open'
	/Users/jmcdonagh/.rvm/rubies/ruby-3.2.8/lib/ruby/3.2.0/fileutils.rb:2290:in `copy_file'
	/Users/jmcdonagh/.rvm/rubies/ruby-3.2.8/lib/ruby/3.2.0/fileutils.rb:1088:in `copy_file'
	/Users/jmcdonagh/.rvm/rubies/ruby-3.2.8/lib/ruby/3.2.0/fileutils.rb:887:in `block in cp'
	/Users/jmcdonagh/.rvm/rubies/ruby-3.2.8/lib/ruby/3.2.0/fileutils.rb:2481:in `block in fu_each_src_dest'
	/Users/jmcdonagh/.rvm/rubies/ruby-3.2.8/lib/ruby/3.2.0/fileutils.rb:2495:in `fu_each_src_dest0'
	/Users/jmcdonagh/.rvm/rubies/ruby-3.2.8/lib/ruby/3.2.0/fileutils.rb:2479:in `fu_each_src_dest'
	/Users/jmcdonagh/.rvm/rubies/ruby-3.2.8/lib/ruby/3.2.0/fileutils.rb:886:in `cp'
	/Users/jmcdonagh/.rvm/gems/ruby-3.2.8@slithernix-monorepo/gems/ceedling-1.0.1/plugins/dependencies/lib/dependencies.rb:345:in `deploy_if_required'

so if you open up dependencies.rb and go to the deploy_if_required function, and add:

src_path = File.join(@dependencies[lib_path][:paths][:artifact], lib_path)

around line 343 before the deploying dependency log message, and change the FileUtils.cp first arg from lib_path to src_path, it works.

Now, what is more confusing here is that in the docs the examples for dynamic_libraries show paths. yet if i use a path, it "double paths" me and compiles every time because, naturally, rake/make need to see the file there to satisfy the target and zarf/vendor/bash/zarf/vendor/bash/bash is not a thing.

BTW- I am a pretty skilled non-Rails Ruby developer. if you need some help here I can probably squeeze you in. This is an awesome tool and actually a way better version of a piece of garbage I threw together for this exact problem (C dependency management in a monorepo).

[mvandervoord]

Hi! Thanks for digging into this issue, reporting it, and offering to help! You're already a superstar in my book. :)

This isn't actually a usecase we had considered when we built the dependencies plugin. Up until now, these dependencies are always things that are getting fetched and/or built for the purpose of becoming part of the final executable and/or tests. Your proposal is a natural extension, though, and I don't see any reason why we shouldn't expand the usecase to include such things and building required tools.

The dependencies plugin is an under-tested plugin for ceedling and, as such, it might indeed have lurking significant issues like the path problem you've cited. I'm a little surprised that something so fundamental is broken... it seems like we would have already noticed it... but this perhaps suggests you're using it slightly differently than others. We'll have to be careful with the repair here. Ideally we build up some more test cases along the way. :)

[jmcdonagh]

let me see what i can do later, gotta learn the codebase a little more, check out the test suite, yadda yadda. i'm also noticing that in this particular use case, only the first "dynamic library" is copied over rather than the entire list here (which would be bash and bashbug). that tells me i need to look at something lower in the call stack. i'll be back.

[jmcdonagh]

so, i've been digging into the codebase some yesterday and today. at first i was going to just try to patch plugin/lib/dependencies.rb but, i am a little obsessive sometimes and started going down the rabbit hole of the whole repo. there's lots of stuff that needs refactoring all over the place to bring the codebase up to a modern, well-factored Ruby standard so I'm going to spend some real time on the codebase at large.

in the end if you like what i've done you can have it, if not, well it'll be a fork.

[mvandervoord]

Hi @jmcdonagh -- here's a thought that might effect this idea. If you look at the history of going from 0.32 to 1.0.n, you'll see that we have large refactorings underway already. There are still waves of this in progress, to bring in functionality that we used to inherit from other projects, or to restructure the way information flows through the system to better meet the needs of the project.

My point in this is that you seem to be a person who is motivated to make some positive change, and we could use more people like that. I feel like it'd be a real lost opportunity to have you running one direction with refactoring and us running a different direction with refactoring, when our goals are the same: a more modern and flexible system upon which to build reliable features.

Instead of you forking and throwing a bunch of ideas together, how do you feel about some planning conversations and we can possibly all chip away at different aspects of a longer-term single direction plan?