merge with coretoolbox, or not

[cgwalters]

Moving this from https://github.com/debarshiray/toolbox/pull/100 which outlines why I created https://github.com/cgwalters/coretoolbox

Let's try to come to an agreed architecture and plan. I want to share efforts - the end goal here is that there's only one toolbox on FCOS/Silverblue systems. (If we can't resolve, we may end up e.g. shipping a different one in FCOS, add an option for Silverblue etc.)

I created coretoolbox because I don't think shell script makes sense for a project that's so critical. From there, we disagree on how to proceed. I think continuing to run podman as a subprocess is quite workable - we can parse JSON. I also am experimenting with varlink. However, upstream podman doesn't have a lot of varlink testing. On the other hand, apparently Cockpit uses it too - we want varlink to work.

So let's call this "option subprocess+varlink".

I think your position is to vendor libpod and use Go. That would make toolbox work more how crio works today. An advantage is it's known to work. As I note above, I think there are a huge amount of disadvantages, starting with having two libpod versions operating on the same storage is simply not at all tested upstream. podman continues to change formats in a way that requires migrations.

[dustymabe]

We discussed this in the Fedora CoreOS meeting today.

  * AGREED: regarding golang vs rust that is up to the author of
    toolbox. we do care about size and exploding the number of
    container-runtime implementations. If Golang is chosen we prefer
    that we don't vendor libpod, but rather either shell out or use the
    varlink API to manage containers.  (dustymabe, 17:33:38)

[dustymabe]

so @debarshiray @cgwalters is it sufficient to say the topics to be investigated:

1. will the debarshiray/toolbox project be ok with not vendoring libpod (i.e. either shell out or use varlink) so as to not blow up the size of toolbox?
2. if we can get consensus on point 1 then @cgwalters do you agree that either rust or golang are sufficient tools (disregarding personal "preferences")

[cgwalters]

if we can get consensus on point 1 then @cgwalters do you agree that either rust or golang are sufficient tools (disregarding personal "preferences")

Yeah, sure.

[debarshiray]

1. will the debarshiray/toolbox project be ok with not vendoring libpod

(i.e. either shell out or use varlink) so as to not blow up the size of toolbox?

I am happy to shell out to Podman from a wrapper Go binary, since the inflated size from vendoring in libpod is a deal breaker for CoreOS.

[cgwalters]

I am happy to shell out to Podman from a wrapper Go binary, since the inflated size from vendoring in libpod is a deal breaker for CoreOS.

If we're rewriting, why not take my existing Rust implementation? That's an important point in this - I have working code already that I actively use.

A good point for Rust BTW is that the upstream varlink stuff has a lot of great Rust code. That said varlink-go also exists.

[cgwalters]

If we're rewriting, why not take my existing Rust implementation? That's an important point in this - I have working code already that I actively use.

To flesh this out more, here's a proposal - we move debarshiray/toolbox to coreos/toolbox, then have a commit that replaces most of the code with cgwalters/coretoolbox - @debarshiray becomes a co-maintainer along with others from the CoreOS group. (Or we could move to fedora-silverblue/ GH org, since it's currently more associated with that, but we want to capture the host root cases too which is more coreos/)

[debarshiray]

I am fed up with this constant passive aggressive tone trying to push coretoolbox everywhere, and behaviour that looks a lot like trying to takeover this project.

Please, stop!

Given how many times we have spoken about this in the recent past, and your strangely evasive and aggressive behaviour, I can no longer assume well. As a long-time free software contributor, you really should know better.

The Toolbox project exists because Silverblue absolutely critically needed a way to offer developers to set up their own development environment. Some of us working on Silverblue, with a lot of help from the Podman team, hacked something up using rootless Podman that gave us what we needed. A huge chunk of this effort was just to get rootless Podman to work, because we can't tell Silverblue users that they now suddenly need to be root to get a shell.

You are probably not aware of reality because coretoolbox showed up only in April 2019. By that time a lot of major turbulence had subsided. However, back in the summer of 2018, when rootless Podman barely even existed, things were very stormy.

Say what you want about shell scripts, it's incredibly easy to share a script and expect the other party to be able to tweak and run it; and when your development workflow primarily involves throwing random Podman incantations and see what sticks and what doesn't and why not, this ease is absolutely critical for making any progress. There's no way you can ask some random person to grab some Rust or Go or whatever code and expect them to try it out in a matter of minutes. Many will probably balk at not knowing how to build it, and you can forget about ever getting to the point of hand holding them through the right Podman build or some configuration fix or whatever.

I will also point out that I have never seen any real feedback from the CoreOS team. Never.

Over the last year, I have been to multiple IRC meetings, spoke with various people in real life, and all I have heard is basically yes, yes, we love Toolbox, we are going to send patches to make it work on CoreOS. Except, I haven't seen a single patch.

The first real feedback was on the 8th of July this year about the inflated dependencies being pulled in by the toolbox RPM, which I chased down and got fixed.

Meanwhile, the Silverblue team has been busy continuously tracking Podman development, testing Podman releases across all supported branches, working on improving automated tests, working through one difficult migration after another... The list goes on.

You think you have working code? Do you have documentation? Do you have tests?

Oh, and how many users do you have? How many users from the wild have you hand held to find out why some old container is no longer working with new tools? How many regressions have you tracked down up and down the stack from GNOME to the OCI runtime?

We have lots of real end-users using Toolbox in the wild. We cannot just up and leave, and suddenly change everything. We have a ton of Cgroups v2 bugs that absolutely and urgently need to be worked out for Fedora 31.

You wonder why Toolbox is still written in POSIX shell? There's your answer - we've been busy.

Silverblue users usually don't expect to have to know how their operating system works or the ins and outs of container technology. They expect to run toolbox enter and get a shell. They don't get a shell, they throw their hands up, and complain.

Every random obstacle in using Silverblue, compared to legacy Workstation, is a hurdle to adoption.

You think you know how critical Toolbox is? Trust me, we know. For Silverblue, Toolbox is almost as critical as Bash.

I have also never seen a concrete list of specifications from CoreOS either. No wonder coreos/fedora-coreos-tracker#38 meanders from one detail to another without ever specifying what the hard requirements are.

Compare that to https://fedoraproject.org/wiki/Workstation/Desktop_Containers

I think it's pretty clear how crucial Toolbox is for Silverblue, possibly a lot more than CoreOS, because ultimately we showed up to do all the grunt work.

We could've easily taken Toolbox in a direction that's fundamentally problematic for CoreOS, like writing it in Go with libpod vendored in or whatever, but we didn't, did we? Even when all you could do was Rust fanboyism without ever specifying what your actual needs are.

We also absorbed a bunch of interesting ideas from coretoolbox with full attribution without you having to lift a single finger.

How about you learn to show some gratitude and say thank you first?

As for the future direction of Toolbox...

Toolbox will be re-written in Go. It is the language the rest of OCI ecosystem, including Podman, is written in. There's no such as thing as working on Toolbox without touching the Podman stack. It is in our best interests to stay as aligned as possible. It would be a needless hurdle to tell contributors that they need to be both Rust and Go literate to work on Toolbox.

We don't want to use Varlink, if we can avoid it. It's not something that we currently use in Silverblue at all, the Podman team isn't very happy with it, and regardless it would be risky to put a fundamentally new piece of technology in the middle of something so crucial. I'd rather focus our efforts on making Podman as robust as possible.

As for vendoring libpod versus shelling out to podman, we will stick to the latter like we discussed at the CoreOS meeting to keep /usr/bin/toolbox as small as possible.

I'd personally be happy to see CoreOS use Toolbox, and will continue to listen and accommodate the needs of CoreOS as much as possible. Of course, contributions welcome and all that. I also won't begrudge you if CoreOS decides not to use Toolbox for whatever reason.

However, I am not going to move this project to some other place for some vague wishy washy reason, or just because somebody on the Internet discovered that rootless Podman is a thing. Those who show up everyday to deal with all the boring difficulty get to decide.

Closing.

[cgwalters]

How about you learn to show some gratitude and say thank you first?

Thank you!

Given how many times we have spoken about this in the recent past, and your strangely evasive and aggressive behaviour, I can no longer assume well.

Hmm, let's change that! I don't know what's evasive. I'm sorry you perceive aggression. I certainly often have strongly held technical opinions, but please don't take anything personally. Most importantly; the reason I continue to have conversations like this is because I want to work together!

I am fed up with this constant passive aggressive tone trying to push coretoolbox everywhere, and behaviour that looks a lot like trying to takeover this project.

Ah but - we're talking about something pretty important here. I absolutely appreciate all of the work you've done. But we have technical disagreements.

Let me state it clearly: I personally absolutely do want more control over the future direction of this project. That's not the same as total control. I believe in "rough consensus and working code". I think we could work together! You and I know each other, I hope you'd agree.

As you know, coretoolbox started because I tried to contribute a patch and it didn't get merged. So let's keep that in mind! I did try hacking up the shell script first.

Oh, and how many users do you have?

Oh, not many. But, there's e.g. this comment.

Even when all you could do was Rust fanboyism without ever specifying what your actual needs are.

Hmm, now that's just unnecessary. We already went over the reasons for using a real language, like option parsing, config files, sane error handling, etc. That said I also do love Rust, it's true 😉

However, I am not going to move this project to some other place for some vague wishy washy reason, or just because somebody on the Internet discovered that rootless Podman is a thing. Those who show up everyday to deal with all the boring difficulty get to decide.

Here's another way to look at it - again, given the criticality, I think the project should move into a maintainership that isn't just you and your github personal domain. I absolutely do want to contribute to such a project.

Toolbox will be re-written in Go. It is the language the rest of OCI ecosystem, including Podman, is written in. There's no such as thing as working on Toolbox without touching the Podman stack. It is in our best interests to stay as aligned as possible. It would be a needless hurdle to tell contributors that they need to be both Rust and Go literate to work on Toolbox.

So, this is obviously where we kept getting stuck - but I think the language issue should come after maintainership. Do you see this project remaining in your personal GH under solely you, or do you see it moving to e.g. the coreos/ GH org or fedora-silverblue/ GH org and having multiple maintainers?

[dhawal55]

What is the best way to run toolbox on FCOS aws ami (v31.20200223.3.0)?

I tried running toolbox but it's not working

toolbox create
toolbox: TOOLBOX_PATH not set

TOOLBOX_PATH=/usr/bin/toolbox toolbox create
toolbox: this is not a toolbox container