Add support in Windows Terminal for Kitty's Graphics Protocol

[WSLUser]

Description of the new feature/enhancement

Support Kitty's graphics rendering protocol. See https://sw.kovidgoyal.net/kitty/graphics-protocol.html. This would provide even more support than Sixel (but would be an additive sequence, not replacement.)

Proposed technical implementation details (optional)

kovidgoyal/kitty#33 provides some discussion and an implementation spec.

[DHowett]

Thanks for the request! I'd rather we not plan support for a third graphics protocol when we don't even support one.

[WSLUser]

Could it be a possible extension idea then? Though based on what I saw from the extension PR, not sure if this is something that could be extended under current spec.

[WSLUser]

@j4james Since Sixel is looking like it's not currently feasible, would Kitty's protocol be more easily adopted/implemented?

[zadjii-msft]

@WSLUser What makes you believe that sixel isn't feasible?

[WSLUser]

#448 (comment) and also seems to be more to it here: hackerb9/lsix#41

[zadjii-msft]

That doesn't sound like "not feasible" to me. That sounds like "more complicated to be fully compatible than you might think". I'm pretty sure the entire saga of PRs (e.g. #10011) that @j4james has been working on has been a long steady march towards sixel support (he can correct me if I'm wrong).

[WSLUser]

Yeah I said currently. I'm hoping for us to have at least something to use and seems like getting Sixel support will be a long way aways as we're waiting for these bugs in other terminals and Sixel applications to be resolved. Since he made it clear he's not really interested in trying to support it further at this time, I thought/hoping he may be interested in trying to support this other protocol instead, which technically is superior to Sixel anyways.

[j4james]

There are two issues involved here:

1. Propagating images over conpty. The conclusion I reached was that the most practical long-term solution would be a pure pass-through implementation of conpty. And this would likely be necessary for any image protocol. I'm not saying it's impossible to achieve with the current conpty implementation, but that would require buffering the images on the conhost side, and then regenerating partial image slices on the fly as the screen was updated. It's complicated, inefficient, and would probably depend on proprietary VT extensions.

   Another approach would be to pass just the image protocol through directly, and hope it doesn't get erased by a subsequent refresh of the text buffer. This might work some of the time, e.g. if you're just outputting an image from the command line, but any reasonably complex application involving a mix of text and images is almost certain to break at some point.

2. The sixel image protocol itself. Personally I would like our sixel implementation to be a true emulation of the original hardware terminals, just with simple backward-compatible extensions (like support for a larger palette size, and optional per-image palettes). Most open source terminals, however, implement a format that (as Douglas Adams would describe it) is almost, but not quite, entirely unlike sixel. And the problem is that at least some modern sixel applications rely on the behaviour of these pseudo-sixel terminals.

   I was hoping there might be a way we could implement the protocol that was both compatible with the original hardware, and also capable of supporting modern sixel applications, but that's not looking promising at the moment. And while I'd personally be quite happy with just a standard version of sixel, I don't think the rest of the WT userbase would necessarily agree, and our users tend to be abusive when they don't get what they want, so I'd rather avoid that fight.

So right now I'm working on point 1, since that I think is a necessity for any image protocol. On point 2, I've put the coding side of things on hold for now, but I'm still working in the background to try and nudge other terminals an apps in the right direction. And even if I don't get my way, there's no reason someone else with different priorities couldn't put together a sixel implementation that is good enough for modern usage.

As for the Kitty Graphics Protocol, I have zero interest in implementing that myself, but if someone did want to take that on, we still need to resolve point 1 first, so it's in no way a quick fix.