Use asyncio

[rwols]

This PR switches the codebase to using async def functions and asyncio. The loop provider is sublime_aio.

close #2863.

should be merged (and released) at the same time as:

• adapt to asyncio changes LSP-rust-analyzer#224
• chore: ensure json/schemaAssociations is sent before textDocument/didOpen LSP-json#237
• chore: make handle_json_schema_uri async LSP-yaml#94
• chore: use session.open_uri LSP-leo#10
• chore: fixups for asyncio LSP-Tinymist#30

The main driver for doing this is to decrease the thread usage of this plugin from O(n) to O(1) threads, where n is the number of language servers running. The secondary driver is syntax sugar.

Why is this PR so large? Please read: What color is your function?

Self-contained bits:

• chore: enhance logging functions #2991
• chore: make promise adhere to the typing.Awaitable protocol #2994
• chore: make the 'P' type var contravariant #2995

Topic	Old Way	New Way
Running a short function on a thread that's not the main thread	sublime.set_timeout_async	LSP.plugin.core.aio.call_soon_threadsafe
Defining a function that's asynchronous	def f() -> Promise[T]: ...	async def f() -> T: ...
Chaining asynchronous functions	Promise.then(lambda x: ...)	x = await f()
Doing something when a server request is done	session.send_request_async(R(), lambda x: ...)	x = await session.request(R())
Doing something when a server request fails	define an on_error callback	use a try ... except ResponseException: block
Handling partial request results	define an on_partial_result callback	async for partial_result in session.stream(R()): (caveat: only works for list[...]-style responses)
Starting a coroutine from a regular function	n/a	LSP.plugin.core.aio.run_coroutine_threadsafe(f())
Awaiting old-style Promise objects	Promise.then	await promise
Waiting for all asynchronous operations to complete	Promise.all	asyncio.gather
Doing something later	sublime.set_timeout_async(f, timeout_ms=1000)	await asyncio.sleep(1)
Enforcing a critical section	use threading.Lock, or write very complicated queueing logic	use asyncio.Lock
Wrapping a new async function in a Promise	n/a	Promise.wrap_task

f calls g	async g	blocking g
async f	async def f(): await g()	async def f(): g()
blocking f, guaranteed called from asyncio thread	use aio.TaskContainer.create_task(g())	def f(): g()
blocking f, any thread	def f(): aio.run_coroutine_threadsafe(g()), or use aio.TaskContainer.create_task_threadsafe(g())	def f(): g()

The Plan

Make "most" code run on the sublime_aio thread

Most code is doing bookkeeping. This type of code used to run on the Sublime "async" thread. It should run on the asyncio loop thread.

Previously, the code attempted to make most code run on the ST async thread. We never really enforced this. We tried to make it clear that a function/method should be running on the ST async thread by suffixing it with _async.

If you have an async def coroutine function, then such a coroutine function is forced to run on the asyncio loop thread. So enforcement becomes automatic.

Keep _async suffixes, assume they run on the asyncio thread

When a method or function has the suffix _async in its name, we tried to ensure these functions run on the ST async thread. These can now be assumed to be running on the asyncio thread.

Make compute-intensive function run on the Sublime "async" thread

The only compute-intensive code we deal with are parsing and emitting JSON. Only the JSON parser/emitter should run on the ST async thread.

Bridging code for existing LSP-* plugins

We made sure that all AbstractPlugin and LspPlugin related (class)methods ran on the ST async thread. I want to now make sure all these (class)methods run on the sublime_aio thread with this pull request.

Certain methods may also be marked async for LspPlugin, most notably on_pre_start and perhaps on_initialize.

The Promise object can be awaited, so older AbstractPlugin/LspPlugin-related functionality returning promises from request handlers work.

[netlify]

✅ Deploy Preview for sublime-lsp ready!

Name	Link
🔨 Latest commit	e6ea018
🔍 Latest deploy log	https://app.netlify.com/projects/sublime-lsp/deploys/6a9b0ac6c9f7ef00083858b4
😎 Deploy Preview	https://deploy-preview-2880--sublime-lsp.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[rchl]

Would be useful to split it into smaller chunks, if possible. For example it would likely be possible to make start_async async by making it return a Promise and then later convert it to asyncio easily.

Lots of assumptions on my side but that's what I feel.

I was actually looking into that before as I wanted to move start_async into dedicated thread so that it doesn't black other plugins (kinda opposite goal of yours but also kinda similar as I guess with asyncio it will also run on dedicated thread). See #2863

It will be hard to review it properly with a big dump of code that refactors most of the code base.

[rchl]

An idea: keep original method names with _async suffix while working on the task and only rename in a dedicated PR after asyncio is actually merged.

Why? Because renaming introduces a significant amount of changes which means:

• higher chance of conflicts with ongoing changes and more work resolving conflicts
• a lot more (distracting) changes to go through when reviewing

[rwols]

Agreed, removing async suffices (suffixes?) was a mistake, I’ll revert that.

[rwols]

I'm recreating the PR because GitHub has trouble rendering the diff...