Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions cmd/silo/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -1917,6 +1917,7 @@ func main() {
},
)
pluginHost = pluginhost.NewHost(pluginhost.Config{
IdleStopAfter: pluginhost.IdleStopAfter,
EventPublisher: eventsHub,
LibraryLister: pluginhost.NewLibraryLister(libDataSource),
CatalogPresence: catalogPresence,
Expand Down
6 changes: 4 additions & 2 deletions docs/admin-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -3440,8 +3440,10 @@ exponential backoff from 1 s to 60 s, and is parked as `failed` after ten consec
failures until restarted or reconfigured), `state` (`stopped`, `starting`, `running`,
`backoff`, `failed`), `restart_count`, `last_error`, `last_started_at` and `next_restart_at`.
Plugins declaring `network_access_provider.v1` are resident; every other plugin starts on
first use and reports `running`, `stopped`, or `failed` when its last launch on the answering
server failed, with `last_error` saying why. A later successful launch, an administrator
first use, is stopped again after 30 minutes
without use, and reports `running`, `stopped`, or `failed` when its last launch on the answering
server failed, with `last_error` saying why. A plugin stopped for being idle reports `stopped`
and starts on its next use. A later successful launch, an administrator
restart, an update or a configuration change clears that failure. A plugin that isn't running
also reports `failed` when the answering server has to restore its files from the stored
archive and the archive can't be read (it won't open or extract, or doesn't match the
Expand Down
76 changes: 76 additions & 0 deletions docs/architecture/plugin-idle-stop.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# Stopping idle plugin processes

A plugin that is not resident starts on its first RPC. The host stops it again
once nothing has used it for 30 minutes, and the next RPC starts it as the first
one did. A server whose libraries never call a plugin (an audiobook metadata
provider on a Movies and Series server) does not keep that process in memory.

The period is fixed (`pluginhost.IdleStopAfter`). Thirty minutes covers a burst
of calls, and a plugin that caches responses for up to half an hour keeps that
cache through a quiet spell. Proxy nodes build their plugin host without an idle
period, so nothing there is stopped for being idle.

## What counts as use

`internal/pluginhost/idle.go` keeps, per process, the time of the last use and
the number of calls in flight:

- A lookup through `Host.Client`, which every RPC to a lazily started plugin
makes through `Service.ensureClient`.
- A host call to the plugin on the `silo.plugin.v1` services, from the moment
it is sent until it returns. Scheduled task runs, event deliveries, plugin
HTTP routes, scan source polls and sign-in exchanges are all such calls. The
idle period counts from the end of the last call.
- A plugin call back to the host (`RuntimeHost`), for the same span. A callback
with no host call in flight is the only sign of work a plugin does on its
own.

The health probe, go-plugin's broker, stdio and controller traffic, and status
reads (`Host.Running`, which backs the admin `runtime.state`) are not use.

## What is never stopped

- Resident plugins (`network_access_provider.v1`). The supervisor owns them and
their RPCs do not start a process. `Service.start` asks for an idle stop only
when the manifest is not resident; the host never decides this by itself.
- Config test runs (negative installation IDs), which their caller stops.
- A process that opened a `silo.plugin.v1` stream. Every capability RPC is
unary today. A streaming one has to count as in flight until it closes
before its plugin can be stopped.

## Invariants

- The idle check and the removal from the host's instance map happen under the
lock `Host.Client` takes to record a use. A caller either finds the process
and keeps it alive for another idle period, or finds none and starts one.
The same decision marks the process as stopping under the lock that admits
plugin callbacks, so a callback the plugin starts afterwards is refused with
`Unavailable` instead of being cut off mid-flight.
- The watcher holds the installation's start slot until the old process has
exited, so a restart never overlaps the process it replaces. A `Start`
already in flight wins: it cancels the watcher. `Stop` and `Shutdown` also
wait for that exit. The one exception is a process that still has not
exited `DefaultIdleStopTimeout` after being killed (stuck in uninterruptible
I/O): the watcher logs a warning and releases the slot, so a restart can
overlap it and `Stop` or `Shutdown` can return before it exits.
- The first call after an idle stop waits for a cold start before it is sent:
the binary is verified again, the process launches, and the host reads its
manifest and configures it. A start that fails is recorded as any failed
lazy start is.
- An idle stop is a deliberate stop. The client is not marked unhealthy, the
exit handler is not called, and no lazy start failure is recorded. The
installation reports `stopped`, and the admin UI shows it as **Active**, like
any other healthy plugin that is not running. The stop is logged at info.
- The process is asked to shut down and is killed if it has not exited after
two seconds. go-plugin's shutdown request has no deadline, so a process that
is frozen when its idle period ends is killed after `DefaultIdleStopTimeout`
instead of holding the start slot, and every caller waiting on it, for good.
- State a plugin keeps in memory does not survive an idle stop. Host-held
state (configuration, instance state, OAuth and device-code flow state, scan
markers, sync cursors) is sent again with each call. A plugin must not rely
on a response cache or a background timer outliving its last call by more
than the idle period.

A caller that resolves a capability client and waits longer than the idle
period before calling it gets a connection error, not a restart. No caller
holds a client across operations; keep it that way.
98 changes: 92 additions & 6 deletions internal/pluginhost/host.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,15 @@ type Config struct {
// ExitCheckInterval is how often the exit watcher polls the plugin
// process for an exit. Zero takes DefaultExitCheckInterval.
ExitCheckInterval time.Duration
// IdleStopAfter is how long a process started with StopWhenIdle may go
// unused before the host stops it. Zero never stops an idle process.
IdleStopAfter time.Duration
// IdleCheckInterval is how often such a process is checked. Zero takes
// DefaultIdleCheckInterval.
IdleCheckInterval time.Duration
// IdleStopTimeout is how long an idle stop waits for the plugin to shut
// down before killing its process. Zero takes DefaultIdleStopTimeout.
IdleStopTimeout time.Duration

// EventPublisher receives events that plugins publish via RuntimeHost.
// Typically silo's *events.Hub. When nil, plugins that try to
Expand Down Expand Up @@ -65,13 +74,23 @@ type StartRequest struct {
BinaryPath string
Manifest *pluginv1.PluginManifest
Config []*pluginv1.ConfigEntry
// StopWhenIdle lets the host stop the process after Config.IdleStopAfter
// without use. Only set it for a plugin whose callers go back through
// Client or Start before every call, so the next one starts it again.
// It is ignored for config test runs (negative installation ids).
StopWhenIdle bool
}

type Host struct {
logger hclog.Logger
healthCheckInterval time.Duration
healthFailureLimit int
exitCheckInterval time.Duration
idleStopAfter time.Duration
idleCheckInterval time.Duration
idleStopTimeout time.Duration
// now is a test seam for the idle clock.
now func() time.Time

// exitHandler is told the installation id of a plugin whose process went
// away on its own (exit or failed health), never one the host stopped on
Expand All @@ -93,7 +112,11 @@ type Host struct {
mu sync.RWMutex
instances map[int]*instance
starting map[int]chan struct{}
startSeq atomic.Uint64
// idleStopping holds, per installation, a channel closed once the idle
// watcher has finished stopping a process it already removed from
// instances.
idleStopping map[int]chan struct{}
startSeq atomic.Uint64
}

type instance struct {
Expand All @@ -102,6 +125,7 @@ type instance struct {
usageOnce sync.Once
protocol plugin.ClientProtocol
client *Client
activity *activity
// installationID lets stopInstance revoke the ingress token of a network
// access provider without a map lookup.
installationID int
Expand Down Expand Up @@ -133,8 +157,20 @@ func NewHost(cfg Config) *Host {
if exitInterval <= 0 {
exitInterval = DefaultExitCheckInterval
}
idleInterval := cfg.IdleCheckInterval
if idleInterval <= 0 {
idleInterval = DefaultIdleCheckInterval
}
idleStopTimeout := cfg.IdleStopTimeout
if idleStopTimeout <= 0 {
idleStopTimeout = DefaultIdleStopTimeout
}

return &Host{
idleStopAfter: max(cfg.IdleStopAfter, 0),
idleCheckInterval: idleInterval,
idleStopTimeout: idleStopTimeout,
now: time.Now,
logger: logger,
healthCheckInterval: interval,
healthFailureLimit: failureLimit,
Expand All @@ -150,6 +186,7 @@ func NewHost(cfg Config) *Host {
networkAccess: cfg.NetworkAccess,
instances: make(map[int]*instance),
starting: make(map[int]chan struct{}),
idleStopping: make(map[int]chan struct{}),
}
}

Expand Down Expand Up @@ -206,11 +243,15 @@ func (h *Host) Start(ctx context.Context, req StartRequest) (*Client, error) {
ingressToken = token
}

used := newActivity(h.now)
command := exec.Command(req.BinaryPath)
command.Env = ProcessEnv()
process := plugin.NewClient(&plugin.ClientConfig{
HandshakeConfig: HandshakeConfig(),
GRPCDialOptions: []grpc.DialOption{grpc.WithChainUnaryInterceptor(observePluginRPC)},
GRPCDialOptions: []grpc.DialOption{
grpc.WithChainUnaryInterceptor(used.unaryClient, observePluginRPC),
grpc.WithChainStreamInterceptor(used.streamClient),
},
AllowedProtocols: []plugin.Protocol{
plugin.ProtocolGRPC,
},
Expand Down Expand Up @@ -257,7 +298,7 @@ func (h *Host) Start(ctx context.Context, req StartRequest) (*Client, error) {
return nil, fmt.Errorf("unexpected plugin runtime client type %T", rawClient)
}

if err := h.bindRuntimeHost(ctx, rpcClient, req.Manifest.GetPluginId(), req.InstallationID, provider, ingressToken, hostInfo, instanceState); err != nil {
if err := h.bindRuntimeHost(ctx, rpcClient, used, req.Manifest.GetPluginId(), req.InstallationID, provider, ingressToken, hostInfo, instanceState); err != nil {
_ = protocol.Close()
process.Kill()
return nil, fmt.Errorf("bind runtime host: %w", err)
Expand Down Expand Up @@ -307,6 +348,7 @@ func (h *Host) Start(ctx context.Context, req StartRequest) (*Client, error) {
command: command,
protocol: protocol,
client: client,
activity: used,
installationID: req.InstallationID,
provider: provider,
ingressToken: ingressToken,
Expand All @@ -320,6 +362,9 @@ func (h *Host) Start(ctx context.Context, req StartRequest) (*Client, error) {

go h.monitorHealth(monitorCtx, req.InstallationID, instance)
go h.watchExit(monitorCtx, req.InstallationID, instance)
if req.StopWhenIdle && req.InstallationID > 0 && h.idleStopAfter > 0 {
go h.watchIdle(monitorCtx, req.InstallationID, instance)
}

return client, nil
}
Expand Down Expand Up @@ -362,9 +407,15 @@ func (h *Host) SetExitHandler(handler func(installationID int)) {
h.exitMu.Unlock()
}

// Client returns the running client for a caller about to use it, which
// counts as a use: the idle watcher takes the write side of h.mu to remove an
// instance, so one returned here is not stopped for another idle period.
func (h *Host) Client(installationID int) (*Client, error) {
h.mu.RLock()
instance, ok := h.instances[installationID]
if ok && instance.activity != nil {
instance.activity.touch()
}
h.mu.RUnlock()
if !ok {
return nil, ErrClientNotFound
Expand All @@ -380,6 +431,20 @@ func (h *Host) Client(installationID int) (*Client, error) {
return instance.client, nil
}

// Running reports whether a healthy process is up for the installation
// without counting as a use of it, for status reads.
func (h *Host) Running(installationID int) bool {
h.mu.RLock()
instance, ok := h.instances[installationID]
h.mu.RUnlock()
if !ok {
return false
}
instance.client.mu.RLock()
defer instance.client.mu.RUnlock()
return !instance.client.unhealthy
}

func (h *Host) NextStartSeq() uint64 { return h.startSeq.Load() }

func (h *Host) Stop(installationID int) error {
Expand All @@ -388,9 +453,15 @@ func (h *Host) Stop(installationID int) error {
if ok {
delete(h.instances, installationID)
}
idleStop := h.idleStopping[installationID]
h.mu.Unlock()

if !ok {
// The process may still be exiting after an idle stop. Callers treat
// a return from Stop as the process being gone.
if idleStop != nil {
<-idleStop
}
return ErrClientNotFound
}
h.stopInstance(instance)
Expand All @@ -406,11 +477,18 @@ func (h *Host) Shutdown(ctx context.Context) error {
delete(h.instances, installationID)
instances = append(instances, instance)
}
idleStops := make([]chan struct{}, 0, len(h.idleStopping))
for _, idleStop := range h.idleStopping {
idleStops = append(idleStops, idleStop)
}
h.mu.Unlock()

for _, instance := range instances {
h.stopInstance(instance)
}
for _, idleStop := range idleStops {
<-idleStop
}
close(done)
}()

Expand Down Expand Up @@ -509,13 +587,21 @@ func (h *Host) retireInstance(ctx context.Context, installationID int, instance
}

func (h *Host) stopInstance(instance *instance) {
h.teardown(instance, false)
}

// teardown stops an instance already removed from the host. A graceful one
// leaves closing the connection to Kill, which then asks the plugin to shut
// down and waits briefly for it to exit by itself, so a routine idle stop is
// not logged and counted as a killed process.
func (h *Host) teardown(instance *instance, graceful bool) {
if instance == nil {
return
}
if instance.cancelMonitors != nil {
instance.cancelMonitors()
}
if instance.protocol != nil {
if instance.protocol != nil && !graceful {
_ = instance.protocol.Close()
}
if instance.process != nil {
Expand All @@ -539,7 +625,7 @@ func (h *Host) stopInstance(instance *instance) {
// tears it down.
//
// Skipped when no RuntimeHost services are configured.
func (h *Host) bindRuntimeHost(ctx context.Context, sdkClient *sdkruntime.Client, pluginID string, installationID int, provider, ingressToken string, hostInfo HostInfoFunc, instanceState InstanceStateStore) error {
func (h *Host) bindRuntimeHost(ctx context.Context, sdkClient *sdkruntime.Client, used *activity, pluginID string, installationID int, provider, ingressToken string, hostInfo HostInfoFunc, instanceState InstanceStateStore) error {
if h.eventPublisher == nil && h.libraryLister == nil && h.catalogPresence == nil && h.installedPlugins == nil && h.globalConfigSetter == nil &&
hostInfo == nil && instanceState == nil && h.networkAccess == nil {
return nil
Expand All @@ -554,7 +640,7 @@ func (h *Host) bindRuntimeHost(ctx context.Context, sdkClient *sdkruntime.Client

streamID := broker.NextId()
go broker.AcceptAndServe(streamID, func(opts []grpc.ServerOption) *grpc.Server {
s := grpc.NewServer(append(opts, grpc.ChainUnaryInterceptor(observePluginCallback))...)
s := grpc.NewServer(append(opts, grpc.ChainUnaryInterceptor(used.unaryServer, observePluginCallback))...)
srv := NewRuntimeHostServerWithOptions(RuntimeHostOptions{
Publisher: h.eventPublisher,
Libraries: h.libraryLister,
Expand Down
Loading