Hi...

[makhnoband]

Hi!

First, thanks for FaceID. I'm using FaceID 0.9.1 with Frigate 0.17 and Home Assistant.

I have a use case where fast recognition is important: when several people approach the entrance together, I use the FaceID MQTT sensor in Home Assistant to identify who entered.

I noticed a case where FaceID recognized some people in the group quickly, but another person was recognized much later.

Looking at the FaceID log, I found what appears to be the reason.

For example, one event showed:

• snapshot face size: 39 px
• min_face_px: 48
• result: face rejected as too small

Later, the recording fallback found a much better frame:

• face size: 91 px
• detection confidence: 0.86
• match: Evgenii (0.502)

So the recording contained a perfectly usable face, but it became available to recognition much later than I need for real-time Home Assistant automations.

My current relevant settings are:

• min_face_px: 48
• det_size: 640
• max_attempts: 6
• clip_fallback: true
• poll_interval: 0
• Frigate detect stream: 1280x720 @ 10 fps
• separate high-quality main stream is used by Frigate for recording

Would it be possible to add an optional live high-resolution fallback?

The idea would be:

1. Receive the normal Frigate event/snapshot.
2. Try the normal fast recognition path.
3. If a face is detected but rejected because it is smaller than min_face_px (or perhaps if no usable face is found after N attempts), do not wait for the normal recording fallback.
4. Immediately fetch/grab a higher-resolution frame from the camera/Frigate main stream or current recording.
5. Run face detection/recognition on that frame.
6. Continue using the existing behavior if the high-resolution attempt also fails.

Conceptually:

Normal path:

Frigate event
→ snapshot/person crop
→ face >= min_face_px
→ recognition

Proposed fallback:

Frigate event
→ snapshot/person crop
→ face < min_face_px
→ immediately get high-resolution frame
→ detect face
→ recognition

In my case this could potentially turn:

39 px face
→ rejected
→ wait/retries
→ recording fallback
→ 91 px face
→ recognized

into:

39 px face
→ too small
→ live high-res fallback
→ larger face
→ recognized immediately

I would prefer this over simply lowering min_face_px, because recognizing very small faces may reduce recognition reliability. The high-resolution fallback could preserve the current quality threshold while improving recognition latency.

It could perhaps be configurable, for example:

• live_hires_fallback: true/false
• live_hires_fallback_after_attempts: 1
• optional camera allowlist
• optional cooldown to avoid excessive CPU/network load

Do you think this is technically possible with the current Frigate integration?

If FaceID already has a mechanism that can achieve the same result, I'd also appreciate advice on the recommended configuration.

Thanks!

[NerdyHank]

Thanks — this is a good report, and the diagnosis in it is correct. Shipped in v0.10.0
as live_hires_fallback, though not quite the way you proposed: I measured your suggested
route first and it turned out not to work.

The recording cannot be fast. Your step 4 was "immediately fetch a higher-resolution
frame from the recording". I checked how fresh a frame Frigate will actually serve, by
requesting increasing ages until one came back:

requested age	result
2s … 30s	nothing
45s and older	frame, 2560x1920

So the recording only starts answering about 45 seconds after the moment. Fetching it
"immediately" would just trade one wait for another — no good for a doorway automation.

go2rtc is the fast path. It ships with Frigate, listens on port 1984, and serves the
main stream directly:

GET http://<frigate-host>:1984/api/frame.jpeg?src=<camera>
→ 2560x1920 in 0.6–1.0s

Frigate's own /api/<camera>/latest.jpg is not a substitute — it comes from the detect
stream, so it is exactly as small as the snapshot that just failed. On my setup it returns
1280x960 against go2rtc's 2560x1920.

Cropping to the person box made things worse. Your step 4 implied using the detection
box, which sounded right to me too. Measured over the same events, three ways:

source	events with a usable face	face size
detect snapshot (today's path)	5/8	50–105px
full-resolution frame	7/8	104–212px
full-resolution + person-box crop	3/8	—

The crop loses because data.box describes one specific moment; by the time a frame is
fetched the person has moved, and the crop cuts them off. Using the frame whole wins, and
it also removes the need to map detect-stream coordinates onto the main stream — so the
feature is simpler than either of us expected.

Worth noting for your case specifically: faces come out roughly twice as large. Your
39px face would land around 80px — comfortably over your min_face_px: 48, without
lowering it. That was your explicit preference and it holds up.

How to enable it

live_hires_fallback: true
live_hires_fallback_cameras: []   # or a list, e.g. [front_door]

Or in Settings → Live hi-res retry. It runs once per event, only when the snapshot
found nothing usable, so there is no repeated cost across max_attempts — which also
covers the cooldown you suggested. clip_fallback stays as the net for whatever the live
attempt misses.

It is off by default because port 1984 is not exposed in every setup, notably behind
some proxy arrangements. FaceID probes go2rtc once at startup and tells you either way, so
you do not have to guess:

go2rtc reachable (2560x1920) — 'live_hires_fallback' would work here

If you instead see that it did not answer, check whether 1984 is reachable from the FaceID
container, and whether your src name matches the camera name. There is also a
go2rtc_url setting under frigate: if it runs somewhere other than the Frigate host.

One caveat, stated plainly: the measurements above come from recorded events, and my
own live path has not yet caught a real person since I enabled it — it is on here now, so
that will follow. The mechanism is verified end to end (frame fetched, face detected,
match published); what I have not yet observed is the full sequence triggered by someone
actually walking up. If you try it, I would like to know what latency you see between the
Frigate event and the MQTT sensor updating — that is the number your use case lives on,
and I cannot measure it as well as you can.

[NerdyHank]

Follow-up: v0.11.0 now recognises every face in that live frame, not just the largest — which turned out to matter directly for your group case.

Measured here, 5 of 12 door events had more than one person in the full frame. best_face() kept the biggest and discarded the rest, so whoever stood closest to the camera won. Frigate raises one event per person, so a group usually still resolves — but an event whose snapshot yields nothing loses that person entirely, even though their face is plainly visible in the frame just fetched for somebody else. That is exactly the "recognised some people quickly, another much later" pattern you described.

Now every recognised person is published and added to the camera's presence sensor, which has always held a set of people. Only the largest face binds to the Frigate event, since sub_label takes one name.

While building it I found a related bug worth mentioning, because it would have bitten you specifically: the per-event "already announced" marker held a single name. Two known people in one frame overwrote each other and got re-announced in turn — duplicate notifications, which is the opposite of what that marker is for. It is a set now.

Still no real person has walked into my live path since enabling it, so the end-to-end latency question from my last comment stands.

[makhnoband]

Продолжение: версия 0.11.0 теперь распознает каждое лицо в этом живом кадре, а не только самое крупное, что, как оказалось, имеет непосредственное значение для вашего группового случая.

Здесь, по нашим измерениям, в 5 из 12 событий с дверями в полный кадр попадало более одного человека. best_face() Мы оставили самые крупные, а остальные отбраковали, так что победил тот, кто стоял ближе всех к камере. Frigate поднимает по одному событию на человека, так что группа обычно все равно распознается, но событие, снимок которого ничего не дает, полностью теряет этого человека, даже если его лицо отчетливо видно в кадре, который только что был получен для кого-то другого. Это именно тот случай, когда «одних людей распознают быстро, а других — гораздо позже».

Теперь каждое распознанное лицо публикуется и добавляется в датчик присутствия камеры, который всегда фиксировал группу людей. К событию Frigate привязывается только самое крупное лицо, поскольку sub_label принимает одно имя.

При его создании я обнаружил связанную с ним ошибку, о которой стоит упомянуть, потому что она могла коснуться именно вас: маркер «уже анонсировано» для каждого события содержал одно имя. Два известных человека в одном кадре перезаписывали друг друга и, в свою очередь, получали повторное уведомление — это противоречит назначению этого маркера. Теперь это набор.

С тех пор как я включил эту функцию, ни один реальный человек не появился на моем пути, так что вопрос о сквозной задержке, который я задал в предыдущем комментарии, остается открытым.

Большое спасибо за то, что так быстро реализовали эти изменения! Я очень ценю, что вы оперативно разобрались в проблеме, протестировали разные подходы и добавили как live high-resolution fallback, так и распознавание нескольких лиц одновременно.

Похоже, это именно то, что было нужно для моего сценария.

Я уже обновил FaceID и включил новую функциональность. Как только у меня появится возможность нормально протестировать её в реальных условиях, я обязательно дам подробную обратную связь — особенно по сценарию, когда несколько человек заходят одновременно.

Ещё раз спасибо за быструю реакцию и реализацию!

[makhnoband]

Thanks again for the additional testing instructions. I’ve confirmed that the startup log shows the go2rtc connection and that the live hi-res fallback is active at 2688×1520.

As promised, I’m going to properly test the current v0.11.0 implementation first and will report back with real-world results and timing.

While thinking through the group-arrival scenario, though, I noticed one possible limitation and wanted to get your thoughts on it.

Suppose three people enter together, but Frigate only detects/tracks two person objects — for example because the third person is partially occluded.

If FaceID successfully recognizes both detected people from their normal Frigate snapshots, the live hi-res fallback is never triggered. In that case, the third person could remain completely invisible to FaceID, even though their face might actually be clearly visible in the full 2688×1520 frame.

On the other hand, if one of those normal snapshots fails and triggers the new live hi-res fallback, FaceID now scans the full frame and, with the new multi-face support, could potentially find all three people — including the person Frigate did not create a separate event for.

That made me wonder whether an optional mode could be useful where the first person event from a camera triggers a full-frame live hi-res scan immediately, rather than using it only as a fallback.

Something conceptually like:

Frigate detects at least one person
↓
FaceID receives the event
↓
one live frame from go2rtc (2688×1520)
↓
detect and recognize ALL faces in the full frame
↓
publish all recognized identities
To avoid doing the same expensive full-frame analysis multiple times when Frigate creates several person events for the same group, there could perhaps be a short per-camera cooldown/debounce — e.g. 1–2 seconds.

So if Frigate generates three person events almost simultaneously:

event #1 → run full-frame hi-res scan
event #2 → skip hi-res scan (cooldown)
event #3 → skip hi-res scan (cooldown)
This would effectively let Frigate act primarily as the trigger (“someone appeared on this camera”), while FaceID determines how many recognizable faces are actually present in the high-resolution frame.

I’m not suggesting replacing the current behavior — the existing snapshot-first approach obviously has performance advantages. I’m thinking of this as an optional mode for use cases where reliably identifying everyone in a group is more important than minimizing processing.

Perhaps something along the lines of:

live_hires_mode: fallback # current behavior

or

live_hires_mode: always

live_hires_cooldown: 2
Does this architecture make sense to you, or is there something about the Frigate event / FaceID pipeline that would make this approach problematic?

In any case, I’ll test the current v0.11.0 implementation first as promised and send you the actual logs and latency results once I have a good real-world test.

[NerdyHank]

Thanks — glad it fits. Two things that will save you a confusing first test:

Check the startup line before anything else. The live path only works if go2rtc answers on port 1984, which not every setup exposes:

go2rtc reachable (2560x1920) — 'live_hires_fallback' would work here
Live hi-res fallback active via go2rtc (2560x1920)

If it instead reports that go2rtc did not answer, nothing else will work regardless of the setting. Frigate's own port (5000) will not do — /api/go2rtc/... returns 404 there, it is a separate listener.

The new path only shows up when the snapshot fails. If a face is recognised the normal way you will just see attempt 1, match <name>, which is a good outcome but tells you nothing about the new code. The line to look for is:

event … (front_door): live frame 2560x1920 has 2 face(s) the snapshot missed (largest 212px, det 0.78, 1.0s)

A count of 2 or more means both new behaviours fired at once.

Since your case is specifically several people arriving together, the number I am most curious about is the one I cannot measure here: the delay between the Frigate event and the MQTT sensor updating, compared to what you saw before. No rush — I would rather have one good real-world observation than a quick answer.

Also worth knowing: your 39px face should land around 80px through this path, so min_face_px: 48 can stay where it is. That was your explicit preference and it holds.

[NerdyHank]

Good catch, and you are right that the fallback structurally cannot reach that case. If
Frigate never tracks the hidden person as their own object, both other snapshots succeed,
the fallback never fires, and that person simply does not exist as far as FaceID is
concerned. Multi-face support does not help, because it only runs once the fallback has
been triggered by someone else's failure.

Shipped as v0.12.0: live_hires_mode: always, plus live_hires_cooldown — essentially
your design.

But I measured it first, and the numbers argue against making it the default. Seven
days on a front-door camera, grouping events whose time ranges overlap, then comparing how
many people Frigate reported against how many faces are in the full frame:

59 events in 31 groups, 18 with more than one simultaneous event

  time          Frigate  faces in full frame
  08.08. 09:57        5          1
  10.08. 10:23        4          2
  09.08. 16:16        3          1
  10.08. 11:55        2          1
  …
  analysable groups: 15
  groups with more faces than events: 0   (0%)

Not once. And the ratio runs the other way — five events for a single visible face —
because Frigate re-tracks the same person as a new object whenever a track breaks. What
looked like "a group of five" was one person, tracked five times.

So on this setup always would have cost ~31 extra frame fetches a week and found nobody.
That is one camera at a quiet suburban front door, though, and your entrance sounds
busier — the case you describe is entirely plausible there. Hence: shipped, off by default,
with the number in the docs so nobody enables it on a hunch.

Two details worth knowing before you switch it on:

The cooldown does what you proposed (default 2s, per camera). One group produces a
burst of events and the frame would be practically identical for each, so it is fetched
once. Verified: three events in a burst, one fetch.

Nothing found in always mode is bound to the Frigate event. The scan runs before
the snapshot has established who the event is about, so the largest face in the frame may
be somebody else entirely. Those people are published and counted towards the camera's
presence sensor, but sub_label and the event's own score stay with the snapshot. Without
that split, always would start writing the wrong name onto events — the opposite of what
you want.

If you want to check your own cameras before switching: group the events by overlapping
time ranges, pull one full frame per group, and compare the face count against the number
of events in that group. If your ratio differs from mine, I would genuinely like to see it
— it would be the first evidence that the mode earns its cost, and I would then reconsider
the default.