Timeline - Improve recording/position changes with keyboard controls

[CDx4f3kCAf3Y]

This is based off of d8a219d.

In that commit, there is the following:

• Improved random seeking for recordings using the new pan and zoom timeline features
• Improved sequential scanning of recordings using the new backward and forward buttons
• Seeking and scanning within a recording using the exiting video controls (the position indicator has to be clicked first)

All of these actions are mostly mouse or touch oriented.

I had been thinking for a while once an interesting section of the timeline has been identified (using the new pan and zoom controls), that it would be great if the user could use the arrow keys on their keyboard to sequentially scan between recordings at a broad level and within a recording at a fine level, while keeping the overall interactions with the page to a minimum.

The changes below are a very initial and rough attempt to address this goal.

With these changes:

• When the Timeline loads, it is in the 'broad' navigation mode.
  • Left/Right arrow keys will move the playhead sequentially backward and forward. (Just like the backward and forward buttons do.)
• The user simply has to click anywhere on the video to switch to the 'fine' navigation mode.
  • Left/Right arrow keys will now move the play position within the video by one second backward and forward.
• The user simply has to click anywhere not on the video to switch back to the 'broad' navigation mode.

Let me know what you think, please.

--- a/web/js/components/preact/timeline/TimelineControls.jsx
+++ b/web/js/components/preact/timeline/TimelineControls.jsx
@@ -22,7 +22,11 @@ import { nowMilliseconds } from '../../../utils/date-utils.js';
  * TimelineControls component
  * @returns {JSX.Element} TimelineControls component
  */
-export function TimelineControls() {
+export function TimelineControls({
+  keySeqNavScale,
+  keySeqNavValue,
+  setKeySeqNavValue,
+}) {
   const [isPlaying, setIsPlaying] = useState(false);
   const [canZoomIn, setCanZoomIn] = useState(true);
   const [canZoomOut, setCanZoomOut] = useState(true);
@@ -464,6 +468,29 @@ export function TimelineControls() {
   const canJumpBackward = activeSegmentIndex > 0;
   const canJumpForward = activeSegmentIndex !== -1 && activeSegmentIndex < segmentCount - 1;
 
+  useEffect(() => {
+    if (keySeqNavScale === 'broad') {
+      if (keySeqNavValue === -1) {
+        console.log('broad - received backward motion');
+        if (canJumpBackward) {
+          console.log('broad - performing backward jump');
+          jumpToAdjacentSegment(-1);
+        }
+        console.log('broad - resetting');
+        setKeySeqNavValue(0);
+      }
+      if (keySeqNavValue === 1) {
+        console.log('broad - received forward motion');
+        if (canJumpForward) {
+          console.log('broad - performing forward jump');
+          jumpToAdjacentSegment(1);
+        }
+        console.log('broad - resetting');
+        setKeySeqNavValue(0);
+      }
+    }
+  }, [canJumpBackward, canJumpForward, jumpToAdjacentSegment, keySeqNavScale, keySeqNavValue, setKeySeqNavValue]);
+
   return (
     <div className="timeline-controls flex justify-between items-center mb-1">
       <div className="flex items-center gap-1.5">

--- a/web/js/components/preact/timeline/TimelinePage.jsx
+++ b/web/js/components/preact/timeline/TimelinePage.jsx
@@ -217,6 +217,8 @@ export function TimelinePage() {
   const [idsSegmentInfo, setIdsSegmentInfo] = useState(null);  // metadata from IDs endpoint
   const [idsTimelineSegments, setIdsTimelineSegments] = useState([]);
   const [isDownloadModalOpen, setIsDownloadModalOpen] = useState(false);
+  const [keySeqNavScale, setKeySeqNavScale] = useState('broad');
+  const [keySeqNavValue, setKeySeqNavValue] = useState(0);
 
   // Refs
   const timelineContainerRef = useRef(null);
@@ -227,6 +229,31 @@ export function TimelinePage() {
   const selectedDateRef = useRef(urlParams.date);
   const videoElementRef = useRef(null);
 
+  useEffect(() => {
+    document.body.addEventListener('click', (e) => {
+      // broad to move +/- 30 seconds via timeline
+      // fine to move +/- 1 seconds via video
+      const value = e.target.tagName === 'VIDEO' ? 'fine' : 'broad';
+      console.log('main - switching scale to ' + value);
+      setKeySeqNavScale(value);
+    });
+    document.body.addEventListener('keydown', (e) => {
+      // -1 for backwards
+      // 0 for no change
+      // 1 for forwards
+      switch (e.keyCode) {
+        case 37: // Left arrow key
+          console.log('main - moving backward, for ' + keySeqNavScale);
+          setKeySeqNavValue(-1);
+          break;
+        case 39: // Right arrow key
+          console.log('main - moving forward, for ' + keySeqNavScale);
+          setKeySeqNavValue(1);
+          break;
+      }
+    });
+  }, [setKeySeqNavScale, setKeySeqNavValue]);
+
   useEffect(() => {
     selectedDateRef.current = selectedDate;
   }, [selectedDate]);
@@ -802,10 +829,19 @@ export function TimelinePage() {
     return (
       <>
         {/* Video player */}
-        <TimelinePlayer videoElementRef={videoElementRef} />
+        <TimelinePlayer
+          videoElementRef={videoElementRef}
+          keySeqNavScale={keySeqNavScale}
+          keySeqNavValue={keySeqNavValue}
+          setKeySeqNavValue={setKeySeqNavValue}
+        />
 
         {/* Playback controls (includes time display) */}
-        <TimelineControls />
+        <TimelineControls
+          keySeqNavScale={keySeqNavScale}
+          keySeqNavValue={keySeqNavValue}
+          setKeySeqNavValue={setKeySeqNavValue}
+        />
 
         {/* Timeline */}
         <div

--- a/web/js/components/preact/timeline/TimelinePlayer.jsx
+++ b/web/js/components/preact/timeline/TimelinePlayer.jsx
@@ -21,7 +21,12 @@ const DETECTION_SCALE_BASE = 400; // Baseline display dimension (px) for detecti
  * TimelinePlayer component
  * @returns {JSX.Element} TimelinePlayer component
  */
-export function TimelinePlayer({ videoElementRef = null }) {
+export function TimelinePlayer({
+  videoElementRef = null,
+  keySeqNavScale,
+  keySeqNavValue,
+  setKeySeqNavValue,
+}) {
   // Local state
   const [currentSegmentIndex, setCurrentSegmentIndex] = useState(-1);
   const [segments, setSegments] = useState([]);
@@ -924,6 +929,29 @@ export function TimelinePlayer({ videoElementRef = null }) {
     }, 'image/jpeg', 0.95);
   }, [segmentRecordingData]);
 
+  useEffect(() => {
+    if (keySeqNavScale === 'fine') {
+      if (keySeqNavValue === -1) {
+        console.log('fine - received backward motion');
+        if (setVideoRefs.current) {
+          console.log('fine - performing backward jump');
+          setVideoRefs.current.currentTime--;
+        }
+        console.log('fine - resetting');
+        setKeySeqNavValue(0);
+      }
+      if (keySeqNavValue === 1) {
+        console.log('fine - processing forward motion');
+        if (setVideoRefs.current) {
+          console.log('fine - performing forward jump');
+          setVideoRefs.current.currentTime++;
+        }
+        console.log('fine - resetting');
+        setKeySeqNavValue(0);
+      }
+    }
+  }, [keySeqNavScale, keySeqNavValue, setKeySeqNavValue]);
+
   return (
     <>
       <div className="timeline-player-container mb-1" id="video-player">

[matteius]

Commit pushed to main for this.

[CDx4f3kCAf3Y]

This is working great in 3bf4298.

I noticed two things that could be improved.

• For the 'fine' and 'broad' modes
  • If you tap the arrow keys, the counter updates and the playhead moves.
  • If you hold down the arrow keys, only the counter updates. The playhead doesn't move until the keys are released.
    • I think the playhead should move as well here.
    • It can be initially confusing (in the 'broad' mode, as there is no video displayed - which is expected), as most users would normally look at the playhead position before they look at the counter, to see where they are.
• For the 'fine mode only
  • If you tap or hold the arrow keys forward through the end of the recording, it will auto advance to the next recording. That is super cool!
  • However, if you do the same going backwards, you are stuck at the beginning of the current recording. It never auto rewinds to be previous recording.
    • Can that functionality be added? It would avoid having to stop and navigate to the previous recording.

[CDx4f3kCAf3Y]

Tested in b5ed773.

This is working amazing! All of it. One more change and I think this is golden.

When in 'fine' mode, could the arrow keys advance/rewind the video by the selected Speed, instead of by a hard coded 1 second?

[matteius]

change pushed.

[CDx4f3kCAf3Y]

Tested in 3345132.

The change works well. Can you make it so clicking the speed buttons does not take the user out of 'fine' mode. That would avoid having to re-click the video to continue at the new speed.

In my excitement, I prematurely said this was done. I found some intermittent instabilities in later testing yesterday that I am trying to figure out how to reliably reproduce:

• Sometimes when advancing forward past the end of a recording, it loads the next recording at it's end.
  • Instead of the expected 1 second movement, it does an unexpected 30 second movement.
• Sometimes when advancing forward, it will spontaneously switch from 'fine' movement to 'broad' movement.
• Clear control over when a video will play vs pause is not certain/established.
  • Sometimes the timeline will enter a fast loop cycle of 'play, pause, repeat'. Breaking out of this can be difficult.
  • Videos seem to start playing when clicking the user clicks on the timeline. I think that should only seek. Needs more testing.
  • Green play button doesn't seem to always work.
    • An improvement for this - it should show green when not playing and red when playing.
  • I think similar to what was done for the arrow keys, the space bar key should also control play/pause regardless if the video is clicked or not (currently, seems to intermittently do that but only when the video control is focused, see above issues)

[matteius]

Changeset pushed.

[CDx4f3kCAf3Y]

Tested in c1a912f.

When the timeline has focus and a key press occurs, a black box appears around it. That should not happen.

When the timeline is clicked and the focus remains on the timeline, a single tap of an arrow key will jump in the expected direction by what appears to be 33 minutes. As long as the video is allowed to complete its loading cycle, this can be repeated infinitely in either direction. If the video is not allowed to complete its loading cycle, the behavior seems random and uncontrollable. Having the timeline focused should not affect keyboard navigation.

The data-keyboard-nav-preserve attribute is a great and elegant way to avoid changing keyboard-nav mode. This needs fine-tuning. This should be set on all controls on the page (excluding the video player). Changing the keyboard-nav mode should be kept as simple as possible for the user - click the video control to switch to 'fine' mode, click the page background to switch to 'broad' mode - clicking anywhere else should not change the mode. Besides the timeline control (described above), the only other control I can see that doesn't initially play well with this concept is the Streams drop down. Perhaps it needs to have blur() called after the change? (In my testing, I was in 'fine' mode and having success changing speed, full screen, snapshot - all working as expected until I went to protect the recording, then I found myself kicked back to 'broad' mode. I also found that clicking the page background between the full screen / snapshot button group on the left and the speed buttons on the right did not switch me to 'broad' mode.)

When the video control is focused, tapping the space bar results in a double play/pause toggle, cancelling each out.

There is still instability in 'fine' mode when sequentially viewing recordings in a forward direction.

• This does not happen in a backwards direction (regardless of gaps in the timeline). To see what I'm talking about, select a recording that follows a gap in the timeline. Hold the left arrow button down and observe how it is stable while rewinding across recordings and across gaps in the timeline. Then hold the right arrow button down and observe the same is not true.
• Two issues with the forward direction that can be reliably reproduced:
  • Loading the next recording at its end when there is no immediate gap in the timeline
    • Perform a fresh load of timeline page
    • Click video control
    • Advance to the end of the current recording so it reads '0:30 / 0:30' (or whatever length the recording is)
    • Tap right arrow key once more
    • Observe the next recording is loaded at its end
  • Auto switching to play when loading the next recording and there is an immediate gap in the timeline
    • Perform a fresh load of timeline page
    • Pan/zoom the timeline until a gap in the recordings is found and zoomed in as far as it will allow
    • Click the timeline just a bit left of the gap
    • Click video control
    • Advance to the end of the current recording so it reads '0:30 / 0:30' (or whatever length the recording is)
    • Tap right arrow key once more
    • Observe the video control enters play mode and the green dot changes to red

The play/pause repeating loop issue still happens (but maybe not as often?). I have not been able to reliably reproduce it but I've seen it occasionally happen while putting together the reproduction steps for the previous issue above (Auto switching to play when loading the next recording and there is an immediate gap in the timeline). I cannot rule out the length of the recording (i.e. not being a full segment length) or the length of the gap between recordings as contributing factors.

Otherwise, I think progress is great!

[matteius]

Thanks for your feedback and requirements -- I keep fighting with these AI suggestions breaking frontend code and they never stop, like they usually make sense which is why I try merigng them or feeding them into my agent to modify the code, and yet the frontend nitty ones tend to break things, so I think I'll stop unless its totally obvious it will improve things.

[CDx4f3kCAf3Y]

I think it is coming along nice. The core features are in place and now everything is being shaken / ironed out / polished. There is a lot of complex code behind all of this, so I'm not surprised the remaining issues tend to be more low level / more difficult to figure out. I think this is going to be a really great tool for reviewing recordings.

[matteius]

Changes pushed

[CDx4f3kCAf3Y]

Tested with commit c39a590.

The ability to switch keyboard navigation modes should be disabled when the Delete and Recording Tag modals are shown.

Otherwise, this is much improved. I will keep testing.

[CDx4f3kCAf3Y]

Here is a wish list item for the Timeline.

Right now, the Refresh button has to be clicked to load in any new recordings. Can that be replaced with something that transparently polls for and adds to the Timeline any new recordings it finds? That would be real useful when reviewing the very-near-recent recordings in full screen mode. It eliminates having to periodically exit out and manually click that button.

If the Timeline was, instead, built from selected recordings in the Table - this transparent polling feature would not be started.

[matteius]

Changes pushed.

[CDx4f3kCAf3Y]

Working great in 9f90854.

I will keep testing.

[CDx4f3kCAf3Y]

Testing in commit 767ed94.

The issue with the seemingly un-endable fast play/pause loop is back. It occurs intermittently. It occurs more often in Firefox than in Chrome.

Usually to trigger this all I have to do is load the timeline, hit spacebar to immediately start playing, then hit space bar again to stop (and enter the loop).

Sometimes, if that doesn't work, I can get it to happen after seeking around a bit with the arrow keys and/or clicking the timeline.

[CDx4f3kCAf3Y]

Tested in commit b624c98.

Have not seen the fast play/pause loop again, but have not tested as much due to issues with the timeline. I will keep monitoring this.

New issue...

• When the timeline page loads tapping the space bar to play or pause works consistently in Chrome/Firefox.
• After clicking the timeline to select a different recording...
  • In Chrome, tapping the space bar seems to work only intermittent. Green icon turns to Red but video does not play.
  • In Firefox, , tapping the space bar seems to not work at all. Green icon turns to Red but video does not play.