Repository navigation
media: s5p-mfc: H.264, HEVC and VP8 encode on the Pixel 7 (gs201 MFC v15) - #256
Conversation
fac4b7a to
b07f960
Compare
|
Thank you, Ángel. The encoder fixes are careful work: each one names the client behaviour it fixes and the measurement that showed it, and the v15 register check against Google's This one follows #250:
Two small things while you're there:
|
b07f960 to
567e192
Compare
|
Thank you, Derek. Both small things are done, and the branch already sits on the reworked #250 (
|
567e192 to
90153c9
Compare
|
Thank you, Ángel. Both small points landed:
Our gate passes on the stack merged onto today's I'll merge #250 first. After that, please rebase this onto |
V4L2_CID_MPEG_VIDEO_HEVC_MAX_QP defaults to 0, and s5p_mfc_set_enc_params_hevc() writes the minimum and maximum QP as the QP bound. With the defaults every HEVC frame is coded at QP 0, whatever bitrate the client asks for, and the frame QP controls are clamped to [0, 0] until the client raises the maximum. Default it to 51, the top of the range, as the H.264 maximum QP. Assisted-by: LLM Signed-off-by: Ángel René <17342133+angelrenefm@users.noreply.github.com>
The other encoders program E_RC_FRAME_RATE from the frame rate set with VIDIOC_S_PARM. HEVC only uses the control V4L2_CID_MPEG_VIDEO_HEVC_FRAME_RATE_RESOLUTION, which defaults to 1. So with frame rate control on, the firmware spreads the bitrate over one frame per second: on the Pixel 7 (MFC v15), 4 Mbit/s asked at 30 fps gave about 18 Mbit/s. Use the S_PARM frame rate when it is set, and the HEVC control otherwise. Assisted-by: LLM Signed-off-by: Ángel René <17342133+angelrenefm@users.noreply.github.com>
s5p_mfc_set_enc_params_vp8() writes the frame rate from reg and then uses reg again to set the frame QP in E_RC_CONFIG. When the client has set a frame rate, E_RC_CONFIG gets the frame rate value with the QP bits replaced, which clears the frame rate control enable bit: on the Pixel 7 (MFC v15) the asked bitrate was ignored. Read E_RC_CONFIG again before setting the frame QP. Assisted-by: LLM Signed-off-by: Ángel René <17342133+angelrenefm@users.noreply.github.com>
The encoder's VIDIOC_S_PARM takes V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE but its VIDIOC_G_PARM only takes V4L2_BUF_TYPE_VIDEO_OUTPUT, and neither reports V4L2_CAP_TIMEPERFRAME. GStreamer reads the parameters first and does not set the frame rate when that fails, so the encoder never gets it. Accept both types in G_PARM and report V4L2_CAP_TIMEPERFRAME. Assisted-by: LLM Signed-off-by: Ángel René <17342133+angelrenefm@users.noreply.github.com>
The encoder never sets the timestamp of a CAPTURE buffer, so every encoded frame comes out with timestamp 0. GStreamer's V4L2 encoders match encoded frames to their input by timestamp and attach all of them to frame 0. Give the encoded buffer the timestamp of the source buffer the firmware reports as done with it. Assisted-by: LLM Signed-off-by: Ángel René <17342133+angelrenefm@users.noreply.github.com>
After V4L2_ENC_CMD_STOP the encoder returns an empty CAPTURE buffer without V4L2_BUF_FLAG_LAST, so clients that wait for that flag (as the stateful encoder interface asks) never see the end of the drain. And when no CAPTURE buffer is queued at that moment, nothing is returned at all. Set V4L2_BUF_FLAG_LAST on that buffer and, when none is queued, return the next queued one empty with the flag. Assisted-by: LLM Signed-off-by: Ángel René <17342133+angelrenefm@users.noreply.github.com>
The encoder's VIDIOC_S_FMT returns -EBUSY when either queue streams. That forbids setting the OUTPUT format while CAPTURE streams, which the stateful encoder interface allows. The decoder got the same fix earlier in this series. Assisted-by: LLM Signed-off-by: Ángel René <17342133+angelrenefm@users.noreply.github.com>
…s not A client may leave the CAPTURE sizeimage to the driver (v4l2-ctl does). The encoder then keeps 0 and fails later when it allocates the buffers. Use the variant's CPB size. Assisted-by: LLM Signed-off-by: Ángel René <17342133+angelrenefm@users.noreply.github.com>
The encoder's VIDIOC_TRY_FMT for the OUTPUT queue leaves field and num_planes as the client passed them. GStreamer reports a driver bug for V4L2_FIELD_ANY, and a client cannot learn the plane count of a format before setting it. Assisted-by: LLM Signed-off-by: Ángel René <17342133+angelrenefm@users.noreply.github.com>
MFC v15 has no E_FRAME_WIDTH and E_FRAME_HEIGHT: 0xF770 and 0xF774 are the decoder's payload registers there. The encoder takes the cropped size only, as in Google's exynos-mfc driver (kernel/google-modules/soc/gs, drivers/media/platform/exynos/mfc/, branch android-gs-pantah-6.1-android16). The other encoder registers that s5p-mfc writes match that driver's mfc_regs.h by address, 76 of 76. Assisted-by: LLM Signed-off-by: Ángel René <17342133+angelrenefm@users.noreply.github.com>
Offer V4L2_PIX_FMT_NV12 in one buffer as the encoder input on MFC v15, laid out as the decoder's NV12 output: 64 byte aligned stride, chroma right after bytesperline * height, and room after it for the rows the MFC reads past the chroma and its 256 byte padding. NV12M and NV21M are not offered on v15. GStreamer always prefers the non-contiguous variant and then places the chroma at stride * height, ignoring the padded luma sizeimage, so it writes past the luma buffer. Assisted-by: LLM Signed-off-by: Ángel René <17342133+angelrenefm@users.noreply.github.com>
On MFC v15: - default V4L2_CID_MPEG_VIDEO_HEADER_MODE to JOINED_WITH_1ST_FRAME. GStreamer expects one CAPTURE buffer per frame: with a separate header buffer it drops the next one, which is the keyframe. - set bit 12 of E_VP8_OPTION. Without it the firmware wraps VP8 in IVF file and frame headers, and V4L2_PIX_FMT_VP8 is bare frames. Google's driver sets it too. There is then no header to return at sequence start, so on v15 do not return an empty buffer that would read as the end of the stream. Other versions return the header buffer as before. Assisted-by: LLM Signed-off-by: Ángel René <17342133+angelrenefm@users.noreply.github.com>
On the Pixel 7 the MFC v15 firmware takes MPEG-4 and H.263 encode sessions but they do not work: FFmpeg's h263_v4l2m2m fails, and mpeg4_v4l2m2m writes a stream with no decodable frame. Do not offer them on v15, as for the decoder. Assisted-by: LLM Signed-off-by: Ángel René <17342133+angelrenefm@users.noreply.github.com>
With the previous commits the encoder works on the Pixel 7. Through GStreamer's v4l2h264enc, v4l2h265enc and v4l2vp8enc, 60 frame 1080p clips are complete in 18 runs out of 18, 90 frame 3840x2160 clips at 20 Mbit/s too, and 1 and 4 Mbit/s asked are met. Through FFmpeg's *_v4l2m2m encoders at 4 Mbit/s, 720p, PSNR against the source is about 43 dB for H.264, HEVC and VP8. Assisted-by: LLM Signed-off-by: Ángel René <17342133+angelrenefm@users.noreply.github.com>
90153c9 to
fb58ec3
Compare
|
Rebased with |
|
Thank you, Ángel. The rebase is exactly right: the 14 encoder commits, unchanged, directly on |
Hardware H.264, HEVC and VP8 encode on the Pixel 7 (gs201 MFC v15), on top of the decoder in #250.
This depends on #250. The branch starts at #250's head (
50d6d6c2a), so the first 22 commits here are #250's.Only the last 14 are new. I can rebase it onto
omamobileonce #250 is merged.Commits 1 to 9 fix the s5p-mfc encoder in general. They are not specific to v15, but I could only test them on v15.
Commits 10 to 14 are for v15.
default the HEVC encoder maximum QP to 51: the HEVC max QP control defaulted to 0, and the driver writes min andmax as the QP bound. So every HEVC frame was coded at QP 0 and the asked bitrate had no effect (measured: 70 dB,
21 Mbit/s with 4 Mbit/s asked).
use the S_PARM frame rate for HEVC rate control: HEVC took its frame rate only fromV4L2_CID_MPEG_VIDEO_HEVC_FRAME_RATE_RESOLUTION, which defaults to 1 fps. With 4 Mbit/s asked at 30 fps it gaveabout 18 Mbit/s.
keep VP8 rate control when a frame rate is set: after writing the frame rate, the VP8 setup reused that value asE_RC_CONFIG, which turned rate control off whenever the client set a frame rate.accept the multiplanar type in the encoder G_PARM: S_PARM tookOUTPUT_MPLANEbut G_PARM did not, and neitherreported
V4L2_CAP_TIMEPERFRAME. GStreamer then never sets the frame rate.copy the source timestamp to encoded buffers: every encoded buffer had timestamp 0. GStreamer matches output toinput by timestamp.
flag the last buffer of an encoder drain: the empty buffer afterV4L2_ENC_CMD_STOPhad noV4L2_BUF_FLAG_LAST. If no CAPTURE buffer was queued when the drain ended, nothing was returned at all. GStreamerhung at EOS, in about 1 run out of 3.
check only the queue being set in the encoder S_FMT: same fix as the decoder's in media: s5p-mfc: H.264, HEVC, VP8 and VP9 decode on the Pixel 7 (gs201 MFC v15) #250.size the encoder bitstream buffer when the client does not: a CAPTURE sizeimage of 0 now gets the CPB size.fill field and num_planes in the encoder TRY_FMT.do not write the encoder frame size on MFC v15: v15 has noE_FRAME_WIDTH/E_FRAME_HEIGHT. 0xF770/0xF774 arethe decoder's payload registers there, and Google's driver writes only the cropped size. The other encoder
registers the driver uses match Google's
mfc_regs.h(76 of 76 by address).single-buffer NV12 encoder input on MFC v15: NV12 in one buffer, with the same layout as the decoder's NV12output. NM12 and NM21 are not offered on v15: GStreamer always picks the non-contiguous variant, and it then puts
the chroma at stride * height. That ignores the padded luma sizeimage, so it wrote past the buffer (SIGSEGV, or
SIGILL in ORC's code).
v15 encoder: header with the first frame, bare VP8:separate header buffer, it dropped the keyframe, and the mp4 files had 47 decodable frames out of 60.
E_VP8_OPTION, as Google's driver does. Without it, the firmware wraps VP8 in IVF headers.offer only H.264, HEVC and VP8 encode on MFC v15: h263_v4l2m2m fails, and mpeg4_v4l2m2m writes no decodableframe.
register the encoder on MFC v15.Tested on my Pixel 7 (kernel
7.3.0-rc6-omamobile-push1-g44048a9aa795). The DTB is #250's (256 MiBmfc-pool, MFCclock).
s5p-mfc.kowas built from this branch's head (fac4b7a64, then on #250's commit 21) and loaded with insmod. GStreamer 1.28.9 andFFmpeg are from Arch Linux ARM.
v4l2*enc1920x1080, 60 frames to mp4/webm, all frames in the file*_v4l2m2m1280x720, 60 frames at 4 Mbit/s, PSNR against the sourceThe first number is the same code before I split it into these commits. The second is the module built from this
branch.
* H.264 was also run 8 times at 720p and 8 times at 1080p earlier, all 60/60.
After encoding, the decoder from #250 still decodes (an HEVC clip encoded here, through
v4l2h265dec). dmesg showsno warning.
After #250 got its commit 22, I rebased this onto it (
git range-diff: the 14 commits are unchanged) and ran theencode test at 1080p for the three codecs again (60/60 each), and 4K VP9 decode (bit-exact). Checked with
tools/kernel-pr-check --buildon the 14 commits: all ok. The device-tree step was skipped becausethis series touches no device tree. checkpatch --strict: no warnings.
Not checked:
videotestsrcon the CPU, and I did not measure the MFC alone.videotestsrc, FFmpeg'stestsrc2and raw files.GStreamer's handling of NM12 with a padded luma plane (commit 11) is a GStreamer issue. I have not reported it yet.
I used AI help (Claude) to work on this.