[quantization] PyTorch-based PTQ for MediaPipe palm detection and hand landmark models

[mhs4670go]

Background

I'm trying to quantize the following MediaPipe Hand Landmarker models to UINT8:

• palm_detection.tflite
• hand_landmarks_detector.tflite

The palm detection model detects hand regions and key points from an input image, while the hand landmark model predicts detailed hand landmarks from the cropped hand ROI.

Motivation

I have already evaluated several PTQ approaches directly on the Circle models, including min/max and percentile-based calibration, bias correction, and distillation. However, none of them produced a meaningful accuracy improvement.

I would therefore like to explore a PyTorch-based workflow. This has several potential benefits:

• Easier integration with TICO’s existing quantization and debugging infrastructure
• More flexible access to intermediate tensors and layer-wise quantization errors
• Support for optimization-based PTQ methods that are difficult to apply directly to Circle models
• An opportunity to extend TICO’s current PyTorch quantization tools beyond Transformer models and improve CNN support
• Better extensibility for future QAT, mixed-precision, and model-specific optimization work

Proposed work

1. Convert both TFLite models to equivalent PyTorch models and verify FP32 output parity.
2. Add or improve TICO support for CNN operators, especially regular, grouped, and depthwise convolutions.
3. Add layer-wise and block-wise quantization debugging, including activation statistics, reconstruction errors, and sensitivity analysis.
4. Evaluate reconstruction-based PTQ methods, such as adaptive weight rounding and block-wise weight/activation optimization.
5. Export the optimized models back to Circle and validate both numerical correctness and end-to-end hand detection/landmark accuracy.

The initial goal is to determine whether a PyTorch-based optimization pipeline can recover more UINT8 accuracy than the current Circle-only PTQ approaches, while also building reusable CNN quantization infrastructure in TICO.