From 1dfe92bc9725d12998b3f5445cfe0833e7674d7f Mon Sep 17 00:00:00 2001 From: Alvin Ji Date: Thu, 3 Sep 2026 16:00:07 -0700 Subject: [PATCH 1/5] Add Text Detection API explainer --- text_detection_explainer.md | 229 ++++++++++++++++++++++++++++++++++++ 1 file changed, 229 insertions(+) create mode 100644 text_detection_explainer.md diff --git a/text_detection_explainer.md b/text_detection_explainer.md new file mode 100644 index 0000000..c48e4d4 --- /dev/null +++ b/text_detection_explainer.md @@ -0,0 +1,229 @@ +# Explainer: Text Detection API + +## Introduction + +The **Text Detection API** enables web applications to detect and recognize text within images, video frames, canvas elements, and other image sources. As part of the broader [WICG Shape Detection API](https://wicg.github.io/shape-detection-api/text.html) initiative, this API exposes a standardized, high-performance text detection interface directly through the Web Platform. + +Extracting textual information from visual media (Optical Character Recognition, or OCR) is a critical requirement across many modern web experiences—such as document scanning, receipt processing, live camera text extraction, image translation, and assistive reading tools. + +--- + +## Problem Statement & Alternatives Today + +Today, web developers needing to detect text in images must choose between two approaches, each carrying substantial tradeoffs: + +### 1. Bundling Client-Side Third-Party Libraries (e.g., WebAssembly / JavaScript OCR) +- **Bandwidth and Payload Overhead:** Full OCR engines require packaging neural network models, dictionaries, and WebAssembly runtimes. Shipping these bundles requires downloading tens of megabytes over the network, dramatically inflating initial page load times and consuming costly mobile bandwidth. +- **CPU, Memory, and Battery Drain:** Executing unoptimized OCR models inside JavaScript or WebAssembly worker threads heavily taxes CPU cores and memory, leading to frame drops, thermal throttling, and battery drain on mobile and laptop devices. +- **Limited Access to Hardware Acceleration:** Web content cannot directly leverage platform-level vision pipelines or hardware accelerators. + +### 2. Using Remote Cloud Vision Services +- **Network Latency:** Uploading high-resolution images or camera frames to a remote cloud API introduces round-trip network delays, making interactive or live viewfinder experiences sluggish. +- **Infrastructure and Financial Cost:** Running or subscribing to cloud vision endpoints incurs ongoing hosting and API costs that increase linearly with application usage. +- **Offline Limitations:** Cloud-based recognition fails completely in low-connectivity, intermittent, or offline environments. +- **Privacy and Data Residency:** Sending sensitive user images—such as receipts, identity documents, bank statements, or private photos—over the network introduces privacy concerns and adds compliance overhead regarding data residency and user consent. + +--- + +## Benefits of a Browser-Provided API + +A native Web Platform Text Detection API addresses these challenges by offering: + +- **Zero Payload Overhead:** The capability is provided by the browser environment, eliminating the need for web applications to bundle and distribute large model weights or runtimes. +- **Optimized Performance:** Browsers can integrate directly with platform-level acceleration, executing text recognition with high efficiency and lower power consumption than user-space scripts. +- **Privacy by Default:** Image data remains within the browser's execution boundary, removing the requirement to transmit user documents or camera feeds across the network to third-party endpoints. + +--- + +## API Design + +The API is exposed in `Window` and `DedicatedWorker` contexts within secure contexts (`https://`). + +### Web IDL + +```webidl +[ + Exposed=(Window,DedicatedWorker), + SecureContext +] interface TextDetector { + // Asynchronously initializes the detector and confirms that any underlying + // platform resources, models, or services are ready before resolving. + static Promise create(); + + // Detects text in an image source. + Promise> detect(ImageBitmapSource image); +}; + +dictionary DetectedText { + required DOMString rawValue; + required DOMRectReadOnly boundingBox; + required sequence cornerPoints; +}; + +dictionary Point2D { + required unrestricted double x; + required unrestricted double y; +}; +``` + +### Asynchronous Initialization (`TextDetector.create()`) + +`TextDetector.create()` initializes the detector and verifies that underlying platform resources or models are available and ready before resolving: + +- **Readiness Verification:** Returns a `Promise` that resolves once the detector engine is ready. If the host environment lacks text detection capabilities or if initialization fails, the promise rejects with a `NotSupportedError` DOMException. +- **Predictable Error Handling:** Web applications can verify support and readiness up front (e.g., before requesting camera permissions or accepting file uploads) and present appropriate fallback user interfaces. + +--- + +## Example Usage + +### 1. Basic Text Detection + +```javascript +// Check for feature availability +if ('TextDetector' in globalThis) { + try { + // Asynchronously create and verify detector readiness + const detector = await TextDetector.create(); + + const imageElement = document.getElementById('scanned-doc'); + const detectedTexts = await detector.detect(imageElement); + + for (const text of detectedTexts) { + console.log(`Detected: "${text.rawValue}"`); + console.log(`Bounding Box: [x: ${text.boundingBox.x}, y: ${text.boundingBox.y}, ` + + `w: ${text.boundingBox.width}, h: ${text.boundingBox.height}]`); + } + } catch (err) { + console.warn('Text detection failed to initialize or detect:', err); + } +} else { + console.log('Text Detection API is not supported in this browser.'); +} +``` + +### 2. Live Camera Viewfinder with Oriented Overlays + +```javascript +const video = document.getElementById('camera-preview'); +const canvas = document.getElementById('overlay-canvas'); +const ctx = canvas.getContext('2d'); + +// Initialize the camera stream +video.srcObject = await navigator.mediaDevices.getUserMedia({ video: true }); +await video.play(); + +const detector = await TextDetector.create(); + +async function processFrame() { + if (video.readyState >= HTMLMediaElement.HAVE_CURRENT_DATA) { + const results = await detector.detect(video); + + ctx.clearRect(0, 0, canvas.width, canvas.height); + + for (const item of results) { + // Draw oriented polygon around text using cornerPoints + const [tl, tr, br, bl] = item.cornerPoints; + ctx.beginPath(); + ctx.moveTo(tl.x, tl.y); + ctx.lineTo(tr.x, tr.y); + ctx.lineTo(br.x, br.y); + ctx.lineTo(bl.x, bl.y); + ctx.closePath(); + + ctx.lineWidth = 2; + ctx.strokeStyle = '#00E676'; + ctx.stroke(); + + // Display detected text + ctx.font = '14px sans-serif'; + ctx.fillStyle = '#00E676'; + ctx.fillText(item.rawValue, tl.x, tl.y - 4); + } + } + requestAnimationFrame(processFrame); +} + +requestAnimationFrame(processFrame); +``` + +### 3. Offloading Processing to a Web Worker + +```javascript +// worker.js +let detector = null; + +self.onmessage = async (event) => { + const { imageBitmap } = event.data; + + if (!detector) { + detector = await TextDetector.create(); + } + + const results = await detector.detect(imageBitmap); + imageBitmap.close(); // Clean up transferable resource + + self.postMessage({ results }); +}; +``` + +--- + +## Example Use Cases + +1. **Document & Receipt Scanning:** + Expense trackers and financial applications can extract vendor names, totals, and line items from camera images without uploading unencrypted receipts to external cloud servers. +2. **Real-World Text Interaction:** + Translators, dictionary lookups, and travel tools can translate signs, menus, and printed text in real time directly from camera feeds. +3. **Form Autofill & Verification:** + Capturing tracking numbers, serial numbers, IBANs, or physical addresses from physical cards or packaging to automatically populate web forms. +4. **Accessibility & Assistive Reading:** + Making text embedded inside images, screenshots, charts, and canvas drawings readable and searchable for screen readers and assistive technologies. +5. **Interactive Video & Canvas Annotations:** + Locating subtitles or graphical text inside video streams to enable in-video search and selectable text highlights. + +--- + +## Future Work & Potential Extensions + +The initial specification prioritizes a minimal, robust API surface (`create()` and `detect()`) that can be implemented cleanly across diverse operating systems and browser engines. Future revisions may explore several natural extensions: + +### 1. Language Negotiation & Availability Checking +While modern vision engines often recognize multilingual text automatically, applications operating in specialized or resource-sensitive environments may benefit from querying language support in advance or hinting preferred languages: + +```webidl +enum Availability { + "unavailable", + "downloadable", + "available" +}; + +dictionary TextDetectorOptions { + required sequence languages; // BCP-47 language tags +}; + +dictionary TextDetectorCreateOptions { + sequence languages; + AbortSignal signal; +}; + +partial interface TextDetector { + static Promise availability(TextDetectorOptions options); + static Promise create(optional TextDetectorCreateOptions options = {}); +}; +``` + +This would allow web applications to query whether specific language packs (e.g., `["ja", "ko"]`) are readily available or require on-demand downloads before initiating recognition. + +### 2. Structural Hierarchy (Blocks, Lines, Words) +The initial API returns recognized text segments at the line level. Future extensions could optionally expose hierarchical segmentation—such as identifying paragraphs, lines, and individual word bounding boxes—to assist advanced document editors and in-place translation overlays. + +--- + +## Privacy & Security Considerations + +- **Secure Contexts Only:** The `TextDetector` interface is restricted to Secure Contexts (`HTTPS`), preventing man-in-the-middle tampering and eavesdropping. +- **Cross-Origin Image Protection (CORS):** To prevent unauthorized reading of cross-origin visual data, `detect()` enforces the same-origin policy on all `ImageBitmapSource` inputs. Passing a cross-origin image or video that has not been granted CORS access rejects the promise with a `SecurityError` DOMException. +- **Data Confidentiality:** Unlike cloud-based OCR services, the API allows text recognition to occur within the browser without transmitting user images or recognition results across network boundaries. + + From 4a27d5e8a43fc1a4d1bae7489c46abf8a1a87d2c Mon Sep 17 00:00:00 2001 From: Alvin Ji Date: Fri, 4 Sep 2026 09:59:09 -0700 Subject: [PATCH 2/5] Address review comments: add downloading state and use inclusive terminology --- text_detection_explainer.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/text_detection_explainer.md b/text_detection_explainer.md index c48e4d4..5052f38 100644 --- a/text_detection_explainer.md +++ b/text_detection_explainer.md @@ -195,6 +195,7 @@ While modern vision engines often recognize multilingual text automatically, app enum Availability { "unavailable", "downloadable", + "downloading", "available" }; @@ -222,7 +223,7 @@ The initial API returns recognized text segments at the line level. Future exten ## Privacy & Security Considerations -- **Secure Contexts Only:** The `TextDetector` interface is restricted to Secure Contexts (`HTTPS`), preventing man-in-the-middle tampering and eavesdropping. +- **Secure Contexts Only:** The `TextDetector` interface is restricted to Secure Contexts (`HTTPS`), preventing person-in-the-middle tampering and eavesdropping. - **Cross-Origin Image Protection (CORS):** To prevent unauthorized reading of cross-origin visual data, `detect()` enforces the same-origin policy on all `ImageBitmapSource` inputs. Passing a cross-origin image or video that has not been granted CORS access rejects the promise with a `SecurityError` DOMException. - **Data Confidentiality:** Unlike cloud-based OCR services, the API allows text recognition to occur within the browser without transmitting user images or recognition results across network boundaries. From c3cc05236cefa66845119bf87aaddf0891f4909b Mon Sep 17 00:00:00 2001 From: Alvin Ji Date: Tue, 8 Sep 2026 11:30:08 -0700 Subject: [PATCH 3/5] Update section title to Live Camera Stream with Text Overlays --- text_detection_explainer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text_detection_explainer.md b/text_detection_explainer.md index 5052f38..edec6fa 100644 --- a/text_detection_explainer.md +++ b/text_detection_explainer.md @@ -102,7 +102,7 @@ if ('TextDetector' in globalThis) { } ``` -### 2. Live Camera Viewfinder with Oriented Overlays +### 2. Live Camera Stream with Text Overlays ```javascript const video = document.getElementById('camera-preview'); From 0333f2a2d53aa125f3338bbc7e264702bdcadd47 Mon Sep 17 00:00:00 2001 From: Alvin Ji Date: Tue, 8 Sep 2026 11:35:15 -0700 Subject: [PATCH 4/5] Remove offline limitations and viewfinder phrasing from explainer --- text_detection_explainer.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/text_detection_explainer.md b/text_detection_explainer.md index edec6fa..250b039 100644 --- a/text_detection_explainer.md +++ b/text_detection_explainer.md @@ -18,9 +18,8 @@ Today, web developers needing to detect text in images must choose between two a - **Limited Access to Hardware Acceleration:** Web content cannot directly leverage platform-level vision pipelines or hardware accelerators. ### 2. Using Remote Cloud Vision Services -- **Network Latency:** Uploading high-resolution images or camera frames to a remote cloud API introduces round-trip network delays, making interactive or live viewfinder experiences sluggish. +- **Network Latency:** Uploading high-resolution images or camera frames to a remote cloud API introduces round-trip network delays, making interactive or live camera experiences sluggish. - **Infrastructure and Financial Cost:** Running or subscribing to cloud vision endpoints incurs ongoing hosting and API costs that increase linearly with application usage. -- **Offline Limitations:** Cloud-based recognition fails completely in low-connectivity, intermittent, or offline environments. - **Privacy and Data Residency:** Sending sensitive user images—such as receipts, identity documents, bank statements, or private photos—over the network introduces privacy concerns and adds compliance overhead regarding data residency and user consent. --- From 1d49cb7d272ffc1aff277db9273d964308af7718 Mon Sep 17 00:00:00 2001 From: Alvin Ji Date: Wed, 9 Sep 2026 12:13:48 -0700 Subject: [PATCH 5/5] Address review feedback: simplify camera example and expand future work --- text_detection_explainer.md | 59 +++++++++++++++++++------------------ 1 file changed, 31 insertions(+), 28 deletions(-) diff --git a/text_detection_explainer.md b/text_detection_explainer.md index 250b039..886697a 100644 --- a/text_detection_explainer.md +++ b/text_detection_explainer.md @@ -115,35 +115,31 @@ await video.play(); const detector = await TextDetector.create(); async function processFrame() { - if (video.readyState >= HTMLMediaElement.HAVE_CURRENT_DATA) { - const results = await detector.detect(video); - - ctx.clearRect(0, 0, canvas.width, canvas.height); - - for (const item of results) { - // Draw oriented polygon around text using cornerPoints - const [tl, tr, br, bl] = item.cornerPoints; - ctx.beginPath(); - ctx.moveTo(tl.x, tl.y); - ctx.lineTo(tr.x, tr.y); - ctx.lineTo(br.x, br.y); - ctx.lineTo(bl.x, bl.y); - ctx.closePath(); - - ctx.lineWidth = 2; - ctx.strokeStyle = '#00E676'; - ctx.stroke(); - - // Display detected text - ctx.font = '14px sans-serif'; - ctx.fillStyle = '#00E676'; - ctx.fillText(item.rawValue, tl.x, tl.y - 4); - } + const results = await detector.detect(video); + + ctx.clearRect(0, 0, canvas.width, canvas.height); + + for (const item of results) { + // Draw bounding box around detected text + // (For tilted/skewed text, item.cornerPoints can be traced instead) + const { x, y, width, height } = item.boundingBox; + ctx.beginPath(); + ctx.rect(x, y, width, height); + ctx.lineWidth = 2; + ctx.strokeStyle = '#00E676'; + ctx.stroke(); + + // Display detected text (simplified demo overlay; not production-ready + // as fixed-size text may render outside the bounding box or canvas) + ctx.font = '14px sans-serif'; + ctx.fillStyle = '#00E676'; + ctx.fillText(item.rawValue, x, y - 4); } - requestAnimationFrame(processFrame); + + video.requestVideoFrameCallback(processFrame); } -requestAnimationFrame(processFrame); +video.requestVideoFrameCallback(processFrame); ``` ### 3. Offloading Processing to a Web Worker @@ -215,8 +211,15 @@ partial interface TextDetector { This would allow web applications to query whether specific language packs (e.g., `["ja", "ko"]`) are readily available or require on-demand downloads before initiating recognition. -### 2. Structural Hierarchy (Blocks, Lines, Words) -The initial API returns recognized text segments at the line level. Future extensions could optionally expose hierarchical segmentation—such as identifying paragraphs, lines, and individual word bounding boxes—to assist advanced document editors and in-place translation overlays. +### 2. Structural Hierarchy & Rich Text Markup +The initial API returns recognized text segments at the line level as plain text. Future extensions could optionally expose hierarchical segmentation—such as identifying paragraphs, lines, and individual word bounding boxes—as well as preserving semantic or stylistic markup (e.g., ``, ``) to assist advanced document editors and in-place translation overlays as more OCR engines support font styling. + +### 3. Task-Specific Detection Modes +Different use cases have varying requirements for recognition versus localization: +- **Pure OCR / Text Extraction:** Applications indexing document text may only need the recognized strings (`rawValue`) without computing geometric bounding boxes. +- **Text Localization / Redaction:** Privacy-preserving workflows (such as blurring sensitive text or license plates in video feeds) may only need geometric coordinates (`boundingBox` / `cornerPoints`) without running full character recognition. + +Future options could allow developers to selectively enable only the capabilities they need as more platform backends expose granular execution modes. ---