Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,6 @@ backend/campaigns/

# Demo before/after renders (regenerated by demo_runner.py)
backend/demo_out/

# Static web demo build (npm run build:demo)
web-dist/
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@
Pixel runs a gaze model over an ad, finds what is pulling the eye away from the brand, and has a team of AI agents redesign it. Every edit is re-scored by the gaze model, and edits that don't raise the score are thrown out.</p>

<p align="center">
<img src="assets/media/deepgaze.gif" width="400" alt="DeepGaze attention heatmaps and predicted fixation order drawn over four sample ads">
<a href="https://sparkylab.web.app/pixel/"><img src="assets/media/deepgaze.gif" width="400" alt="DeepGaze attention heatmaps and predicted fixation order drawn over four sample ads"></a><br>
<a href="https://sparkylab.web.app/pixel/"><b>Try the demo</b></a>
</p>

<p align="center">Built in one day at Multimodal Hacks (NY Tech Week, June 6 2026) with <a href="https://github.com/rishis123">Rishi Shah</a>.</p>
Expand Down Expand Up @@ -85,6 +86,8 @@ npm run dev # http://localhost:5173

Keys go in git-ignored files: `GEMINI_API_KEY`, `PINECONE_API_KEY` and `PINECONE_INDEX` in `backend/.env`, and `VITE_CLERK_PUBLISHABLE_KEY` in `frontend/.env.local`. Seed the competitor index once with `python backend/pinecone_seed.py`. Without a Gemini key the app still analyzes ads, but edits come back unchanged. DeepGaze runs on the GPU when there is one and on the CPU otherwise.

The [live demo](https://sparkylab.web.app/pixel/) is the same frontend replaying runs recorded on the real backend, so it needs no server and no keys. `python scripts/record_demo.py` records them and `npm run build:demo` in `frontend` writes the static site to `web-dist/`.

To rebuild the figures in this README:

```bash
Expand Down
1 change: 1 addition & 0 deletions frontend/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"scripts": {
"dev": "vite",
"build": "vite build",
"build:demo": "vite build --mode demo",
"preview": "vite preview"
},
"dependencies": {
Expand Down
Binary file added frontend/public/replay/apple/heat.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added frontend/public/replay/apple/image.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
61 changes: 61 additions & 0 deletions frontend/public/replay/apple/predict.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
{
"width": 1400,
"height": 2100,
"attention_score": 0.7581,
"heatmap_png": "replay/apple/heat.png",
"target_box": [
0.32,
0.2,
0.36,
0.62
],
"scanpath": [
{
"x": 0.484,
"y": 0.371,
"order": 1
},
{
"x": 0.523,
"y": 0.634,
"order": 2
},
{
"x": 0.478,
"y": 0.942,
"order": 3
},
{
"x": 0.521,
"y": 0.511,
"order": 4
},
{
"x": 0.158,
"y": 0.489,
"order": 5
}
],
"distractors": [
{
"region": [
0.3809,
0.8443,
0.2,
0.2
],
"share": 0.09,
"desc": "Smartwatch band"
},
{
"region": [
0.0,
0.3912,
0.2,
0.2
],
"share": 0.027,
"desc": "Abstract blurred image"
}
]
}
Binary file added frontend/public/replay/apple/step0-heat.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added frontend/public/replay/apple/step0.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added frontend/public/replay/apple/step1-heat.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added frontend/public/replay/apple/step1.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added frontend/public/replay/apple/step2-heat.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added frontend/public/replay/apple/step2.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added frontend/public/replay/apple/step3-heat.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added frontend/public/replay/apple/step3.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
82 changes: 82 additions & 0 deletions frontend/public/replay/apple/steps.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
[
{
"step": 0,
"directive": "make SEVERAL coordinated changes at once to turn this into a polished campaign: remove the clutter and any objects, hands, props or stray text crowding or blocking the Apple logo and product; reframe it head-on and enlarge it as the clear hero; clean and simplify the background; and add a bold, legible on-brand headline, call-to-action and wordmark on the Apple logo and product",
"variant_png": "replay/apple/step0.jpg",
"current_score": 0.7581,
"new_score": 0.6945,
"delta": -0.0636,
"target_salience_before": 0.6983,
"target_salience_after": 0.5065,
"judge": 0.8,
"judge_reason": "The creative exhibits strong brand-fit for Apple in terms of its aesthetic, messaging, and overall feel. The minimalist composition, aspirational slogan ('Experience More'), and clear call to action ('Learn More') are all characteristic of Apple's advertising. The imagery of the floating iPhone above a hand with an Apple Watch subtly highlights the ecosystem and a sense of effortless interaction or wonder. The photography quality and typography also align well with Apple's premium brand image. The main factor reducing a perfect score is the use of an iPhone X, which is an older model; current Apple ads almost exclusively feature their latest hardware. However, if this ad were created when the iPhone X was current, it would be an excellent example of an Apple campaign.",
"vetoed": false,
"guard": "reject",
"guard_reasons": [
"attention score did not improve"
],
"improved": false,
"n_directives": 8,
"variant_heatmap": "replay/apple/step0-heat.png"
},
{
"step": 1,
"directive": "add a bold, legible, on-brand headline and call-to-action and strengthen the logo/wordmark right at the Apple logo and product as a strong, high-contrast focal point",
"variant_png": "replay/apple/step1.jpg",
"current_score": 0.7581,
"new_score": 0.6853,
"delta": -0.0728,
"target_salience_before": 0.6983,
"target_salience_after": 0.4851,
"judge": 0.6,
"judge_reason": "The ad features core Apple products (iPhone, Apple Watch, logo) and a relevant tagline 'CAPTURE YOUR WORLD' with a clear call to action, aligning with a key iPhone feature (camera). The composition is clean and the lighting is natural. However, it's unlikely to run as a real Apple ad due to several factors: the iPhone model appears to be an older generation (likely an iPhone X), whereas Apple typically features its latest models prominently in new campaigns. The concept of a levitating phone, while common in tech ads, lacks the unique, minimalist, and often more abstract or experiential creativity typically associated with official Apple campaigns. Furthermore, the overall production quality, while decent, doesn't quite reach the hyper-polished, studio-level photography and precise typography standards Apple is known for.",
"vetoed": false,
"guard": "reject",
"guard_reasons": [
"attention score did not improve"
],
"improved": false,
"n_directives": 8,
"variant_heatmap": "replay/apple/step1-heat.png"
},
{
"step": 2,
"directive": "clearly enlarge, brighten and sharpen the Apple logo and product so it becomes the single biggest, boldest focal element while gently dimming and de-cluttering the surroundings \u2014 a polished, real ad",
"variant_png": "replay/apple/step2.jpg",
"current_score": 0.7581,
"new_score": 0.7858,
"delta": 0.0277,
"target_salience_before": 0.6983,
"target_salience_after": 0.8174,
"judge": 0.3,
"judge_reason": "This creative attempts to capture Apple's focus on user experience, using a clean font and the iconic logo. However, several factors make it unlikely to run as a real Apple ad: \n1. **Outdated Product:** The ad features an iPhone X, which is an older model. Apple typically highlights its latest products in primary ad campaigns.\n2. **Quality & Polish:** The 'floating' effect and the glow around the phone appear somewhat amateurish and not seamlessly integrated. Apple ads are known for their extremely high production value, precision, and minimalist perfection, which this creative doesn't quite achieve.\n3. **Generic Message:** While 'EXPERIENCE' is a core tenet of Apple's brand, a single, broad word like this is often too generic for a standalone ad creative. Apple usually pairs its general ethos with specific features, innovations, or emotional benefits ('Privacy. That's iPhone,' 'Shot on iPhone,' 'The new way to interact with your phone').\n4. **Background:** The blurred background, while aiming for a lifestyle feel, lacks the distinctive or aspirational quality typical of Apple's curated environmental shots or their signature clean, often studio-like aesthetic.\n5. **Brand Fit (Aesthetics):** While the Apple logo and text font are on-brand, the overall execution lacks the sophisticated, premium feel Apple meticulously maintains in its advertising.",
"vetoed": true,
"guard": "accept",
"guard_reasons": [
"perceptible, localized/justified, and target salience improved"
],
"improved": false,
"n_directives": 8,
"variant_heatmap": "replay/apple/step2-heat.png"
},
{
"step": 3,
"directive": "remove or clean away any objects, hands, props or background obstructions that crowd or block the Apple logo and product so the product is fully visible, unobstructed, and the clear hero of the shot",
"variant_png": "replay/apple/step3.jpg",
"current_score": 0.7581,
"new_score": 0.7711,
"delta": 0.013,
"target_salience_before": 0.6983,
"target_salience_after": 0.7506,
"judge": 0.9,
"judge_reason": "This creative demonstrates strong brand-fit for Apple. The minimalist aesthetic, featuring a clean product shot against a plain background, is a hallmark of Apple's advertising. The typography is simple and elegant, and the concise tagline 'CAPTURE THE MOMENT' directly highlights a key iPhone feature (the camera) and a core user benefit, aligning perfectly with Apple's benefit-driven messaging. While the phone is an older model (iPhone X/XS era), if evaluated within its historical context, the quality of the image and the clarity of the text are high. It's a straightforward, effective, and visually consistent ad that Apple would very likely run to promote the iPhone's camera capabilities.",
"vetoed": false,
"guard": "accept",
"guard_reasons": [
"perceptible, localized/justified, and target salience improved"
],
"improved": true,
"n_directives": 8,
"variant_heatmap": "replay/apple/step3-heat.png"
}
]
Binary file added frontend/public/replay/apple/thumb.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added frontend/public/replay/coca-cola/heat.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added frontend/public/replay/coca-cola/image.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
61 changes: 61 additions & 0 deletions frontend/public/replay/coca-cola/predict.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
{
"width": 1600,
"height": 2000,
"attention_score": 0.8026,
"heatmap_png": "replay/coca-cola/heat.png",
"target_box": [
0.33,
0.28,
0.33,
0.64
],
"scanpath": [
{
"x": 0.429,
"y": 0.605,
"order": 1
},
{
"x": 0.532,
"y": 0.426,
"order": 2
},
{
"x": 0.496,
"y": 0.767,
"order": 3
},
{
"x": 0.53,
"y": 0.999,
"order": 4
},
{
"x": 0.495,
"y": 0.318,
"order": 5
}
],
"distractors": [
{
"region": [
0.4299,
0.899,
0.2,
0.2
],
"share": 0.073,
"desc": "Wet soda can top"
},
{
"region": [
0.4397,
0.8346,
0.2,
0.2
],
"share": 0.11,
"desc": "Wet soda can body"
}
]
}
Binary file added frontend/public/replay/coca-cola/step0-heat.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added frontend/public/replay/coca-cola/step0.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added frontend/public/replay/coca-cola/step1.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added frontend/public/replay/coca-cola/step2-heat.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added frontend/public/replay/coca-cola/step2.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added frontend/public/replay/coca-cola/step3-heat.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added frontend/public/replay/coca-cola/step3.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
82 changes: 82 additions & 0 deletions frontend/public/replay/coca-cola/steps.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
[
{
"step": 0,
"directive": "make SEVERAL coordinated changes at once to turn this into a polished campaign: remove the clutter and any objects, hands, props or stray text crowding or blocking the Coca-Cola logo and product; reframe it head-on and enlarge it as the clear hero; clean and simplify the background; and add a bold, legible on-brand headline, call-to-action and wordmark on the Coca-Cola logo and product",
"variant_png": "replay/coca-cola/step0.jpg",
"current_score": 0.8026,
"new_score": 0.5641,
"delta": -0.2385,
"target_salience_before": 0.8585,
"target_salience_after": 0.2733,
"judge": 0.98,
"judge_reason": "This Coca-Cola ad creative is exceptionally high quality and demonstrates outstanding brand-fit. The photography of the can is professional, showcasing the product with appealing condensation that conveys extreme coldness and refreshment. The vibrant red background is the iconic Coca-Cola color, instantly recognizable. Both taglines, 'REFRESH YOUR MOMENT' and especially 'TASTE THE FEELING.' (which is Coca-Cola's actual global marketing tagline since 2016), are perfectly aligned with the brand's messaging of refreshment, experience, and emotional connection. The composition is clean, direct, and impactful, making the product the clear hero. This creative would absolutely run as a real ad; it embodies the quintessential Coca-Cola brand aesthetic and message.",
"vetoed": false,
"guard": "reject",
"guard_reasons": [
"attention score did not improve"
],
"improved": false,
"n_directives": 7,
"variant_heatmap": "replay/coca-cola/step0-heat.png"
},
{
"step": 1,
"directive": "add a bold, legible, on-brand headline and call-to-action and strengthen the logo/wordmark right at the Coca-Cola logo and product as a strong, high-contrast focal point",
"variant_png": "replay/coca-cola/step1.jpg",
"current_score": 0.8026,
"new_score": 0.5539,
"delta": -0.2487,
"target_salience_before": 0.8585,
"target_salience_after": 0.2623,
"judge": 0.9,
"judge_reason": "This creative is highly effective and demonstrates strong brand fit for Coca-Cola. The use of the iconic red background, a perfectly photographed cold can with condensation, and the prominent display of the Coca-Cola logo immediately establish brand identity. The headline 'REFRESH YOUR MOMENT' aligns perfectly with Coca-Cola's core messaging around refreshment and experience. The tagline 'TASTE THE FEELING. BUY NOW.' directly incorporates a well-known global Coca-Cola campaign tagline ('Taste the Feeling') and a clear call to action. While the headline font isn't the classic Coca-Cola Spencerian script, it's a stylistic choice that doesn't detract from the brand's essence, and the overall composition is clean, direct, and professional. This ad would very likely run as a real ad in various formats (digital, print, in-store) due to its clarity, strong branding, and direct appeal.",
"vetoed": false,
"guard": "reject",
"guard_reasons": [
"attention score did not improve"
],
"improved": false,
"n_directives": 7,
"variant_heatmap": "replay/coca-cola/step1-heat.png"
},
{
"step": 2,
"directive": "clearly enlarge, brighten and sharpen the Coca-Cola logo and product so it becomes the single biggest, boldest focal element while gently dimming and de-cluttering the surroundings \u2014 a polished, real ad",
"variant_png": "replay/coca-cola/step2.jpg",
"current_score": 0.8026,
"new_score": 0.6652,
"delta": -0.1374,
"target_salience_before": 0.8585,
"target_salience_after": 0.4196,
"judge": 1.0,
"judge_reason": "This Coca-Cola ad creative is exceptionally high quality and a perfect brand fit. The image features a perfectly rendered, condensation-laden Coca-Cola can against an iconic red background, immediately evoking refreshment. The use of the official global campaign tagline, 'TASTE THE FEELING,' further solidifies its authenticity and brand alignment. The minimalist yet powerful composition is characteristic of Coca-Cola's advertising, making it highly effective and absolutely suitable to run as a real ad.",
"vetoed": false,
"guard": "reject",
"guard_reasons": [
"attention score did not improve"
],
"improved": false,
"n_directives": 7,
"variant_heatmap": "replay/coca-cola/step2-heat.png"
},
{
"step": 3,
"directive": "remove or clean away any objects, hands, props or background obstructions that crowd or block the Coca-Cola logo and product so the product is fully visible, unobstructed, and the clear hero of the shot",
"variant_png": "replay/coca-cola/step3.jpg",
"current_score": 0.8026,
"new_score": 0.6691,
"delta": -0.1335,
"target_salience_before": 0.8585,
"target_salience_after": 0.427,
"judge": 1.0,
"judge_reason": "This creative is of excellent quality and perfectly aligns with the Coca-Cola brand. The vibrant red background, the perfectly chilled can with condensation, and the iconic 'Taste the Feeling' tagline are all quintessential Coca-Cola. The photography/rendering is professional and appealing, making the product look refreshing. The minimalist composition focuses entirely on the product and brand message. This would undoubtedly run as a real ad, fitting seamlessly into their established advertising aesthetic.",
"vetoed": false,
"guard": "reject",
"guard_reasons": [
"attention score did not improve"
],
"improved": false,
"n_directives": 7,
"variant_heatmap": "replay/coca-cola/step3-heat.png"
}
]
Binary file added frontend/public/replay/coca-cola/thumb.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
15 changes: 15 additions & 0 deletions frontend/public/replay/manifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"recorded": "2026-10-01",
"engine": "deepgaze-iie",
"device": "cuda",
"samples": [
"apple",
"coca-cola",
"mcdonalds",
"nike",
"pepsi",
"red-bull",
"spotify",
"the-ordinary"
]
}
Binary file added frontend/public/replay/mcdonalds/heat.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added frontend/public/replay/mcdonalds/image.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
40 changes: 40 additions & 0 deletions frontend/public/replay/mcdonalds/predict.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
{
"width": 1600,
"height": 1600,
"attention_score": 0.8383,
"heatmap_png": "replay/mcdonalds/heat.png",
"target_box": [
0.26,
0.31,
0.52,
0.33
],
"scanpath": [
{
"x": 0.354,
"y": 0.557,
"order": 1
},
{
"x": 0.618,
"y": 0.551,
"order": 2
},
{
"x": 0.512,
"y": 0.551,
"order": 3
},
{
"x": 0.724,
"y": 0.547,
"order": 4
},
{
"x": 0.458,
"y": 0.355,
"order": 5
}
],
"distractors": []
}
Binary file added frontend/public/replay/mcdonalds/step0-heat.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added frontend/public/replay/mcdonalds/step0.jpg
Binary file added frontend/public/replay/mcdonalds/step1-heat.png
Binary file added frontend/public/replay/mcdonalds/step1.jpg
Binary file added frontend/public/replay/mcdonalds/step2-heat.png
Binary file added frontend/public/replay/mcdonalds/step2.jpg
Binary file added frontend/public/replay/mcdonalds/step3-heat.png
Binary file added frontend/public/replay/mcdonalds/step3.jpg
Loading
Loading