Pattern Over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation

The problem

Plausible code that ignores the pixels

Screenshot-to-code benchmarks mostly measure page-level fidelity: does the output look like the input? But a model can produce globally plausible HTML/CSS while silently normalizing away a small, intentional local deviation — a card that is deliberately wider, a nav item deliberately larger.

These are exactly the errors that matter in front-end work. The output looks broadly correct, so the mistake survives review, yet the generated code fails to preserve the precise design detail the developer intended.

What the figure shows. One post card has width:__ while every sibling is width:100%. Sonnet‑4.6 observes that the card extends past its container and computes the right ratio (≈118% → 120%) — then discards its own calculation to “match the others.”

A model computes the correct card width of 120 percent, then discards its own answer to match the repeated pattern.

Abstract

Multimodal large language models (MLLMs) are increasingly used to translate webpage screenshots into front-end code, but repeated UI patterns may sway them toward visually incorrect yet pattern-consistent outputs. We test how repeated webpage patterns hurt MLLM accuracy on an objective screenshot-to-code fill-in-the-blank task. We introduce the first benchmark for visual pattern-completion bias, where one localized element in a repeated UI pattern is perturbed and the model must recover the masked width or font-size value from the screenshot and HTML context. Starting from 30 webpages curated from Design2Code, we build 1,440 evaluated screenshots spanning structural card and text-style patterns under standard and noise-overlaid conditions. We evaluate five frontier MLLMs and find that all are strongly biased toward the repeated baseline. Mean bias rate reaches 69.78% on card-width perturbations and 80.22% on text font-size perturbations, while mean accuracy is only 21.17% and 7.89%, respectively. Reasoning analysis shows that greater reasoning effort correlates with lower bias, yet qualitative evidence reveals that models can identify the anomalous element and still override it with the pattern-consistent answer.

The benchmark

Pattern2Code

We start from 484 Design2Code webpages and keep the 30 with clear repeated UI structures. From each page we extract one card pattern and one text pattern, normalize every element in the pattern to 100%, then perturb exactly one element.

real webpages

720

masked instances

1,440

screenshots / model

frontier MLLMs

The task

The model sees a rendered screenshot and the element's HTML with the target value masked:

You are doing a visual code fill-in-the-blank task. Given the webpage
screenshot and HTML snippet below, fill the blank token __ with ONLY the
missing CSS/HTML value. The goal is to predict the masked value to recreate
the {pattern} in the webpage design. Return the final answer in JSON format:
{"answer":"<value>"}. The blank must be a percentage value divisible by 10.

Correct — the true perturbed value Bias-aligned — the repeated baseline 100% Other error — neither

Model	Card patterns — `width`	Text patterns — `font-size`
Code-specialized
Codex-5.3	68.61	26.39	5.00	13.89	70.56	15.56
Opus-4.6	6.39	87.78	5.83	5.56	88.06	6.39
General-purpose
ChatGPT-5.3	8.33	69.44	22.22	10.28	66.94	22.78
Sonnet-4.6	8.06	83.33	8.61	8.89	79.44	11.67
Flash-3.0	14.44	81.94	3.61	0.83	96.11	3.06
Mean	21.17	69.78	9.06	7.89	80.22	11.89

Analysis

Bias tracks visual saliency

Three independent knobs — noise, magnitude, position — all modulate the same underlying variable: how hard the deviation is to see. Each one below pairs the claim with a demonstration built from real benchmark screenshots and the measured result across all five models.

Noise amplifies bias

Overlaying eight opaque rectangles raises mean card bias 69.78% → 73.89% (+4.11). Text bias is already near ceiling, so it moves less (+1.84).

The rectangles never cover the target element — they only make the page noisier to read. The layout is byte-for-byte identical in both renders.

Standard

Noise-overlaid

The same forum thread with opaque rectangles overlaid at random positions.

Show the underlying numbers (bias rate, %)

Model	Card bias rate (%)			Text bias rate (%)
Model	Standard	Noise	Δ	Standard	Noise	Δ
Code-specialized
Codex-5.3	26.39	30.83	+4.44	70.56	75.00	+4.44
Opus-4.6	87.78	90.00	+2.22	88.06	90.28	+2.22
General-purpose
ChatGPT-5.3	69.44	67.78	−1.67	66.94	68.33	+1.39
Sonnet-4.6	83.33	89.44	+6.11	79.44	80.56	+1.12
Flash-3.0	81.94	91.39	+9.44	96.11	96.11	0.00
Mean	69.78	73.89	+4.11	80.22	82.06	+1.84

Subtler perturbations are worse

On cards, mean bias climbs 50.7% → 82.2% as the perturbation shrinks from 80% to 110%. Widening is harder to spot than shrinking.

The same element at all four magnitudes, shown at 1:1 scale beneath the unperturbed neighbour above it. At 80% the gap is obvious; at 110% it is a few pixels.

80%

The target post rendered at 80 percent width, outlined in red, below an unperturbed neighbour.

90% · nearest the baseline

The target post rendered at 90 percent width, outlined in red, below an unperturbed neighbour.

110% · nearest the baseline

The target post rendered at 110 percent width, outlined in red, below an unperturbed neighbour.

120%

The target post rendered at 120 percent width, outlined in red, below an unperturbed neighbour.

Show the underlying numbers (mean bias rate, %)

Family	80%	90%	110%	120%
Cards (`width`)	50.7	69.3	82.2	76.9
Text (`font-size`)	69.8	88.2	87.4	75.6

Boundaries hide deviations

Middle elements are flanked by pattern-consistent neighbors on both sides, making them most salient: lowest bias 61.67% vs. 73.83% at either boundary.

The same page with the same 120% perturbation moved to each position. Only the middle one can be compared against a neighbour above and below.

First

Middle

Last

Show the underlying numbers (accuracy / bias, %)

Model	First		Middle		Last
Model	Acc.	Bias	Acc.	Bias	Acc.	Bias
Code-specialized
Codex-5.3	62.50	33.33	74.17	20.00	69.17	25.83
Opus-4.6	5.83	87.50	9.17	85.83	4.17	90.00
General-purpose
ChatGPT-5.3	7.50	62.50	10.83	64.17	6.67	81.67
Sonnet-4.6	3.33	90.00	13.33	74.17	7.50	85.83
Flash-3.0	3.33	95.83	29.17	64.17	10.83	85.83
Mean	16.50	73.83	27.33	61.67	19.67	73.83

Reasoning effort

More reasoning, much less bias

Only the two OpenAI models report usage_reasoning_tokens, so this analysis covers ChatGPT-5.3 and Codex-5.3. We bin every example by how many reasoning tokens the model spent on it.

Show the underlying numbers

Model	Mean tokens	Examples	Accuracy	Bias
ChatGPT-5.3	0	806	2.2	93.2
ChatGPT-5.3	~65	498	21.3	40.8
ChatGPT-5.3	~150	136	21.3	39.7
Codex-5.3	~98	361	20.8	78.1
Codex-5.3	~201	360	45.6	48.1
Codex-5.3	~322	360	47.5	40.3
Codex-5.3	~612	359	44.0	39.3

Answering without thinking is answering with the pattern

ChatGPT-5.3 spends zero reasoning tokens on 806 of 1,440 examples (56.0%). Those answers are biased 93.2% of the time and correct only 2.2% of the time.

A little deliberation halves the bias

Once reasoning kicks in at ~65 tokens, ChatGPT-5.3's bias drops to 40.8% and accuracy rises to 21.3%. Codex-5.3 falls from 78.1% bias at ~98 tokens to 40.3% at ~322.

But it plateaus — and stops paying

Beyond the threshold the curves flatten. Codex-5.3's longest bin (~612 tokens) is worse on accuracy than its ~322-token bin (44.0% vs. 47.5%), while bias barely moves.

Less bias is not the same as more accuracy. Extra deliberation reliably pulls these models away from the reflexive 100% answer — but past a point it redirects them toward a different wrong value rather than toward the pixels. Note also that reasoning length is observed, not controlled: the models choose how much to reason per example, so this is a correlation, not the effect of forcing longer reasoning.

Mechanism

Why do they fail?

We classified all 237 biased responses that contain a free-text rationale using an LLM judge (gpt-5.4-mini, temperature 0), validated against 30 blind hand-labeled traces (29/30 agreement).

26.2%

MODE A

Code-anchored

Reasons only from the HTML and never references the screenshot. The baseline answer is a default, not a visual judgment.

“All other boxes use width:100%, so the blank is 100%.”

59.1%

MODE B

Perceived-consistent

Looks at the screenshot but reports the element as visually matching its neighbors. The deviation is in the pixels but never registers.

“The three buttons appear the same width, so 100% to match others.”

14.8%

MODE C

Observed-then-overrode

Explicitly notes that the element looks different — sometimes computing the right value — then answers the baseline anyway.

“The post extends beyond the container (~120%); 100% is most consistent.”

The bias is mostly perceptual, not deliberate. In ~74% of verbalized biased responses the model demonstrably engages the screenshot — it simply fails to register the deviation (mode B). The dramatic “saw it and discarded it” override (mode C) is real, but rarer and highly model-dependent (45.5% for Opus-4.6 vs. 4.3% for Flash-3.0).

Qualitative evidence

Caught in the act

Models perceive the deviation, say so out loud, and override it anyway.

Screenshot in which the Translators panel is rendered narrower than its sibling panels.

“The Translators card appears to be about 80% of the full width compared to the other cards which span 100%. […] However, examining more carefully […] It's most likely also 100% to match the consistent layout of all four cards.”

Ground truth 80% Answered 100%

(a) Opus-4.6 estimates the width correctly, then talks itself out of it.

Screenshot in which the Message input field is rendered narrower than the fields above it.

“The ‘Message’ field's input is visibly narrower than the input fields above it. […] it spans a significantly smaller portion of the row […] 100% is the most consistent choice despite the visual rendering of the box inside.”

Ground truth 80% Answered 100%

(b) Flash-3.0 names the deviation, then explicitly overrules the pixels.

Interactive

Try it yourself

Each page below contains one element that deviates from its repeated pattern. Can you spot it before you reveal it? Then copy the prompt and see whether your own model follows the pixels or the pattern.

cards/standard/360_card-mid_w-120pct.png

Forum thread in which one post block is rendered wider than the others.

Masked HTML snippet

<div class="post" style="width:100% !important;">
    <div class="posttop">
     <div class="username">
      xlandguy
     </div>
     <div class="date">
      06-14-2012, 12:49 PM
     </div>
    </div>
    <div class="posttext">
     Please see my post HERE (http://forums.offshoreelectrics.com/showthread.php?35318-Gsbuikman-scammed-me-then-closes-threads), thank you.
    </div>
   </div>
<div class="post" style="width:100% !important;">
    <div class="posttop">
     <div class="username">
      Rumdog
     </div>
     <div class="date">
      06-14-2012, 04:23 PM
     </div>
    </div>
    <div class="posttext">
     again...?
    </div>
   </div>
<div class="post" style="width:100% !important;">
    <div class="posttop">
     <div class="username">
      SFC K
     </div>
     <div class="date">
      06-14-2012, 04:28 PM
     </div>
    </div>
    <div class="posttext">
     Did he finally get in touch with you to make you change this? I read it earlier this morning and looked at the pictures again and the way you described the boat and pictures something was up.
    </div>
   </div>
<div class="post" style="width:__ !important;">
    <div class="posttop">
     <div class="username">
      xlandguy
     </div>
     <div class="date">
      06-14-2012, 05:35 PM
     </div>
    </div>
    <div class="posttext">
     Yes, well at first it seemed as though I couldnt get any response. Turns out he was busy doing something, grandma sick.
    </div>
   </div>
<div class="post" style="width:100% !important;">
    <div class="posttop">
     <div class="username">
      wht73
     </div>
     <div class="date">
      06-15-2012, 04:21 PM
     </div>
    </div>
    <div class="posttext">
     I had similar problem with another member and suggested the same thing.  I was rewarded with no reply to suggestion of small partial refund 2x.
     <br>
     <br>
     I feel ya brother.
    </div>
   </div>
<div class="post" style="width:100% !important;">
    <div class="posttop">
     <div class="username">
      ron1950
     </div>
     <div class="date">
      06-17-2012, 12:44 AM
     </div>
    </div>
    <div class="posttext">
     wow miust have really missed something here
    </div>
   </div>
<div class="post" style="width:100% !important;">
    <div class="posttop">
     <div class="username">
      xlandguy
     </div>
     <div class="date">
      06-26-2012, 08:25 PM
     </div>
    </div>
    <div class="posttext">
     wow miust have really missed something here
     <br>
     <br>
     Well you can catch it all right HERE (http://forums.offshoreelectrics.com/showthread.php?35318-Gsbuikman-scammed-me-then-closes-threads).
    </div>
   </div>

One post block is rendered at width: 120%; every sibling is 100%. Ground truth 120%

cards/standard/6998_card-mid_w-80pct.png

Library record page in which one information panel is rendered narrower than the others.

Masked HTML snippet

<li class="list-group-item level-2 panel-color-top" style="width:100% !important;">
          <h4 class="list-group-item-heading">
           Authors
          </h4>
          <ul>
           <li>
            <a>
             Raymond  Frederick Surburg
            </a>
           </li>
           <li>
            <a>
             David P. Scaer
            </a>
           </li>
           <li>
            <a>
             Eugene F. Klug
            </a>
           </li>
           <li>
            <a>
             Harold Herman Buls
            </a>
           </li>
          </ul>
         </li>
<li class="list-group-item level-2 panel-color-top" style="width:100% !important;">
          <h4 class="list-group-item-heading">
           Editors
          </h4>
          <p>
           There are no editors for this item.
          </p>
         </li>
<li class="list-group-item level-2 panel-color-top" style="width:__ !important; position:relative !important; z-index:20 !important;">
          <h4 class="list-group-item-heading">
           Translators
          </h4>
          <p>
           There are no translators for this item.
          </p>
         </li>
<li class="list-group-item level-2 panel-color-top" style="width:100% !important;">
          <h4 class="list-group-item-heading">
           Compilers
          </h4>
          <p>
           There are no compilers for this item.
          </p>
         </li>

One information panel is rendered at width: 80%; every sibling is 100%. Ground truth 80%

texts/standard/60_navtext-first_fs-120pct.png

Landing page whose first navigation item is rendered in a larger font than its siblings.

Masked HTML snippet

<a class="featuresLink" title="Check out other great Tripod features" style="font-size:__ !important;">
         Features
        </a>
<a class="pricingLink" title="Tripod plan comparisons and pricing" style="font-size:100% !important;">
         Pricing
        </a>
<a title="Tripod's Team Blog for the latest Tripod updates" style="font-size:100% !important;">
         Blog
        </a>
<a class="helpLink" title="Help and support for your Tripod and Zeeblio websites" style="font-size:100% !important;">
         Help
        </a>

One nav item is rendered at font-size: 120%; every sibling is 100%. Ground truth 120%

texts/standard/360_navtext-mid_fs-80pct.png

Forum page whose middle breadcrumb item is rendered in a smaller font than its siblings.

Masked HTML snippet

<a style="font-size:100% !important;">
     Offshore Electrics Forums
    </a>
<a style="font-size:100% !important;">
     Swap Shop
    </a>
<a style="font-size:__ !important;">
     Items for Sale
    </a>
<a style="font-size:100% !important;">
     Seller feedback
    </a>
<span class="breadcrumb-current" style="font-size:100% !important;">Not real happy</span>

One breadcrumb item is rendered at font-size: 80%; every sibling is 100%. Ground truth 80%

Prompt

You are doing a visual code fill-in-the-blank task. Given the webpage screenshot and HTML snippet below, fill the blank token __ with ONLY the missing CSS/HTML value. The goal is to predict the masked value to recreate the {pattern} in the webpage design. Return the final answer in JSON format: {"answer":"<value>"}. The blank must be a percentage value divisible by 10.

All 720 instances and 1,440 screenshots are on Hugging Face.

Pattern Over Pixels Measuring Pattern Completion Bias in Multimodal Code Generation

Plausible code that ignores the pixels

Pattern2Code

The task

All five models default to the pattern

Bias tracks visual saliency

Noise amplifies bias

Subtler perturbations are worse

Boundaries hide deviations

More reasoning, much less bias

Answering without thinking is answering with the pattern

A little deliberation halves the bias

But it plateaus — and stops paying

Why do they fail?

Code-anchored

Perceived-consistent

Observed-then-overrode

Caught in the act

Try it yourself

Masked HTML snippet

Masked HTML snippet

Masked HTML snippet

Masked HTML snippet

Prompt

What this means

Trust coarse layout, verify fine styling

Page-level metrics hide this

Verification is cheap

BibTeX