Experimental design: LLM interpretation of design-system property and enum-value names
This design is constrained by the verified evidence base: causal effects are established for tool/component and parameter names with fixed descriptions; enum-value naming and misleading-vs-opaque comparisons remain unestablished; name–description conflict and pretrained semantic anchors are documented in adjacent settings; hard generation-time constraints impose a validity–correctness tradeoff on smaller models.
A. Experimental hypothesis set
Primary question
When an LLM must select Button property values from a fixed component schema, how do semantic alignment, opacity, and semantic conflict in property and enum names affect selection accuracy?
Operational definitions of alignment
| Form | Operational definition | How tested |
|---|---|---|
| Human-semantic alignment | Independent human raters (designers / product people) judge that the name accurately describes the concept’s behaviour | Pre-registration rating; not used as primary IV |
| Model-semantic alignment | Zero-context elicitation: given only the name, the model’s most probable behavioural interpretation matches the local definition | Pre-treatment elicitation battery (see §7) |
| Local-system alignment | The schema’s own description of the value matches the frozen behavioural concept | Always true by construction once names are mapped onto concept IDs |
These three can diverge. The experiment primarily manipulates model-semantic alignment relative to local-system alignment, while holding local definitions fixed.
Hypotheses
H0 (null)
Enum-value name type (aligned / opaque / misleading) has no effect on exact intended value selection when behavioural descriptions are identical.
H1 (enum main effect)
Aligned enum-value names yield higher exact-match accuracy than opaque names, which in turn differ from misleading names, with descriptions held fixed.
H2 (misleading vs opaque)
Misleading familiar names produce lower accuracy than opaque identifiers under identical descriptions (C < B).
If rejected in favour of B ≈ C or B < C, the “opacity is safer than conflict” conjecture is weakened.
H3 (description moderation)
Full behavioural descriptions reduce (but do not necessarily eliminate) the accuracy gap between name types (name × description interaction).
H4 (property-name effect)
Property name (model-aligned vs opaque vs model-conflicting) affects value selection accuracy even when value names and descriptions are fixed.
H5 (model-size interaction)
Name-type effects are smaller or absent on ~3B-class models and larger on frontier models (motivated by Hood’s capability threshold and PA-Tool’s SLM focus).
H6 (conflict / prior capture)
Under misleading names, a measurable share of errors are “followed pretrained meaning instead of local definition” (failure taxonomy code P1), even when descriptions are present and correct.
All hypotheses are pre-registered. H2 is the critical gap relative to the current literature.
B. Canonical Button concept model
Behaviour is frozen before any experimental names are assigned. Concepts receive neutral IDs only.
Core concepts (illustrative; final set fixed before naming)
| ID | Behavioural definition (name-free) |
|---|---|
| C1 | The single most important affirmative action on the surface; advances the primary user goal; at most one per view region |
| C2 | Affirmative but secondary; supports or accompanies C1; never the sole primary goal |
| C3 | Action that permanently destroys data or cannot be undone without explicit recovery; requires heightened caution |
| C4 | Dismisses or aborts the current flow without committing side effects |
| C5 | Navigates to another context without committing the current form/state |
| C6 | Confirms a previously staged decision (e.g. after a summary step) |
| C7 | Submits form data that creates or updates a record (reversible or not depending on paired flags) |
Orthogonal axes (also ID-only until naming)
- Hierarchy rank: H1 (sole primary) / H2 (supporting) / H3 (tertiary)
- Reversibility: R1 (easily reversible) / R2 (hard or irreversible)
- Visual weight: W1 (highest emphasis) / W2 (medium) / W3 (lowest) — only if the experiment includes a visual-emphasis property; otherwise omit to avoid confounding hierarchy with paint
Rule: No experimental condition may use the strings primary, secondary, destructive, variant, etc. in the canonical model. Those strings appear only after mapping in the treatment matrix.
C. Treatment matrix
What stays fixed across all conditions
- Canonical concept set and behavioural definitions
- User-intent scenarios (wording never contains target enum labels)
- Schema structure (one component, fixed properties, same number of values)
- Model decoding settings (temperature 0 primary; limited multi-sample for consistency)
- Absence of few-shot examples that name the target values (unless an explicit follow-up block)
- No default values that reveal the intended choice
- Validation schema used only for scoring, not as soft guidance in the main generation pass (except in a dedicated constraint-delivery block)
What changes
Factor E — Enum-value name type (primary)
| Level | Mapping rule | Example mapping onto C1 / C2 |
|---|---|---|
| E1 Aligned | Model-semantic prior matches local behaviour | C1→main, C2→support (or other pairs that pass elicitation) |
| E2 Opaque | No useful prior | C1→v1, C2→v2 |
| E3 Misleading | Model-semantic prior conflicts with local behaviour | C1→ label whose zero-context meaning is “secondary/supporting”; C2→ label whose zero-context meaning is “main/primary” |
Factor P — Property name (secondary; reduced levels)
| Level | Example family (final strings chosen via elicitation) |
|---|---|
| P1 Model-aligned | Name whose prior suggests “visual/ behavioural style of the control” |
| P2 Opaque | opt, p1, or similar |
| P3 Model-conflicting | Name whose prior suggests a different axis (e.g. urgency, size, or priority) while local meaning is hierarchy |
Factor D — Description richness (moderator)
| Level | Content |
|---|---|
| D0 | Property present; values listed with no per-value descriptions |
| D1 | Concise behavioural description per value (identical wording across E levels) |
| D2 | Full behavioural + when-to-use / when-not-to-use (still identical across E levels) |
Descriptions are written against concept IDs first, then attached to whatever label is assigned in that condition. Wording never changes with the label.
Recommended design structure (avoids explosion)
| Block | Factors | Role |
|---|---|---|
| Core factorial | E (3) × D (3) × Model (3–4) | Estimates enum main effect, description effect, E×D interaction, model-size interaction |
| Property block | P (3) × E (3) at D1 only | Estimates property-name main effect and P×E; held at one description level |
| Constraint block (optional follow-up) | Constraint delivery (prose only vs schema-visible constraints vs post-validator + retry) at E1/E2/E3 × D1 | Tests architecture hypothesis; not required for naming hypotheses |
Full cross of E×P×D×Model is unnecessary for the first study. Core + property block answers H1–H5 with manageable size.
D. Scenario set design
Scenarios are written against concept IDs. The correct answer is always a concept ID; labels are applied only at scoring time.
Scenario types (representative; not the full bank)
- Obvious hierarchy — One clear main goal and one supporting action on the same surface.
- Subtle hierarchy — Two affirmative actions; only discourse structure or goal priority distinguishes C1 from C2.
- Near-boundary — Action that is primary in one reading and supporting in another; description must decide.
- Misleading lexical cues — User wording contains words that match a wrong name condition (e.g. “secondary step” when C1 is correct).
- Destructive — Irreversible delete/remove; must map to C3.
- Cancel / dismiss — Abort without commit → C4.
- Navigation — Leave flow without submit → C5.
- Confirmation — Affirm staged decision → C6.
- Form submit — Commit form data → C7 (with reversibility flag if tested).
- Multiple primaries pressure — Copy that tempts two C1s; schema allows only one; tests constraint adherence.
- Repeated / bulk action — Same action over many items; hierarchy still applies.
- Mixed irreversible + reversible — Two actions; only one is R2.
Leakage rule: Scenario text must not contain any experimental enum or property string. Pilot with a held-out model: if accuracy collapses when names are removed from the schema, scenarios are too easy or leaky.
Size target: 24–36 scenarios after piloting (2–3 per type), enough for stable condition means without publication-scale volume.
E. Controls and randomization plan
| Leakage source | Control |
|---|---|
| User-task wording | Ban target labels; pilot for accidental synonyms that match one name condition only |
| Property / enum descriptions | Written on concept IDs; identical text across E levels |
| Examples / few-shots | None in main experiment |
| Schema order | Randomize property order and enum order per trial |
| Default values | None, or identical neutral defaults that do not encode the answer |
| Component docs | Minimal fixed blurb; no value names |
| Surrounding code | Not provided |
| Enum ordering | Randomized; report order as covariate if needed |
| Position bias | Randomize order of candidate values in schema |
Randomization
- Scenario order shuffled per model run
- Name mapping (which opaque string is C1 vs C2) fixed within a condition but counterbalanced across seeds if multiple opaque pairs are used
- Temperature 0 for primary metric; optional 3-sample majority for consistency secondary metric
Pre-treatment isolation
Elicitation of model-semantic priors (to build E1/E3) uses a separate prompt suite and must not reuse evaluation scenarios.
F. Metrics and statistical plan
Metrics
| Type | Metric | Definition |
|---|---|---|
| Primary | Exact intended value | Selected enum value maps to correct concept ID |
| Secondary | Schema-valid | Output parses and satisfies schema (types, required, enum membership) |
| Secondary | Invented value | Token not in the allowed enum |
| Secondary | Invalid combination | Violates frozen co-occurrence rules (if any) |
| Secondary | Confidence | Prefer log-prob of chosen value under the schema constraint if available; else self-report as secondary only |
| Secondary | Consistency | Agreement across repeated identical trials (temp 0 or fixed seed) |
| Secondary | Retries | Count of validator failures before valid+correct (constraint block only) |
| Secondary | Tokens / latency | Cost controls |
Scoring rule: Schema-valid but wrong concept = incorrect on primary metric.
Statistical plan
- Unit of analysis: scenario × model × condition trial
- Scenarios: 24–36
- Repeats: 1 primary (temp 0); optional 3 for consistency
- Models: see §9
- Expected effect sizes (from adjacent literature, not guarantees): method-name effects 10–29 pp (Hood); noise-induced drops tens of points (RoTBench); description-swap collapses 39–43 pp. Enum-value effects are unknown; plan for detecting ≥8–10 pp with the chosen N.
- Model: Mixed-effects logistic regression
- Fixed: E, D, E×D, model size band, (in property block) P, P×E
- Random: scenario intercept; optional model intercept
- Contrasts: E1 vs E2; E2 vs E3; E1 vs E3; interaction tests for H3, H5
- Multiple comparisons: Pre-specified primary contrast family (E levels at D1); Holm or similar within family; exploratory contrasts secondary
- Power: Prioritize precise estimation of E main effect and E×D; property and constraint blocks are secondary
G. Failure taxonomy
Every error is coded with one primary code (and optional secondary).
| Code | Name | Meaning |
|---|---|---|
| P1 | Prior capture | Choice matches pretrained meaning of the label, not local definition |
| P2 | Description ignore | Local description clearly entailed a different value; model did not follow it |
| P3 | Property misread | Correct value family chosen under wrong property interpretation |
| P4 | Invented value | Output not in enum |
| P5 | Valid wrong combo | Schema-legal but concept-wrong combination |
| P6 | Lexical match to user | Matched surface words in the user prompt rather than behaviour |
| P7 | Schema misunderstanding | Malformed structure, wrong property filled, missing required |
| P8 | Inconsistent / residual | Temp-0 disagreement across repeats or no clear mechanism |
P1 is the critical code for H6. High P1 rate under E3 supports “misleading names pull toward pretrained meaning.”
H. Threats to validity
| Threat | Mitigation |
|---|---|
| Pretrained exposure to common design-system terms | Opaque and misleading conditions; elicitation-based label choice; report if frontier models show ceiling on E1 |
| Vendor / benchmark contamination | Prefer scenarios not scraped from public DS docs; avoid copying Storybook examples |
| Prompt sensitivity | Fixed prompt template; ablate only in a small sensitivity check |
| Schema-order / position effects | Randomize order every trial |
| Cross-model semantic differences | Per-model elicitation for building E1/E3; do not assume one misleading set fits all families |
| Human-semantic vs model-semantic mismatch | Document both; primary IV is model-semantic |
| Overfitting to Button | Pre-register that generalization to other components is out of scope for this study |
| Constraint tax confounding naming | Main naming analyses use unconstrained or lightly formatted generation; hard constraints isolated in optional block |
I. What the experiment would establish
| Outcome pattern | Defensible conclusion |
|---|---|
| E1 > E2 ≈ E3 | Aligned names help; opacity and conflict are similarly harmful (or help is alignment, not “avoid opacity”) |
| E1 > E2 > E3 | Aligned best; opaque intermediate; misleading worse than opaque (supports H2) |
| E1 ≈ E2 > E3 | Only conflict hurts; opacity is acceptable |
| E1 ≈ E2 ≈ E3 at D2, gaps at D0 | Descriptions largely override name effects |
| Gaps persist at D2 | Descriptions do not fully override name priors (especially if P1 errors dominate under E3) |
| Effects only on frontier, null at ~3B | Replicates capability-threshold pattern for enum values |
| High P1 under E3 | Errors are specifically prior-driven, not generic noise |
| Property block: P3 hurts | Property name can shift interpretation of the same values |
| Constraint block: post-validator recovers validity without naming effect change | Naming affects selection; constraints affect validity — separable concerns |
Non-conclusions (still off-limits)
- Which production PDS names to adopt
- That JSON is inherently better than prose
- That the same pattern holds for every component without replication
Model set (item 9)
| Band | Role | Motivation from verified literature |
|---|---|---|
| ~3B-class (one open model) | Lower end of Hood null effect | Test whether enum-value naming effects require frontier capacity |
| Mid (7–8B class, one family) | SLM regime of PA-Tool / RoTBench | Bridge |
| Frontier (two families if feasible, e.g. one Anthropic-class, one OpenAI-class or open frontier) | Where method-name effects were large | Estimate upper-bound naming sensitivity |
Interaction of interest: name-type × size band (H5).
Defining misleading names rigorously (item 7)
Procedure (run before evaluation)
- Candidate pool: human-proposed names that might conflict (and aligned/opaque candidates).
- Zero-context elicitation (per model family):
“In a user interface, what does a control labelled [NAME] usually mean? Answer in one short behavioural sentence.” - Score elicitation outputs against frozen concept definitions (blind raters or entailment check).
- E1: name whose dominant elicitation matches C1 (or C2).
- E3: name whose dominant elicitation matches the wrong concept (e.g. elicits C2 behaviour but will be bound to C1).
- E2: strings with no consistent behavioural elicitation (or only generic “option/version” readings).
- Require multi-model agreement for E3 assignment where possible; document per-model exceptions.
- Never use evaluation scenarios in this phase.
This makes “misleading” a measured property of model priors, not designer intuition.
This design converts the verified evidence gap into a falsifiable test of enum-value and property naming under controlled descriptions, with leakage controls, a name-free behavioural core, and an analysis plan that can support or reject the specific claim that misleading familiar names are worse than opaque identifiers for design-system value selection.