button-enum-naming-experiment-v0.2

Button enum-value and property-name experiment v0.2

Type: Experiment
Status: Active draft
Canonical for: The sole current preregistration design for testing Button enum-value and property-name effects
Depends on: Research-bundle reconciliation, label pretesting, gold-policy review and power simulation

It is not a production naming recommendation and should not be run until the label pretest, gold-policy review and power simulation are complete.

1. Causal question

When an LLM selects a Button action-hierarchy value from a fixed schema, do enum-value names and the property name affect correct selection when the underlying concepts, descriptions, scenarios and legal structure are held constant?

This study tests schema interpretation, not visual design quality or component selection.

2. Evidence boundary

Direct causal evidence exists for tool names, parameter names and schema-key wording in adjacent tasks. Direct isolated evidence does not exist for design-system enum-value names. The experiment is designed to close that gap.

It does not test whether an AI-ready contract should contain particular fields, whether PDS should rename variants or whether one serialization format is superior.

3. Frozen concept model

Use one semantic axis only: action hierarchy within a declared surface.

Concept ID	Name-free behavioral definition
H1	The single action that most directly advances the main affirmative goal of the current action group. At most one H1 is allowed in that group.
H2	An affirmative supporting action in the same group. It may accompany H1 but is not the group’s main goal.
H3	An optional or supplementary action that should not compete with H1 or H2.

The definitions must not use any experimental property or enum labels.

Excluded from the first study:

• destructive consequence;
• cancel/dismiss role;
• navigation versus action;
• form submission;
• confirmation stage;
• tone, color and visual treatment;
• loading, disabled and selected state.

Those are future replications on distinct semantic axes.

4. Primary hypotheses

• H0: Enum-name treatment does not change exact concept selection when descriptions are fixed.
• H1: At least one of aligned, opaque and misleading enum-name treatments differs from another.
• H2: Misleading familiar names reduce accuracy relative to opaque identifiers under identical descriptions.
• H3: Description richness moderates enum-name effects.
• H4: Property-name treatment changes value selection when enum names and descriptions are fixed.
• H5: Enum/property-name effects interact with model family or capability. Direction is exploratory.
• H6: Under misleading names, errors disproportionately follow the name’s pretested conventional interpretation rather than the local definition.

H2 is the central unresolved comparison. H5 must not be directional.

5. Treatments

Enum-name treatment E

Level	Definition
E1 aligned	Pretested conventional interpretation matches the assigned hierarchy concept.
E2 opaque	Pretest yields no stable hierarchy interpretation.
E3 misleading	Pretested interpretation reliably matches a different hierarchy concept from the assigned local definition.

Use at least three independently constructed label sets per level where feasible. Counterbalance mappings. Treat label set/mapping as a random effect or blocking factor. Do not allow one memorable string to stand for an entire treatment.

Description treatment D

Level	Information
D0	Values listed with no per-value descriptions.
D1	Concise behavior definition for each value.
D2	Same definition plus scoped use/avoid boundaries and closest alternative.

Descriptions are written against H1–H3 first, then attached unchanged to the labels in every E condition. Match information content and ordering across conditions.

Property-name treatment P

Level	Definition
P1 aligned	Pretested as denoting action hierarchy/emphasis in the declared scope.
P2 opaque	Carries no stable semantic interpretation.
P3 conflicting	Pretested as denoting another axis, such as urgency or size.

Final strings must come from the same pretest procedure as enum labels. Do not assume variant, emphasis, priority or tone belongs to a treatment without measurement.

6. Study structure

Study A: enum naming and descriptions

Cross E (3) × D (3) × model. Keep the component name and property name fixed.

Primary contrasts:

• E1 versus E2 at D1;
• E2 versus E3 at D1;
• E1 versus E3 at D1;
• E × D interaction.

Study B: property naming

Cross P (3) × selected E levels at D1. Use a smaller preregistered block. This isolates property-name interpretation without the full factorial explosion.

Later study: constraint delivery

Do not include constraint architecture in the primary naming study. A separately powered follow-up may compare:

• prose guidance;
• schema-visible constraints;
• semantic generation followed by deterministic validation and repair;
• hard constrained decoding.

That study must measure validity and semantic correctness separately.

7. Label pretest

1. Create a broad candidate pool from human proposals and cross-system terminology.
2. In a separate zero-context prompt, ask each target model what behavioral hierarchy the candidate usually denotes in a UI.
3. Map responses to H1–H3 or “no stable hierarchy meaning” using blinded coding.
4. Measure agreement across prompts, seeds and models.
5. Select shared label sets that meet preregistered alignment thresholds where possible.
6. Record per-model alignment strength as a manipulation check.
7. Never use evaluation scenarios during the pretest.

If a label is aligned for one model and misleading or opaque for another, retain that fact in analysis rather than silently assigning different experimental meanings without reporting it.

8. Scenario bank and gold policy

Create scenarios only after the hierarchy policy is frozen.

Include:

• clear H1/H2/H3 choices;
• two affirmative actions where discourse establishes hierarchy;
• local action groups nested within a larger page;
• optional actions that are useful but not goal-advancing;
• boundary cases where clarification is the correct response;
• wording that could lexically attract an incorrect experimental label without containing that label;
• cases that pressure the model to assign more than one H1.

Do not copy public design-system examples. Avoid consequence, navigation and dismissal cues in the first study.

Independent design-system experts should assign concept IDs while blind to treatment labels. Retain only scenarios meeting a preregistered agreement threshold. Ambiguous cases should have a gold “clarify” outcome rather than a forced hierarchy value.

9. Fixed elements and leakage controls

Hold constant:

• Button component name and description;
• concept definitions;
• scenario text;
• number and order-randomized set of values;
• schema structure;
• prompt instructions;
• model decoding configuration;
• absence of target-label examples;
• no informative defaults;
• scoring validator.

Randomize:

• value order;
• scenario order;
• label-set/mapping block;
• presentation order where the provider permits it.

Check scenario text for differential similarity to treatment labels using both human review and embedding/lexical diagnostics. A label-free scenario should still contain enough task information to determine H1–H3 from the frozen policy.

10. Models and runs

Use at minimum:

• one smaller open model;
• one mid-sized open model;
• two frontier models from different families, if feasible;
• the actual production agent harness as a separate deployment condition.

Model-family interaction is exploratory. Record exact model IDs, provider versions, structured-output settings, system prompts, retries and dates.

Temperature-zero runs are primary. Repetition at temperature zero measures service/harness nondeterminism, not stochastic sampling in the usual sense. A separately declared nonzero-temperature block may measure stability.

11. Outcomes

Primary

• exact returned enum label mapped to the correct hierarchy concept ID;
• appropriate clarification for gold-ambiguous scenarios.

Secondary

• schema-valid output;
• invented/out-of-enum value;
• missing or wrong property;
• multiple-H1 policy violation;
• consistency across paraphrases and runs;
• high-confidence error rate where comparable token probabilities exist;
• tokens, latency and retries.

Self-reported confidence is exploratory and should not be compared directly across providers.

12. Error taxonomy

Code	Meaning
P1 prior capture	Choice matches the pretested conventional meaning of the label, not its local definition.
P2 description ignored	Description clearly supports another concept.
P3 property-axis error	Model interprets the property as a different semantic axis.
P4 invented value	Output is outside the allowed enum.
P5 valid semantic error	Output is schema-valid but maps to the wrong concept.
P6 task lexical capture	Choice follows superficial task wording rather than the frozen policy.
P7 structural error	Missing/wrong property or malformed output.
P8 clarification failure	Model guesses when the gold policy requires clarification, or clarifies unnecessarily.
P9 residual	Error cannot be assigned reliably.

Coders should be blind to the hypothesis and use the recorded label-pretest interpretation for P1. Report inter-rater agreement.

13. Statistical plan

Use a mixed-effects logistic model or Bayesian hierarchical equivalent.

Fixed effects:

• E, D and E × D in Study A;
• P, E and P × E in Study B;
• model/family interactions as exploratory.

Random effects should include scenario and label set/mapping. Model may be a fixed factor when the small named set is itself of interest.

Set the smallest effect of practical interest before power calculation. Use simulation based on plausible baseline ranges rather than importing tool-name effect sizes as expected enum-name effects. Determine scenario count, label-set count and repetitions from that simulation.

Pre-register the primary contrast family and multiplicity correction. Report effect sizes and uncertainty, not only significance.

14. Interpretation

Observed pattern	Supported interpretation
E1 > E2 > E3	Alignment helps and conflict is worse than opacity under tested conditions.
E1 ≈ E2 > E3	Conflict hurts; opacity is not materially worse than alignment.
E1 > E2 ≈ E3	Alignment helps, but the study does not distinguish opacity from conflict.
No difference at D2 but differences at D0/D1	Rich descriptions compensate under tested conditions.
Differences persist at D2	Names remain an independent signal despite rich descriptions.
High P1 concentration under E3	Errors are consistent with prior capture.
Property effects vary in sign across models	Public API wording cannot be optimized from one model result.
No material effects	Naming may be lower leverage than descriptions/retrieval for this task.

No result from this hierarchy-only study establishes behavior for destructive actions, role, state, tone, tokens or cross-component composition.

15. Stop condition

The study is ready to run only when:

• hierarchy concepts and gold policy are frozen;
• scenarios pass expert agreement and leakage review;
• at least three label sets per treatment pass the manipulation pretest or a justified alternative is documented;
• provider schemas can represent the identical experiment;
• the smallest effect of practical interest and power simulation are complete;
• prompts, scoring and analysis are preregistered;
• no production naming decision is coupled to a desired outcome.