Training Taste - Thais Castello Branco, Taste Labs

AI slop in subjective domains like design can be quantitatively measured and systematically reduced through structured decomposition of taste into verifiable...

By Sean Weldon

Training Taste: Quantifying and Reducing AI Slop in Design Systems

Abstract

This synthesis examines a research program that reframes AI slop - the homogeneous, low-effort output increasingly common in generative design - as a measurable and tractable engineering problem rather than an irreducible matter of taste. The approach decomposes subjective quality into verifiable components, operationalized through a three-characteristic framework: repetition, lack of fit, and low intent interpretation. This framework was validated against a corpus exceeding two million websites collected over ten years, alongside synthetically generated AI-designed sites. Lightweight classifiers ("probes") trained on structured design features outperformed LLM-as-a-judge baselines in slop prediction. Two intervention layers are examined - model training and inference-time application - with concrete tooling proposed for the latter, including a Creativity API and a Brand API. Findings suggest that as production cost approaches zero, judgment becomes the scarce and valuable resource, with practical implications for evaluation design and agentic systems.

1. Introduction

The marginal cost of producing a website, brand identity, or visual artifact has collapsed. Generative models now allow any user to produce polished-looking design output in seconds, without the years of deliberate practice that professional designers typically accumulate. The consequence of this cost collapse is not simply an increase in volume but a systematic drift toward the statistical mean of the training distribution - a phenomenon colloquially termed AI slop.

Slop is defined by three characteristics: repetition (recurring patterns across otherwise unrelated contexts), lack of fit (mismatch between output and the context it is meant to serve), and low intent interpretation (failure to capture what a user actually needed). This stands in contrast to greatness, which in subjective domains is characterized by uniqueness, care, attention to detail, and authenticity. Unlike mathematical problems, subjective domains admit no single correct answer for what constitutes "great." However, a key methodological claim underlying this work is that slop is considerably easier to define and agree upon than greatness - observers share an intuitive sense of repetition and soullessness even when they diverge on what constitutes excellence. This asymmetry supplies a tractable optimization target: rather than attempting to specify greatness directly, the research program targets the elimination of slop's identifiable characteristics.

This analysis covers the empirical measurement of slop across a large web corpus, the construction of classifiers that outperform LLM-based evaluation, and two intervention layers - model training and inference-time tooling - through which slop reduction is operationalized. Design is treated as the first of several planned "pillars," with writing and other subjective domains to follow.

2. Background and Related Work

Slop is not a phenomenon unique to AI. Homogenization of internet design predates generative models, driven by engagement-optimized social media platforms and templated design systems that converged toward shared visual conventions. Corpus analysis of website design over the past decade confirms this pre-existing homogenization trend. What generative AI contributes is acceleration: two mechanisms compound the prior trend. First, the near-zero cost of production means users without developed design taste can now generate finished-looking artifacts. Second, off-the-shelf generative models exhibit a tendency toward distributional collapse, defaulting to the stylistic mean of their training data absent explicit steering.

Prior evaluative practice in subjective domains has relied substantially on LLM-as-a-judge methods, in which a frontier model scores outputs against a natural-language rubric. This approach inherits the evaluator's own distributional biases and, as demonstrated in this research, underperforms simpler, structured alternatives when tasked with detecting slop specifically.

3. Core Analysis

3.1 Measuring Slop at Scale

The empirical foundation of this work rests on analysis of more than 2 million websites spanning ten years, functioning as a Wayback-Machine-style longitudinal dataset. This corpus was supplemented with synthetically generated AI-designed websites, enabling direct comparison between human-made and AI-generated design artifacts. The analysis confirms that internet homogenization began prior to the introduction of generative AI tools, but that AI has measurably increased repetition - notably across site categories that share no obvious functional or audience relationship, indicating that the convergence is stylistic rather than context-driven.

3.2 Probes as an Alternative to LLM-as-a-Judge

To operationalize slop detection, the research program constructed probes - lightweight classifiers trained on structured, extracted design features including color palettes, typography, layout structure, and inferred audience. Individually, these probes function as narrow pattern detectors; combined, they achieved higher accuracy in predicting slop than LLM-as-a-judge approaches. This finding is significant because it suggests that decomposing a subjective quality judgment into structured, verifiable sub-components yields more reliable evaluation than asking a general-purpose model to render a holistic aesthetic verdict.

3.3 Two Layers of Intervention

The research distinguishes two loci at which slop can be addressed: the model layer, involving frontier model training data and reinforcement learning environments, and the inference layer, involving application-level agents operating on fixed model weights. The analysis argues that inference-time solutions carry equal or greater importance than model-layer fixes, since this is the layer at which actual user interaction and contextual specificity occur. Off-the-shelf models, absent inference-time steering, tend to collapse toward stylistic averages regardless of underlying training quality - making application-layer intervention a necessary complement to model-level improvement rather than a stopgap.

3.4 Structured Tooling for Out-of-Distribution Generation

Two concrete tools instantiate the inference-layer strategy. The Creativity API is designed as an "inspiration machine" that intentionally produces out-of-distribution outputs while maintaining adherence to a given category - distinguished explicitly from simply increasing model temperature, which introduces randomness rather than purposeful deviation. The Brand API extracts brand identity from a URL into structured components that an agent can follow and be judged against; it is currently in beta with design partners. A demonstrated case study - generation of a slide deck for a fictional "General Intelligence Company of New York" using Claude - showed markedly higher fidelity to original branding when brand extraction was used compared to default generation, illustrating the practical value of structured brand representation over unconstrained generation.

4. Technical Insights

Several implementation-level findings merit attention. First, feature decomposition - breaking an aesthetic judgment into color, typography, layout, and audience components - proves more tractable for automated evaluation than holistic LLM scoring, suggesting that evaluation pipelines for subjective domains should favor structured feature extraction over end-to-end judgment models. Second, the distinction between randomness and purposeful creativity is architecturally significant: naively increasing sampling temperature does not reliably produce desirable out-of-distribution outputs, since it does not preserve fit to context or category constraints. Third, not all design elements require the same treatment - attributes such as contrast and alignment approach near-deterministic solutions once problem context is well-specified, while aesthetic judgment continues to show legitimate expert disagreement and is better addressed through data-driven, corpus-grounded approaches than through fixed rules. Finally, a proposed brand repository - a retrievable index of pre-created, cohesive brand systems (e.g., a "dreamy" aesthetic) - offers an alternative to generating brand identity from scratch for users lacking an existing brand, trading generative flexibility for consistency and reduced slop risk.

5. Discussion

These findings carry implications beyond design. The success of structured probes over LLM-as-a-judge suggests that evaluation of subjective quality in other domains - writing, audio, code aesthetics - may similarly benefit from decomposing holistic judgments into narrower, verifiable sub-tasks rather than relying on general-purpose models to render single-shot verdicts. This has direct relevance to the broader challenge of building reliable reward models and RL environments for creative tasks, where reward hacking toward distributional means is a known failure mode.

The emphasis on inference-layer intervention also reflects a broader industry trend: as foundation models become commoditized, differentiation increasingly occurs in the application layer, where context, constraints, and user intent are actually available. The claim that "judgment becomes more valuable as production cost goes to zero" positions this work within a wider economic argument about AI's effect on skilled evaluation labor - production is being automated, but discernment is not.

A notable gap remains around greatness itself. While the tractability of slop as a target is well-argued, the analysis stops short of proposing a comparably rigorous framework for cultivating or measuring positive creative excellence, treating slop-avoidance as a necessary but not sufficient condition for quality output.

6. Conclusion

This research program demonstrates that AI slop, despite arising in a subjective domain, can be defined with sufficient precision to be measured empirically and reduced systematically. Through large-scale corpus analysis, structured feature-based classifiers, and targeted inference-layer tooling (Creativity API, Brand API), the work shows that slop reduction does not require solving the harder problem of defining greatness. Practical next steps include extending the three-characteristic framework and probe-based evaluation methodology to writing and other subjective pillars, and further investigating how brand and creativity tooling generalizes across agentic design workflows.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub