Feature Flags for AI: The Deployment Pattern That Saved Us From 3 Production Disasters

Pattern 1: Model Migration Flag

You're running llama-3.1-8b-instant and want to migrate to claude-haiku-4-5 for better extraction quality. The naive approach: update the constant, deploy, pray.

The flag approach:

@Service
@Slf4j
public class DocumentExtractor {

    private final ChatClient chatClient;
    private final AIFeatureFlags flags;
    private final MeterRegistry metrics;

    public ExtractionResult extract(String documentText, String userId) {
        String model = flags.extractModel(userId);

        log.debug("extraction model={} userId={}", model, userId);

        Timer.Sample timer = Timer.start(metrics);

        ChatResponse response = chatClient
                .prompt()
                .system(EXTRACTION_SYSTEM_PROMPT)
                .user(documentText)
                .options(ChatOptionsBuilder.builder().model(model).build())
                .call()
                .chatResponse();

        timer.stop(metrics.timer("ai.extract.latency", "model", model));

        // Model is tagged on every metric — you can compare in Grafana
        metrics.counter("ai.extract.tokens.input",  "model", model)
               .increment(response.getMetadata().getUsage().getPromptTokens());
        metrics.counter("ai.extract.tokens.output", "model", model)
               .increment(response.getMetadata().getUsage().getGenerationTokens());

        return parseResponse(response);
    }
}

In Unleash, configure ai.extract.use-haiku:

• Start at 0% rollout
• Ramp to 5% — watch P95 latency and error rate by model in Grafana
• Ramp to 25%, 50%, 100% as metrics look good
• If anything degrades — flip to 0% in the Unleash UI. No redeploy, propagates in 10 seconds

What to compare: latency P95 per model, error rate per model, token cost per model, and — critically — your quality metric (RAGAS score, human thumbs up rate, or whatever you use). The migration is complete when the new model is better or equivalent on all of them.

Pattern 2: Prompt A/B Testing

A new system prompt is not less risky than a code change. It deserves the same scientific treatment.

@Component
public class SummaryPromptRegistry {

    private static final Map<String, String> PROMPTS = Map.of(

        "control", """
            Summarize the following document in under 150 words.
            Focus on the key decisions, numbers, and outcomes.
            """,

        "variant-a", """
            You are a senior analyst. Summarize the following document for an executive
            who has 30 seconds to read it. Lead with the most important insight.
            Use bullet points for supporting facts. 150 words maximum.
            """,

        "variant-b", """
            Summarize this document. Structure your response as:
            HEADLINE: One sentence capturing the core finding.
            KEY FACTS: 3 bullet points, each under 20 words.
            CONTEXT: One sentence on why this matters.
            """
    );

    public String get(String variant) {
        return PROMPTS.getOrDefault(variant, PROMPTS.get("control"));
    }
}

@Service
public class SummaryService {

    private final AIFeatureFlags flags;
    private final SummaryPromptRegistry prompts;
    private final ChatClient chatClient;
    private final MeterRegistry metrics;

    public SummaryResult summarize(String content, String userId) {
        String variant = flags.summaryPromptVariant(userId);    // "control" | "variant-a" | "variant-b"
        String prompt  = prompts.get(variant);

        ChatResponse response = chatClient.prompt()
                .system(prompt)
                .user(content)
                .call()
                .chatResponse();

        // Tag every quality signal with the variant — A/B analysis in Grafana
        metrics.counter("ai.summary.requests", "variant", variant).increment();

        return new SummaryResult(
            response.getResult().getOutput().getContent(),
            variant  // return variant so callers can record user satisfaction per-variant
        );
    }
}

In Unleash, configure ai.summary.prompt-variant as a Variants toggle:

• control — 50% of users
• variant-a — 25% of users
• variant-b — 25% of users

After 7 days and sufficient sample size, compare thumbs-up rate per variant in your analytics. Promote the winner to 100% by setting all weight to the winning variant, then eventually replace control with it.

[!NOTE] Tag every downstream metric — user satisfaction scores, task completion rates, support tickets — with the prompt variant. The quality metric that matters is business outcome, not LLM eval score.

Pattern 3: Graceful Degradation via Kill Switch

When an AI feature breaks, you need to serve something useful while you fix it. A kill switch paired with a fallback path is the pattern.

@Service
public class SmartSearchService {

    private final AIFeatureFlags     flags;
    private final AISearchService    aiSearch;       // semantic vector search
    private final KeywordSearchService keywordSearch; // traditional Elasticsearch

    public SearchResults search(SearchQuery query, String userId) {
        if (!flags.isAiSearchEnabled(userId)) {
            // AI search is killed — fall back to keyword search
            log.info("ai-search disabled — falling back to keyword search userId={}", userId);
            return keywordSearch.search(query);
        }

        try {
            return aiSearch.search(query);
        } catch (Exception e) {
            // Unexpected failure — fall back and alert, but don't surface the error to users
            log.error("ai-search failed unexpectedly — falling back: {}", e.getMessage());
            metrics.counter("ai.search.fallback", "reason", "exception").increment();
            return keywordSearch.search(query);
        }
    }
}

The kill switch gives you three modes:

1. Normal — AI search for everyone
2. Partial kill — AI search for 0% of users (flag disabled), keyword search for all. No service degradation, just capability downgrade.
3. Automatic fallback — AI search throws an unexpected error, keyword search runs transparently

The key design rule: the fallback path must exist before you ship the AI feature. "We'll add fallback later" means you have no kill switch on day one — exactly when you're most likely to need it.

Pattern 4: User-Segment Targeting for Beta Rollouts

Before a public rollout, you want your internal team (and willing beta customers) to dog-food the feature.

@Service
public class AIFeatureFlags {

    // Internal users get new AI features first
    public boolean isNewExtractorEnabled(String userId, Set<String> userRoles) {
        UnleashContext ctx = UnleashContext.newBuilder()
                .userId(userId)
                .addProperty("role", String.join(",", userRoles))
                .build();
        return unleash.isEnabled("ai.extractor.v2", ctx);
    }
}

In Unleash, configure ai.extractor.v2 with activation strategies:

1. UserWithId — list specific internal user IDs: always enabled for the team
2. GradualRolloutUserId — 5% of everyone else

This means your engineers and internal QA users see the new extractor immediately. External users get it at 5% rollout. You run this for 2 weeks, then ramp external users to 25%, 50%, 100%.

// Usage in a service — no change needed as rollout percentage increases
public ExtractionResult extract(Document doc, Authentication auth) {
    String userId = auth.getName();
    Set<String> roles = auth.getAuthorities().stream()
            .map(GrantedAuthority::getAuthority)
            .collect(Collectors.toSet());

    if (!flags.isNewExtractorEnabled(userId, roles)) {
        return legacyExtractor.extract(doc);
    }
    return newExtractor.extract(doc);
}

Observability: Linking Flag State to Metrics

Flags are useless without observability. Every request that touches an AI feature should emit metrics tagged with the active flag variant. This is what makes "is variant-A performing better than control?" answerable in Grafana.

@Aspect
@Component
public class AIFlagMetricsAspect {

    private final MeterRegistry metrics;
    private final AIFeatureFlags flags;

    @Around("@annotation(TrackedAIFeature)")
    public Object trackWithFlagContext(ProceedingJoinPoint pjp) throws Throwable {
        // Extract userId from first String argument (by convention)
        String userId = Arrays.stream(pjp.getArgs())
                .filter(a -> a instanceof String)
                .map(a -> (String) a)
                .findFirst().orElse("unknown");

        String model   = flags.summaryModel(userId);
        String variant = flags.summaryPromptVariant(userId);

        Timer.Sample timer = Timer.start(metrics);
        try {
            Object result = pjp.proceed();
            timer.stop(metrics.timer("ai.feature.duration",
                    "model", model, "variant", variant, "status", "ok"));
            return result;
        } catch (Throwable t) {
            timer.stop(metrics.timer("ai.feature.duration",
                    "model", model, "variant", variant, "status", "error"));
            throw t;
        }
    }
}

In Grafana, you now have a single dashboard panel that shows latency by model and variant — so model migration and prompt A/B testing are both visible without extra instrumentation.

The four queries you always want:

# Error rate by variant
rate(ai_feature_duration_seconds_count{status="error"}[5m]) 
  / rate(ai_feature_duration_seconds_count[5m])
  by (variant)

# P95 latency by model  
histogram_quantile(0.95, 
  rate(ai_feature_duration_seconds_bucket[5m]))
  by (model, le)

# Cost by model (tokens × price)
rate(ai_tokens_input_total[1h]) by (model)

# Rollout progress (% of requests per variant)
rate(ai_feature_duration_seconds_count[5m]) by (variant)
  / ignoring(variant) group_left
  sum(rate(ai_feature_duration_seconds_count[5m]))

The Three Disasters We Avoided

Disaster 1 — Model degradation on a specific doc type. Our summarization model started hallucinating numbers in financial documents after a provider-side update. Kill switch disabled AI summarization for affected document categories within 45 seconds of the on-call alert. Users saw the keyword-based fallback. No customer data corrupted.

Disaster 2 — Prompt regression. A new prompt that performed well in offline evals caused a 12% drop in thumbs-up rate in production. We caught it at 5% rollout (affecting ~800 users) instead of rolling it to all 16,000. Rolled back to control variant in the Unleash UI.

Disaster 3 — Cost spike. A batch job that used a new prompt sent to 100% of traffic consumed 4× the expected tokens due to an accidental context window issue. The cost alert fired at $180 (2× rolling average). Kill switch → model flag reverted to the cheaper model → cost normalized in under 2 minutes.

In all three cases: no redeploy, no hotfix PR, no incident spanning multiple engineers. One person, one flag change, one minute.

Anti-Patterns

Checking flags at the wrong layer. Putting flag checks inside the LLM integration class couples your infrastructure to your feature logic. Put flags in the service layer, let the LLM client be a dumb executor.

One flag for the whole AI system. A single ai.enabled kill switch means you can't surgically kill summarization while keeping classification running. One flag per feature, or per feature + model combination.

No fallback for the kill switch. A kill switch that returns an error instead of a fallback response is just a way to 500 your users more deterministically. The fallback must exist before the flag.

Forgetting to tag metrics with flag state. If your metrics don't tell you which variant produced which latency, you're flying blind. Tag everything.

Long-lived feature branches. Don't keep AI behavior in a Git branch while waiting for the flag rollout to complete. Merge to main, deploy behind a 0% flag. The code is in production; the behavior is off. This is trunk-based development for AI.

Starting Simple: The Minimum Viable AI Flag

If you're not ready for Unleash or LaunchDarkly, this works:

// Minimal flag store — a database table, or even a config map
@Component
public class SimpleAIFlags {

    @Value("${ai.summary.enabled:true}")
    private boolean summaryEnabled;

    @Value("${ai.summary.model:llama-3.1-8b-instant}")
    private String summaryModel;

    @Value("${ai.summary.rollout-percentage:100}")
    private int rolloutPercentage;

    public boolean isSummaryEnabled(String userId) {
        if (!summaryEnabled) return false;
        // Deterministic hash-based assignment so the same user always gets the same experience
        return Math.abs(userId.hashCode() % 100) < rolloutPercentage;
    }

    public String getSummaryModel() { return summaryModel; }
}

With Spring Boot Actuator, you can update @Value properties at runtime via /actuator/env + /actuator/refresh (with @RefreshScope). That's your 60-second kill switch without any external system.

Start here. Add Unleash when you have more than 3 AI features and more than one person managing flag changes.

Key Takeaways

• AI features fail silently. Flags let you catch silent failures at 5% before they affect 100% of users.
• Prompt changes are deployments. Treat them with the same rigor: staging, gradual rollout, observability, rollback.
• Build the fallback first. A kill switch that errors instead of degrading gracefully is worse than no kill switch.
• Tag every metric with flag state. Without this, you can't answer "is the new model better?" from production data.
• Model providers change under you. Your flag infrastructure needs to handle changes you didn't trigger, not just changes you did.
• The kill switch window matters. 60 seconds to production effect (flag change → all pods updated) is the target. Anything above 5 minutes is too slow for an active incident.

The engineering investment is modest — a few days to wire up Unleash and build the flag service. The return is the ability to ship AI features confidently and fix them without waking up your team at 3AM.

The next part of this series covers AI observability specifically: the metrics, traces, and dashboards that make "what exactly did the model return for user X?" answerable in under 30 seconds. If you're already doing feature flags and want to pair them with tighter observability, that's where to look next.