Implementing feature flags for controlled rollouts and experimentation in production

# Implementing Feature Flags for Controlled Rollouts and Experimentation in Production

Feature flags have transformed how engineering teams deploy and test new functionality. At Spotify, we reduced deployment risks by 80% after implementing a robust feature flagging system. Netflix reports that feature flags enable them to run over 250 experiments simultaneously, leading to a 30% increase in user engagement metrics for successful features.

This post explores how to implement feature flags effectively, from basic toggles to sophisticated experimentation platforms. I'll share code examples, architectural patterns, and lessons from implementing feature flags across multiple organizations.

## What Are Feature Flags?

Feature flags (also called feature toggles or switches) are a software development technique that allows teams to modify system behavior without changing code. They create decision points in code that determine whether a feature is enabled for specific users.

```java
// Simple feature flag example
if (featureFlags.isEnabled("new-checkout-flow")) {
    // New implementation
    return newCheckoutFlow(user);
} else {
    // Current implementation
    return currentCheckoutFlow(user);
}

Types of Feature Flags

Feature flags serve different purposes throughout the software development lifecycle:

Release Flags: Control feature visibility in production
Experiment Flags: Support A/B testing
Ops Flags: Manage operational aspects like performance settings
Permission Flags: Control access based on user attributes

Building a Basic Feature Flag System

Let's start by implementing a simple feature flag system in a Spring Boot application:

@Service
public class FeatureFlagService {
    private final Map<String, Boolean> flags = new ConcurrentHashMap<>();

    public FeatureFlagService() {
        // Initialize with default flags
        flags.put("new-search-algorithm", false);
        flags.put("enhanced-recommendations", false);
        flags.put("dark-mode", true);
    }

    public boolean isFeatureEnabled(String featureName) {
        return flags.getOrDefault(featureName, false);
    }

    public boolean isFeatureEnabledForUser(String featureName, User user) {
        if (!flags.getOrDefault(featureName, false)) {
            return false;
        }

        // Check if user is in beta group
        return user.isBetaTester();
    }

    public void setFeatureFlag(String featureName, boolean enabled) {
        flags.put(featureName, enabled);
    }
}

This basic implementation works for small applications but lacks persistence, user targeting, and remote configuration capabilities.

Advanced Feature Flag Implementation

For production systems, we need more sophisticated capabilities:

Persistent storage
User targeting rules
Gradual rollouts
Remote configuration
Monitoring and analytics

Here's an enhanced implementation:

@Service
public class EnhancedFeatureFlagService {
    private final FeatureFlagRepository repository;
    private final UserService userService;
    private final MetricsService metricsService;

    @Autowired
    public EnhancedFeatureFlagService(
            FeatureFlagRepository repository,
            UserService userService,
            MetricsService metricsService) {
        this.repository = repository;
        this.userService = userService;
        this.metricsService = metricsService;
    }

    public boolean isFeatureEnabled(String featureName, String userId) {
        FeatureFlag flag = repository.findByName(featureName)
            .orElse(new FeatureFlag(featureName, false, 0, Collections.emptyList()));

        if (!flag.isEnabled()) {
            return false;
        }

        User user = userService.findById(userId);

        // Check if user is in target groups
        if (!flag.getTargetGroups().isEmpty() && 
            !flag.getTargetGroups().contains(user.getUserGroup())) {
            return false;
        }

        // Check percentage rollout
        if (flag.getRolloutPercentage() < 100) {
            int userHash = Math.abs(userId.hashCode() % 100);
            if (userHash >= flag.getRolloutPercentage()) {
                return false;
            }
        }

        // Record feature usage for analytics
        metricsService.recordFeatureUsage(featureName, userId);

        return true;
    }

    public void updateFeatureFlag(FeatureFlag flag) {
        repository.save(flag);
    }
}

Implementing Feature Flags in a React Frontend

Frontend applications also benefit from feature flags. Here's how to implement them in React:

// feature-flags.js
import { useState, useEffect, createContext, useContext } from 'react';

const FeatureFlagContext = createContext({});

export function FeatureFlagProvider({ children }) {
  const [flags, setFlags] = useState({});
  const [loading, setLoading] = useState(true);

  useEffect(() => {
    async function fetchFlags() {
      try {
        const response = await fetch('/api/feature-flags');
        const data = await response.json();
        setFlags(data);
      } catch (error) {
        console.error('Failed to fetch feature flags:', error);
      } finally {
        setLoading(false);
      }
    }

    fetchFlags();
  }, []);

  return (
    <FeatureFlagContext.Provider value={{ flags, loading }}>
      {children}
    </FeatureFlagContext.Provider>
  );
}

export function useFeatureFlag(flagName) {
  const { flags, loading } = useContext(FeatureFlagContext);
  return {
    enabled: flags[flagName] === true,
    loading
  };
}

Using the feature flag in a component:

import { useFeatureFlag } from './feature-flags';

function SearchComponent() {
  const { enabled: newSearchEnabled, loading } = useFeatureFlag('new-search-algorithm');

  if (loading) return <div>Loading...</div>;

  return (
    <div>
      {newSearchEnabled ? (
        <NewSearchInterface />
      ) : (
        <LegacySearchInterface />
      )}
    </div>
  );
}

Feature Flag Management Systems

While custom implementations work for smaller teams, larger organizations typically use dedicated feature flag management systems:

LaunchDarkly: Enterprise-grade feature management
Split.io: Feature experimentation platform
Flagsmith: Open-source feature flag solution
Unleash: Self-hosted feature toggle service

Let's look at integrating LaunchDarkly into a Node.js application:

const LaunchDarkly = require('launchdarkly-node-server-sdk');

// Initialize the LaunchDarkly client
const ldClient = LaunchDarkly.init('sdk-key-123abc');

ldClient.once('ready', () => {
  console.log('LaunchDarkly SDK is ready');
});

async function checkFeatureFlag(userId, flagName) {
  // Wait for client initialization
  await ldClient.waitForInitialization();

  // Create a user context
  const user = {
    key: userId,
    email: `${userId}@example.com`,
    custom: {
      groups: ['beta-testers'],
      plan: 'premium'
    }
  };

  // Evaluate the flag
  const flagValue = await ldClient.variation(flagName, user, false);

  return flagValue;
}

// Example usage
app.get('/api/checkout', async (req, res) => {
  const userId = req.user.id;
  const useNewCheckout = await checkFeatureFlag(userId, 'new-checkout-flow');

  if (useNewCheckout) {
    return res.json(newCheckoutProcess(req.body));
  } else {
    return res.json(legacyCheckoutProcess(req.body));
  }
});

Implementing Experimentation with Feature Flags

Feature flags enable powerful experimentation capabilities. Here's how to implement A/B testing:

@Service
public class ExperimentService {
    private final FeatureFlagService featureFlagService;
    private final AnalyticsService analyticsService;

    @Autowired
    public ExperimentService(
            FeatureFlagService featureFlagService,
            AnalyticsService analyticsService) {
        this.featureFlagService = featureFlagService;
        this.analyticsService = analyticsService;
    }

    public String getExperimentVariant(String experimentName, String userId) {
        Experiment experiment = experimentRepository.findByName(experimentName)
            .orElseThrow(() -> new ExperimentNotFoundException(experimentName));

        if (!experiment.isActive()) {
            return "control";
        }

        // Deterministic variant assignment based on user ID
        int variantIndex = Math.abs(userId.hashCode() % experiment.getVariants().size());
        String variant = experiment.getVariants().get(variantIndex);

        // Track exposure to experiment
        analyticsService.trackExperimentExposure(experimentName, variant, userId);

        return variant;
    }

    public void recordConversion(String experimentName, String userId) {
        String variant = getExperimentVariant(experimentName, userId);
        analyticsService.trackExperimentConversion(experimentName, variant, userId);
    }
}

Using this in a controller:

@RestController
@RequestMapping("/api/recommendations")
public class RecommendationController {
    private final RecommendationService recommendationService;
    private final ExperimentService experimentService;

    @Autowired
    public RecommendationController(
            RecommendationService recommendationService,
            ExperimentService experimentService) {
        this.recommendationService = recommendationService;
        this.experimentService = experimentService;
    }

    @GetMapping
    public List<Recommendation> getRecommendations(@RequestParam String userId) {
        String variant = experimentService.getExperimentVariant("recommendation-algorithm", userId);

        List<Recommendation> recommendations;
        switch (variant) {
            case "collaborative-filtering":
                recommendations = recommendationService.getCollaborativeFilteringRecommendations(userId);
                break;
            case "content-based":
                recommendations = recommendationService.getContentBasedRecommendations(userId);
                break;
            default: // control
                recommendations = recommendationService.getStandardRecommendations(userId);
        }

        return recommendations;
    }

    @PostMapping("/{recommendationId}/click")
    public void trackClick(@PathVariable String recommendationId, @RequestParam String userId) {
        // Record conversion for the experiment
        experimentService.recordConversion("recommendation-algorithm", userId);

        // Normal click tracking logic
        recommendationService.trackClick(recommendationId, userId);
    }
}

Feature Flag Best Practices

After implementing feature flags at multiple companies, I've identified these best practices:

1. Flag Naming Conventions

Establish a consistent naming convention:

[team]-[feature]-[purpose]

Examples: - search-autocomplete-release - payments-crypto-experiment - infra-cache-ops

2. Flag Lifecycle Management

Feature flags should be temporary. Implement a process to clean up flags:

@Scheduled(cron = "0 0 0 * * *") // Run daily at midnight
public void cleanupStaleFlags() {
    List<FeatureFlag> staleFlags = featureFlagRepository.findByLastUpdatedBefore(
        LocalDateTime.now().minusDays(90));

    for (FeatureFlag flag : staleFlags) {
        if (flag.isEnabled()) {
            // Send notification about enabled stale flag
            notificationService.sendStaleFeatureFlagAlert(flag);
        } else {
            // Archive disabled stale flags
            flag.setArchived(true);
            featureFlagRepository.save(flag);

            // Send notification about archived flag
            notificationService.sendArchivedFeatureFlagNotification(flag);
        }
    }
}

3. Testing with Feature Flags

Feature flags complicate testing. Here's how to handle them in tests:

@Test
public void testCheckoutWithNewFlowEnabled() {
    // Mock feature flag service
    when(featureFlagService.isFeatureEnabled("new-checkout-flow", anyString()))
        .thenReturn(true);

    // Test the new checkout flow
    CheckoutResult result = checkoutService.processCheckout(createSampleOrder(), "user123");

    // Assertions for new flow
    assertEquals(CheckoutStatus.COMPLETED, result.getStatus());
    assertTrue(result.isExpressShippingAvailable());
}

@Test
public void testCheckoutWithNewFlowDisabled() {
    // Mock feature flag service
    when(featureFlagService.isFeatureEnabled("new-checkout-flow", anyString()))
        .thenReturn(false);

    // Test the legacy checkout flow
    CheckoutResult result = checkoutService.processCheckout(createSampleOrder(), "user123");

    // Assertions for legacy flow
    assertEquals(CheckoutStatus.PENDING, result.getStatus());
    assertFalse(result.isExpressShippingAvailable());
}

4. Monitoring and Alerting

Monitor feature flag usage and set up alerts for unexpected behavior:

@Aspect
@Component
public class FeatureFlagMonitoringAspect {
    private final MetricsService metricsService;
    private final AlertService alertService;

    @Autowired
    public FeatureFlagMonitoringAspect(
            MetricsService metricsService,
            AlertService alertService) {
        this.metricsService = metricsService;
        this.alertService = alertService;
    }

    @Around("execution(* com.example.service.FeatureFlagService.isFeatureEnabled(..)) && args(featureName, userId)")
    public Object monitorFeatureFlag(ProceedingJoinPoint joinPoint, String featureName, String userId) throws Throwable {
        long startTime = System.currentTimeMillis();
        boolean result = false;

        try {
            result = (boolean) joinPoint.proceed();
            return result;
        } finally {
            long executionTime = System.currentTimeMillis() - startTime;

            // Record metrics
            metricsService.recordFeatureFlagCheck(featureName, result, executionTime);

            // Check for anomalies
            if (executionTime > 100) { // More than 100ms is slow
                alertService.sendSlowFeatureFlagAlert(featureName, executionTime);
            }
        }
    }
}

Real-World Feature Flag Architecture

For large-scale applications, a comprehensive feature flag architecture includes:

Flag Management Service: Central service for flag configuration
Flag Evaluation Service: High-performance service for flag evaluation
Flag SDK: Client libraries for different platforms
Analytics Integration: Track feature usage and experiment results
Admin UI: Interface for managing flags

Here's a diagram of this architecture:

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│                 │     │                 │     │                 │
│  Admin UI       │────▶│  Flag           │◀────│  Analytics      │
│                 │     │  Management     │     │  Service        │
└─────────────────┘     │  Service        │     └─────────────────┘
                        │                 │
                        └────────┬────────┘
                                 │
                                 ▼
                        ┌─────────────────┐
                        │                 │
                        │  Flag           │
                        │  Evaluation     │
                        │  Service        │
                        │                 │
                        └────────┬────────┘
                                 │
                 ┌───────────────┼───────────────┐
                 │               │               │
                 ▼               ▼               ▼
        ┌─────────────┐  ┌─────────────┐  ┌─────────────┐
        │             │  │             │  │             │
        │  Web SDK    │  │  Mobile SDK │  │  Server SDK │
        │             │  │             │  │             │
        └─────────────┘  └─────────────┘  └─────────────┘

Case Study: Gradual Database Migration

One powerful use case for feature flags is database migrations. At a fintech company, we used feature flags to migrate from MongoDB to PostgreSQL without downtime:

@Service
public class UserDataService {
    private final MongoUserRepository mongoRepo;
    private final PostgresUserRepository postgresRepo;
    private final FeatureFlagService featureFlagService;

    @Autowired
    public UserDataService(
            MongoUserRepository mongoRepo,
            PostgresUserRepository postgresRepo,
            FeatureFlagService featureFlagService) {
        this.mongoRepo = mongoRepo;
        this.postgresRepo = postgresRepo;
        this.featureFlagService = featureFlagService;
    }

    public User getUserById(String userId) {
        boolean usePostgres = featureFlagService.isFeatureEnabled("use-postgres-db", userId);

        if (usePostgres) {
            return postgresRepo.findById(userId)
                .orElseGet(() -> {
                    // Fallback to MongoDB if not found in Postgres
                    User user = mongoRepo.findById(userId).orElse(null);
                    if (user != null) {
                        // Backfill to Postgres
                        postgresRepo.save(user);
                    }
                    return user;
                });
        } else {
            return mongoRepo.findById(userId).orElse(null);
        }
    }

    public User saveUser(User user) {
        boolean usePostgres = featureFlagService.isFeatureEnabled("use-postgres-db", user.getId());

        // Always save to MongoDB during migration
        mongoRepo.save(user);

        // Conditionally save to Postgres
        if (usePostgres) {
            postgresRepo.save(user);
        }

        return user;
    }
}

This approach allowed us to: 1. Start with 0% of traffic on PostgreSQL 2. Gradually increase to 100% over two weeks 3. Monitor performance and errors at each step 4. Roll back instantly if issues occurred

The migration was completed with zero downtime and no customer impact.

Conclusion

Feature flags have evolved from simple if/else statements to sophisticated systems that enable controlled rollouts, experimentation, and operational flexibility. Implementing them effectively requires careful planning, proper architecture, and disciplined management.

Start small with a basic implementation, then expand as your needs grow. Remember that feature flags introduce complexity, so use them judiciously and clean them up when they're no longer needed.

By following the patterns and practices outlined in this post, you can build a feature flag system that empowers your team to deploy with confidence, experiment with new ideas, and respond quickly to changing requirements. ```

Implementing feature flags for controlled rollouts and experimentation in production