Article
In today's distributed systems, services depend on numerous external dependencies—databases, APIs, third-party services, and caches. While these dependencies work flawlessly in ideal conditions, real-world networks introduce latency, timeouts, and slowdowns. Latency Injection is a crucial fault-tolerance practice that deliberately introduces artificial delays into a system to test, validate, and improve its resilience under non-ideal conditions.
This article will explore why latency injection matters, various implementation strategies in Java, and how to integrate it into your development and testing workflows.
What is Latency Injection and Why Use It?
Latency injection is the practice of artificially adding delays to method calls, network requests, or other operations to simulate real-world network conditions and service degradation.
Key Benefits:
- Resilience Testing: Verify that your system handles slow responses gracefully without cascading failures.
- Timeout Validation: Test whether your timeout configurations are appropriate.
- Circuit Breaker Verification: Ensure circuit breakers trip correctly under slow response conditions.
- Performance Profiling: Understand how your application behaves under load with slow dependencies.
- Chaos Engineering: Proactively discover weaknesses before they cause production incidents.
Implementation Strategies in Java
Here are several practical approaches to implement latency injection, from simple manual techniques to sophisticated framework-based solutions.
1. Manual Latency Injection
The most straightforward approach using Thread.sleep() or more precise timing mechanisms.
public class ManualLatencyInjector {
public static void injectLatency(long milliseconds) {
try {
Thread.sleep(milliseconds);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new RuntimeException("Latency injection interrupted", e);
}
}
public static <T> T withLatency(Callable<T> operation, long latencyMs) throws Exception {
injectLatency(latencyMs);
return operation.call();
}
}
// Usage example
public class PaymentService {
public PaymentResult processPayment(PaymentRequest request) {
// Simulate network latency to payment gateway
ManualLatencyInjector.injectLatency(150); // 150ms delay
// Actual payment processing logic
return paymentGateway.charge(request);
}
}
2. Configurable Latency Injection
A more flexible approach that allows runtime configuration.
@Component
public class ConfigurableLatencyInjector {
private final LatencyConfig config;
private final Random random = new Random();
public ConfigurableLatencyInjector(LatencyConfig config) {
this.config = config;
}
public void injectLatency(String operationName) {
if (!config.isEnabled()) {
return;
}
LatencyProfile profile = config.getProfile(operationName);
if (profile != null && shouldInject(profile.getProbability())) {
long delay = calculateDelay(profile);
sleepUninterruptibly(delay, TimeUnit.MILLISECONDS);
}
}
private boolean shouldInject(double probability) {
return random.nextDouble() < probability;
}
private long calculateDelay(LatencyProfile profile) {
long baseDelay = profile.getBaseLatencyMs();
long jitter = (long) (random.nextDouble() * profile.getJitterMs());
return baseDelay + jitter;
}
private void sleepUninterruptibly(long duration, TimeUnit unit) {
try {
Thread.sleep(unit.toMillis(duration));
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
// Configuration classes
@Configuration
@ConfigurationProperties(prefix = "latency")
public class LatencyConfig {
private boolean enabled = false;
private Map<String, LatencyProfile> profiles = new HashMap<>();
// Getters and setters
public boolean isEnabled() { return enabled; }
public void setEnabled(boolean enabled) { this.enabled = enabled; }
public Map<String, LatencyProfile> getProfiles() { return profiles; }
public void setProfiles(Map<String, LatencyProfile> profiles) { this.profiles = profiles; }
public LatencyProfile getProfile(String operationName) {
return profiles.get(operationName);
}
}
public class LatencyProfile {
private long baseLatencyMs;
private long jitterMs;
private double probability = 1.0; // 0.0 to 1.0
// Constructors, getters, setters
public LatencyProfile(long baseLatencyMs, long jitterMs, double probability) {
this.baseLatencyMs = baseLatencyMs;
this.jitterMs = jitterMs;
this.probability = probability;
}
// ... getters and setters
}
Configuration in application.yml:
latency: enabled: true profiles: database-query: base-latency-ms: 100 jitter-ms: 50 probability: 0.3 # 30% chance to inject latency external-api-call: base-latency-ms: 300 jitter-ms: 200 probability: 0.5 cache-lookup: base-latency-ms: 10 jitter-ms: 5 probability: 0.1
3. Aspect-Oriented Programming (AOP) Approach
Using Spring AOP to automatically inject latency based on annotations.
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface InjectLatency {
long value() default 100; // default latency in ms
double probability() default 1.0;
}
@Aspect
@Component
@ConditionalOnProperty(name = "latency.injection.enabled", havingValue = "true")
public class LatencyInjectionAspect {
private final Random random = new Random();
@Around("@annotation(injectLatency)")
public Object injectLatency(ProceedingJoinPoint joinPoint, InjectLatency injectLatency)
throws Throwable {
if (shouldInject(injectLatency.probability())) {
long delay = injectLatency.value();
Thread.sleep(delay);
}
return joinPoint.proceed();
}
private boolean shouldInject(double probability) {
return random.nextDouble() < probability;
}
}
// Usage on service methods
@Service
public class UserService {
@InjectLatency(value = 200, probability = 0.2)
public User findUserById(Long userId) {
// This method will have 200ms latency injected 20% of the time
return userRepository.findById(userId).orElse(null);
}
@InjectLatency(50) // 50ms latency, 100% probability
public List<User> findUsersByStatus(String status) {
return userRepository.findByStatus(status);
}
}
4. HTTP Client Latency Injection
Injecting latency at the HTTP client level for external API calls.
@Component
public class LatencyAwareRestTemplate {
private final RestTemplate restTemplate;
private final ConfigurableLatencyInjector latencyInjector;
public LatencyAwareRestTemplate(RestTemplate restTemplate,
ConfigurableLatencyInjector latencyInjector) {
this.restTemplate = restTemplate;
this.latencyInjector = latencyInjector;
}
public <T> T getForObject(String url, Class<T> responseType, Object... uriVariables) {
latencyInjector.injectLatency("http-client");
return restTemplate.getForObject(url, responseType, uriVariables);
}
public <T> ResponseEntity<T> exchange(String url, HttpMethod method,
HttpEntity<?> requestEntity,
Class<T> responseType,
Object... uriVariables) {
latencyInjector.injectLatency("http-client");
return restTemplate.exchange(url, method, requestEntity, responseType, uriVariables);
}
}
// Alternatively, using ClientHttpRequestInterceptor
@Component
public class LatencyInjectionInterceptor implements ClientHttpRequestInterceptor {
private final LatencyConfig config;
private final Random random = new Random();
@Override
public ClientHttpResponse intercept(HttpRequest request, byte[] body,
ClientHttpRequestExecution execution) throws IOException {
// Inject latency based on the target service
String serviceName = extractServiceName(request.getURI());
injectServiceLatency(serviceName);
return execution.execute(request, body);
}
private void injectServiceLatency(String serviceName) {
long latency = calculateLatencyForService(serviceName);
if (latency > 0) {
try {
Thread.sleep(latency);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
private long calculateLatencyForService(String serviceName) {
// Implementation based on configuration
return 150; // Example fixed latency
}
private String extractServiceName(URI uri) {
return uri.getHost();
}
}
5. Database Latency Injection
Using Hibernate interceptors to simulate database latency.
@Component
public class DatabaseLatencyInterceptor extends EmptyInterceptor {
private final LatencyInjector latencyInjector;
@Override
public String onPrepareStatement(String sql) {
latencyInjector.injectLatency("database-query");
return super.onPrepareStatement(sql);
}
}
// Or using Spring Data JPA repositories with AOP
@Repository
public interface UserRepository extends JpaRepository<User, Long> {
@InjectLatency(100)
@Query("SELECT u FROM User u WHERE u.email = :email")
User findByEmail(@Param("email") String email);
}
6. Comprehensive Testing with JUnit 5
Writing tests to verify system behavior under latency conditions.
@SpringBootTest
@TestPropertySource(properties = {
"latency.enabled=true",
"latency.profiles.database-query.base-latency-ms=500",
"latency.profiles.database-query.probability=1.0"
})
public class UserServiceLatencyTest {
@Autowired
private UserService userService;
@Test
public void testUserServiceHandlesLatencyGracefully() {
// Given - latency is enabled via configuration
// When - calling a method that should have latency injected
long startTime = System.currentTimeMillis();
User user = userService.findUserById(1L);
long endTime = System.currentTimeMillis();
// Then - verify the call took at least the expected latency
long duration = endTime - startTime;
assertThat(duration).isGreaterThanOrEqualTo(450); // 500ms ± jitter
assertThat(user).isNotNull();
}
@Test
public void testCircuitBreakerTripsUnderSustainedLatency() {
// Make multiple calls to trigger circuit breaker
for (int i = 0; i < 10; i++) {
try {
userService.findUserById((long) i);
} catch (Exception e) {
// Expected when circuit breaker opens
}
}
// Verify circuit breaker is open
// This depends on your circuit breaker implementation
}
}
7. Production-Grade Latency Injection Framework
For enterprise use, consider building a more sophisticated framework.
@Component
public class ProductionLatencyEngine {
private final Map<String, LatencyStrategy> strategies;
private final LatencyTelemetry telemetry;
public ProductionLatencyEngine(LatencyTelemetry telemetry) {
this.telemetry = telemetry;
this.strategies = Map.of(
"fixed", new FixedLatencyStrategy(),
"gaussian", new GaussianLatencyStrategy(),
"spike", new SpikeLatencyStrategy()
);
}
public <T> T executeWithLatency(String operation, String strategyName,
Callable<T> operation) throws Exception {
LatencyStrategy strategy = strategies.get(strategyName);
long latency = strategy.calculateLatency(operation);
if (latency > 0) {
telemetry.recordLatencyInjection(operation, latency);
sleepUninterruptibly(latency, TimeUnit.MILLISECONDS);
}
return operation.call();
}
}
public interface LatencyStrategy {
long calculateLatency(String operation);
}
public class SpikeLatencyStrategy implements LatencyStrategy {
private final Random random = new Random();
private boolean spikeActive = false;
private long spikeStartTime = 0;
@Override
public long calculateLatency(String operation) {
// 1% chance to start a spike that lasts for 30 seconds
if (!spikeActive && random.nextDouble() < 0.01) {
spikeActive = true;
spikeStartTime = System.currentTimeMillis();
}
// End spike after 30 seconds
if (spikeActive &&
System.currentTimeMillis() - spikeStartTime > 30000) {
spikeActive = false;
}
return spikeActive ? 2000 + random.nextInt(3000) : 0; // 2-5 seconds during spike
}
}
Best Practices and Considerations
- Safety First: Always gate latency injection with feature flags or environment checks. Never enable in production by default.
- Gradual Rollout: Start with low probabilities and small latencies, then gradually increase.
- Monitoring: Track when latency is injected and monitor system metrics during injection periods.
- Clean Separation: Keep latency injection code separate from business logic.
- Team Awareness: Ensure the entire team understands when and why latency injection is active.
When to Use Latency Injection
- Development: Test timeout configurations and circuit breakers locally.
- CI/CD Pipelines: Automated resilience testing.
- Staging Environments: Validate system behavior before production.
- Chaos Engineering: Controlled experiments in production-like environments.
Conclusion
Latency injection is a powerful technique for building and validating resilient Java applications. By systematically introducing artificial delays, you can proactively discover weaknesses, validate fault-tolerance mechanisms, and build confidence in your system's ability to handle real-world network conditions.
The approaches range from simple Thread.sleep() calls to sophisticated framework-based solutions that can be controlled via configuration. By integrating latency injection into your development workflow, you shift resilience testing from being an afterthought to a first-class concern, ultimately leading to more robust and reliable distributed systems.