Network partition testing (also known as split-brain testing) is crucial for ensuring distributed systems can handle network failures gracefully. It involves simulating network failures between system components to verify fault tolerance and recovery mechanisms.
Core Concepts
What are Network Partitions?
- Network Isolation: When network segments become disconnected
- Split-Brain: Multiple components think they're the primary/leader
- Partition Tolerance: System's ability to continue operating during partitions
CAP Theorem Implications
- Consistency: All nodes see the same data
- Availability: Every request receives a response
- Partition Tolerance: System continues despite network partitions
Testing Approaches and Tools
1. Using Toxiproxy for Network Testing
Dependencies
<dependency> <groupId>org.testcontainers</groupId> <artifactId>toxiproxy</artifactId> <version>1.19.0</version> <scope>test</scope> </dependency> <dependency> <groupId>eu.rekawek.toxiproxy</groupId> <artifactId>toxiproxy-java</artifactId> <version>2.1.7</version> <scope>test</scope> </dependency>
Example 1: Basic Toxiproxy Testing
import eu.rekawek.toxiproxy.Proxy;
import eu.rekawek.toxiproxy.ToxiproxyClient;
import eu.rekawek.toxiproxy.model.ToxicDirection;
import org.junit.jupiter.api.Test;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.containers.Network;
import org.testcontainers.containers.ToxiproxyContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import java.io.IOException;
@Testcontainers
public class ToxiproxyNetworkPartitionTest {
private static final Network network = Network.newNetwork();
@Container
private static final ToxiproxyContainer toxiproxy = new ToxiproxyContainer("ghcr.io/shopify/toxiproxy:2.5.0")
.withNetwork(network);
@Container
private static final GenericContainer<?> redis = new GenericContainer<>("redis:7-alpine")
.withNetwork(network)
.withNetworkAliases("redis")
.withExposedPorts(6379);
@Test
void testRedisConnectionDuringNetworkPartition() throws IOException {
// Create Toxiproxy client
ToxiproxyClient client = new ToxiproxyClient(toxiproxy.getHost(), toxiproxy.getControlPort());
// Create proxy for Redis
Proxy proxy = client.createProxy("redis-proxy",
"0.0.0.0:8666",
"redis:6379");
// Get proxied Redis connection details
String redisHost = toxiproxy.getHost();
int redisPort = toxiproxy.getMappedPort(8666);
// Test normal connection
RedisClient redisClient = new RedisClient(redisHost, redisPort);
assertThat(redisClient.ping()).isTrue();
// Simulate network partition - add latency and packet loss
proxy.toxics()
.latency("network-latency", ToxicDirection.DOWNSTREAM, 5000) // 5 seconds latency
.setJitter(1000);
proxy.toxics()
.timeout("connection-timeout", ToxicDirection.DOWNSTREAM, 10000); // 10 seconds timeout
// Test behavior during partition
assertThatThrownBy(() -> redisClient.set("key", "value"))
.isInstanceOf(RedisConnectionException.class);
// Remove network issues
proxy.toxics().get("network-latency").remove();
proxy.toxics().get("connection-timeout").remove();
// Test recovery
await().atMost(30, TimeUnit.SECONDS)
.untilAsserted(() -> {
assertThat(redisClient.ping()).isTrue();
redisClient.set("key", "value");
assertThat(redisClient.get("key")).isEqualTo("value");
});
}
}
2. Custom Network Partition Simulator
Example 2: Java-based Network Partition Simulator
@Component
@Slf4j
public class NetworkPartitionSimulator {
private final Map<String, PartitionRule> activePartitions = new ConcurrentHashMap<>();
private final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(5);
public static class PartitionRule {
private final String sourceComponent;
private final String targetComponent;
private final PartitionType type;
private final Duration duration;
private final double packetLossPercentage;
private final Duration latency;
public PartitionRule(String sourceComponent, String targetComponent,
PartitionType type, Duration duration) {
this(sourceComponent, targetComponent, type, duration, 0.0, Duration.ZERO);
}
public PartitionRule(String sourceComponent, String targetComponent,
PartitionType type, Duration duration,
double packetLossPercentage, Duration latency) {
this.sourceComponent = sourceComponent;
this.targetComponent = targetComponent;
this.type = type;
this.duration = duration;
this.packetLossPercentage = packetLossPercentage;
this.latency = latency;
}
// getters
}
public enum PartitionType {
COMPLETE, // No communication
LATENCY, // Delayed communication
PACKET_LOSS, // Random packet loss
UNRELIABLE // Combination of issues
}
@PreDestroy
public void cleanup() {
scheduler.shutdown();
try {
if (!scheduler.awaitTermination(30, TimeUnit.SECONDS)) {
scheduler.shutdownNow();
}
} catch (InterruptedException e) {
scheduler.shutdownNow();
Thread.currentThread().interrupt();
}
}
public String createPartition(PartitionRule rule) {
String partitionId = UUID.randomUUID().toString();
activePartitions.put(partitionId, rule);
log.info("Created network partition: {} between {} and {}",
partitionId, rule.getSourceComponent(), rule.getTargetComponent());
// Schedule automatic removal if duration is specified
if (!rule.getDuration().isZero()) {
scheduler.schedule(() -> removePartition(partitionId),
rule.getDuration().toMillis(), TimeUnit.MILLISECONDS);
}
return partitionId;
}
public void removePartition(String partitionId) {
PartitionRule removed = activePartitions.remove(partitionId);
if (removed != null) {
log.info("Removed network partition: {} between {} and {}",
partitionId, removed.getSourceComponent(), removed.getTargetComponent());
}
}
public boolean shouldBlockCommunication(String fromComponent, String toComponent) {
return activePartitions.values().stream()
.anyMatch(rule -> matchesRule(rule, fromComponent, toComponent));
}
public boolean shouldDelayCommunication(String fromComponent, String toComponent) {
return activePartitions.values().stream()
.filter(rule -> matchesRule(rule, fromComponent, toComponent))
.anyMatch(rule -> rule.getType() == PartitionType.LATENCY ||
rule.getType() == PartitionType.UNRELIABLE);
}
public Duration getCommunicationDelay(String fromComponent, String toComponent) {
return activePartitions.values().stream()
.filter(rule -> matchesRule(rule, fromComponent, toComponent))
.filter(rule -> rule.getLatency() != null && !rule.getLatency().isZero())
.map(PartitionRule::getLatency)
.findFirst()
.orElse(Duration.ZERO);
}
public boolean shouldDropPacket(String fromComponent, String toComponent) {
return activePartitions.values().stream()
.filter(rule -> matchesRule(rule, fromComponent, toComponent))
.filter(rule -> rule.getPacketLossPercentage() > 0.0)
.anyMatch(rule -> Math.random() < rule.getPacketLossPercentage() / 100.0);
}
private boolean matchesRule(PartitionRule rule, String from, String to) {
return (rule.getSourceComponent().equals(from) && rule.getTargetComponent().equals(to)) ||
(rule.getSourceComponent().equals(to) && rule.getTargetComponent().equals(from));
}
public List<PartitionRule> getActivePartitions() {
return new ArrayList<>(activePartitions.values());
}
public void clearAllPartitions() {
log.info("Clearing all network partitions");
activePartitions.clear();
}
}
// HTTP Client with Partition Awareness
@Component
@Slf4j
public class PartitionAwareHttpClient {
private final NetworkPartitionSimulator partitionSimulator;
private final RestTemplate restTemplate;
private final ObjectMapper objectMapper;
public PartitionAwareHttpClient(NetworkPartitionSimulator partitionSimulator,
RestTemplate restTemplate,
ObjectMapper objectMapper) {
this.partitionSimulator = partitionSimulator;
this.restTemplate = restTemplate;
this.objectMapper = objectMapper;
}
public <T> T executeWithPartitionAwareness(String serviceName, String url,
HttpMethod method, Object request,
Class<T> responseType) {
String currentService = getCurrentServiceName();
// Check if communication should be blocked
if (partitionSimulator.shouldBlockCommunication(currentService, serviceName)) {
log.warn("Network partition blocked communication from {} to {}",
currentService, serviceName);
throw new NetworkPartitionException(
"Network partition prevents communication to " + serviceName);
}
// Check for packet loss
if (partitionSimulator.shouldDropPacket(currentService, serviceName)) {
log.warn("Packet loss simulated from {} to {}", currentService, serviceName);
throw new NetworkPartitionException(
"Packet loss simulated for communication to " + serviceName);
}
// Apply latency if needed
Duration delay = partitionSimulator.getCommunicationDelay(currentService, serviceName);
if (!delay.isZero()) {
log.info("Applying network latency of {}ms from {} to {}",
delay.toMillis(), currentService, serviceName);
try {
Thread.sleep(delay.toMillis());
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new NetworkPartitionException("Latency simulation interrupted");
}
}
// Execute the actual HTTP call
try {
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
HttpEntity<Object> entity = new HttpEntity<>(request, headers);
ResponseEntity<String> response = restTemplate.exchange(
url, method, entity, String.class);
return objectMapper.readValue(response.getBody(), responseType);
} catch (ResourceAccessException e) {
log.error("Network error calling {}: {}", serviceName, e.getMessage());
throw new NetworkPartitionException("Network error: " + e.getMessage(), e);
} catch (Exception e) {
log.error("Unexpected error calling {}: {}", serviceName, e.getMessage());
throw new RuntimeException("Service call failed", e);
}
}
private String getCurrentServiceName() {
// In real implementation, this would come from configuration
return System.getProperty("service.name", "unknown-service");
}
}
// Custom Exception
class NetworkPartitionException extends RuntimeException {
public NetworkPartitionException(String message) {
super(message);
}
public NetworkPartitionException(String message, Throwable cause) {
super(message, cause);
}
}
3. Database Partition Testing
Example 3: Database Connection Partition Testing
@SpringBootTest
@Testcontainers
@Slf4j
public class DatabasePartitionTest {
@Container
private static final PostgreSQLContainer<?> postgres =
new PostgreSQLContainer<>("postgres:15-alpine");
@Container
private static final ToxiproxyContainer toxiproxy =
new ToxiproxyContainer("ghcr.io/shopify/toxiproxy:2.5.0");
private static Proxy dbProxy;
@Autowired
private DataSource dataSource;
@BeforeAll
static void setup() throws Exception {
// Create proxy for database
ToxiproxyClient client = new ToxiproxyClient(
toxiproxy.getHost(), toxiproxy.getControlPort());
dbProxy = client.createProxy("postgres-proxy",
"0.0.0.0:8666",
postgres.getHost() + ":" + postgres.getFirstMappedPort());
}
@DynamicPropertySource
static void configureProperties(DynamicPropertyRegistry registry) {
// Use proxied database connection
registry.add("spring.datasource.url", () ->
String.format("jdbc:postgresql://%s:%d/%s",
toxiproxy.getHost(),
toxiproxy.getMappedPort(8666),
postgres.getDatabaseName()));
registry.add("spring.datasource.username", postgres::getUsername);
registry.add("spring.datasource.password", postgres::getPassword);
}
@Test
void testDatabaseOperationsDuringNetworkPartition() throws Exception {
// Setup test data
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
jdbcTemplate.execute("CREATE TABLE IF NOT EXISTS test_table (id SERIAL PRIMARY KEY, data VARCHAR)");
jdbcTemplate.execute("INSERT INTO test_table (data) VALUES ('test-data')");
// Test normal operation
String result = jdbcTemplate.queryForObject(
"SELECT data FROM test_table WHERE id = 1", String.class);
assertThat(result).isEqualTo("test-data");
// Simulate network partition - high latency and timeout
dbProxy.toxics()
.latency("db-latency", ToxicDirection.DOWNSTREAM, 10000) // 10 seconds
.setJitter(2000);
dbProxy.toxics()
.timeout("db-timeout", ToxicDirection.DOWNSTREAM, 5000); // 5 seconds timeout
// Test behavior during partition
long startTime = System.currentTimeMillis();
assertThatThrownBy(() -> {
jdbcTemplate.queryForObject(
"SELECT data FROM test_table WHERE id = 1", String.class);
}).isInstanceOf(DataAccessResourceFailureException.class);
long duration = System.currentTimeMillis() - startTime;
assertThat(duration).isGreaterThanOrEqualTo(4900); // Should timeout around 5 seconds
// Remove network issues
dbProxy.toxics().get("db-latency").remove();
dbProxy.toxics().get("db-timeout").remove();
// Test recovery
await().atMost(30, TimeUnit.SECONDS)
.untilAsserted(() -> {
String recoveredResult = jdbcTemplate.queryForObject(
"SELECT data FROM test_table WHERE id = 1", String.class);
assertThat(recoveredResult).isEqualTo("test-data");
});
}
@Test
void testTransactionBehaviorDuringPartition() throws Exception {
TransactionTemplate transactionTemplate =
new TransactionTemplate(transactionManager);
// Simulate partition during transaction commit
dbProxy.toxics()
.latency("commit-latency", ToxicDirection.DOWNSTREAM, 15000);
assertThatThrownBy(() -> {
transactionTemplate.execute(status -> {
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
jdbcTemplate.execute("INSERT INTO test_table (data) VALUES ('transaction-test')");
// This should fail due to timeout
return null;
});
}).isInstanceOf(TransactionTimedOutException.class);
// Verify transaction was rolled back
dbProxy.toxics().get("commit-latency").remove();
Integer count = new JdbcTemplate(dataSource).queryForObject(
"SELECT COUNT(*) FROM test_table WHERE data = 'transaction-test'", Integer.class);
assertThat(count).isZero(); // Transaction should have rolled back
}
}
4. Microservices Partition Testing
Example 4: Spring Cloud Microservices Partition Testing
@SpringBootTest
@ActiveProfiles("test")
@Testcontainers
@Slf4j
public class MicroservicesPartitionTest {
@Container
private static final ToxiproxyContainer toxiproxy =
new ToxiproxyContainer("ghcr.io/shopify/toxiproxy:2.5.0");
@Container
private static final GenericContainer<?> userService =
new GenericContainer<>("user-service:latest")
.withNetwork(Network.SHARED)
.withNetworkAliases("user-service")
.withExposedPorts(8080);
@Container
private static final GenericContainer<?> orderService =
new GenericContainer<>("order-service:latest")
.withNetwork(Network.SHARED)
.withNetworkAliases("order-service")
.withExposedPorts(8080);
@Container
private static final GenericContainer<?> paymentService =
new GenericContainer<>("payment-service:latest")
.withNetwork(Network.SHARED)
.withNetworkAliases("payment-service")
.withExposedPorts(8080);
private static Proxy userServiceProxy;
private static Proxy orderServiceProxy;
private static Proxy paymentServiceProxy;
@BeforeAll
static void setupProxies() throws Exception {
ToxiproxyClient client = new ToxiproxyClient(
toxiproxy.getHost(), toxiproxy.getControlPort());
userServiceProxy = client.createProxy("user-service-proxy",
"0.0.0.0:8667", "user-service:8080");
orderServiceProxy = client.createProxy("order-service-proxy",
"0.0.0.0:8668", "order-service:8080");
paymentServiceProxy = client.createProxy("payment-service-proxy",
"0.0.0.0:8669", "payment-service:8080");
}
@Test
void testOrderProcessingDuringServicePartition() throws Exception {
// Create partition between order service and payment service
paymentServiceProxy.toxics()
.latency("payment-partition", ToxicDirection.DOWNSTREAM, 30000) // 30 seconds
.timeout("payment-timeout", ToxicDirection.DOWNSTREAM, 10000); // 10 seconds timeout
// Test order creation
OrderServiceClient orderClient = new OrderServiceClient(
"http://" + toxiproxy.getHost() + ":" + toxiproxy.getMappedPort(8668));
CreateOrderRequest orderRequest = new CreateOrderRequest(
"user-123", List.of("item-1", "item-2"), new BigDecimal("99.99"));
// This should fail due to payment service being unavailable
assertThatThrownBy(() -> orderClient.createOrder(orderRequest))
.isInstanceOf(ServiceUnavailableException.class);
// Verify circuit breaker pattern
await().atMost(10, TimeUnit.SECONDS)
.until(() -> orderClient.getCircuitBreakerState() == CircuitBreakerState.OPEN);
// Remove partition
paymentServiceProxy.toxics().get("payment-partition").remove();
paymentServiceProxy.toxics().get("payment-timeout").remove();
// Verify recovery and circuit breaker closing
await().atMost(30, TimeUnit.SECONDS)
.until(() -> orderClient.getCircuitBreakerState() == CircuitBreakerState.CLOSED);
// Test successful order creation after recovery
OrderResponse order = orderClient.createOrder(orderRequest);
assertThat(order.getStatus()).isEqualTo(OrderStatus.COMPLETED);
}
@Test
void testDataConsistencyDuringPartition() throws Exception {
// Simulate partition between services and their databases
userServiceProxy.toxics()
.latency("user-db-partition", ToxicDirection.DOWNSTREAM, 15000);
// Try to update user profile
UserServiceClient userClient = new UserServiceClient(
"http://" + toxiproxy.getHost() + ":" + toxiproxy.getMappedPort(8667));
UpdateUserRequest updateRequest = new UpdateUserRequest("user-456", "[email protected]");
// This should timeout
assertThatThrownBy(() -> userClient.updateUser(updateRequest))
.isInstanceOf(ServiceUnavailableException.class);
// Remove partition
userServiceProxy.toxics().get("user-db-partition").remove();
// Verify eventual consistency
await().atMost(30, TimeUnit.SECONDS)
.untilAsserted(() -> {
UserProfile profile = userClient.getUser("user-456");
assertThat(profile.getEmail()).isEqualTo("[email protected]");
});
}
}
// Service Client with Resilience Patterns
@Component
@Slf4j
public class OrderServiceClient {
private final RestTemplate restTemplate;
private final CircuitBreaker circuitBreaker;
private final Retry retry;
private final String serviceUrl;
public OrderServiceClient(@Value("${order.service.url}") String serviceUrl,
RestTemplate restTemplate,
CircuitBreakerRegistry circuitBreakerRegistry,
RetryRegistry retryRegistry) {
this.serviceUrl = serviceUrl;
this.restTemplate = restTemplate;
this.circuitBreaker = circuitBreakerRegistry.circuitBreaker("order-service");
this.retry = retryRegistry.retry("order-service");
}
public OrderResponse createOrder(CreateOrderRequest request) {
return circuitBreaker.executeSupplier(() ->
retry.executeSupplier(() -> {
try {
ResponseEntity<OrderResponse> response = restTemplate.postForEntity(
serviceUrl + "/api/orders", request, OrderResponse.class);
return response.getBody();
} catch (ResourceAccessException e) {
log.warn("Network error calling order service: {}", e.getMessage());
throw new ServiceUnavailableException("Order service unavailable", e);
}
})
);
}
public CircuitBreakerState getCircuitBreakerState() {
return circuitBreaker.getState();
}
}
5. Advanced Partition Scenarios
Example 5: Chaos Engineering with Custom Partitions
@Service
@Slf4j
public class ChaosEngineeringService {
private final NetworkPartitionSimulator partitionSimulator;
private final ScheduledExecutorService chaosExecutor;
private final Random random = new Random();
public ChaosEngineeringService(NetworkPartitionSimulator partitionSimulator) {
this.partitionSimulator = partitionSimulator;
this.chaosExecutor = Executors.newScheduledThreadPool(3);
}
public void startRandomPartitions(ChaosConfig config) {
log.info("Starting chaos engineering with config: {}", config);
// Schedule random partitions
chaosExecutor.scheduleAtFixedRate(() -> {
if (random.nextDouble() < config.getPartitionProbability()) {
createRandomPartition(config);
}
}, 0, config.getIntervalSeconds(), TimeUnit.SECONDS);
}
public void stopChaos() {
log.info("Stopping chaos engineering");
chaosExecutor.shutdown();
partitionSimulator.clearAllPartitions();
}
private void createRandomPartition(ChaosConfig config) {
List<String> services = config.getTargetServices();
if (services.size() < 2) return;
// Pick two random services to partition
String service1 = services.get(random.nextInt(services.size()));
String service2;
do {
service2 = services.get(random.nextInt(services.size()));
} while (service1.equals(service2));
PartitionType type = getRandomPartitionType();
Duration duration = Duration.ofSeconds(
config.getMinPartitionDuration() +
random.nextInt(config.getMaxPartitionDuration() - config.getMinPartitionDuration()));
double packetLoss = type == PartitionType.PACKET_LOSS || type == PartitionType.UNRELIABLE ?
random.nextDouble() * 50.0 : 0.0; // Up to 50% packet loss
Duration latency = type == PartitionType.LATENCY || type == PartitionType.UNRELIABLE ?
Duration.ofMillis(1000 + random.nextInt(9000)) : Duration.ZERO; // 1-10 seconds latency
PartitionRule rule = new PartitionRule(service1, service2, type, duration, packetLoss, latency);
partitionSimulator.createPartition(rule);
log.info("Created random partition: {} <-> {} (type: {}, duration: {}s)",
service1, service2, type, duration.getSeconds());
}
private PartitionType getRandomPartitionType() {
PartitionType[] types = PartitionType.values();
return types[random.nextInt(types.length)];
}
@PreDestroy
public void cleanup() {
stopChaos();
}
}
// Chaos Configuration
@Data
@ConfigurationProperties(prefix = "chaos")
public class ChaosConfig {
private boolean enabled = false;
private double partitionProbability = 0.1; // 10% chance
private int intervalSeconds = 60; // Check every minute
private int minPartitionDuration = 30; // seconds
private int maxPartitionDuration = 300; // seconds
private List<String> targetServices = List.of();
}
// Chaos Controller for Testing
@RestController
@RequestMapping("/api/chaos")
@Slf4j
@ConditionalOnProperty(name = "chaos.enabled", havingValue = "true")
public class ChaosController {
private final ChaosEngineeringService chaosService;
private final NetworkPartitionSimulator partitionSimulator;
public ChaosController(ChaosEngineeringService chaosService,
NetworkPartitionSimulator partitionSimulator) {
this.chaosService = chaosService;
this.partitionSimulator = partitionSimulator;
}
@PostMapping("/start")
public ResponseEntity<String> startChaos(@RequestBody ChaosConfig config) {
chaosService.startRandomPartitions(config);
return ResponseEntity.ok("Chaos engineering started");
}
@PostMapping("/stop")
public ResponseEntity<String> stopChaos() {
chaosService.stopChaos();
return ResponseEntity.ok("Chaos engineering stopped");
}
@PostMapping("/partitions")
public ResponseEntity<String> createPartition(@RequestBody PartitionRequest request) {
PartitionRule rule = new PartitionRule(
request.getSourceService(),
request.getTargetService(),
request.getType(),
Duration.ofSeconds(request.getDurationSeconds()),
request.getPacketLossPercentage(),
Duration.ofMillis(request.getLatencyMs())
);
String partitionId = partitionSimulator.createPartition(rule);
return ResponseEntity.ok("Partition created: " + partitionId);
}
@DeleteMapping("/partitions/{partitionId}")
public ResponseEntity<String> removePartition(@PathVariable String partitionId) {
partitionSimulator.removePartition(partitionId);
return ResponseEntity.ok("Partition removed: " + partitionId);
}
@GetMapping("/partitions")
public ResponseEntity<List<PartitionRule>> getActivePartitions() {
return ResponseEntity.ok(partitionSimulator.getActivePartitions());
}
}
@Data
class PartitionRequest {
private String sourceService;
private String targetService;
private NetworkPartitionSimulator.PartitionType type;
private int durationSeconds;
private double packetLossPercentage;
private long latencyMs;
}
6. Monitoring and Observability
Example 6: Partition-Aware Monitoring
@Component
@Slf4j
public class PartitionMonitoringService {
private final MeterRegistry meterRegistry;
private final NetworkPartitionSimulator partitionSimulator;
private final Map<String, Timer> serviceCallTimers = new ConcurrentHashMap<>();
private final Counter partitionEventsCounter;
private final Counter failedCallsCounter;
public PartitionMonitoringService(MeterRegistry meterRegistry,
NetworkPartitionSimulator partitionSimulator) {
this.meterRegistry = meterRegistry;
this.partitionSimulator = partitionSimulator;
this.partitionEventsCounter = Counter.builder("network.partition.events")
.description("Number of network partition events")
.register(meterRegistry);
this.failedCallsCounter = Counter.builder("service.calls.failed")
.description("Number of failed service calls")
.register(meterRegistry);
}
public void recordServiceCall(String fromService, String toService,
Duration duration, boolean success) {
String timerName = "service.call.duration";
Timer timer = serviceCallTimers.computeIfAbsent(fromService + "." + toService,
key -> Timer.builder(timerName)
.tag("from", fromService)
.tag("to", toService)
.register(meterRegistry));
timer.record(duration);
if (!success) {
failedCallsCounter.increment();
}
// Check if partition affected this call
if (partitionSimulator.shouldBlockCommunication(fromService, toService) {
partitionEventsCounter.increment();
log.warn("Service call {} -> {} failed due to network partition",
fromService, toService);
}
}
public void recordPartitionEvent(String source, String target,
NetworkPartitionSimulator.PartitionType type) {
Counter.builder("network.partition.created")
.tag("source", source)
.tag("target", target)
.tag("type", type.name())
.register(meterRegistry)
.increment();
}
public Map<String, Object> getPartitionMetrics() {
Map<String, Object> metrics = new HashMap<>();
metrics.put("activePartitions", partitionSimulator.getActivePartitions().size());
metrics.put("totalPartitionEvents", partitionEventsCounter.count());
metrics.put("failedServiceCalls", failedCallsCounter.count());
// Add service-specific metrics
serviceCallTimers.forEach((servicePair, timer) -> {
Timer.Snapshot snapshot = timer.takeSnapshot();
metrics.put(servicePair + ".mean", snapshot.mean());
metrics.put(servicePair + ".max", snapshot.max());
});
return metrics;
}
}
// Health Check with Partition Awareness
@Component
@Slf4j
public class PartitionAwareHealthIndicator implements HealthIndicator {
private final NetworkPartitionSimulator partitionSimulator;
private final List<ServiceHealthChecker> serviceHealthCheckers;
public PartitionAwareHealthIndicator(NetworkPartitionSimulator partitionSimulator,
List<ServiceHealthChecker> serviceHealthCheckers) {
this.partitionSimulator = partitionSimulator;
this.serviceHealthCheckers = serviceHealthCheckers;
}
@Override
public Health health() {
Map<String, Object> details = new HashMap<>();
boolean healthy = true;
// Check active partitions
List<NetworkPartitionSimulator.PartitionRule> activePartitions =
partitionSimulator.getActivePartitions();
details.put("activePartitions", activePartitions.size());
if (!activePartitions.isEmpty()) {
healthy = false;
details.put("partitionDetails", activePartitions.stream()
.map(rule -> rule.getSourceComponent() + " <-> " + rule.getTargetComponent())
.collect(Collectors.toList()));
}
// Check service health considering partitions
for (ServiceHealthChecker checker : serviceHealthCheckers) {
String serviceName = checker.getServiceName();
boolean serviceHealthy = checker.isHealthy();
// If there's a partition to this service, mark it as degraded
if (partitionSimulator.shouldBlockCommunication("current-service", serviceName)) {
details.put(serviceName + ".status", "DEGRADED");
details.put(serviceName + ".reason", "Network partition");
healthy = false;
} else if (!serviceHealthy) {
details.put(serviceName + ".status", "DOWN");
healthy = false;
} else {
details.put(serviceName + ".status", "UP");
}
}
if (healthy) {
return Health.up().withDetails(details).build();
} else {
return Health.down().withDetails(details).build();
}
}
}
Best Practices for Network Partition Testing
1. Test Scenarios to Cover
public class NetworkPartitionTestScenarios {
/**
* Test scenarios for comprehensive partition testing
*/
public static final List<PartitionScenario> SCENARIOS = List.of(
// Basic partition scenarios
new PartitionScenario("complete-partition",
"Complete network isolation between services",
PartitionType.COMPLETE, Duration.ofMinutes(5)),
// High latency scenarios
new PartitionScenario("high-latency",
"High network latency (5-10 seconds)",
PartitionType.LATENCY, Duration.ofMinutes(3)),
// Packet loss scenarios
new PartitionScenario("packet-loss",
"Random packet loss (10-30%)",
PartitionType.PACKET_LOSS, Duration.ofMinutes(2)),
// Unreliable network scenarios
new PartitionScenario("unreliable-network",
"Combination of latency and packet loss",
PartitionType.UNRELIABLE, Duration.ofMinutes(4)),
// Cascading failure scenarios
new PartitionScenario("cascading-failure",
"Partition that triggers cascading failures",
PartitionType.COMPLETE, Duration.ofMinutes(10))
);
@Data
@AllArgsConstructor
public static class PartitionScenario {
private String name;
private String description;
private NetworkPartitionSimulator.PartitionType type;
private Duration duration;
}
}
2. Automated Partition Testing Pipeline
@Component
@Slf4j
public class AutomatedPartitionTesting {
private final NetworkPartitionSimulator partitionSimulator;
private final TestScenarioExecutor testExecutor;
private final ResultsCollector resultsCollector;
public AutomatedPartitionTesting(NetworkPartitionSimulator partitionSimulator,
TestScenarioExecutor testExecutor,
ResultsCollector resultsCollector) {
this.partitionSimulator = partitionSimulator;
this.testExecutor = testExecutor;
this.resultsCollector = resultsCollector;
}
public void runPartitionTestSuite() {
log.info("Starting automated partition test suite");
for (PartitionTestScenario scenario : getTestScenarios()) {
runPartitionTest(scenario);
}
log.info("Partition test suite completed");
generateTestReport();
}
private void runPartitionTest(PartitionTestScenario scenario) {
log.info("Running partition test: {}", scenario.getName());
try {
// Setup partition
String partitionId = partitionSimulator.createPartition(scenario.getPartitionRule());
// Run tests during partition
TestResults results = testExecutor.executeTests(scenario.getTests());
// Remove partition
partitionSimulator.removePartition(partitionId);
// Verify recovery
TestResults recoveryResults = testExecutor.executeTests(scenario.getRecoveryTests());
// Collect results
resultsCollector.recordScenarioResults(scenario, results, recoveryResults);
} catch (Exception e) {
log.error("Partition test failed: {}", scenario.getName(), e);
resultsCollector.recordTestFailure(scenario, e);
}
}
private List<PartitionTestScenario> getTestScenarios() {
// Define comprehensive test scenarios
return List.of(
new PartitionTestScenario("database-partition",
"Partition between application and database",
new NetworkPartitionSimulator.PartitionRule("app", "database",
PartitionType.COMPLETE, Duration.ofMinutes(2)),
List.of("testDatabaseOperations", "testTransactionHandling"),
List.of("testRecovery", "testDataConsistency")),
new PartitionTestScenario("service-mesh-partition",
"Partition between microservices",
new NetworkPartitionSimulator.PartitionRule("order-service", "payment-service",
PartitionType.UNRELIABLE, Duration.ofMinutes(3), 20.0, Duration.ofSeconds(5)),
List.of("testOrderProcessing", "testCircuitBreaker"),
List.of("testServiceRecovery", "testMessageReconciliation"))
);
}
private void generateTestReport() {
TestReport report = resultsCollector.generateReport();
log.info("Partition Test Report:\n{}", report);
// Alert if critical tests failed
if (report.getFailureCount() > 0) {
log.error("CRITICAL: {} partition tests failed", report.getFailureCount());
}
}
}
Conclusion
Network partition testing is essential for building resilient distributed systems:
Key Testing Strategies:
- Toxiproxy: For realistic network failure simulation
- Custom Simulators: For fine-grained control over partition behavior
- Chaos Engineering: For automated, random partition testing
- Monitoring Integration: For observability during partitions
Critical Test Scenarios:
- Complete Isolation: Verify graceful degradation
- High Latency: Test timeout and retry mechanisms
- Packet Loss: Validate message reliability
- Cascading Failures: Ensure failure containment
- Recovery Testing: Verify system restoration
Best Practices:
- Start with controlled environments before production
- Use feature flags to enable/disable partition testing
- Monitor system behavior comprehensively during tests
- Test recovery mechanisms thoroughly
- Document expected behaviors and failure modes
Network partition testing helps build systems that can withstand real-world network failures, ensuring business continuity and customer satisfaction even under adverse conditions.