Toxiproxy is a powerful TCP proxy for simulating network conditions in testing environments. It allows you to test your application's resilience against various network failures, latency, and other problematic conditions.

Understanding Toxiproxy

What is Toxiproxy?

• A framework for simulating network conditions
• Proxy that sits between your application and dependent services
• Can introduce latency, timeouts, bandwidth limits, and network failures
• Essential for chaos engineering and resilience testing

Key Features:

• Toxic Types: latency, bandwidth_limit, timeout, reset_peer, slow_close, slicer
• HTTP API for dynamic configuration
• Easy integration with existing applications
• Multiple language clients including Java

Setup and Dependencies

1. Running Toxiproxy

# Using Docker docker run --name toxiproxy -d -p 8474:8474 -p 6379:6379 ghcr.io/shopify/toxiproxy:latest # Verify it's running curl http://localhost:8474/version

2. Maven Dependencies

<properties> <toxiproxy-java.version>2.1.7</toxiproxy-java.version> <junit.version>5.9.2</junit.version> <testcontainers.version>1.18.3</testcontainers.version> </properties> <dependencies> <!-- Toxiproxy Java Client --> <dependency> <groupId>eu.rekawek.toxiproxy</groupId> <artifactId>toxiproxy-java</artifactId> <version>${toxiproxy-java.version}</version> <scope>test</scope> </dependency> <!-- TestContainers for Toxiproxy --> <dependency> <groupId>org.testcontainers</groupId> <artifactId>testcontainers</artifactId> <version>${testcontainers.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>org.testcontainers</groupId> <artifactId>junit-jupiter</artifactId> <version>${testcontainers.version}</version> <scope>test</scope> </dependency> </dependencies>

Core Toxiproxy Integration

1. Basic Toxiproxy Client Setup

import eu.rekawek.toxiproxy.Proxy; import eu.rekawek.toxiproxy.ToxiproxyClient; import eu.rekawek.toxiproxy.model.ToxicDirection; public class ToxiproxyManager { private final ToxiproxyClient client; private Proxy proxy; public ToxiproxyManager(String toxiproxyHost, int toxiproxyPort) { this.client = new ToxiproxyClient(toxiproxyHost, toxiproxyPort); } public Proxy createProxy(String proxyName, String upstreamHost, int upstreamPort, String listenHost, int listenPort) throws IOException { this.proxy = client.createProxy(proxyName, listenHost, listenPort, upstreamHost, upstreamPort); return this.proxy; } public void deleteProxy(String proxyName) throws IOException { client.deleteProxy(proxyName); } public List<Proxy> getProxies() throws IOException { return client.getProxies(); } }

2. Toxic Types Configuration

public class ToxicConfigurator { private final Proxy proxy; public ToxicConfigurator(Proxy proxy) { this.proxy = proxy; } // Add latency to network calls public void addLatency(String toxicName, long latencyMs, long jitterMs) throws IOException { proxy.toxics() .latency(toxicName, ToxicDirection.DOWNSTREAM, latencyMs) .setJitter(jitterMs); } // Limit bandwidth public void addBandwidthLimit(String toxicName, long rateInBytesPerSecond) throws IOException { proxy.toxics() .bandwidthLimit(toxicName, ToxicDirection.DOWNSTREAM, rateInBytesPerSecond); } // Add timeout public void addTimeout(String toxicName, long timeoutMs) throws IOException { proxy.toxics() .timeout(toxicName, ToxicDirection.DOWNSTREAM, timeoutMs); } // Reset peer connection (simulate connection resets) public void addResetPeer(String toxicName, long timeoutMs) throws IOException { proxy.toxics() .resetPeer(toxicName, ToxicDirection.DOWNSTREAM, timeoutMs); } // Slow connection close public void addSlowClose(String toxicName, long delayMs) throws IOException { proxy.toxics() .slowClose(toxicName, ToxicDirection.DOWNSTREAM, delayMs); } // Remove a specific toxic public void removeToxic(String toxicName) throws IOException { proxy.toxics().get(toxicName).remove(); } // Remove all toxics public void clearAllToxics() throws IOException { for (var toxic : proxy.toxics().getAll()) { toxic.remove(); } } }

Integration Testing with Toxiproxy

1. Database Resilience Testing

import org.junit.jupiter.api.*; import org.testcontainers.containers.GenericContainer; import org.testcontainers.containers.PostgreSQLContainer; import org.testcontainers.junit.jupiter.Container; import org.testcontainers.junit.jupiter.Testcontainers; import java.sql.Connection; import java.sql.DriverManager; import java.sql.Statement; @Testcontainers @TestInstance(TestInstance.Lifecycle.PER_CLASS) public class DatabaseResilienceTest { @Container private static final GenericContainer<?> toxiproxyContainer = new GenericContainer<>("ghcr.io/shopify/toxiproxy:latest") .withExposedPorts(8474, 6379); @Container private static final PostgreSQLContainer<?> postgresContainer = new PostgreSQLContainer<>("postgres:15-alpine") .withDatabaseName("testdb") .withUsername("testuser") .withPassword("testpass"); private ToxiproxyManager toxiproxyManager; private ToxicConfigurator toxicConfigurator; private String proxiedPostgresUrl; @BeforeAll void setup() throws Exception { // Get Toxiproxy container details String toxiproxyHost = toxiproxyContainer.getHost(); int toxiproxyPort = toxiproxyContainer.getMappedPort(8474); // Initialize Toxiproxy manager toxiproxyManager = new ToxiproxyManager(toxiproxyHost, toxiproxyPort); // Create proxy for PostgreSQL String postgresHost = postgresContainer.getHost(); int postgresPort = postgresContainer.getMappedPort(5432); Proxy proxy = toxiproxyManager.createProxy( "postgres-proxy", postgresHost, postgresPort, "0.0.0.0", 54320 // Listen on this port ); toxicConfigurator = new ToxicConfigurator(proxy); // Build proxied connection URL proxiedPostgresUrl = String.format( "jdbc:postgresql://%s:%d/testdb", toxiproxyContainer.getHost(), 54320 // The proxy listen port ); } @Test void testDatabaseConnectionWithLatency() throws Exception { // Test normal connection first testDatabaseConnection(); // Add 2 seconds latency toxicConfigurator.addLatency("high-latency", 2000, 500); // Test with latency - should still work but slower long startTime = System.currentTimeMillis(); testDatabaseConnection(); long endTime = System.currentTimeMillis(); Assertions.assertTrue((endTime - startTime) >= 2000, "Query should take at least 2 seconds with latency"); // Clean up toxicConfigurator.removeToxic("high-latency"); } @Test void testDatabaseConnectionTimeout() throws Exception { // Add timeout that will cause connection failures toxicConfigurator.addTimeout("connection-timeout", 100); // This should fail due to timeout Assertions.assertThrows(Exception.class, this::testDatabaseConnection); // Clean up toxicConfigurator.removeToxic("connection-timeout"); } private void testDatabaseConnection() throws Exception { try (Connection conn = DriverManager.getConnection( proxiedPostgresUrl, "testuser", "testpass"); Statement stmt = conn.createStatement()) { var rs = stmt.executeQuery("SELECT 1"); Assertions.assertTrue(rs.next()); Assertions.assertEquals(1, rs.getInt(1)); } } @AfterAll void cleanup() throws Exception { if (toxiproxyManager != null) { toxiproxyManager.deleteProxy("postgres-proxy"); } } }

2. HTTP Service Resilience Testing

@Testcontainers @TestInstance(TestInstance.Lifecycle.PER_CLASS) public class HttpClientResilienceTest { private final HttpClient httpClient = HttpClient.newBuilder() .connectTimeout(Duration.ofSeconds(5)) .build(); @Container private static final GenericContainer<?> toxiproxyContainer = new GenericContainer<>("ghcr.io/shopify/toxiproxy:latest") .withExposedPorts(8474); @Container private static final GenericContainer<?> mockServerContainer = new GenericContainer<>("mockserver/mockserver:latest") .withExposedPorts(1080) .withCommand("-serverPort 1080 -logLevel INFO"); private ToxiproxyManager toxiproxyManager; private ToxicConfigurator toxicConfigurator; private String proxiedBaseUrl; @BeforeAll void setup() throws Exception { // Setup Toxiproxy String toxiproxyHost = toxiproxyContainer.getHost(); int toxiproxyPort = toxiproxyContainer.getMappedPort(8474); toxiproxyManager = new ToxiproxyManager(toxiproxyHost, toxiproxyPort); // Create proxy for mock server String mockServerHost = mockServerContainer.getHost(); int mockServerPort = mockServerContainer.getMappedPort(1080); Proxy proxy = toxiproxyManager.createProxy( "http-proxy", mockServerHost, mockServerPort, "0.0.0.0", 10800 ); toxicConfigurator = new ToxicConfigurator(proxy); proxiedBaseUrl = String.format("http://%s:%d", toxiproxyContainer.getHost(), 10800); // Setup mock responses setupMockResponses(); } @Test void testHttpClientWithBandwidthLimit() throws Exception { // Limit bandwidth to simulate slow network toxicConfigurator.addBandwidthLimit("slow-bandwidth", 1024); // 1 KB/s HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(proxiedBaseUrl + "/api/data")) .GET() .build(); long startTime = System.currentTimeMillis(); HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()); long endTime = System.currentTimeMillis(); Assertions.assertEquals(200, response.statusCode()); System.out.printf("Request completed in %d ms with bandwidth limit%n", endTime - startTime); toxicConfigurator.removeToxic("slow-bandwidth"); } @Test void testHttpClientWithConnectionReset() throws Exception { // Simulate connection resets toxicConfigurator.addResetPeer("random-reset", 5000); HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(proxiedBaseUrl + "/api/data")) .GET() .build(); // This might succeed or fail depending on when reset happens try { HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()); Assertions.assertEquals(200, response.statusCode()); } catch (IOException e) { // Expected when connection is reset System.out.println("Connection reset as expected: " + e.getMessage()); } toxicConfigurator.removeToxic("random-reset"); } private void setupMockResponses() { // This would typically use MockServer client to setup expectations // Simplified for example purposes } }

3. Advanced Resilience Patterns with Retry

@Component public class ResilientHttpClient { private final HttpClient httpClient; private final RetryConfig retryConfig; public ResilientHttpClient() { this.httpClient = HttpClient.newBuilder() .connectTimeout(Duration.ofSeconds(3)) .build(); this.retryConfig = RetryConfig.custom() .maxAttempts(3) .waitDuration(Duration.ofMillis(500)) .retryOnException(this::isRetryableException) .build(); } public String executeWithRetry(String url) { Retry retry = Retry.of("http-call", retryConfig); return Retry.decorateSupplier(retry, () -> { HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(url)) .GET() .build(); HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()); if (response.statusCode() >= 500) { throw new RuntimeException("Server error: " + response.statusCode()); } return response.body(); }).get(); } private boolean isRetryableException(Throwable throwable) { return throwable instanceof IOException || throwable instanceof TimeoutException || (throwable instanceof RuntimeException && throwable.getMessage().contains("Server error")); } }

4. Spring Boot Integration

@Configuration @Profile("test") public class ToxiproxyTestConfig { @Bean @Primary public RestTemplate proxiedRestTemplate(ToxiproxyManager toxiproxyManager) throws IOException { // Create proxy for external service Proxy proxy = toxiproxyManager.createProxy( "external-service-proxy", "external-service.com", 443, "0.0.0.0", 8443 ); // Return RestTemplate that uses proxied connection return new RestTemplate(); } @Bean public ToxicConfigurator toxicConfigurator(ToxiproxyManager toxiproxyManager) throws IOException { Proxy proxy = toxiproxyManager.getProxies().stream() .filter(p -> p.getName().equals("external-service-proxy")) .findFirst() .orElseThrow(); return new ToxicConfigurator(proxy); } } @Service public class ChaosEngineeringService { private final ToxicConfigurator toxicConfigurator; public void simulateNetworkChaos(ChaosConfig config) throws IOException { toxicConfigurator.clearAllToxics(); if (config.getLatencyMs() > 0) { toxicConfigurator.addLatency("chaos-latency", config.getLatencyMs(), config.getJitterMs()); } if (config.getTimeoutMs() > 0) { toxicConfigurator.addTimeout("chaos-timeout", config.getTimeoutMs()); } if (config.isResetConnections()) { toxicConfigurator.addResetPeer("chaos-reset", 3000); } } public void resetNetwork() throws IOException { toxicConfigurator.clearAllToxics(); } }

5. Test Controller for Chaos Engineering

@RestController @RequestMapping("/chaos") @Profile("test") public class ChaosEngineeringController { private final ChaosEngineeringService chaosService; @PostMapping("/network") public ResponseEntity<String> configureNetworkChaos(@RequestBody ChaosConfig config) { try { chaosService.simulateNetworkChaos(config); return ResponseEntity.ok("Network chaos configured: " + config); } catch (IOException e) { return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR) .body("Failed to configure chaos: " + e.getMessage()); } } @PostMapping("/reset") public ResponseEntity<String> resetNetwork() { try { chaosService.resetNetwork(); return ResponseEntity.ok("Network conditions reset to normal"); } catch (IOException e) { return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR) .body("Failed to reset network: " + e.getMessage()); } } }

Best Practices

1. Use in Test Environments Only: Never use Toxiproxy in production
2. Clean Up Resources: Always remove proxies and toxics after tests
3. Isolate Tests: Ensure tests don't interfere with each other
4. Monitor Performance: Track how your application behaves under different conditions
5. Test Retry Mechanisms: Verify your retry logic works correctly
6. Combine with Monitoring: Use metrics to understand impact on application performance

// Example of comprehensive test cleanup @AfterEach void cleanupToxics() throws IOException { if (toxicConfigurator != null) { toxicConfigurator.clearAllToxics(); } }

Conclusion

Toxiproxy integration in Java provides a powerful way to:

• Test resilience of your applications under realistic network conditions
• Validate retry mechanisms and circuit breaker patterns
• Build confidence in your application's ability to handle failures
• Implement chaos engineering practices in a controlled manner

By integrating Toxiproxy into your testing strategy, you can proactively identify and fix resilience issues before they impact users in production environments. The combination of Toxiproxy with TestContainers and modern testing frameworks creates a robust foundation for building reliable distributed systems.