Comprehensive Load Testing: Using Artillery for Java Applications

Article

Load testing is crucial for ensuring Java applications can handle production traffic while maintaining performance and reliability. Artillery is a modern, powerful load testing tool that enables developers to create realistic load tests for HTTP, WebSocket, and other protocols. When combined with Java applications, it provides a complete solution for performance validation.

What is Artillery?

Artillery is an open-source load testing toolkit designed for distributed load testing and smoke testing of applications. It uses YAML/JSON for test configuration and JavaScript for complex test logic, making it highly flexible and developer-friendly.

Key Advantages for Java Teams:

Simple YAML Configuration: Easy to write and maintain test scenarios
JavaScript Extensibility: Custom logic for complex test flows
Multiple Protocols: HTTP, WebSocket, Socket.IO, and more
Rich Metrics: Detailed performance reports and metrics
CI/CD Integration: Easy integration with testing pipelines
Distributed Testing: Scale tests across multiple machines

Installation and Setup

1. Install Artillery

# Install via npm (requires Node.js) npm install -g artillery # Or as development dependency npm install --save-dev artillery # Verify installation artillery --version # Install plugins npm install -g artillery-plugin-metrics-by-endpoint npm install -g artillery-plugin-ensure

2. Docker Installation

# Run Artillery via Docker docker run --rm -it -v $(pwd):/tests artilleryio/artillery:latest run /tests/load-test.yml

3. Java Application Setup

Ensure your Java application is running and accessible:

# Spring Boot example ./mvnw spring-boot:run # Or run built JAR java -jar target/my-java-app-1.0.0.jar

Basic Load Test Configuration

1. Simple HTTP Load Test

basic-load-test.yml:

config: target: "http://localhost:8080" phases: - duration: 60 arrivalRate: 10 name: "Warm up phase" - duration: 120 arrivalRate: 50 name: "Load phase" - duration: 60 arrivalRate: 10 name: "Cool down phase" plugins: metrics-by-endpoint: {} ensure: {} ensure: thresholds: - http.response_time.p99: 500 - http.response_time.median: 200 - http.requests: 5000 scenarios: - name: "Basic API Test" flow: - get: url: "/api/health" capture: json: "$.status" as: "health_status" expect: - statusCode: 200 - contentType: "application/json" - think: 2 # Wait 2 seconds - get: url: "/api/users" expect: - statusCode: 200 - post: url: "/api/users" json: name: "Test User" email: "[email protected]" expect: - statusCode: 201

2. Run the Test

# Run basic test artillery run basic-load-test.yml # Run with JSON output artillery run basic-load-test.yml --output report.json # Generate HTML report artillery run basic-load-test.yml --output report.json artillery report report.json

Advanced Java Application Testing

1. Spring Boot REST API Load Test

spring-boot-load-test.yml:

config: target: "http://localhost:8080" phases: - duration: 300 arrivalRate: 20 rampTo: 100 name: "Ramp up to peak load" plugins: metrics-by-endpoint: {} ensure: {} publish-metrics: - type: "datadog" apiKey: "${DATADOG_API_KEY}" appKey: "${DATADOG_APP_KEY}" ensure: thresholds: - http.response_time.p99: 1000 - http.response_time.median: 300 - http.downloaded_bytes: 500000 payload: path: "test-data/users.csv" fields: - "username" - "email" - "userId" order: "random" scenarios: - name: "Spring Boot REST API Test" flow: # Health check - get: url: "/actuator/health" name: "Health Check" expect: - statusCode: 200 - hasProperty: "status", equals: "UP" # Get all users - get: url: "/api/v1/users" name: "Get Users" query: page: 0 size: 10 expect: - statusCode: 200 - contentType: "application/json" # Create user with dynamic data - post: url: "/api/v1/users" name: "Create User" json: username: "{{ username }}" email: "{{ email }}" role: "USER" capture: json: "$.id" as: "userId" expect: - statusCode: 201 # Get specific user - get: url: "/api/v1/users/{{ userId }}" name: "Get User by ID" expect: - statusCode: 200 # Update user - put: url: "/api/v1/users/{{ userId }}" name: "Update User" json: username: "{{ username }}_updated" email: "updated_{{ email }}" expect: - statusCode: 200 # Delete user - delete: url: "/api/v1/users/{{ userId }}" name: "Delete User" expect: - statusCode: 204 # Search users - get: url: "/api/v1/users/search" name: "Search Users" query: query: "test" expect: - statusCode: 200

2. Test Data File (users.csv)

username,email,userId john_doe,[email protected],1001 jane_smith,[email protected],1002 bob_wilson,[email protected],1003 alice_brown,[email protected],1004 charlie_davis,[email protected],1005

Java-Specific Test Scenarios

1. JWT Authentication Flow

jwt-auth-test.yml:

config: target: "https://api.myjavaapp.com" phases: - duration: 180 arrivalRate: 30 name: "Authentication Load Test" processor: "./auth-hooks.js" scenarios: - name: "JWT Authentication Flow" flow: # Login and get token - post: url: "/api/auth/login" json: username: "{{ username }}" password: "{{ password }}" capture: json: "$.accessToken" as: "authToken" expect: - statusCode: 200 - function: "setAuthHeader" # Access protected endpoints - get: url: "/api/protected/user-profile" expect: - statusCode: 200 - get: url: "/api/protected/orders" expect: - statusCode: 200 # Refresh token - post: url: "/api/auth/refresh" json: refreshToken: "{{ refreshToken }}" capture: json: "$.accessToken" as: "authToken" expect: - statusCode: 200 - function: "setAuthHeader"

auth-hooks.js:

module.exports = { setAuthHeader, generateTestData }; function setAuthHeader(requestParams, context, ee, next) { // Add Authorization header with JWT token requestParams.headers = requestParams.headers || {}; requestParams.headers['Authorization'] = `Bearer ${context.vars.authToken}`; return next(); } function generateTestData(requestParams, context, ee, next) { // Generate dynamic test data context.vars.username = `user_${Math.floor(Math.random() * 10000)}`; context.vars.password = `pass_${Math.floor(Math.random() * 100000)}`; context.vars.refreshToken = `refresh_${Math.floor(Math.random() * 1000000)}`; return next(); }

2. Database-Backed API Test

database-api-test.yml:

config: target: "http://localhost:8080" phases: - duration: 300 arrivalRate: 50 name: "Database API Load" payload: path: "test-data/products.csv" fields: - "productId" - "name" - "price" order: "random" scenarios: - name: "Product API Load Test" flow: # Get all products - get: url: "/api/products" name: "Get Products" expect: - statusCode: 200 # Get specific product - get: url: "/api/products/{{ productId }}" name: "Get Product by ID" expect: - statusCode: 200 # Create order - post: url: "/api/orders" name: "Create Order" json: productId: "{{ productId }}" quantity: 1 customerEmail: "[email protected]" capture: json: "$.orderId" as: "orderId" expect: - statusCode: 201 # Get order status - get: url: "/api/orders/{{ orderId }}" name: "Get Order Status" expect: - statusCode: 200

WebSocket Testing for Java Applications

1. Spring WebSocket Load Test

websocket-test.yml:

config: target: "ws://localhost:8080" phases: - duration: 120 arrivalRate: 10 name: "WebSocket Connection Test" engine: "ws" scenarios: - name: "Real-time Chat Load Test" engine: "ws" flow: # Connect to WebSocket endpoint - connect: "/ws/chat" # Send authentication message - send: data: | { "type": "AUTH", "username": "user_{{ $randomNumber(1000, 9999) }}", "token": "test_token" } # Wait for authentication response - think: 1 # Join chat room - send: data: | { "type": "JOIN", "room": "general" } # Send multiple chat messages - loop: - send: data: | { "type": "MESSAGE", "content": "Test message {{ $loopCount }}", "room": "general" } - think: 2 count: 5 # Listen for incoming messages - think: 10 # Disconnect - close: true

2. Socket.IO Testing

socketio-test.yml:

config: target: "http://localhost:3000" phases: - duration: 180 arrivalRate: 20 name: "Socket.IO Load Test" engine: "socketio" scenarios: - name: "Real-time Notifications" engine: "socketio" flow: - emit: channel: "authenticate" data: token: "user_token_{{ $randomNumber(1000, 9999) }}" - emit: channel: "join_room" data: room: "notifications" - think: 2 - emit: channel: "send_notification" data: message: "Test notification {{ $timestamp }}" type: "info" - on: channel: "new_notification" capture: json: "$.message" as: "notification_message"

Custom Metrics and Hooks for Java Apps

1. Custom Response Validation

custom-validation.js:

module.exports = { validateSpringResponse, captureBusinessMetrics, logSlowRequests }; function validateSpringResponse(requestParams, response, context, ee, next) { // Validate Spring Boot specific responses if (response.body) { try { const body = JSON.parse(response.body); // Check for Spring Data REST specific structures if (body._embedded) { context.vars.hasEmbedded = true; } // Check for Spring HATEOAS links if (body._links) { context.vars.hasLinks = true; } // Validate pagination structure if (body.page) { const { size, totalElements, totalPages } = body.page; if (totalElements > 10000) { console.warn(`Large dataset detected: ${totalElements} elements`); } } } catch (e) { // Ignore JSON parse errors } } return next(); } function captureBusinessMetrics(requestParams, response, context, ee, next) { // Capture business-specific metrics if (response.timings) { const responseTime = response.timings.phases.firstByte; // Categorize response times if (responseTime > 1000) { ee.emit('counter', 'slow_responses', 1); } else if (responseTime < 100) { ee.emit('counter', 'fast_responses', 1); } // Track endpoint-specific metrics const endpoint = requestParams.url || 'unknown'; ee.emit('histogram', `response_time.${endpoint}`, responseTime); } return next(); } function logSlowRequests(requestParams, response, context, ee, next) { const slowThreshold = 2000; // 2 seconds if (response.timings && response.timings.phases.firstByte > slowThreshold) { console.log(`SLOW REQUEST: ${requestParams.method} ${requestParams.url} - ${response.timings.phases.firstByte}ms`); } return next(); }

2. Enhanced Load Test Configuration

advanced-java-test.yml:

config: target: "http://localhost:8080" phases: - duration: 60 arrivalRate: 5 name: "Smoke test" - duration: 300 arrivalRate: 10 rampTo: 100 name: "Load test" - duration: 60 arrivalRate: 200 name: "Spike test" processor: "./custom-validation.js" plugins: metrics-by-endpoint: {} ensure: {} apdex: {} apdex: threshold: 500 ensure: thresholds: - http.response_time.p95: 800 - http.response_time.median: 300 - http.request_rate: 50 - apdex: 0.9 scenarios: - name: "Comprehensive API Test" beforeRequest: "captureBusinessMetrics" afterResponse: "validateSpringResponse" flow: - get: url: "/api/health" afterResponse: "logSlowRequests" - post: url: "/api/auth/login" json: username: "load_test_user" password: "test_password" capture: json: "$.token" as: "authToken"

Distributed Load Testing

1. Multi-Node Testing Setup

docker-compose-distributed.yml:

version: '3.8' services: artillery-primary: image: artilleryio/artillery:latest volumes: - ./tests:/tests environment: - ARTILLERY_REDIS_HOST=redis command: run /tests/distributed-test.yml --config /tests/artillery-config.yml depends_on: - redis networks: - load-test artillery-worker-1: image: artilleryio/artillery:latest volumes: - ./tests:/tests environment: - ARTILLERY_REDIS_HOST=redis command: run-worker --config /tests/artillery-config.yml depends_on: - redis networks: - load-test artillery-worker-2: image: artilleryio/artillery:latest volumes: - ./tests:/tests environment: - ARTILLERY_REDIS_HOST=redis command: run-worker --config /tests/artillery-config.yml depends_on: - redis networks: - load-test redis: image: redis:7-alpine networks: - load-test networks: load-test: driver: bridge

2. Distributed Test Configuration

artillery-config.yml:

config: environments: primary: target: "http://java-app:8080" phases: - duration: 600 arrivalRate: 100 processor: "./distributed-hooks.js" worker: target: "http://java-app:8080" redis: host: "redis" port: 6379 plugins: publish-metrics: - type: "datadog" apiKey: "${DATADOG_API_KEY}" - type: "prometheus" pushgateway: "http://prometheus:9091"

CI/CD Integration

1. GitHub Actions Workflow

name: Load Test Java Application on: push: branches: [ main ] schedule: - cron: '0 2 * * *' # Daily at 2 AM jobs: load-test: runs-on: ubuntu-latest services: redis: image: redis:7-alpine ports: - 6379:6379 steps: - uses: actions/checkout@v3 - name: Setup Node.js uses: actions/setup-node@v3 with: node-version: '18' - name: Install Artillery run: npm install -g artillery artillery-plugin-metrics-by-endpoint - name: Start Java Application run: | docker run -d -p 8080:8080 --name java-app my-company/my-java-app:latest sleep 30 # Wait for app to start - name: Run Smoke Test run: artillery run tests/smoke-test.yml - name: Run Load Test run: artillery run tests/load-test.yml --output results.json - name: Generate Report run: artillery report results.json --output report.html - name: Upload Report uses: actions/upload-artifact@v3 with: name: load-test-report path: report.html - name: Check Performance Thresholds run: | artillery run tests/load-test.yml --tags "thresholds"

2. Jenkins Pipeline

pipeline { agent any environment { ARTILLERY_CONFIG = 'tests/ci-config.yml' } stages { stage('Build Java App') { steps { sh 'mvn clean package' } } stage('Deploy to Test') { steps { sh 'docker-compose -f docker-compose.test.yml up -d' sh 'sleep 60' // Wait for app to start } } stage('Run Load Tests') { steps { sh ''' npm install -g artillery artillery run ${ARTILLERY_CONFIG} --output test-results.json artillery report test-results.json ''' } } stage('Performance Gates') { steps { sh ''' artillery run ${ARTILLERY_CONFIG} --tags "thresholds" ''' } } } post { always { sh 'docker-compose -f docker-compose.test.yml down' archiveArtifacts artifacts: 'test-results.json, report.html' } } }

Best Practices for Java Load Testing

1. Test Data Management

config: payload: path: "test-data/{{ $environment }}-users.csv" fields: - "username" - "userId" order: "random"

2. Environment-Specific Configuration

config: environments: dev: target: "http://dev-api:8080" phases: - duration: 60 arrivalRate: 10 staging: target: "https://staging-api.company.com" phases: - duration: 300 arrivalRate: 50 prod: target: "https://api.company.com" phases: - duration: 600 arrivalRate: 100

3. Monitoring Integration

config: plugins: publish-metrics: - type: "datadog" apiKey: "${DATADOG_API_KEY}" - type: "prometheus" pushgateway: "http://prometheus:9091" - type: "influxdb" url: "http://influxdb:8086" database: "load_tests"

Conclusion

Artillery provides Java development teams with a powerful, flexible load testing solution that integrates seamlessly into modern development workflows. Key advantages include:

Developer-Friendly: YAML configuration with JavaScript extensibility
Java Ecosystem Integration: Perfect for Spring Boot, Micronaut, Quarkus applications
Comprehensive Protocol Support: HTTP, WebSocket, Socket.IO for full-stack testing
CI/CD Ready: Easy integration with Jenkins, GitHub Actions, and other pipelines
Production Insights: Rich metrics and reporting for performance analysis

For Java applications, Artillery enables teams to proactively identify performance bottlenecks, validate scalability requirements, and ensure reliable operation under production loads, making it an essential tool for modern Java development.