Comprehensive Load Testing: Using Artillery for Java Applications

Table of Contents

Article

Load testing is crucial for ensuring Java applications can handle production traffic while maintaining performance and reliability. Artillery is a modern, powerful load testing tool that enables developers to create realistic load tests for HTTP, WebSocket, and other protocols. When combined with Java applications, it provides a complete solution for performance validation.

What is Artillery?

Artillery is an open-source load testing toolkit designed for distributed load testing and smoke testing of applications. It uses YAML/JSON for test configuration and JavaScript for complex test logic, making it highly flexible and developer-friendly.

Key Advantages for Java Teams:

Simple YAML Configuration: Easy to write and maintain test scenarios
JavaScript Extensibility: Custom logic for complex test flows
Multiple Protocols: HTTP, WebSocket, Socket.IO, and more
Rich Metrics: Detailed performance reports and metrics
CI/CD Integration: Easy integration with testing pipelines
Distributed Testing: Scale tests across multiple machines

Installation and Setup

1. Install Artillery

# Install via npm (requires Node.js)
npm install -g artillery
# Or as development dependency
npm install --save-dev artillery
# Verify installation
artillery --version
# Install plugins
npm install -g artillery-plugin-metrics-by-endpoint
npm install -g artillery-plugin-ensure

2. Docker Installation

# Run Artillery via Docker
docker run --rm -it -v $(pwd):/tests artilleryio/artillery:latest run /tests/load-test.yml

3. Java Application Setup

Ensure your Java application is running and accessible:

# Spring Boot example
./mvnw spring-boot:run
# Or run built JAR
java -jar target/my-java-app-1.0.0.jar

Basic Load Test Configuration

1. Simple HTTP Load Test

basic-load-test.yml:

config:
target: "http://localhost:8080"
phases:
- duration: 60
arrivalRate: 10
name: "Warm up phase"
- duration: 120
arrivalRate: 50
name: "Load phase"
- duration: 60
arrivalRate: 10
name: "Cool down phase"
plugins:
metrics-by-endpoint: {}
ensure: {}
ensure:
thresholds:
- http.response_time.p99: 500
- http.response_time.median: 200
- http.requests: 5000
scenarios:
- name: "Basic API Test"
flow:
- get:
url: "/api/health"
capture:
json: "$.status"
as: "health_status"
expect:
- statusCode: 200
- contentType: "application/json"
- think: 2  # Wait 2 seconds
- get:
url: "/api/users"
expect:
- statusCode: 200
- post:
url: "/api/users"
json:
name: "Test User"
email: "[email protected]"
expect:
- statusCode: 201

2. Run the Test

# Run basic test
artillery run basic-load-test.yml
# Run with JSON output
artillery run basic-load-test.yml --output report.json
# Generate HTML report
artillery run basic-load-test.yml --output report.json
artillery report report.json

Advanced Java Application Testing

1. Spring Boot REST API Load Test

spring-boot-load-test.yml:

config:
target: "http://localhost:8080"
phases:
- duration: 300
arrivalRate: 20
rampTo: 100
name: "Ramp up to peak load"
plugins:
metrics-by-endpoint: {}
ensure: {}
publish-metrics:
- type: "datadog"
apiKey: "${DATADOG_API_KEY}"
appKey: "${DATADOG_APP_KEY}"
ensure:
thresholds:
- http.response_time.p99: 1000
- http.response_time.median: 300
- http.downloaded_bytes: 500000
payload:
path: "test-data/users.csv"
fields:
- "username"
- "email"
- "userId"
order: "random"
scenarios:
- name: "Spring Boot REST API Test"
flow:
# Health check
- get:
url: "/actuator/health"
name: "Health Check"
expect:
- statusCode: 200
- hasProperty: "status", equals: "UP"
# Get all users
- get:
url: "/api/v1/users"
name: "Get Users"
query:
page: 0
size: 10
expect:
- statusCode: 200
- contentType: "application/json"
# Create user with dynamic data
- post:
url: "/api/v1/users"
name: "Create User"
json:
username: "{{ username }}"
email: "{{ email }}"
role: "USER"
capture:
json: "$.id"
as: "userId"
expect:
- statusCode: 201
# Get specific user
- get:
url: "/api/v1/users/{{ userId }}"
name: "Get User by ID"
expect:
- statusCode: 200
# Update user
- put:
url: "/api/v1/users/{{ userId }}"
name: "Update User"
json:
username: "{{ username }}_updated"
email: "updated_{{ email }}"
expect:
- statusCode: 200
# Delete user
- delete:
url: "/api/v1/users/{{ userId }}"
name: "Delete User"
expect:
- statusCode: 204
# Search users
- get:
url: "/api/v1/users/search"
name: "Search Users"
query:
query: "test"
expect:
- statusCode: 200

2. Test Data File (users.csv)

username,email,userId
john_doe,[email protected],1001
jane_smith,[email protected],1002
bob_wilson,[email protected],1003
alice_brown,[email protected],1004
charlie_davis,[email protected],1005

Java-Specific Test Scenarios

1. JWT Authentication Flow

jwt-auth-test.yml:

config:
target: "https://api.myjavaapp.com"
phases:
- duration: 180
arrivalRate: 30
name: "Authentication Load Test"
processor: "./auth-hooks.js"
scenarios:
- name: "JWT Authentication Flow"
flow:
# Login and get token
- post:
url: "/api/auth/login"
json:
username: "{{ username }}"
password: "{{ password }}"
capture:
json: "$.accessToken"
as: "authToken"
expect:
- statusCode: 200
- function: "setAuthHeader"
# Access protected endpoints
- get:
url: "/api/protected/user-profile"
expect:
- statusCode: 200
- get:
url: "/api/protected/orders"
expect:
- statusCode: 200
# Refresh token
- post:
url: "/api/auth/refresh"
json:
refreshToken: "{{ refreshToken }}"
capture:
json: "$.accessToken"
as: "authToken"
expect:
- statusCode: 200
- function: "setAuthHeader"

auth-hooks.js:

module.exports = { 
setAuthHeader,
generateTestData 
};
function setAuthHeader(requestParams, context, ee, next) {
// Add Authorization header with JWT token
requestParams.headers = requestParams.headers || {};
requestParams.headers['Authorization'] = `Bearer ${context.vars.authToken}`;
return next();
}
function generateTestData(requestParams, context, ee, next) {
// Generate dynamic test data
context.vars.username = `user_${Math.floor(Math.random() * 10000)}`;
context.vars.password = `pass_${Math.floor(Math.random() * 100000)}`;
context.vars.refreshToken = `refresh_${Math.floor(Math.random() * 1000000)}`;
return next();
}

2. Database-Backed API Test

database-api-test.yml:

config:
target: "http://localhost:8080"
phases:
- duration: 300
arrivalRate: 50
name: "Database API Load"
payload:
path: "test-data/products.csv"
fields:
- "productId"
- "name"
- "price"
order: "random"
scenarios:
- name: "Product API Load Test"
flow:
# Get all products
- get:
url: "/api/products"
name: "Get Products"
expect:
- statusCode: 200
# Get specific product
- get:
url: "/api/products/{{ productId }}"
name: "Get Product by ID"
expect:
- statusCode: 200
# Create order
- post:
url: "/api/orders"
name: "Create Order"
json:
productId: "{{ productId }}"
quantity: 1
customerEmail: "[email protected]"
capture:
json: "$.orderId"
as: "orderId"
expect:
- statusCode: 201
# Get order status
- get:
url: "/api/orders/{{ orderId }}"
name: "Get Order Status"
expect:
- statusCode: 200

WebSocket Testing for Java Applications

1. Spring WebSocket Load Test

websocket-test.yml:

config:
target: "ws://localhost:8080"
phases:
- duration: 120
arrivalRate: 10
name: "WebSocket Connection Test"
engine: "ws"
scenarios:
- name: "Real-time Chat Load Test"
engine: "ws"
flow:
# Connect to WebSocket endpoint
- connect: "/ws/chat"
# Send authentication message
- send:
data: |
{
"type": "AUTH",
"username": "user_{{ $randomNumber(1000, 9999) }}",
"token": "test_token"
}
# Wait for authentication response
- think: 1
# Join chat room
- send:
data: |
{
"type": "JOIN",
"room": "general"
}
# Send multiple chat messages
- loop:
- send:
data: |
{
"type": "MESSAGE",
"content": "Test message {{ $loopCount }}",
"room": "general"
}
- think: 2
count: 5
# Listen for incoming messages
- think: 10
# Disconnect
- close: true

2. Socket.IO Testing

socketio-test.yml:

config:
target: "http://localhost:3000"
phases:
- duration: 180
arrivalRate: 20
name: "Socket.IO Load Test"
engine: "socketio"
scenarios:
- name: "Real-time Notifications"
engine: "socketio"
flow:
- emit:
channel: "authenticate"
data: 
token: "user_token_{{ $randomNumber(1000, 9999) }}"
- emit:
channel: "join_room"
data:
room: "notifications"
- think: 2
- emit:
channel: "send_notification"
data:
message: "Test notification {{ $timestamp }}"
type: "info"
- on:
channel: "new_notification"
capture:
json: "$.message"
as: "notification_message"

Custom Metrics and Hooks for Java Apps

1. Custom Response Validation

custom-validation.js:

module.exports = {
validateSpringResponse,
captureBusinessMetrics,
logSlowRequests
};
function validateSpringResponse(requestParams, response, context, ee, next) {
// Validate Spring Boot specific responses
if (response.body) {
try {
const body = JSON.parse(response.body);
// Check for Spring Data REST specific structures
if (body._embedded) {
context.vars.hasEmbedded = true;
}
// Check for Spring HATEOAS links
if (body._links) {
context.vars.hasLinks = true;
}
// Validate pagination structure
if (body.page) {
const { size, totalElements, totalPages } = body.page;
if (totalElements > 10000) {
console.warn(`Large dataset detected: ${totalElements} elements`);
}
}
} catch (e) {
// Ignore JSON parse errors
}
}
return next();
}
function captureBusinessMetrics(requestParams, response, context, ee, next) {
// Capture business-specific metrics
if (response.timings) {
const responseTime = response.timings.phases.firstByte;
// Categorize response times
if (responseTime > 1000) {
ee.emit('counter', 'slow_responses', 1);
} else if (responseTime < 100) {
ee.emit('counter', 'fast_responses', 1);
}
// Track endpoint-specific metrics
const endpoint = requestParams.url || 'unknown';
ee.emit('histogram', `response_time.${endpoint}`, responseTime);
}
return next();
}
function logSlowRequests(requestParams, response, context, ee, next) {
const slowThreshold = 2000; // 2 seconds
if (response.timings && response.timings.phases.firstByte > slowThreshold) {
console.log(`SLOW REQUEST: ${requestParams.method} ${requestParams.url} - ${response.timings.phases.firstByte}ms`);
}
return next();
}

2. Enhanced Load Test Configuration

advanced-java-test.yml:

config:
target: "http://localhost:8080"
phases:
- duration: 60
arrivalRate: 5
name: "Smoke test"
- duration: 300
arrivalRate: 10
rampTo: 100
name: "Load test"
- duration: 60
arrivalRate: 200
name: "Spike test"
processor: "./custom-validation.js"
plugins:
metrics-by-endpoint: {}
ensure: {}
apdex: {}
apdex:
threshold: 500
ensure:
thresholds:
- http.response_time.p95: 800
- http.response_time.median: 300
- http.request_rate: 50
- apdex: 0.9
scenarios:
- name: "Comprehensive API Test"
beforeRequest: "captureBusinessMetrics"
afterResponse: "validateSpringResponse"
flow:
- get:
url: "/api/health"
afterResponse: "logSlowRequests"
- post:
url: "/api/auth/login"
json:
username: "load_test_user"
password: "test_password"
capture:
json: "$.token"
as: "authToken"

Distributed Load Testing

1. Multi-Node Testing Setup

docker-compose-distributed.yml:

version: '3.8'
services:
artillery-primary:
image: artilleryio/artillery:latest
volumes:
- ./tests:/tests
environment:
- ARTILLERY_REDIS_HOST=redis
command: run /tests/distributed-test.yml --config /tests/artillery-config.yml
depends_on:
- redis
networks:
- load-test
artillery-worker-1:
image: artilleryio/artillery:latest
volumes:
- ./tests:/tests
environment:
- ARTILLERY_REDIS_HOST=redis
command: run-worker --config /tests/artillery-config.yml
depends_on:
- redis
networks:
- load-test
artillery-worker-2:
image: artilleryio/artillery:latest
volumes:
- ./tests:/tests
environment:
- ARTILLERY_REDIS_HOST=redis
command: run-worker --config /tests/artillery-config.yml
depends_on:
- redis
networks:
- load-test
redis:
image: redis:7-alpine
networks:
- load-test
networks:
load-test:
driver: bridge

2. Distributed Test Configuration

artillery-config.yml:

config:
environments:
primary:
target: "http://java-app:8080"
phases:
- duration: 600
arrivalRate: 100
processor: "./distributed-hooks.js"
worker:
target: "http://java-app:8080"
redis:
host: "redis"
port: 6379
plugins:
publish-metrics:
- type: "datadog"
apiKey: "${DATADOG_API_KEY}"
- type: "prometheus"
pushgateway: "http://prometheus:9091"

CI/CD Integration

1. GitHub Actions Workflow

name: Load Test Java Application
on:
push:
branches: [ main ]
schedule:
- cron: '0 2 * * *'  # Daily at 2 AM
jobs:
load-test:
runs-on: ubuntu-latest
services:
redis:
image: redis:7-alpine
ports:
- 6379:6379
steps:
- uses: actions/checkout@v3
- name: Setup Node.js
uses: actions/setup-node@v3
with:
node-version: '18'
- name: Install Artillery
run: npm install -g artillery artillery-plugin-metrics-by-endpoint
- name: Start Java Application
run: |
docker run -d -p 8080:8080 --name java-app my-company/my-java-app:latest
sleep 30  # Wait for app to start
- name: Run Smoke Test
run: artillery run tests/smoke-test.yml
- name: Run Load Test
run: artillery run tests/load-test.yml --output results.json
- name: Generate Report
run: artillery report results.json --output report.html
- name: Upload Report
uses: actions/upload-artifact@v3
with:
name: load-test-report
path: report.html
- name: Check Performance Thresholds
run: |
artillery run tests/load-test.yml --tags "thresholds"

2. Jenkins Pipeline

pipeline {
agent any
environment {
ARTILLERY_CONFIG = 'tests/ci-config.yml'
}
stages {
stage('Build Java App') {
steps {
sh 'mvn clean package'
}
}
stage('Deploy to Test') {
steps {
sh 'docker-compose -f docker-compose.test.yml up -d'
sh 'sleep 60' // Wait for app to start
}
}
stage('Run Load Tests') {
steps {
sh '''
npm install -g artillery
artillery run ${ARTILLERY_CONFIG} --output test-results.json
artillery report test-results.json
'''
}
}
stage('Performance Gates') {
steps {
sh '''
artillery run ${ARTILLERY_CONFIG} --tags "thresholds"
'''
}
}
}
post {
always {
sh 'docker-compose -f docker-compose.test.yml down'
archiveArtifacts artifacts: 'test-results.json, report.html'
}
}
}

Best Practices for Java Load Testing

1. Test Data Management

config:
payload:
path: "test-data/{{ $environment }}-users.csv"
fields:
- "username"
- "userId"
order: "random"

2. Environment-Specific Configuration

config:
environments:
dev:
target: "http://dev-api:8080"
phases:
- duration: 60
arrivalRate: 10
staging:
target: "https://staging-api.company.com"
phases:
- duration: 300
arrivalRate: 50
prod:
target: "https://api.company.com"
phases:
- duration: 600
arrivalRate: 100

3. Monitoring Integration

config:
plugins:
publish-metrics:
- type: "datadog"
apiKey: "${DATADOG_API_KEY}"
- type: "prometheus"
pushgateway: "http://prometheus:9091"
- type: "influxdb"
url: "http://influxdb:8086"
database: "load_tests"

Conclusion

Artillery provides Java development teams with a powerful, flexible load testing solution that integrates seamlessly into modern development workflows. Key advantages include:

Developer-Friendly: YAML configuration with JavaScript extensibility
Java Ecosystem Integration: Perfect for Spring Boot, Micronaut, Quarkus applications
Comprehensive Protocol Support: HTTP, WebSocket, Socket.IO for full-stack testing
CI/CD Ready: Easy integration with Jenkins, GitHub Actions, and other pipelines
Production Insights: Rich metrics and reporting for performance analysis

For Java applications, Artillery enables teams to proactively identify performance bottlenecks, validate scalability requirements, and ensure reliable operation under production loads, making it an essential tool for modern Java development.