Article
In the world of Java development, we rigorously test our code with JUnit and ensure API correctness with integration tests. But how does our Spring Boot service behave when 1000 concurrent users hit the /checkout endpoint? Does the JVM garbage collector cause latency spikes under load? Does our database connection pool hold up?
Artillery is a modern, open-source load testing tool designed to answer these critical questions. It allows Java developers to script complex user flows and simulate real-world traffic patterns to validate performance, stability, and scalability before hitting production.
What is Artillery?
Artillery is a powerful, code-oriented load testing tool built in Node.js but agnostic to the application being tested. It's perfect for testing any Java application (Spring Boot, Quarkus, Micronaut) that exposes HTTP, WebSocket, or Socket.io endpoints.
Key Advantages for Java Teams:
- YAML/JS Configuration: Define test scenarios in a clean, declarative YAML format, making it easy to version and collaborate.
- Rich Scenarios: Model complex, multi-step user journeys (e.g., login -> browse -> add to cart -> checkout).
- Realistic Load: Generate dynamic data and create realistic traffic shapes (ramps, spikes, sustained load).
- Extensible: Use JavaScript hooks for custom logic, request pre-processing, and response validation.
- CI/CD Friendly: Runs as a single CLI command, making it ideal for performance gates in your Jenkins or GitHub Actions pipelines.
Core Concepts: The artillery.yml File
The entire test is defined in a configuration file, typically artillery.yml. This file has three main sections: configuration, scenarios, and phases.
# artillery.yml
# 1. CONFIGURATION
config:
target: "http://localhost:8080" # Your Java app's base URL
phases:
- duration: 60 # Ramp up traffic for 60 seconds
arrivalRate: 1 # Start with 1 user per second
name: "Warm up phase"
- duration: 120
arrivalRate: 10 # Increase to 10 new users per second
rampTo: 50 # ...and ramp up to 50 by the end of the phase
name: "Ramp to peak load"
- duration: 60
arrivalRate: 50 # Sustain 50 new users/sec for 1 minute
name: "Sustained peak load"
# Plugins and payloads for dynamic data
plugins:
ensure: {} # Allows for assertion checks
metrics-by-endpoint: {} # Breaks down metrics by endpoint
payload:
- path: "test-data/users.csv"
fields:
- "username"
- "password"
order: "random"
# 2. SCENARIOS (The User Journey)
scenarios:
- name: "Authenticated User Browse and Purchase Flow"
flow:
# Step 1: Get a CSRF token (common in Spring Security)
- get:
url: "/csrf"
capture:
- json: "$.token"
as: "csrfToken"
# Step 2: Login with dynamic data from the CSV
- post:
url: "/api/login"
beforeRequest: "setLoginBody" # A JS function to build the request
json:
username: "{{ username }}"
password: "{{ password }}"
capture:
- json: "$.authToken"
as: "jwtToken"
ensure:
- statusCode: 200
# Step 3: Set the JWT token for subsequent requests
- function: "setAuthHeader"
# Step 4: Browse products
- get:
url: "/api/products"
ensure:
- statusCode: 200
- hasProperty: "$[0].id" # Assert response structure
# Step 5: Add a product to cart
- post:
url: "/api/cart/items"
json:
productId: 12345
quantity: 1
ensure:
- statusCode: 201
# Step 6: Think like a user (pause)
- think: 3 # Pause for 3 seconds
# Step 7: Checkout
- post:
url: "/api/checkout"
ensure:
- statusCode: 200
JavaScript Hooks for Complex Logic
For dynamic behavior (like setting headers or generating data), you use a separate functions.js file.
// functions.js
// Helper to set the login request body dynamically
module.exports = { setLoginBody, setAuthHeader };
function setLoginBody(requestParams, context, ee, next) {
requestParams.json = {
username: context.vars.username,
password: context.vars.password
};
return next();
}
// Helper to set the Authorization header after login
function setAuthHeader(userContext, events, done) {
userContext.vars.headers = {
'Authorization': `Bearer ${userContext.vars.jwtToken}`,
'X-CSRF-TOKEN': userContext.vars.csrfToken
};
return done();
}
Running the Test and Analyzing Results
- Install Artillery:
npm install -g artillery - Create your test data CSV:
csv username,password user1,pass123 user2,pass456 user3,pass789 - Run the test:
artillery run artillery.yml - Generate a report:
artillery run artillery.yml --output report.json && artillery report report.json
Artillery provides a comprehensive summary in the console and a detailed HTML report:
-------------------------------------- Metrics for period to: 13:45:22(-0500) (60s) -------------------------------------- http.codes.200: ................................................................ 2850 http.codes.201: ................................................................ 950 http.request_rate: ............................................................. 63/sec http.requests: ................................................................. 3800 http.response_time: min: ......................................................................... 12 max: ......................................................................... 1890 median: ...................................................................... 45.3 p95: ......................................................................... 98.2 p99: ......................................................................... 245.1 http.responses: ................................................................ 3800 vusers.completed: .............................................................. 950 vusers.created: ................................................................ 950 vusers.created_by_name.Authenticated User Browse and Purchase Flow: .......... 950 vusers.failed: ................................................................. 0 vusers.session_length: min: ......................................................................... 5123 max: ......................................................................... 8921 median: ...................................................................... 6544.1 p95: ......................................................................... 7854.3 p99: ......................................................................... 8654.8
Java-Specific Testing Considerations
- JVM Warm-Up: Always include a warm-up phase in your load test. The JIT compiler needs time to optimize hot paths. Testing on a "cold" JVM gives unrealistic performance data.
- Garbage Collection: Monitor GC logs during the test. A rising p95/p99 latency often correlates with major GC cycles. Correlate Artillery metrics with your APM tool (e.g., Datadog, New Relic).
- Database Connection Pools: High error rates or timeouts under load might indicate your HikariCP or Tomcat JDBC pool is too small. Artillery helps you find the right size.
- Testing Resilience: Use Artillery to intentionally overwhelm your service and verify that circuit breakers (e.g., Resilience4j) and rate limiters kick in correctly.
Integrating with CI/CD
You can fail a build if performance degrades by using the ensure plugin and setting thresholds.
config:
target: "http://localhost:8080"
phases:
- duration: 60
arrivalRate: 20
plugins:
ensure: {}
ensure:
thresholds:
- "http.response_time.p95": 100 # Fail if p95 latency > 100ms
- "http.codes.200": 0.95 # Fail if success rate < 95%
scenarios:
- flow:
- get:
url: "/api/health"
Run with: artillery run --quiet artillery.yml
Conclusion
For Java developers, Artillery fills a critical gap in the testing pyramid. It moves performance testing from an afterthought to an integral part of the development lifecycle. By modeling real user behavior and applying sustained load, you can uncover JVM-specific bottlenecks, validate scaling assumptions, and build confidence that your Java application will perform reliably under pressure.
Adopting Artillery empowers Java teams to ship faster, knowing that performance regressions will be caught early and often.
Pyroscope Profiling in Java
Explains how to use Pyroscope for continuous profiling in Java applications, helping developers analyze CPU and memory usage patterns to improve performance and identify bottlenecks.
https://macronepal.com/blog/pyroscope-profiling-in-java/
OpenTelemetry Metrics in Java: Comprehensive Guide
Provides a complete guide to collecting and exporting metrics in Java using OpenTelemetry, including counters, histograms, gauges, and integration with monitoring tools. (MACRO NEPAL)
https://macronepal.com/blog/opentelemetry-metrics-in-java-comprehensive-guide/
OTLP Exporter in Java: Complete Guide for OpenTelemetry
Explains how to configure OTLP exporters in Java to send telemetry data such as traces, metrics, and logs to monitoring systems using HTTP or gRPC protocols. (MACRO NEPAL)
https://macronepal.com/blog/otlp-exporter-in-java-complete-guide-for-opentelemetry/
Thanos Integration in Java: Global View of Metrics
Explains how to integrate Thanos with Java monitoring systems to create a scalable global metrics view across multiple Prometheus instances.
https://macronepal.com/blog/thanos-integration-in-java-global-view-of-metrics
Time Series with InfluxDB in Java: Complete Guide (Version 2)
Explains how to manage time-series data using InfluxDB in Java applications, including storing, querying, and analyzing metrics data.
https://macronepal.com/blog/time-series-with-influxdb-in-java-complete-guide-2
Time Series with InfluxDB in Java: Complete Guide
Provides an overview of integrating InfluxDB with Java for time-series data handling, including monitoring applications and managing performance metrics.
https://macronepal.com/blog/time-series-with-influxdb-in-java-complete-guide
Implementing Prometheus Remote Write in Java (Version 2)
Explains how to configure Java applications to send metrics data to Prometheus-compatible systems using the remote write feature for scalable monitoring.
https://macronepal.com/blog/implementing-prometheus-remote-write-in-java-a-complete-guide-2
Implementing Prometheus Remote Write in Java: Complete Guide
Provides instructions for sending metrics from Java services to Prometheus servers, enabling centralized monitoring and real-time analytics.
https://macronepal.com/blog/implementing-prometheus-remote-write-in-java-a-complete-guide
Building a TileServer GL in Java: Vector and Raster Tile Server
Explains how to build a TileServer GL in Java for serving vector and raster map tiles, useful for geographic visualization and mapping applications.
https://macronepal.com/blog/building-a-tileserver-gl-in-java-vector-and-raster-tile-server
Indoor Mapping in Java
Explains how to create indoor mapping systems in Java, including navigation inside buildings, spatial data handling, and visualization techniques.