SkyWalking is an open-source Application Performance Monitoring (APM) tool and distributed tracing system designed for microservices, cloud-native, and container-based architectures. The SkyWalking Java Agent enables automatic instrumentation of Java applications without code modification.
Overview of SkyWalking
SkyWalking provides:
- Distributed Tracing: End-to-end request tracking across microservices
- Performance Metrics: Application performance indicators and JVM metrics
- Topology Mapping: Visual service dependency graphs
- Service Mesh Observability: Integration with service mesh frameworks
- Database Monitoring: SQL query performance and database connections
Agent Setup and Configuration
Installation Methods
Method 1: Download and Setup
# Download latest SkyWalking agent wget https://archive.apache.org/dist/skywalking/9.4.0/apache-skywalking-apm-9.4.0.tar.gz tar -xzf apache-skywalking-apm-9.4.0.tar.gz cd apache-skywalking-apm-9.4.0 # Agent directory structure agent/ ├── config/ # Configuration files ├── plugins/ # Auto-instrumentation plugins ├── optional-plugins/ # Optional plugins ├── bootstrap-plugins/ # Bootstrap plugins └── skywalking-agent.jar # Main agent JAR
Method 2: Docker Setup
FROM openjdk:11-jre-slim # Copy SkyWalking agent COPY skywalking-agent/ /skywalking-agent/ # Application JAR COPY target/my-app.jar /app.jar # Run with SkyWalking agent ENTRYPOINT ["java", "-javaagent:/skywalking-agent/skywalking-agent.jar", \ "-Dskywalking.agent.service_name=my-service", \ "-Dskywalking.collector.backend_service=collector:11800", \ "-jar", "/app.jar"]
Basic Configuration
agent.config
# Agent basic configuration
agent.service_name=${SW_AGENT_NAME:My-Java-Service}
agent.sample_n_per_3_secs=${SW_AGENT_SAMPLE:1000}
agent.force_reconnection_period=${SW_AGENT_FORCE_RECONNECTION_PERIOD:1}
# Collector configuration
collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICE:127.0.0.1:11800}
collector.grpc_channel_check_interval=${SW_GRPC_CHANNEL_CHECK_INTERVAL:30}
collector.get_profile_task_interval=${SW_GET_PROFILE_TASK_INTERVAL:20}
# Logging configuration
logging.file_name=${SW_LOGGING_FILE_NAME:skywalking-api.log}
logging.level=${SW_LOGGING_LEVEL:INFO}
logging.dir=${SW_LOGGING_DIR:logs}
# JVM configuration
plugin.jvm=${SW_PLUGIN_JVM:true}
# Plugin configurations
plugin.springmvc=${SW_PLUGIN_SPRINGMVC:true}
plugin.tomcat=${SW_PLUGIN_TOMCAT:true}
plugin.httpclient=${SW_PLUGIN_HTTPCLIENT:true}
plugin.mysql=${SW_PLUGIN_MYSQL:true}
plugin.redis=${SW_PLUGIN_REDIS:true}
plugin.kafka=${SW_PLUGIN_KAFKA:true}
# Ignore suffixes
plugin.ignore_suffix=${SW_PLUGIN_IGNORE_SUFFIX:.jpg,.jpeg,.png,.gif,.css,.js}
# Correlation configuration
correlation.element_max_number=${SW_CORRELATION_ELEMENT_MAX_NUMBER:3}
correlation.value_max_length=${SW_CORRELATION_VALUE_MAX_LENGTH:128}
Application Integration Examples
Example 1: Spring Boot Application
application.yml with SkyWalking
spring: application: name: user-service datasource: url: jdbc:mysql://localhost:3306/user_db username: user password: pass management: endpoints: web: exposure: include: health,metrics,info endpoint: health: show-details: always # SkyWalking specific properties (optional) skywalking: logging: level: INFO agent: service_name: user-service collector: backend_service: localhost:11800
Spring Boot Startup Script
#!/bin/bash # start-springboot-app.sh export SW_AGENT_NAME=user-service export SW_AGENT_COLLECTOR_BACKEND_SERVICE=skywalking-collector:11800 export SW_AGENT_SPRING_PLUGINS=true export SW_LOGGING_LEVEL=INFO java -javaagent:/path/to/skywalking-agent/skywalking-agent.jar \ -Dspring.profiles.active=prod \ -jar target/user-service.jar
Example 2: Manual Instrumentation
import org.apache.skywalking.apm.toolkit.trace.ActiveSpan;
import org.apache.skywalking.apm.toolkit.trace.Trace;
import org.apache.skywalking.apm.toolkit.trace.TraceContext;
@Service
public class UserService {
private final UserRepository userRepository;
private final AuditService auditService;
public UserService(UserRepository userRepository, AuditService auditService) {
this.userRepository = userRepository;
this.auditService = auditService;
}
// Automatic tracing with @Trace annotation
@Trace
public User createUser(CreateUserRequest request) {
// Add custom tag to current span
ActiveSpan.tag("user.email", request.getEmail());
ActiveSpan.tag("user.role", request.getRole());
// Log custom event
ActiveSpan.info("Creating new user: " + request.getUsername());
try {
User user = userRepository.save(mapToUser(request));
auditService.logUserCreation(user.getId());
return user;
} catch (Exception e) {
// Mark span as error
ActiveSpan.error(e);
throw e;
}
}
// Custom operation name
@Trace(operationName = "userService/findUserByEmail")
public User findUserByEmail(String email) {
ActiveSpan.tag("search.email", email);
return userRepository.findByEmail(email)
.orElseThrow(() -> new UserNotFoundException("User not found"));
}
// Get trace ID for logging correlation
public String getCurrentTraceId() {
return TraceContext.traceId();
}
// Manual span creation
public void batchProcessUsers(List<String> userIds) {
for (String userId : userIds) {
// Create custom span for each processing operation
try (ActiveSpan span = ActiveSpan.debug("processUser", userId)) {
span.tag("user.id", userId);
processSingleUser(userId);
}
}
}
private void processSingleUser(String userId) {
// Business logic here
ActiveSpan.info("Processing user: " + userId);
}
}
Example 3: Web Application with HTTP Tracing
@RestController
@RequestMapping("/api/users")
public class UserController {
private final UserService userService;
public UserController(UserService userService) {
this.userService = userService;
}
@GetMapping("/{id}")
public ResponseEntity<User> getUser(@PathVariable String id) {
// Trace context is automatically propagated
ActiveSpan.tag("user.id", id);
User user = userService.findUserById(id);
return ResponseEntity.ok(user);
}
@PostMapping
public ResponseEntity<User> createUser(@RequestBody @Valid CreateUserRequest request) {
// Add custom business attributes
ActiveSpan.tag("user.registration.source", "web-api");
ActiveSpan.tag("request.timestamp", Instant.now().toString());
User user = userService.createUser(request);
return ResponseEntity.status(HttpStatus.CREATED).body(user);
}
@GetMapping("/search")
public ResponseEntity<List<User>> searchUsers(
@RequestParam String query,
@RequestParam(defaultValue = "0") int page,
@RequestParam(defaultValue = "10") int size) {
// Custom tags for search parameters
ActiveSpan.tag("search.query", query);
ActiveSpan.tag("search.page", String.valueOf(page));
ActiveSpan.tag("search.size", String.valueOf(size));
List<User> users = userService.searchUsers(query, page, size);
return ResponseEntity.ok(users);
}
}
// Custom filter for additional HTTP tracing
@Component
public class CustomTracingFilter implements Filter {
@Override
public void doFilter(ServletRequest request, ServletResponse response,
FilterChain chain) throws IOException, ServletException {
HttpServletRequest httpRequest = (HttpServletRequest) request;
// Add custom tags for HTTP request
ActiveSpan.tag("http.client_ip", getClientIp(httpRequest));
ActiveSpan.tag("http.user_agent", httpRequest.getHeader("User-Agent"));
ActiveSpan.tag("http.referer", httpRequest.getHeader("Referer"));
chain.doFilter(request, response);
HttpServletResponse httpResponse = (HttpServletResponse) response;
ActiveSpan.tag("http.response_code", String.valueOf(httpResponse.getStatus()));
}
private String getClientIp(HttpServletRequest request) {
String xForwardedFor = request.getHeader("X-Forwarded-For");
if (xForwardedFor != null && !xForwardedFor.isEmpty()) {
return xForwardedFor.split(",")[0];
}
return request.getRemoteAddr();
}
}
Example 4: Database and External Calls Tracing
@Repository
public class UserRepository {
private final JdbcTemplate jdbcTemplate;
private final RestTemplate restTemplate;
public UserRepository(JdbcTemplate jdbcTemplate, RestTemplate restTemplate) {
this.jdbcTemplate = jdbcTemplate;
this.restTemplate = restTemplate;
}
@Trace
public User save(User user) {
String sql = "INSERT INTO users (id, username, email, created_at) VALUES (?, ?, ?, ?)";
// SQL will be automatically traced by SkyWalking
jdbcTemplate.update(sql,
user.getId(), user.getUsername(), user.getEmail(), user.getCreatedAt());
return user;
}
@Trace
public Optional<User> findByEmail(String email) {
String sql = "SELECT * FROM users WHERE email = ?";
// SQL tracing with parameters
ActiveSpan.tag("db.query.type", "SELECT");
ActiveSpan.tag("db.query.table", "users");
try {
User user = jdbcTemplate.queryForObject(sql, new UserRowMapper(), email);
return Optional.ofNullable(user);
} catch (EmptyResultDataAccessException e) {
return Optional.empty();
}
}
@Trace(operationName = "external/call-audit-service")
public void callAuditService(String userId, String action) {
String auditUrl = "http://audit-service/api/audit";
AuditRequest auditRequest = new AuditRequest(userId, action);
// HTTP call will be automatically traced
ResponseEntity<String> response = restTemplate.postForEntity(
auditUrl, auditRequest, String.class);
if (!response.getStatusCode().is2xxSuccessful()) {
ActiveSpan.error(new RuntimeException("Audit service call failed"));
}
}
}
// Custom RowMapper
class UserRowMapper implements RowMapper<User> {
@Override
public User mapRow(ResultSet rs, int rowNum) throws SQLException {
return new User(
rs.getString("id"),
rs.getString("username"),
rs.getString("email"),
rs.getTimestamp("created_at").toInstant()
);
}
}
Example 5: Messaging with Kafka Tracing
@Service
public class UserEventService {
private final KafkaTemplate<String, Object> kafkaTemplate;
public UserEventService(KafkaTemplate<String, Object> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
@Trace
public void sendUserCreatedEvent(User user) {
UserCreatedEvent event = new UserCreatedEvent(
user.getId(),
user.getUsername(),
user.getEmail()
);
// Kafka message sending will be traced
kafkaTemplate.send("user-events", user.getId(), event)
.addCallback(
result -> ActiveSpan.info("User event sent successfully"),
ex -> ActiveSpan.error(ex)
);
}
@KafkaListener(topics = "user-events")
@Trace
public void handleUserEvent(UserCreatedEvent event) {
// Kafka message consumption will be traced
ActiveSpan.tag("kafka.topic", "user-events");
ActiveSpan.tag("kafka.event.type", "USER_CREATED");
try {
processUserEvent(event);
ActiveSpan.info("User event processed successfully");
} catch (Exception e) {
ActiveSpan.error(e);
throw e;
}
}
private void processUserEvent(UserCreatedEvent event) {
// Business logic for processing user events
ActiveSpan.tag("event.user.id", event.getUserId());
}
}
Advanced Configuration
Custom Plugin Configuration
Optional Plugins Activation
# Enable specific optional plugins cp /skywalking-agent/optional-plugins/apm-spring-cloud-gateway-2.1.x-plugin.jar \ /skywalking-agent/plugins/ cp /skywalking-agent/optional-plugins/apm-elasticsearch-6.x-plugin.jar \ /skywalking-agent/plugins/
Custom Configuration File
# config/custom_agent.config
# Custom sampling rate
agent.sample_n_per_3_secs=100
# Ignore health check endpoints
plugin.springmvc.use_qualified_name_as_endpoint_name=true
plugin.tomcat.collect_http_params=false
# Database specific configurations
plugin.mysql.trace_sql_parameters=true
plugin.mysql.sql_parameters_max_length=512
# Redis configuration
plugin.redis.trace_redis_parameters=true
plugin.redis.redis_parameter_max_length=128
# Kafka configuration
plugin.kafka.bootstrap_servers=${KAFKA_BOOTSTRAP_SERVERS:localhost:9092}
# Custom ignore paths
plugin.ignore_path=/health,/metrics,/info
# Profile configuration
agent.active_v2_header=true
agent.cause_exception_depth=5
Docker Compose Setup
# docker-compose.yml version: '3.8' services: skywalking-oap: image: apache/skywalking-oap-server:9.4.0 container_name: skywalking-oap ports: - "11800:11800" # gRPC API for agent - "12800:12800" # HTTP API for UI environment: - SW_STORAGE=elasticsearch - SW_STORAGE_ES_CLUSTER_NODES=elasticsearch:9200 - TZ=UTC depends_on: - elasticsearch skywalking-ui: image: apache/skywalking-ui:9.4.0 container_name: skywalking-ui ports: - "8080:8080" environment: - SW_OAP_ADDRESS=skywalking-oap:12800 depends_on: - skywalking-oap elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0 container_name: elasticsearch ports: - "9200:9200" environment: - discovery.type=single-node - xpack.security.enabled=false - "ES_JAVA_OPTS=-Xms512m -Xmx512m" my-java-app: build: . container_name: my-java-app ports: - "8080:8080" environment: - SW_AGENT_NAME=my-java-app - SW_AGENT_COLLECTOR_BACKEND_SERVICE=skywalking-oap:11800 - JAVA_OPTS=-javaagent:/skywalking-agent/skywalking-agent.jar depends_on: - skywalking-oap
Monitoring and Troubleshooting
Agent Logs Analysis
// Custom logging integration
@Component
public class SkyWalkingLogbackIntegration {
@PostConstruct
public void setupLogging() {
// Add trace ID to logs
LoggerContext context = (LoggerContext) LoggerFactory.getILoggerFactory();
PatternLayoutEncoder encoder = new PatternLayoutEncoder();
encoder.setPattern("%d{yyyy-MM-dd HH:mm:ss} [%thread] [%X{tid}] %-5level %logger{36} - %msg%n");
encoder.setContext(context);
encoder.start();
// Console appender with trace context
ConsoleAppender<ILoggingEvent> consoleAppender = new ConsoleAppender<>();
consoleAppender.setEncoder(encoder);
consoleAppender.setContext(context);
consoleAppender.start();
Logger logger = context.getLogger(Logger.ROOT_LOGGER_NAME);
logger.addAppender(consoleAppender);
}
}
Health Check Endpoint
@RestController
public class MonitoringController {
@GetMapping("/health")
public ResponseEntity<HealthResponse> health() {
HealthResponse health = new HealthResponse(
"UP",
TraceContext.traceId(),
System.currentTimeMillis()
);
return ResponseEntity.ok(health);
}
@GetMapping("/metrics/skywalking")
public ResponseEntity<Map<String, Object>> skywalkingMetrics() {
Map<String, Object> metrics = new HashMap<>();
metrics.put("traceId", TraceContext.traceId());
metrics.put("agentVersion", getAgentVersion());
metrics.put("serviceName", System.getProperty("skywalking.agent.service_name"));
return ResponseEntity.ok(metrics);
}
private String getAgentVersion() {
try {
Package pkg = ActiveSpan.class.getPackage();
return pkg.getImplementationVersion();
} catch (Exception e) {
return "unknown";
}
}
}
Best Practices
1. Naming Conventions
# Use meaningful service names agent.service_name=payment-service-v1 # Use environment-specific names agent.service_name=user-service-prod
2. Performance Optimization
# Adjust sampling for high-throughput services agent.sample_n_per_3_secs=1000 # Limit SQL parameter collection plugin.mysql.sql_parameters_max_length=512 # Disable unnecessary plugins plugin.rabbitmq=false
3. Security Considerations
# Don't collect sensitive HTTP parameters plugin.tomcat.collect_http_params=false plugin.springmvc.collect_http_params=false # Limit SQL parameter exposure plugin.mysql.trace_sql_parameters=false
Conclusion
SkyWalking Java Agent provides comprehensive observability for Java applications with:
- Zero-code instrumentation for popular frameworks
- Distributed tracing across microservices
- Performance metrics and JVM monitoring
- Database and external call tracking
- Powerful visualization and analysis tools
Key benefits:
- Minimal performance overhead (typically 1-3%)
- Automatic context propagation across service boundaries
- Rich plugin ecosystem for common frameworks
- Production-ready with enterprise-grade features
For modern microservices architectures, SkyWalking provides essential observability capabilities that help teams understand system behavior, troubleshoot issues, and optimize performance across distributed systems.