SkyWalking Agent: Distributed Tracing and APM for Java Applications

SkyWalking is an open-source Application Performance Monitoring (APM) tool and distributed tracing system designed for microservices, cloud-native, and container-based architectures. The SkyWalking Java Agent enables automatic instrumentation of Java applications without code modification.

Overview of SkyWalking

SkyWalking provides:

Distributed Tracing: End-to-end request tracking across microservices
Performance Metrics: Application performance indicators and JVM metrics
Topology Mapping: Visual service dependency graphs
Service Mesh Observability: Integration with service mesh frameworks
Database Monitoring: SQL query performance and database connections

Agent Setup and Configuration

Installation Methods

Method 1: Download and Setup

# Download latest SkyWalking agent wget https://archive.apache.org/dist/skywalking/9.4.0/apache-skywalking-apm-9.4.0.tar.gz tar -xzf apache-skywalking-apm-9.4.0.tar.gz cd apache-skywalking-apm-9.4.0 # Agent directory structure agent/ ├── config/ # Configuration files ├── plugins/ # Auto-instrumentation plugins ├── optional-plugins/ # Optional plugins ├── bootstrap-plugins/ # Bootstrap plugins └── skywalking-agent.jar # Main agent JAR

Method 2: Docker Setup

FROM openjdk:11-jre-slim # Copy SkyWalking agent COPY skywalking-agent/ /skywalking-agent/ # Application JAR COPY target/my-app.jar /app.jar # Run with SkyWalking agent ENTRYPOINT ["java", "-javaagent:/skywalking-agent/skywalking-agent.jar", \ "-Dskywalking.agent.service_name=my-service", \ "-Dskywalking.collector.backend_service=collector:11800", \ "-jar", "/app.jar"]

Basic Configuration

agent.config

# Agent basic configuration agent.service_name=${SW_AGENT_NAME:My-Java-Service} agent.sample_n_per_3_secs=${SW_AGENT_SAMPLE:1000} agent.force_reconnection_period=${SW_AGENT_FORCE_RECONNECTION_PERIOD:1} # Collector configuration collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICE:127.0.0.1:11800} collector.grpc_channel_check_interval=${SW_GRPC_CHANNEL_CHECK_INTERVAL:30} collector.get_profile_task_interval=${SW_GET_PROFILE_TASK_INTERVAL:20} # Logging configuration logging.file_name=${SW_LOGGING_FILE_NAME:skywalking-api.log} logging.level=${SW_LOGGING_LEVEL:INFO} logging.dir=${SW_LOGGING_DIR:logs} # JVM configuration plugin.jvm=${SW_PLUGIN_JVM:true} # Plugin configurations plugin.springmvc=${SW_PLUGIN_SPRINGMVC:true} plugin.tomcat=${SW_PLUGIN_TOMCAT:true} plugin.httpclient=${SW_PLUGIN_HTTPCLIENT:true} plugin.mysql=${SW_PLUGIN_MYSQL:true} plugin.redis=${SW_PLUGIN_REDIS:true} plugin.kafka=${SW_PLUGIN_KAFKA:true} # Ignore suffixes plugin.ignore_suffix=${SW_PLUGIN_IGNORE_SUFFIX:.jpg,.jpeg,.png,.gif,.css,.js} # Correlation configuration correlation.element_max_number=${SW_CORRELATION_ELEMENT_MAX_NUMBER:3} correlation.value_max_length=${SW_CORRELATION_VALUE_MAX_LENGTH:128}

Application Integration Examples

Example 1: Spring Boot Application

application.yml with SkyWalking

spring: application: name: user-service datasource: url: jdbc:mysql://localhost:3306/user_db username: user password: pass management: endpoints: web: exposure: include: health,metrics,info endpoint: health: show-details: always # SkyWalking specific properties (optional) skywalking: logging: level: INFO agent: service_name: user-service collector: backend_service: localhost:11800

Spring Boot Startup Script

#!/bin/bash # start-springboot-app.sh export SW_AGENT_NAME=user-service export SW_AGENT_COLLECTOR_BACKEND_SERVICE=skywalking-collector:11800 export SW_AGENT_SPRING_PLUGINS=true export SW_LOGGING_LEVEL=INFO java -javaagent:/path/to/skywalking-agent/skywalking-agent.jar \ -Dspring.profiles.active=prod \ -jar target/user-service.jar

Example 2: Manual Instrumentation

import org.apache.skywalking.apm.toolkit.trace.ActiveSpan; import org.apache.skywalking.apm.toolkit.trace.Trace; import org.apache.skywalking.apm.toolkit.trace.TraceContext; @Service public class UserService { private final UserRepository userRepository; private final AuditService auditService; public UserService(UserRepository userRepository, AuditService auditService) { this.userRepository = userRepository; this.auditService = auditService; } // Automatic tracing with @Trace annotation @Trace public User createUser(CreateUserRequest request) { // Add custom tag to current span ActiveSpan.tag("user.email", request.getEmail()); ActiveSpan.tag("user.role", request.getRole()); // Log custom event ActiveSpan.info("Creating new user: " + request.getUsername()); try { User user = userRepository.save(mapToUser(request)); auditService.logUserCreation(user.getId()); return user; } catch (Exception e) { // Mark span as error ActiveSpan.error(e); throw e; } } // Custom operation name @Trace(operationName = "userService/findUserByEmail") public User findUserByEmail(String email) { ActiveSpan.tag("search.email", email); return userRepository.findByEmail(email) .orElseThrow(() -> new UserNotFoundException("User not found")); } // Get trace ID for logging correlation public String getCurrentTraceId() { return TraceContext.traceId(); } // Manual span creation public void batchProcessUsers(List<String> userIds) { for (String userId : userIds) { // Create custom span for each processing operation try (ActiveSpan span = ActiveSpan.debug("processUser", userId)) { span.tag("user.id", userId); processSingleUser(userId); } } } private void processSingleUser(String userId) { // Business logic here ActiveSpan.info("Processing user: " + userId); } }

Example 3: Web Application with HTTP Tracing

@RestController @RequestMapping("/api/users") public class UserController { private final UserService userService; public UserController(UserService userService) { this.userService = userService; } @GetMapping("/{id}") public ResponseEntity<User> getUser(@PathVariable String id) { // Trace context is automatically propagated ActiveSpan.tag("user.id", id); User user = userService.findUserById(id); return ResponseEntity.ok(user); } @PostMapping public ResponseEntity<User> createUser(@RequestBody @Valid CreateUserRequest request) { // Add custom business attributes ActiveSpan.tag("user.registration.source", "web-api"); ActiveSpan.tag("request.timestamp", Instant.now().toString()); User user = userService.createUser(request); return ResponseEntity.status(HttpStatus.CREATED).body(user); } @GetMapping("/search") public ResponseEntity<List<User>> searchUsers( @RequestParam String query, @RequestParam(defaultValue = "0") int page, @RequestParam(defaultValue = "10") int size) { // Custom tags for search parameters ActiveSpan.tag("search.query", query); ActiveSpan.tag("search.page", String.valueOf(page)); ActiveSpan.tag("search.size", String.valueOf(size)); List<User> users = userService.searchUsers(query, page, size); return ResponseEntity.ok(users); } } // Custom filter for additional HTTP tracing @Component public class CustomTracingFilter implements Filter { @Override public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { HttpServletRequest httpRequest = (HttpServletRequest) request; // Add custom tags for HTTP request ActiveSpan.tag("http.client_ip", getClientIp(httpRequest)); ActiveSpan.tag("http.user_agent", httpRequest.getHeader("User-Agent")); ActiveSpan.tag("http.referer", httpRequest.getHeader("Referer")); chain.doFilter(request, response); HttpServletResponse httpResponse = (HttpServletResponse) response; ActiveSpan.tag("http.response_code", String.valueOf(httpResponse.getStatus())); } private String getClientIp(HttpServletRequest request) { String xForwardedFor = request.getHeader("X-Forwarded-For"); if (xForwardedFor != null && !xForwardedFor.isEmpty()) { return xForwardedFor.split(",")[0]; } return request.getRemoteAddr(); } }

Example 4: Database and External Calls Tracing

@Repository public class UserRepository { private final JdbcTemplate jdbcTemplate; private final RestTemplate restTemplate; public UserRepository(JdbcTemplate jdbcTemplate, RestTemplate restTemplate) { this.jdbcTemplate = jdbcTemplate; this.restTemplate = restTemplate; } @Trace public User save(User user) { String sql = "INSERT INTO users (id, username, email, created_at) VALUES (?, ?, ?, ?)"; // SQL will be automatically traced by SkyWalking jdbcTemplate.update(sql, user.getId(), user.getUsername(), user.getEmail(), user.getCreatedAt()); return user; } @Trace public Optional<User> findByEmail(String email) { String sql = "SELECT * FROM users WHERE email = ?"; // SQL tracing with parameters ActiveSpan.tag("db.query.type", "SELECT"); ActiveSpan.tag("db.query.table", "users"); try { User user = jdbcTemplate.queryForObject(sql, new UserRowMapper(), email); return Optional.ofNullable(user); } catch (EmptyResultDataAccessException e) { return Optional.empty(); } } @Trace(operationName = "external/call-audit-service") public void callAuditService(String userId, String action) { String auditUrl = "http://audit-service/api/audit"; AuditRequest auditRequest = new AuditRequest(userId, action); // HTTP call will be automatically traced ResponseEntity<String> response = restTemplate.postForEntity( auditUrl, auditRequest, String.class); if (!response.getStatusCode().is2xxSuccessful()) { ActiveSpan.error(new RuntimeException("Audit service call failed")); } } } // Custom RowMapper class UserRowMapper implements RowMapper<User> { @Override public User mapRow(ResultSet rs, int rowNum) throws SQLException { return new User( rs.getString("id"), rs.getString("username"), rs.getString("email"), rs.getTimestamp("created_at").toInstant() ); } }

Example 5: Messaging with Kafka Tracing

@Service public class UserEventService { private final KafkaTemplate<String, Object> kafkaTemplate; public UserEventService(KafkaTemplate<String, Object> kafkaTemplate) { this.kafkaTemplate = kafkaTemplate; } @Trace public void sendUserCreatedEvent(User user) { UserCreatedEvent event = new UserCreatedEvent( user.getId(), user.getUsername(), user.getEmail() ); // Kafka message sending will be traced kafkaTemplate.send("user-events", user.getId(), event) .addCallback( result -> ActiveSpan.info("User event sent successfully"), ex -> ActiveSpan.error(ex) ); } @KafkaListener(topics = "user-events") @Trace public void handleUserEvent(UserCreatedEvent event) { // Kafka message consumption will be traced ActiveSpan.tag("kafka.topic", "user-events"); ActiveSpan.tag("kafka.event.type", "USER_CREATED"); try { processUserEvent(event); ActiveSpan.info("User event processed successfully"); } catch (Exception e) { ActiveSpan.error(e); throw e; } } private void processUserEvent(UserCreatedEvent event) { // Business logic for processing user events ActiveSpan.tag("event.user.id", event.getUserId()); } }

Advanced Configuration

Custom Plugin Configuration

Optional Plugins Activation

# Enable specific optional plugins cp /skywalking-agent/optional-plugins/apm-spring-cloud-gateway-2.1.x-plugin.jar \ /skywalking-agent/plugins/ cp /skywalking-agent/optional-plugins/apm-elasticsearch-6.x-plugin.jar \ /skywalking-agent/plugins/

Custom Configuration File

# config/custom_agent.config # Custom sampling rate agent.sample_n_per_3_secs=100 # Ignore health check endpoints plugin.springmvc.use_qualified_name_as_endpoint_name=true plugin.tomcat.collect_http_params=false # Database specific configurations plugin.mysql.trace_sql_parameters=true plugin.mysql.sql_parameters_max_length=512 # Redis configuration plugin.redis.trace_redis_parameters=true plugin.redis.redis_parameter_max_length=128 # Kafka configuration plugin.kafka.bootstrap_servers=${KAFKA_BOOTSTRAP_SERVERS:localhost:9092} # Custom ignore paths plugin.ignore_path=/health,/metrics,/info # Profile configuration agent.active_v2_header=true agent.cause_exception_depth=5

Docker Compose Setup

# docker-compose.yml version: '3.8' services: skywalking-oap: image: apache/skywalking-oap-server:9.4.0 container_name: skywalking-oap ports: - "11800:11800" # gRPC API for agent - "12800:12800" # HTTP API for UI environment: - SW_STORAGE=elasticsearch - SW_STORAGE_ES_CLUSTER_NODES=elasticsearch:9200 - TZ=UTC depends_on: - elasticsearch skywalking-ui: image: apache/skywalking-ui:9.4.0 container_name: skywalking-ui ports: - "8080:8080" environment: - SW_OAP_ADDRESS=skywalking-oap:12800 depends_on: - skywalking-oap elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0 container_name: elasticsearch ports: - "9200:9200" environment: - discovery.type=single-node - xpack.security.enabled=false - "ES_JAVA_OPTS=-Xms512m -Xmx512m" my-java-app: build: . container_name: my-java-app ports: - "8080:8080" environment: - SW_AGENT_NAME=my-java-app - SW_AGENT_COLLECTOR_BACKEND_SERVICE=skywalking-oap:11800 - JAVA_OPTS=-javaagent:/skywalking-agent/skywalking-agent.jar depends_on: - skywalking-oap

Monitoring and Troubleshooting

Agent Logs Analysis

// Custom logging integration @Component public class SkyWalkingLogbackIntegration { @PostConstruct public void setupLogging() { // Add trace ID to logs LoggerContext context = (LoggerContext) LoggerFactory.getILoggerFactory(); PatternLayoutEncoder encoder = new PatternLayoutEncoder(); encoder.setPattern("%d{yyyy-MM-dd HH:mm:ss} [%thread] [%X{tid}] %-5level %logger{36} - %msg%n"); encoder.setContext(context); encoder.start(); // Console appender with trace context ConsoleAppender<ILoggingEvent> consoleAppender = new ConsoleAppender<>(); consoleAppender.setEncoder(encoder); consoleAppender.setContext(context); consoleAppender.start(); Logger logger = context.getLogger(Logger.ROOT_LOGGER_NAME); logger.addAppender(consoleAppender); } }

Health Check Endpoint

@RestController public class MonitoringController { @GetMapping("/health") public ResponseEntity<HealthResponse> health() { HealthResponse health = new HealthResponse( "UP", TraceContext.traceId(), System.currentTimeMillis() ); return ResponseEntity.ok(health); } @GetMapping("/metrics/skywalking") public ResponseEntity<Map<String, Object>> skywalkingMetrics() { Map<String, Object> metrics = new HashMap<>(); metrics.put("traceId", TraceContext.traceId()); metrics.put("agentVersion", getAgentVersion()); metrics.put("serviceName", System.getProperty("skywalking.agent.service_name")); return ResponseEntity.ok(metrics); } private String getAgentVersion() { try { Package pkg = ActiveSpan.class.getPackage(); return pkg.getImplementationVersion(); } catch (Exception e) { return "unknown"; } } }

Best Practices

1. Naming Conventions

# Use meaningful service names agent.service_name=payment-service-v1 # Use environment-specific names agent.service_name=user-service-prod

2. Performance Optimization

# Adjust sampling for high-throughput services agent.sample_n_per_3_secs=1000 # Limit SQL parameter collection plugin.mysql.sql_parameters_max_length=512 # Disable unnecessary plugins plugin.rabbitmq=false

3. Security Considerations

# Don't collect sensitive HTTP parameters plugin.tomcat.collect_http_params=false plugin.springmvc.collect_http_params=false # Limit SQL parameter exposure plugin.mysql.trace_sql_parameters=false

Conclusion

SkyWalking Java Agent provides comprehensive observability for Java applications with:

Zero-code instrumentation for popular frameworks
Distributed tracing across microservices
Performance metrics and JVM monitoring
Database and external call tracking
Powerful visualization and analysis tools

Key benefits:

Minimal performance overhead (typically 1-3%)
Automatic context propagation across service boundaries
Rich plugin ecosystem for common frameworks
Production-ready with enterprise-grade features

For modern microservices architectures, SkyWalking provides essential observability capabilities that help teams understand system behavior, troubleshoot issues, and optimize performance across distributed systems.