Building CI/CD Agents in Java: Creating a Custom Buildkite Agent

Article

Buildkite is a powerful CI/CD platform that uses agents to run build jobs. While Buildkite provides official agents in various languages, there are scenarios where you might want to build a custom agent in Java—to integrate with specific Java tooling, leverage existing Java infrastructure, or implement custom job handling logic. This article explores how to build a Buildkite-compatible agent in Java that can connect to the Buildkite platform and execute build jobs.

Understanding Buildkite Agent Architecture

Key Components:

Agent Registration: Connects to Buildkite API and registers itself
Job Polling: Continuously checks for available jobs
Job Execution: Runs build jobs in isolated environments
Log Streaming: Streams job output back to Buildkite
Artifact Handling: Uploads build artifacts
Heartbeat: Maintains connection with Buildkite API

Building a Java-Based Buildkite Agent

1. Core Dependencies

<properties> <jackson.version>2.15.2</jackson.version> <okhttp.version>4.11.0</okhttp.version> <jsch.version>0.1.55</jsch.version> </properties> <dependencies> <!-- HTTP client for Buildkite API --> <dependency> <groupId>com.squareup.okhttp3</groupId> <artifactId>okhttp</artifactId> <version>${okhttp.version}</version> </dependency> <!-- JSON processing --> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>${jackson.version}</version> </dependency> <!-- SSH for git operations --> <dependency> <groupId>com.jcraft</groupId> <artifactId>jsch</artifactId> <version>${jsch.version}</version> </dependency> <!-- Process execution --> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-exec</artifactId> <version>1.3</version> </dependency> <!-- Logging --> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>2.0.7</version> </dependency> <!-- Configuration --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter</artifactId> <version>3.1.0</version> <optional>true</optional> </dependency> </dependencies>

2. Core Configuration Model

@Data @ConfigurationProperties(prefix = "buildkite.agent") public class BuildkiteAgentConfig { private String agentToken; private String endpoint = "https://agent.buildkite.com/v3"; private String name = "java-agent"; private String version = "1.0.0"; private String[] tags = {"java", "custom"}; private int heartbeatInterval = 30; // seconds private int jobPollInterval = 3; // seconds private String workDir = "./buildkite-work"; private String buildsPath = "./buildkite-builds"; private String hooksPath = "./buildkite-hooks"; private boolean debug = false; // Git configuration private GitConfig git = new GitConfig(); @Data public static class GitConfig { private String sshKeyPath; private String username; private String email; private int timeout = 300; // seconds } } @Data public class AgentMetadata { private String name; private String version; private String pid; private String hostname; private String[] tags; private Map<String, String> metaData = new HashMap<>(); public AgentMetadata() { this.pid = String.valueOf(ProcessHandle.current().pid()); try { this.hostname = InetAddress.getLocalHost().getHostName(); } catch (UnknownHostException e) { this.hostname = "unknown"; } } }

3. Buildkite API Client

@Component @Slf4j public class BuildkiteApiClient { private final BuildkiteAgentConfig config; private final OkHttpClient httpClient; private final ObjectMapper objectMapper; private String agentId; private String agentAccessToken; public BuildkiteApiClient(BuildkiteAgentConfig config) { this.config = config; this.objectMapper = new ObjectMapper(); this.httpClient = new OkHttpClient.Builder() .addInterceptor(chain -> { Request original = chain.request(); Request request = original.newBuilder() .header("Authorization", "Token " + config.getAgentToken()) .header("User-Agent", "BuildkiteJavaAgent/" + config.getVersion()) .header("Content-Type", "application/json") .build(); return chain.proceed(request); }) .build(); } public AgentRegistration registerAgent(AgentMetadata metadata) throws IOException { Map<String, Object> payload = Map.of( "name", metadata.getName(), "version", metadata.getVersion(), "pid", metadata.getPid(), "hostname", metadata.getHostname(), "tags", metadata.getTags() ); Request request = new Request.Builder() .url(config.getEndpoint() + "/agents") .post(createJsonBody(payload)) .build(); try (Response response = httpClient.newCall(request).execute()) { if (response.isSuccessful()) { JsonNode responseBody = objectMapper.readTree(response.body().string()); this.agentId = responseBody.get("id").asText(); this.agentAccessToken = responseBody.get("access_token").asText(); log.info("Agent registered successfully: {}", agentId); return new AgentRegistration(agentId, agentAccessToken); } else { throw new IOException("Failed to register agent: " + response.code()); } } } public List<BuildJob> getJobs() throws IOException { Request request = new Request.Builder() .url(config.getEndpoint() + "/jobs") .header("Authorization", "Bearer " + agentAccessToken) .get() .build(); try (Response response = httpClient.newCall(request).execute()) { if (response.isSuccessful()) { String responseBody = response.body().string(); return objectMapper.readValue(responseBody, objectMapper.getTypeFactory().constructCollectionType(List.class, BuildJob.class)); } return List.of(); } } public void acceptJob(String jobId) throws IOException { Request request = new Request.Builder() .url(config.getEndpoint() + "/jobs/" + jobId + "/accept") .header("Authorization", "Bearer " + agentAccessToken) .put(createJsonBody(Map.of())) .build(); try (Response response = httpClient.newCall(request).execute()) { if (!response.isSuccessful()) { throw new IOException("Failed to accept job: " + response.code()); } } } public void startJob(String jobId) throws IOException { Request request = new Request.Builder() .url(config.getEndpoint() + "/jobs/" + jobId + "/start") .header("Authorization", "Bearer " + agentAccessToken) .put(createJsonBody(Map.of())) .build(); executeRequest(request); } public void finishJob(String jobId, int exitStatus, String output) throws IOException { Map<String, Object> payload = Map.of( "exit_status", exitStatus, "output", output ); Request request = new Request.Builder() .url(config.getEndpoint() + "/jobs/" + jobId + "/finish") .header("Authorization", "Bearer " + agentAccessToken) .put(createJsonBody(payload)) .build(); executeRequest(request); } public void sendJobOutput(String jobId, String output) throws IOException { Map<String, Object> payload = Map.of( "output", output ); Request request = new Request.Builder() .url(config.getEndpoint() + "/jobs/" + jobId + "/output") .header("Authorization", "Bearer " + agentAccessToken) .post(createJsonBody(payload)) .build(); executeRequest(request); } public void sendHeartbeat() throws IOException { Request request = new Request.Builder() .url(config.getEndpoint() + "/agents/" + agentId + "/heartbeat") .header("Authorization", "Bearer " + agentAccessToken) .post(createJsonBody(Map.of())) .build(); executeRequest(request); } private RequestBody createJsonBody(Object data) throws IOException { String json = objectMapper.writeValueAsString(data); return RequestBody.create(json, MediaType.parse("application/json")); } private void executeRequest(Request request) throws IOException { try (Response response = httpClient.newCall(request).execute()) { if (!response.isSuccessful()) { throw new IOException("Request failed: " + response.code()); } } } } @Data @AllArgsConstructor class AgentRegistration { private String agentId; private String accessToken; } @Data class BuildJob { private String id; private String state; private Map<String, Object> env; private JobConfig config; private String repository; private String commit; private String branch; private String[] command; @Data public static class JobConfig { private Map<String, Object> plugins; private Map<String, String> environment; private String[] commands; } }

4. Job Execution Engine

@Component @Slf4j public class JobExecutor { private final BuildkiteAgentConfig config; private final BuildkiteApiClient apiClient; private final GitService gitService; public JobExecutor(BuildkiteAgentConfig config, BuildkiteApiClient apiClient, GitService gitService) { this.config = config; this.apiClient = apiClient; this.gitService = gitService; } public JobExecutionResult executeJob(BuildJob job) { String jobId = job.getId(); Path buildPath = null; try { log.info("Starting job execution: {}", jobId); apiClient.startJob(jobId); // Create build directory buildPath = createBuildDirectory(jobId); // Checkout repository log.info("Checking out repository: {}", job.getRepository()); gitService.checkoutRepository(job.getRepository(), job.getCommit(), buildPath); // Execute build commands List<CommandResult> commandResults = new ArrayList<>(); for (String command : job.getConfig().getCommands()) { CommandResult result = executeCommand(command, buildPath, job.getEnv()); commandResults.add(result); apiClient.sendJobOutput(jobId, result.getOutput()); if (result.getExitCode() != 0) { log.warn("Command failed with exit code: {}", result.getExitCode()); break; } } // Collect artifacts if any uploadArtifacts(buildPath, jobId); int finalExitCode = commandResults.stream() .filter(r -> r.getExitCode() != 0) .findFirst() .map(CommandResult::getExitCode) .orElse(0); String finalOutput = commandResults.stream() .map(CommandResult::getOutput) .collect(Collectors.joining("\n")); return new JobExecutionResult(finalExitCode, finalOutput); } catch (Exception e) { log.error("Job execution failed: {}", jobId, e); return new JobExecutionResult(1, "Job execution failed: " + e.getMessage()); } finally { if (buildPath != null) { cleanupBuildDirectory(buildPath); } } } private Path createBuildDirectory(String jobId) throws IOException { Path buildPath = Paths.get(config.getBuildsPath(), jobId); Files.createDirectories(buildPath); return buildPath; } private CommandResult executeCommand(String command, Path workingDir, Map<String, Object> environment) { try { ProcessBuilder processBuilder = new ProcessBuilder(); // Set up command (handle shell commands) if (isWindows()) { processBuilder.command("cmd.exe", "/c", command); } else { processBuilder.command("sh", "-c", command); } // Set working directory processBuilder.directory(workingDir.toFile()); // Set environment variables Map<String, String> env = processBuilder.environment(); if (environment != null) { environment.forEach((key, value) -> env.put(key, value != null ? value.toString() : "")); } // Execute process Process process = processBuilder.start(); // Capture output StringBuilder output = new StringBuilder(); try (BufferedReader reader = new BufferedReader( new InputStreamReader(process.getInputStream()))) { String line; while ((line = reader.readLine()) != null) { output.append(line).append("\n"); log.info("[JOB] {}", line); } } // Capture error output try (BufferedReader errorReader = new BufferedReader( new InputStreamReader(process.getErrorStream()))) { String line; while ((line = errorReader.readLine()) != null) { output.append("[ERROR] ").append(line).append("\n"); log.error("[JOB-ERROR] {}", line); } } int exitCode = process.waitFor(); return new CommandResult(exitCode, output.toString()); } catch (IOException | InterruptedException e) { Thread.currentThread().interrupt(); return new CommandResult(1, "Command execution failed: " + e.getMessage()); } } private void uploadArtifacts(Path buildPath, String jobId) { // Look for common artifact patterns try { List<Path> artifacts = Files.walk(buildPath) .filter(path -> Files.isRegularFile(path)) .filter(path -> isArtifactFile(path)) .collect(Collectors.toList()); if (!artifacts.isEmpty()) { log.info("Found {} artifacts to upload", artifacts.size()); // In a real implementation, upload to Buildkite artifacts API } } catch (IOException e) { log.warn("Failed to scan for artifacts", e); } } private boolean isArtifactFile(Path path) { String fileName = path.getFileName().toString(); return fileName.endsWith(".jar") || fileName.endsWith(".war") || fileName.endsWith(".zip") || fileName.contains("target") || fileName.contains("dist"); } private void cleanupBuildDirectory(Path buildPath) { try { Files.walk(buildPath) .sorted(Comparator.reverseOrder()) .map(Path::toFile) .forEach(File::delete); } catch (IOException e) { log.warn("Failed to cleanup build directory: {}", buildPath, e); } } private boolean isWindows() { return System.getProperty("os.name").toLowerCase().contains("win"); } } @Data @AllArgsConstructor class CommandResult { private int exitCode; private String output; } @Data @AllArgsConstructor class JobExecutionResult { private int exitCode; private String output; }

5. Git Service for Repository Management

@Component @Slf4j public class GitService { private final BuildkiteAgentConfig config; public GitService(BuildkiteAgentConfig config) { this.config = config; setupGitConfig(); } public void checkoutRepository(String repositoryUrl, String commit, Path checkoutPath) { try { // Clone repository executeGitCommand(checkoutPath, "clone", repositoryUrl, "."); // Checkout specific commit executeGitCommand(checkoutPath, "checkout", commit); // Fetch additional info if needed executeGitCommand(checkoutPath, "fetch", "--all"); log.info("Successfully checked out {} at {}", repositoryUrl, commit); } catch (Exception e) { throw new RuntimeException("Git checkout failed", e); } } private void executeGitCommand(Path workingDir, String... commands) { List<String> commandList = new ArrayList<>(); commandList.add("git"); commandList.addAll(Arrays.asList(commands)); ProcessBuilder processBuilder = new ProcessBuilder(commandList); processBuilder.directory(workingDir.toFile()); // Set up SSH key if configured if (config.getGit().getSshKeyPath() != null) { Map<String, String> env = processBuilder.environment(); env.put("GIT_SSH_COMMAND", "ssh -i " + config.getGit().getSshKeyPath()); } try { Process process = processBuilder.start(); int exitCode = process.waitFor(); if (exitCode != 0) { String errorOutput = readStream(process.getErrorStream()); throw new IOException("Git command failed: " + errorOutput); } } catch (IOException | InterruptedException e) { Thread.currentThread().interrupt(); throw new RuntimeException("Git command execution failed", e); } } private void setupGitConfig() { if (config.getGit().getUsername() != null) { executeGlobalGitCommand("config", "user.name", config.getGit().getUsername()); } if (config.getGit().getEmail() != null) { executeGlobalGitCommand("config", "user.email", config.getGit().getEmail()); } } private void executeGlobalGitCommand(String... commands) { List<String> commandList = new ArrayList<>(); commandList.add("git"); commandList.addAll(Arrays.asList(commands)); try { Process process = new ProcessBuilder(commandList).start(); process.waitFor(); } catch (IOException | InterruptedException e) { log.warn("Failed to execute global git command", e); } } private String readStream(InputStream inputStream) throws IOException { try (BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream))) { return reader.lines().collect(Collectors.joining("\n")); } } }

6. Main Agent Service

@Service @Slf4j public class BuildkiteAgentService { private final BuildkiteAgentConfig config; private final BuildkiteApiClient apiClient; private final JobExecutor jobExecutor; private final AgentMetadata metadata; private volatile boolean running = false; private final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(3); public BuildkiteAgentService(BuildkiteAgentConfig config, BuildkiteApiClient apiClient, JobExecutor jobExecutor) { this.config = config; this.apiClient = apiClient; this.jobExecutor = jobExecutor; this.metadata = createAgentMetadata(); } public void start() { log.info("Starting Buildkite Java Agent: {}", metadata.getName()); try { // Register agent with Buildkite apiClient.registerAgent(metadata); // Start heartbeat scheduler.scheduleAtFixedRate(this::sendHeartbeat, 0, config.getHeartbeatInterval(), TimeUnit.SECONDS); // Start job polling scheduler.scheduleAtFixedRate(this::pollForJobs, 0, config.getJobPollInterval(), TimeUnit.SECONDS); running = true; log.info("Buildkite Java Agent started successfully"); } catch (Exception e) { log.error("Failed to start Buildkite agent", e); stop(); } } public void stop() { log.info("Stopping Buildkite Java Agent"); running = false; scheduler.shutdown(); try { if (!scheduler.awaitTermination(10, TimeUnit.SECONDS)) { scheduler.shutdownNow(); } } catch (InterruptedException e) { scheduler.shutdownNow(); Thread.currentThread().interrupt(); } } private void sendHeartbeat() { if (!running) return; try { apiClient.sendHeartbeat(); log.debug("Heartbeat sent successfully"); } catch (Exception e) { log.error("Failed to send heartbeat", e); } } private void pollForJobs() { if (!running) return; try { List<BuildJob> jobs = apiClient.getJobs(); for (BuildJob job : jobs) { if ("scheduled".equals(job.getState())) { log.info("Found scheduled job: {}", job.getId()); executeJobAsync(job); } } } catch (Exception e) { log.error("Failed to poll for jobs", e); } } private void executeJobAsync(BuildJob job) { CompletableFuture.runAsync(() -> { try { // Accept the job apiClient.acceptJob(job.getId()); // Execute the job JobExecutionResult result = jobExecutor.executeJob(job); // Finish the job apiClient.finishJob(job.getId(), result.getExitCode(), result.getOutput()); log.info("Job completed: {} with exit code: {}", job.getId(), result.getExitCode()); } catch (Exception e) { log.error("Job execution failed: {}", job.getId(), e); try { apiClient.finishJob(job.getId(), 1, "Job execution failed: " + e.getMessage()); } catch (IOException ex) { log.error("Failed to report job failure", ex); } } }); } private AgentMetadata createAgentMetadata() { AgentMetadata metadata = new AgentMetadata(); metadata.setName(config.getName()); metadata.setVersion(config.getVersion()); metadata.setTags(config.getTags()); // Add system information metadata.getMetaData().put("java.version", System.getProperty("java.version")); metadata.getMetaData().put("os.name", System.getProperty("os.name")); metadata.getMetaData().put("os.arch", System.getProperty("os.arch")); return metadata; } public boolean isRunning() { return running; } }

7. Spring Boot Application

@SpringBootApplication @EnableConfigurationProperties(BuildkiteAgentConfig.class) @Slf4j public class BuildkiteJavaAgentApplication implements CommandLineRunner { @Autowired private BuildkiteAgentService agentService; @Autowired private BuildkiteAgentConfig config; public static void main(String[] args) { SpringApplication.run(BuildkiteJavaAgentApplication.class, args); } @Override public void run(String... args) { log.info("Initializing Buildkite Java Agent with config: {}", config); // Add shutdown hook Runtime.getRuntime().addShutdownHook(new Thread(() -> { log.info("Shutdown signal received"); agentService.stop(); })); // Start the agent agentService.start(); // Keep the application running while (agentService.isRunning()) { try { Thread.sleep(1000); } catch (InterruptedException e) { Thread.currentThread().interrupt(); break; } } } @Bean public BuildkiteApiClient buildkiteApiClient(BuildkiteAgentConfig config) { return new BuildkiteApiClient(config); } @Bean public GitService gitService(BuildkiteAgentConfig config) { return new GitService(config); } @Bean public JobExecutor jobExecutor(BuildkiteAgentConfig config, BuildkiteApiClient apiClient, GitService gitService) { return new JobExecutor(config, apiClient, gitService); } @Bean public BuildkiteAgentService buildkiteAgentService(BuildkiteAgentConfig config, BuildkiteApiClient apiClient, JobExecutor jobExecutor) { return new BuildkiteAgentService(config, apiClient, jobExecutor); } }

8. Configuration

application.yml:

buildkite: agent: agent-token: ${BUILDKITE_AGENT_TOKEN:your-agent-token-here} name: "java-build-agent" version: "1.0.0" tags: ["java", "maven", "gradle", "custom"] heartbeat-interval: 30 job-poll-interval: 3 work-dir: "./buildkite-work" builds-path: "./buildkite-builds" hooks-path: "./buildkite-hooks" debug: true git: ssh-key-path: "${HOME}/.ssh/id_rsa" username: "buildkite-agent" email: "[email protected]" timeout: 300 logging: level: com.yourcompany.buildkite: DEBUG pattern: console: "%d{yyyy-MM-dd HH:mm:ss} - %-5level - %logger{36} - %msg%n"

Building and Running

Build the application:

mvn clean package

Run the agent:

export BUILDKITE_AGENT_TOKEN="your-actual-token" java -jar buildkite-java-agent.jar

Features and Capabilities

This Java-based Buildkite agent provides:

Agent Registration: Registers with Buildkite platform
Job Polling: Continuously checks for new jobs
Git Integration: Clones repositories and checks out commits
Command Execution: Runs build commands with environment variables
Log Streaming: Streams build output to Buildkite in real-time
Artifact Detection: Identifies and can upload build artifacts
Heartbeat Monitoring: Maintains connection with Buildkite
Graceful Shutdown: Handles shutdown signals properly

Use Cases

Java-Specific Builds: Optimized for Java/Maven/Gradle builds
Custom Tooling Integration: Integrate with internal Java-based tools
Enhanced Security: Run in secured Java environments
Existing Infrastructure: Leverage existing Java deployment infrastructure
Custom Job Processing: Implement specialized job handling logic

Conclusion

Building a custom Buildkite agent in Java provides flexibility and control over your CI/CD pipeline execution. While the official agents are robust, a Java-based agent can be tailored to specific organizational needs, integrated with existing Java infrastructure, and extended with custom functionality. This implementation provides a solid foundation that can be extended with additional features like Docker support, advanced artifact handling, or custom plugin systems.