Article
Buildkite is a powerful CI/CD platform that uses agents to run build jobs. While Buildkite provides official agents in various languages, there are scenarios where you might want to build a custom agent in Java—to integrate with specific Java tooling, leverage existing Java infrastructure, or implement custom job handling logic. This article explores how to build a Buildkite-compatible agent in Java that can connect to the Buildkite platform and execute build jobs.
Understanding Buildkite Agent Architecture
Key Components:
- Agent Registration: Connects to Buildkite API and registers itself
- Job Polling: Continuously checks for available jobs
- Job Execution: Runs build jobs in isolated environments
- Log Streaming: Streams job output back to Buildkite
- Artifact Handling: Uploads build artifacts
- Heartbeat: Maintains connection with Buildkite API
Building a Java-Based Buildkite Agent
1. Core Dependencies
<properties>
<jackson.version>2.15.2</jackson.version>
<okhttp.version>4.11.0</okhttp.version>
<jsch.version>0.1.55</jsch.version>
</properties>
<dependencies>
<!-- HTTP client for Buildkite API -->
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>okhttp</artifactId>
<version>${okhttp.version}</version>
</dependency>
<!-- JSON processing -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.version}</version>
</dependency>
<!-- SSH for git operations -->
<dependency>
<groupId>com.jcraft</groupId>
<artifactId>jsch</artifactId>
<version>${jsch.version}</version>
</dependency>
<!-- Process execution -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-exec</artifactId>
<version>1.3</version>
</dependency>
<!-- Logging -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>2.0.7</version>
</dependency>
<!-- Configuration -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
<version>3.1.0</version>
<optional>true</optional>
</dependency>
</dependencies>
2. Core Configuration Model
@Data
@ConfigurationProperties(prefix = "buildkite.agent")
public class BuildkiteAgentConfig {
private String agentToken;
private String endpoint = "https://agent.buildkite.com/v3";
private String name = "java-agent";
private String version = "1.0.0";
private String[] tags = {"java", "custom"};
private int heartbeatInterval = 30; // seconds
private int jobPollInterval = 3; // seconds
private String workDir = "./buildkite-work";
private String buildsPath = "./buildkite-builds";
private String hooksPath = "./buildkite-hooks";
private boolean debug = false;
// Git configuration
private GitConfig git = new GitConfig();
@Data
public static class GitConfig {
private String sshKeyPath;
private String username;
private String email;
private int timeout = 300; // seconds
}
}
@Data
public class AgentMetadata {
private String name;
private String version;
private String pid;
private String hostname;
private String[] tags;
private Map<String, String> metaData = new HashMap<>();
public AgentMetadata() {
this.pid = String.valueOf(ProcessHandle.current().pid());
try {
this.hostname = InetAddress.getLocalHost().getHostName();
} catch (UnknownHostException e) {
this.hostname = "unknown";
}
}
}
3. Buildkite API Client
@Component
@Slf4j
public class BuildkiteApiClient {
private final BuildkiteAgentConfig config;
private final OkHttpClient httpClient;
private final ObjectMapper objectMapper;
private String agentId;
private String agentAccessToken;
public BuildkiteApiClient(BuildkiteAgentConfig config) {
this.config = config;
this.objectMapper = new ObjectMapper();
this.httpClient = new OkHttpClient.Builder()
.addInterceptor(chain -> {
Request original = chain.request();
Request request = original.newBuilder()
.header("Authorization", "Token " + config.getAgentToken())
.header("User-Agent", "BuildkiteJavaAgent/" + config.getVersion())
.header("Content-Type", "application/json")
.build();
return chain.proceed(request);
})
.build();
}
public AgentRegistration registerAgent(AgentMetadata metadata) throws IOException {
Map<String, Object> payload = Map.of(
"name", metadata.getName(),
"version", metadata.getVersion(),
"pid", metadata.getPid(),
"hostname", metadata.getHostname(),
"tags", metadata.getTags()
);
Request request = new Request.Builder()
.url(config.getEndpoint() + "/agents")
.post(createJsonBody(payload))
.build();
try (Response response = httpClient.newCall(request).execute()) {
if (response.isSuccessful()) {
JsonNode responseBody = objectMapper.readTree(response.body().string());
this.agentId = responseBody.get("id").asText();
this.agentAccessToken = responseBody.get("access_token").asText();
log.info("Agent registered successfully: {}", agentId);
return new AgentRegistration(agentId, agentAccessToken);
} else {
throw new IOException("Failed to register agent: " + response.code());
}
}
}
public List<BuildJob> getJobs() throws IOException {
Request request = new Request.Builder()
.url(config.getEndpoint() + "/jobs")
.header("Authorization", "Bearer " + agentAccessToken)
.get()
.build();
try (Response response = httpClient.newCall(request).execute()) {
if (response.isSuccessful()) {
String responseBody = response.body().string();
return objectMapper.readValue(responseBody,
objectMapper.getTypeFactory().constructCollectionType(List.class, BuildJob.class));
}
return List.of();
}
}
public void acceptJob(String jobId) throws IOException {
Request request = new Request.Builder()
.url(config.getEndpoint() + "/jobs/" + jobId + "/accept")
.header("Authorization", "Bearer " + agentAccessToken)
.put(createJsonBody(Map.of()))
.build();
try (Response response = httpClient.newCall(request).execute()) {
if (!response.isSuccessful()) {
throw new IOException("Failed to accept job: " + response.code());
}
}
}
public void startJob(String jobId) throws IOException {
Request request = new Request.Builder()
.url(config.getEndpoint() + "/jobs/" + jobId + "/start")
.header("Authorization", "Bearer " + agentAccessToken)
.put(createJsonBody(Map.of()))
.build();
executeRequest(request);
}
public void finishJob(String jobId, int exitStatus, String output) throws IOException {
Map<String, Object> payload = Map.of(
"exit_status", exitStatus,
"output", output
);
Request request = new Request.Builder()
.url(config.getEndpoint() + "/jobs/" + jobId + "/finish")
.header("Authorization", "Bearer " + agentAccessToken)
.put(createJsonBody(payload))
.build();
executeRequest(request);
}
public void sendJobOutput(String jobId, String output) throws IOException {
Map<String, Object> payload = Map.of(
"output", output
);
Request request = new Request.Builder()
.url(config.getEndpoint() + "/jobs/" + jobId + "/output")
.header("Authorization", "Bearer " + agentAccessToken)
.post(createJsonBody(payload))
.build();
executeRequest(request);
}
public void sendHeartbeat() throws IOException {
Request request = new Request.Builder()
.url(config.getEndpoint() + "/agents/" + agentId + "/heartbeat")
.header("Authorization", "Bearer " + agentAccessToken)
.post(createJsonBody(Map.of()))
.build();
executeRequest(request);
}
private RequestBody createJsonBody(Object data) throws IOException {
String json = objectMapper.writeValueAsString(data);
return RequestBody.create(json, MediaType.parse("application/json"));
}
private void executeRequest(Request request) throws IOException {
try (Response response = httpClient.newCall(request).execute()) {
if (!response.isSuccessful()) {
throw new IOException("Request failed: " + response.code());
}
}
}
}
@Data
@AllArgsConstructor
class AgentRegistration {
private String agentId;
private String accessToken;
}
@Data
class BuildJob {
private String id;
private String state;
private Map<String, Object> env;
private JobConfig config;
private String repository;
private String commit;
private String branch;
private String[] command;
@Data
public static class JobConfig {
private Map<String, Object> plugins;
private Map<String, String> environment;
private String[] commands;
}
}
4. Job Execution Engine
@Component
@Slf4j
public class JobExecutor {
private final BuildkiteAgentConfig config;
private final BuildkiteApiClient apiClient;
private final GitService gitService;
public JobExecutor(BuildkiteAgentConfig config, BuildkiteApiClient apiClient,
GitService gitService) {
this.config = config;
this.apiClient = apiClient;
this.gitService = gitService;
}
public JobExecutionResult executeJob(BuildJob job) {
String jobId = job.getId();
Path buildPath = null;
try {
log.info("Starting job execution: {}", jobId);
apiClient.startJob(jobId);
// Create build directory
buildPath = createBuildDirectory(jobId);
// Checkout repository
log.info("Checking out repository: {}", job.getRepository());
gitService.checkoutRepository(job.getRepository(), job.getCommit(), buildPath);
// Execute build commands
List<CommandResult> commandResults = new ArrayList<>();
for (String command : job.getConfig().getCommands()) {
CommandResult result = executeCommand(command, buildPath, job.getEnv());
commandResults.add(result);
apiClient.sendJobOutput(jobId, result.getOutput());
if (result.getExitCode() != 0) {
log.warn("Command failed with exit code: {}", result.getExitCode());
break;
}
}
// Collect artifacts if any
uploadArtifacts(buildPath, jobId);
int finalExitCode = commandResults.stream()
.filter(r -> r.getExitCode() != 0)
.findFirst()
.map(CommandResult::getExitCode)
.orElse(0);
String finalOutput = commandResults.stream()
.map(CommandResult::getOutput)
.collect(Collectors.joining("\n"));
return new JobExecutionResult(finalExitCode, finalOutput);
} catch (Exception e) {
log.error("Job execution failed: {}", jobId, e);
return new JobExecutionResult(1, "Job execution failed: " + e.getMessage());
} finally {
if (buildPath != null) {
cleanupBuildDirectory(buildPath);
}
}
}
private Path createBuildDirectory(String jobId) throws IOException {
Path buildPath = Paths.get(config.getBuildsPath(), jobId);
Files.createDirectories(buildPath);
return buildPath;
}
private CommandResult executeCommand(String command, Path workingDir,
Map<String, Object> environment) {
try {
ProcessBuilder processBuilder = new ProcessBuilder();
// Set up command (handle shell commands)
if (isWindows()) {
processBuilder.command("cmd.exe", "/c", command);
} else {
processBuilder.command("sh", "-c", command);
}
// Set working directory
processBuilder.directory(workingDir.toFile());
// Set environment variables
Map<String, String> env = processBuilder.environment();
if (environment != null) {
environment.forEach((key, value) ->
env.put(key, value != null ? value.toString() : ""));
}
// Execute process
Process process = processBuilder.start();
// Capture output
StringBuilder output = new StringBuilder();
try (BufferedReader reader = new BufferedReader(
new InputStreamReader(process.getInputStream()))) {
String line;
while ((line = reader.readLine()) != null) {
output.append(line).append("\n");
log.info("[JOB] {}", line);
}
}
// Capture error output
try (BufferedReader errorReader = new BufferedReader(
new InputStreamReader(process.getErrorStream()))) {
String line;
while ((line = errorReader.readLine()) != null) {
output.append("[ERROR] ").append(line).append("\n");
log.error("[JOB-ERROR] {}", line);
}
}
int exitCode = process.waitFor();
return new CommandResult(exitCode, output.toString());
} catch (IOException | InterruptedException e) {
Thread.currentThread().interrupt();
return new CommandResult(1, "Command execution failed: " + e.getMessage());
}
}
private void uploadArtifacts(Path buildPath, String jobId) {
// Look for common artifact patterns
try {
List<Path> artifacts = Files.walk(buildPath)
.filter(path -> Files.isRegularFile(path))
.filter(path -> isArtifactFile(path))
.collect(Collectors.toList());
if (!artifacts.isEmpty()) {
log.info("Found {} artifacts to upload", artifacts.size());
// In a real implementation, upload to Buildkite artifacts API
}
} catch (IOException e) {
log.warn("Failed to scan for artifacts", e);
}
}
private boolean isArtifactFile(Path path) {
String fileName = path.getFileName().toString();
return fileName.endsWith(".jar") ||
fileName.endsWith(".war") ||
fileName.endsWith(".zip") ||
fileName.contains("target") ||
fileName.contains("dist");
}
private void cleanupBuildDirectory(Path buildPath) {
try {
Files.walk(buildPath)
.sorted(Comparator.reverseOrder())
.map(Path::toFile)
.forEach(File::delete);
} catch (IOException e) {
log.warn("Failed to cleanup build directory: {}", buildPath, e);
}
}
private boolean isWindows() {
return System.getProperty("os.name").toLowerCase().contains("win");
}
}
@Data
@AllArgsConstructor
class CommandResult {
private int exitCode;
private String output;
}
@Data
@AllArgsConstructor
class JobExecutionResult {
private int exitCode;
private String output;
}
5. Git Service for Repository Management
@Component
@Slf4j
public class GitService {
private final BuildkiteAgentConfig config;
public GitService(BuildkiteAgentConfig config) {
this.config = config;
setupGitConfig();
}
public void checkoutRepository(String repositoryUrl, String commit, Path checkoutPath) {
try {
// Clone repository
executeGitCommand(checkoutPath, "clone", repositoryUrl, ".");
// Checkout specific commit
executeGitCommand(checkoutPath, "checkout", commit);
// Fetch additional info if needed
executeGitCommand(checkoutPath, "fetch", "--all");
log.info("Successfully checked out {} at {}", repositoryUrl, commit);
} catch (Exception e) {
throw new RuntimeException("Git checkout failed", e);
}
}
private void executeGitCommand(Path workingDir, String... commands) {
List<String> commandList = new ArrayList<>();
commandList.add("git");
commandList.addAll(Arrays.asList(commands));
ProcessBuilder processBuilder = new ProcessBuilder(commandList);
processBuilder.directory(workingDir.toFile());
// Set up SSH key if configured
if (config.getGit().getSshKeyPath() != null) {
Map<String, String> env = processBuilder.environment();
env.put("GIT_SSH_COMMAND", "ssh -i " + config.getGit().getSshKeyPath());
}
try {
Process process = processBuilder.start();
int exitCode = process.waitFor();
if (exitCode != 0) {
String errorOutput = readStream(process.getErrorStream());
throw new IOException("Git command failed: " + errorOutput);
}
} catch (IOException | InterruptedException e) {
Thread.currentThread().interrupt();
throw new RuntimeException("Git command execution failed", e);
}
}
private void setupGitConfig() {
if (config.getGit().getUsername() != null) {
executeGlobalGitCommand("config", "user.name", config.getGit().getUsername());
}
if (config.getGit().getEmail() != null) {
executeGlobalGitCommand("config", "user.email", config.getGit().getEmail());
}
}
private void executeGlobalGitCommand(String... commands) {
List<String> commandList = new ArrayList<>();
commandList.add("git");
commandList.addAll(Arrays.asList(commands));
try {
Process process = new ProcessBuilder(commandList).start();
process.waitFor();
} catch (IOException | InterruptedException e) {
log.warn("Failed to execute global git command", e);
}
}
private String readStream(InputStream inputStream) throws IOException {
try (BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream))) {
return reader.lines().collect(Collectors.joining("\n"));
}
}
}
6. Main Agent Service
@Service
@Slf4j
public class BuildkiteAgentService {
private final BuildkiteAgentConfig config;
private final BuildkiteApiClient apiClient;
private final JobExecutor jobExecutor;
private final AgentMetadata metadata;
private volatile boolean running = false;
private final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(3);
public BuildkiteAgentService(BuildkiteAgentConfig config, BuildkiteApiClient apiClient,
JobExecutor jobExecutor) {
this.config = config;
this.apiClient = apiClient;
this.jobExecutor = jobExecutor;
this.metadata = createAgentMetadata();
}
public void start() {
log.info("Starting Buildkite Java Agent: {}", metadata.getName());
try {
// Register agent with Buildkite
apiClient.registerAgent(metadata);
// Start heartbeat
scheduler.scheduleAtFixedRate(this::sendHeartbeat, 0,
config.getHeartbeatInterval(), TimeUnit.SECONDS);
// Start job polling
scheduler.scheduleAtFixedRate(this::pollForJobs, 0,
config.getJobPollInterval(), TimeUnit.SECONDS);
running = true;
log.info("Buildkite Java Agent started successfully");
} catch (Exception e) {
log.error("Failed to start Buildkite agent", e);
stop();
}
}
public void stop() {
log.info("Stopping Buildkite Java Agent");
running = false;
scheduler.shutdown();
try {
if (!scheduler.awaitTermination(10, TimeUnit.SECONDS)) {
scheduler.shutdownNow();
}
} catch (InterruptedException e) {
scheduler.shutdownNow();
Thread.currentThread().interrupt();
}
}
private void sendHeartbeat() {
if (!running) return;
try {
apiClient.sendHeartbeat();
log.debug("Heartbeat sent successfully");
} catch (Exception e) {
log.error("Failed to send heartbeat", e);
}
}
private void pollForJobs() {
if (!running) return;
try {
List<BuildJob> jobs = apiClient.getJobs();
for (BuildJob job : jobs) {
if ("scheduled".equals(job.getState())) {
log.info("Found scheduled job: {}", job.getId());
executeJobAsync(job);
}
}
} catch (Exception e) {
log.error("Failed to poll for jobs", e);
}
}
private void executeJobAsync(BuildJob job) {
CompletableFuture.runAsync(() -> {
try {
// Accept the job
apiClient.acceptJob(job.getId());
// Execute the job
JobExecutionResult result = jobExecutor.executeJob(job);
// Finish the job
apiClient.finishJob(job.getId(), result.getExitCode(), result.getOutput());
log.info("Job completed: {} with exit code: {}", job.getId(), result.getExitCode());
} catch (Exception e) {
log.error("Job execution failed: {}", job.getId(), e);
try {
apiClient.finishJob(job.getId(), 1, "Job execution failed: " + e.getMessage());
} catch (IOException ex) {
log.error("Failed to report job failure", ex);
}
}
});
}
private AgentMetadata createAgentMetadata() {
AgentMetadata metadata = new AgentMetadata();
metadata.setName(config.getName());
metadata.setVersion(config.getVersion());
metadata.setTags(config.getTags());
// Add system information
metadata.getMetaData().put("java.version", System.getProperty("java.version"));
metadata.getMetaData().put("os.name", System.getProperty("os.name"));
metadata.getMetaData().put("os.arch", System.getProperty("os.arch"));
return metadata;
}
public boolean isRunning() {
return running;
}
}
7. Spring Boot Application
@SpringBootApplication
@EnableConfigurationProperties(BuildkiteAgentConfig.class)
@Slf4j
public class BuildkiteJavaAgentApplication implements CommandLineRunner {
@Autowired
private BuildkiteAgentService agentService;
@Autowired
private BuildkiteAgentConfig config;
public static void main(String[] args) {
SpringApplication.run(BuildkiteJavaAgentApplication.class, args);
}
@Override
public void run(String... args) {
log.info("Initializing Buildkite Java Agent with config: {}", config);
// Add shutdown hook
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
log.info("Shutdown signal received");
agentService.stop();
}));
// Start the agent
agentService.start();
// Keep the application running
while (agentService.isRunning()) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
break;
}
}
}
@Bean
public BuildkiteApiClient buildkiteApiClient(BuildkiteAgentConfig config) {
return new BuildkiteApiClient(config);
}
@Bean
public GitService gitService(BuildkiteAgentConfig config) {
return new GitService(config);
}
@Bean
public JobExecutor jobExecutor(BuildkiteAgentConfig config, BuildkiteApiClient apiClient,
GitService gitService) {
return new JobExecutor(config, apiClient, gitService);
}
@Bean
public BuildkiteAgentService buildkiteAgentService(BuildkiteAgentConfig config,
BuildkiteApiClient apiClient,
JobExecutor jobExecutor) {
return new BuildkiteAgentService(config, apiClient, jobExecutor);
}
}
8. Configuration
application.yml:
buildkite:
agent:
agent-token: ${BUILDKITE_AGENT_TOKEN:your-agent-token-here}
name: "java-build-agent"
version: "1.0.0"
tags: ["java", "maven", "gradle", "custom"]
heartbeat-interval: 30
job-poll-interval: 3
work-dir: "./buildkite-work"
builds-path: "./buildkite-builds"
hooks-path: "./buildkite-hooks"
debug: true
git:
ssh-key-path: "${HOME}/.ssh/id_rsa"
username: "buildkite-agent"
email: "[email protected]"
timeout: 300
logging:
level:
com.yourcompany.buildkite: DEBUG
pattern:
console: "%d{yyyy-MM-dd HH:mm:ss} - %-5level - %logger{36} - %msg%n"
Building and Running
Build the application:
mvn clean package
Run the agent:
export BUILDKITE_AGENT_TOKEN="your-actual-token" java -jar buildkite-java-agent.jar
Features and Capabilities
This Java-based Buildkite agent provides:
- Agent Registration: Registers with Buildkite platform
- Job Polling: Continuously checks for new jobs
- Git Integration: Clones repositories and checks out commits
- Command Execution: Runs build commands with environment variables
- Log Streaming: Streams build output to Buildkite in real-time
- Artifact Detection: Identifies and can upload build artifacts
- Heartbeat Monitoring: Maintains connection with Buildkite
- Graceful Shutdown: Handles shutdown signals properly
Use Cases
- Java-Specific Builds: Optimized for Java/Maven/Gradle builds
- Custom Tooling Integration: Integrate with internal Java-based tools
- Enhanced Security: Run in secured Java environments
- Existing Infrastructure: Leverage existing Java deployment infrastructure
- Custom Job Processing: Implement specialized job handling logic
Conclusion
Building a custom Buildkite agent in Java provides flexibility and control over your CI/CD pipeline execution. While the official agents are robust, a Java-based agent can be tailored to specific organizational needs, integrated with existing Java infrastructure, and extended with custom functionality. This implementation provides a solid foundation that can be extended with additional features like Docker support, advanced artifact handling, or custom plugin systems.