Static Application Security Testing (SAST) has become essential in modern software development, and GitHub's CodeQL is one of the most powerful SAST tools available. However, the real power emerges when you automate the process of uploading and analyzing CodeQL results. This article explores how to programmatically upload SARIF (Static Analysis Results Interchange Format) files in Java, enabling seamless integration of CodeQL into your CI/CD pipelines and security workflows.
Understanding the CodeQL SARIF Ecosystem
Key Components:
- CodeQL: GitHub's semantic code analysis engine
- SARIF: Standardized format for static analysis results (OASIS Standard)
- GitHub Code Scanning API: REST API for uploading SARIF files to GitHub
- CodeQL CLI: Command-line interface for running CodeQL analysis
The Workflow:
CodeQL Analysis → SARIF Results → Upload via API → GitHub Code Scanning
Prerequisites
Before implementing SARIF upload, ensure you have:
- GitHub Personal Access Token with
security_eventswrite permission - CodeQL CLI installed and configured
- SARIF file generated from CodeQL analysis
- Java 11+ with HTTP client capabilities
Manual SARIF Upload Process
First, let's understand the manual process:
# Run CodeQL analysis codeql database create java-db --language=java --source-root=. codeql database analyze java-db --format=sarif-latest --output=results.sarif java-security-and-quality.qls # Upload via GitHub CLI gh api \ -H "Accept: application/vnd.github.v3+json" \ /repos/owner/repo/code-scanning/sarifs \ -f commit_sha='abc123' \ -f ref='refs/heads/main' \ -f [email protected]
Now, let's automate this in Java!
Java Implementation for SARIF Upload
Maven Dependencies:
<dependencies> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.15.2</version> </dependency> <dependency> <groupId>org.apache.httpcomponents.client5</groupId> <artifactId>httpclient5</artifactId> <version>5.2.1</version> </dependency> <dependency> <groupId>org.kohsuke</groupId> <artifactId>github-api</artifactId> <version>1.315</version> </dependency> </dependencies>
Core SARIF Upload Service
package com.example.codeql.upload;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.hc.client5.http.classic.methods.HttpPost;
import org.apache.hc.client5.http.entity.mime.MultipartEntityBuilder;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.core5.http.ContentType;
import org.apache.hc.core5.http.io.entity.EntityUtils;
import org.apache.hc.core5.http.io.entity.StringEntity;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Base64;
public class SarifUploadService {
private final String githubToken;
private final String repository;
private final String apiUrl;
private final ObjectMapper objectMapper;
public SarifUploadService(String githubToken, String repositoryOwner, String repositoryName) {
this.githubToken = githubToken;
this.repository = repositoryOwner + "/" + repositoryName;
this.apiUrl = "https://api.github.com";
this.objectMapper = new ObjectMapper();
}
/**
* Uploads SARIF file to GitHub Code Scanning
*/
public UploadResult uploadSarif(File sarifFile, String commitSha, String ref)
throws IOException, InterruptedException {
String sarifContent = Files.readString(sarifFile.toPath());
String encodedSarif = Base64.getEncoder().encodeToString(sarifContent.getBytes());
// Create upload request payload
String payload = String.format(
"{\"commit_sha\":\"%s\",\"ref\":\"%s\",\"sarif\":\"%s\"}",
commitSha, ref, encodedSarif
);
String uploadUrl = apiUrl + "/repos/" + repository + "/code-scanning/sarifs";
try (CloseableHttpClient httpClient = HttpClients.createDefault()) {
HttpPost httpPost = new HttpPost(uploadUrl);
// Set headers
httpPost.setHeader("Accept", "application/vnd.github.v3+json");
httpPost.setHeader("Authorization", "token " + githubToken);
httpPost.setHeader("Content-Type", "application/json");
httpPost.setHeader("X-GitHub-Api-Version", "2022-11-28");
// Set payload
httpPost.setEntity(new StringEntity(payload, ContentType.APPLICATION_JSON));
// Execute request
try (CloseableHttpResponse response = httpClient.execute(httpPost)) {
String responseBody = EntityUtils.toString(response.getEntity());
int statusCode = response.getCode();
if (statusCode == 202) {
JsonNode jsonResponse = objectMapper.readTree(responseBody);
String id = jsonResponse.get("id").asText();
String url = jsonResponse.get("url").asText();
return new UploadResult(true, id, url, "Upload initiated successfully");
} else {
JsonNode errorResponse = objectMapper.readTree(responseBody);
String errorMessage = errorResponse.get("message").asText();
return new UploadResult(false, null, null,
"Upload failed: " + errorMessage);
}
}
}
}
/**
* Checks the status of a SARIF upload
*/
public ProcessingStatus checkUploadStatus(String uploadId) throws IOException {
String statusUrl = apiUrl + "/repos/" + repository + "/code-scanning/sarifs/" + uploadId;
try (CloseableHttpClient httpClient = HttpClients.createDefault()) {
HttpGet httpGet = new HttpGet(statusUrl);
httpGet.setHeader("Accept", "application/vnd.github.v3+json");
httpGet.setHeader("Authorization", "token " + githubToken);
httpGet.setHeader("X-GitHub-Api-Version", "2022-11-28");
try (CloseableHttpResponse response = httpClient.execute(httpGet)) {
String responseBody = EntityUtils.toString(response.getEntity());
JsonNode jsonResponse = objectMapper.readTree(responseBody);
String status = jsonResponse.get("processing_status").asText();
String message = jsonResponse.path("error").asText(null);
return new ProcessingStatus(status, message);
}
}
}
// Data classes for results
public static class UploadResult {
private final boolean success;
private final String id;
private final String url;
private final String message;
public UploadResult(boolean success, String id, String url, String message) {
this.success = success;
this.id = id;
this.url = url;
this.message = message;
}
// Getters
public boolean isSuccess() { return success; }
public String getId() { return id; }
public String getUrl() { return url; }
public String getMessage() { return message; }
}
public static class ProcessingStatus {
private final String status;
private final String error;
public ProcessingStatus(String status, String error) {
this.status = status;
this.error = error;
}
// Getters
public String getStatus() { return status; }
public String getError() { return error; }
public boolean isComplete() { return "complete".equals(status); }
public boolean isPending() { return "pending".equals(status); }
public boolean isFailed() { return "failed".equals(status); }
}
}
Advanced SARIF Upload with Retry Logic
package com.example.codeql.upload;
import java.io.File;
import java.io.IOException;
import java.time.Duration;
import java.time.Instant;
import java.util.concurrent.TimeUnit;
public class AdvancedSarifUploader {
private final SarifUploadService uploadService;
private final int maxRetries;
private final long retryDelayMs;
public AdvancedSarifUploader(SarifUploadService uploadService, int maxRetries, long retryDelayMs) {
this.uploadService = uploadService;
this.maxRetries = maxRetries;
this.retryDelayMs = retryDelayMs;
}
/**
* Uploads SARIF with retry logic and status polling
*/
public void uploadWithRetry(File sarifFile, String commitSha, String ref,
long timeoutSeconds) throws IOException, InterruptedException {
SarifUploadService.UploadResult uploadResult = null;
// Retry upload on failure
for (int attempt = 1; attempt <= maxRetries; attempt++) {
try {
System.out.printf("Upload attempt %d of %d%n", attempt, maxRetries);
uploadResult = uploadService.uploadSarif(sarifFile, commitSha, ref);
if (uploadResult.isSuccess()) {
System.out.println("Upload initiated successfully. ID: " + uploadResult.getId());
break;
} else if (attempt == maxRetries) {
throw new IOException("Final upload attempt failed: " + uploadResult.getMessage());
}
} catch (Exception e) {
if (attempt == maxRetries) {
throw new IOException("All upload attempts failed", e);
}
}
TimeUnit.MILLISECONDS.sleep(retryDelayMs);
}
// Poll for processing completion
if (uploadResult != null && uploadResult.isSuccess()) {
waitForProcessing(uploadResult.getId(), timeoutSeconds);
}
}
private void waitForProcessing(String uploadId, long timeoutSeconds)
throws IOException, InterruptedException {
Instant start = Instant.now();
Duration timeout = Duration.ofSeconds(timeoutSeconds);
while (Duration.between(start, Instant.now()).compareTo(timeout) < 0) {
SarifUploadService.ProcessingStatus status = uploadService.checkUploadStatus(uploadId);
switch (status.getStatus()) {
case "complete":
System.out.println("SARIF processing completed successfully");
return;
case "failed":
throw new IOException("SARIF processing failed: " + status.getError());
case "pending":
System.out.println("SARIF processing still pending...");
break;
default:
System.out.println("Unknown status: " + status.getStatus());
break;
}
TimeUnit.SECONDS.sleep(5); // Wait 5 seconds between checks
}
throw new IOException("SARIF processing timeout exceeded");
}
}
Integrating with CodeQL Analysis
package com.example.codeql.analysis;
import java.io.*;
import java.nio.file.*;
import java.util.concurrent.TimeUnit;
public class CodeQLAnalyzer {
private final String codeqlPath;
public CodeQLAnalyzer(String codeqlPath) {
this.codeqlPath = codeqlPath;
}
/**
* Runs complete CodeQL analysis and returns SARIF file
*/
public File runAnalysis(Path sourceDir, String language, String outputDir)
throws IOException, InterruptedException {
Path dbPath = Paths.get(outputDir, "codeql-db");
Path sarifPath = Paths.get(outputDir, "results.sarif");
// Create CodeQL database
createDatabase(sourceDir, dbPath.toString(), language);
// Analyze database
analyzeDatabase(dbPath.toString(), sarifPath.toString(), language);
return sarifPath.toFile();
}
private void createDatabase(Path sourceDir, String dbPath, String language)
throws IOException, InterruptedException {
ProcessBuilder builder = new ProcessBuilder(
codeqlPath, "database", "create",
dbPath,
"--language=" + language,
"--source-root", sourceDir.toString(),
"--overwrite"
);
Process process = builder.start();
boolean completed = process.waitFor(10, TimeUnit.MINUTES);
if (!completed || process.exitValue() != 0) {
throw new IOException("CodeQL database creation failed");
}
}
private void analyzeDatabase(String dbPath, String outputPath, String language)
throws IOException, InterruptedException {
String queries = "java-security-and-quality.qls";
ProcessBuilder builder = new ProcessBuilder(
codeqlPath, "database", "analyze",
dbPath,
"--format=sarif-latest",
"--output=" + outputPath,
queries
);
Process process = builder.start();
// Capture output for debugging
try (BufferedReader reader = new BufferedReader(
new InputStreamReader(process.getInputStream()))) {
String line;
while ((line = reader.readLine()) != null) {
System.out.println("CodeQL: " + line);
}
}
boolean completed = process.waitFor(15, TimeUnit.MINUTES);
if (!completed || process.exitValue() != 0) {
throw new IOException("CodeQL analysis failed");
}
}
}
Complete End-to-End Example
package com.example.codeql;
import com.example.codeql.analysis.CodeQLAnalyzer;
import com.example.codeql.upload.AdvancedSarifUploader;
import com.example.codeql.upload.SarifUploadService;
import java.io.File;
import java.nio.file.Path;
import java.nio.file.Paths;
public class CodeQLSarifPipeline {
public static void main(String[] args) {
try {
// Configuration
String githubToken = System.getenv("GITHUB_TOKEN");
String repositoryOwner = "your-org";
String repositoryName = "your-repo";
String commitSha = System.getenv("GIT_COMMIT_SHA");
String branchRef = "refs/heads/main";
String codeqlPath = "/opt/codeql/codeql";
// Initialize services
SarifUploadService uploadService = new SarifUploadService(
githubToken, repositoryOwner, repositoryName);
AdvancedSarifUploader advancedUploader = new AdvancedSarifUploader(
uploadService, 3, 5000);
CodeQLAnalyzer analyzer = new CodeQLAnalyzer(codeqlPath);
// Run analysis
Path sourceDir = Paths.get(".");
Path outputDir = Paths.get("codeql-output");
Files.createDirectories(outputDir);
System.out.println("Starting CodeQL analysis...");
File sarifFile = analyzer.runAnalysis(
sourceDir, "java", outputDir.toString());
System.out.println("Uploading SARIF results...");
advancedUploader.uploadWithRetry(sarifFile, commitSha, branchRef, 300);
System.out.println("CodeQL SARIF pipeline completed successfully!");
} catch (Exception e) {
System.err.println("Pipeline failed: " + e.getMessage());
e.printStackTrace();
System.exit(1);
}
}
}
CI/CD Integration Example
GitHub Actions Workflow:
name: CodeQL SARIF Upload
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
analyze-upload:
runs-on: ubuntu-latest
permissions:
security-events: write
contents: read
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up JDK 17
uses: actions/setup-java@v3
with:
java-version: '17'
distribution: 'temurin'
- name: Setup CodeQL
uses: github/codeql-action/init@v2
with:
languages: java
- name: Build application
run: mvn compile -DskipTests
- name: Run Custom SARIF Upload
run: |
java -jar codeql-uploader.jar
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GIT_COMMIT_SHA: ${{ github.sha }}
Error Handling and Best Practices
1. Validate SARIF File:
public class SarifValidator {
public static boolean isValidSarif(File sarifFile) throws IOException {
String content = Files.readString(sarifFile.toPath());
return content.contains("\"$schema\"") &&
content.contains("\"version\"") &&
content.contains("\"runs\"");
}
}
2. Handle Rate Limiting:
private void handleRateLimit(CloseableHttpResponse response) {
String remaining = response.getFirstHeader("X-RateLimit-Remaining").getValue();
String resetTime = response.getFirstHeader("X-RateLimit-Reset").getValue();
if ("0".equals(remaining)) {
long resetTimestamp = Long.parseLong(resetTime) * 1000;
long waitTime = resetTimestamp - System.currentTimeMillis();
if (waitTime > 0) {
try {
Thread.sleep(waitTime);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
}
Benefits of Programmatic SARIF Upload
- Custom Integration: Embed security scanning in custom tools
- Batch Processing: Upload multiple SARIF files programmatically
- Conditional Uploads: Only upload when specific conditions are met
- Advanced Analytics: Process results before uploading
- Hybrid Pipelines: Combine multiple security tools
Conclusion
Programmatic SARIF upload in Java provides powerful capabilities for integrating CodeQL security scanning into complex development workflows. By using the approaches outlined in this article, you can:
- Automate Security Scanning: Integrate CodeQL into existing CI/CD pipelines
- Handle Complex Scenarios: Implement retry logic, status polling, and error handling
- Customize Processing: Add pre-upload validation and filtering
- Scale Security: Process multiple repositories and analysis results
This approach moves security from being a manual, after-the-fact process to an automated, integral part of your software development lifecycle, helping you catch vulnerabilities earlier and maintain higher security standards across your Java applications.