Introduction to PMTiles
PMTiles (Protomaps Tiles) is an efficient file format for storing large sets of map tiles in a single file. Unlike traditional tile storage that uses thousands of individual files, PMTiles bundles all tiles into one optimized archive with a built-in spatial index, making it ideal for distribution and serving map data.
Key Advantages of PMTiles
- Single File Distribution: All tiles packaged in one file
- Efficient Random Access: Built-in spatial index for fast tile retrieval
- HTTP Range Request Support: Can be served from simple static hosting
- Compression Support: Optional compression to reduce file size
- Metadata Storage: Flexible metadata system for storing additional information
Java Implementation Overview
Let's build a complete Java implementation for reading PMTiles files.
Core Dependencies
First, add these dependencies to your pom.xml:
<dependencies> <dependency> <groupId>com.google.protobuf</groupId> <artifactId>protobuf-java</artifactId> <version>3.25.1</version> </dependency> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-compress</artifactId> <version>1.24.0</version> </dependency> </dependencies>
PMTiles Protobuf Definitions
Create the protocol buffer definitions based on the PMTiles specification:
syntax = "proto3";
message Header {
uint64 tile_data_offset = 1;
uint64 tile_data_length = 2;
uint64 tile_entries_offset = 3;
uint64 tile_entries_length = 4;
uint64 tile_contents_offset = 5;
uint64 tile_contents_length = 6;
uint64 num_tiles = 7;
uint32 min_zoom = 8;
uint32 max_zoom = 9;
float min_lat = 10;
float min_lon = 11;
float max_lat = 12;
float max_lon = 13;
uint32 tile_compression = 14;
uint32 tile_type = 15;
string metadata = 16;
}
message DirectoryEntry {
uint64 tile_id = 1;
uint64 offset = 2;
uint32 length = 3;
uint32 run_length = 4;
}
message TileEntry {
uint64 tile_id = 1;
uint64 offset = 2;
uint32 length = 3;
}
Core PMTiles Reader Implementation
import java.io.*;
import java.n.ByteBuffer;
import java.nio.channels.FileChannel;
import java.util.*;
import java.util.zip.Inflater;
public class PMTilesReader implements AutoCloseable {
private final RandomAccessFile file;
private final FileChannel channel;
private Header header;
private Map<Long, DirectoryEntry> directory;
public PMTilesReader(String filePath) throws IOException {
this.file = new RandomAccessFile(filePath, "r");
this.channel = file.getChannel();
readHeader();
readDirectory();
}
private void readHeader() throws IOException {
// Read the first 127 bytes for header
ByteBuffer buffer = ByteBuffer.allocate(127);
channel.read(buffer, 0);
buffer.flip();
// Check magic number
byte[] magic = new byte[7];
buffer.get(magic);
if (!"PMTiles".equals(new String(magic))) {
throw new IOException("Invalid PMTiles file: magic number mismatch");
}
// Parse header using Protobuf (simplified here)
this.header = parseHeader(buffer);
}
private Header parseHeader(ByteBuffer buffer) {
// Simplified header parsing - in real implementation, use generated Protobuf classes
Header header = new Header();
// Parse header fields from buffer according to PMTiles spec
return header;
}
private void readDirectory() throws IOException {
this.directory = new HashMap<>();
ByteBuffer dirBuffer = ByteBuffer.allocate((int) header.tile_entries_length);
channel.read(dirBuffer, header.tile_entries_offset);
dirBuffer.flip();
// Parse directory entries
while (dirBuffer.remaining() >= 17) { // Each entry is 17 bytes
DirectoryEntry entry = parseDirectoryEntry(dirBuffer);
directory.put(entry.tile_id, entry);
}
}
private DirectoryEntry parseDirectoryEntry(ByteBuffer buffer) {
DirectoryEntry entry = new DirectoryEntry();
entry.tile_id = readVarint(buffer);
entry.offset = readVarint(buffer);
entry.length = (int) readVarint(buffer);
entry.run_length = (int) readVarint(buffer);
return entry;
}
private long readVarint(ByteBuffer buffer) {
long result = 0;
int shift = 0;
byte b;
do {
b = buffer.get();
result |= (long) (b & 0x7F) << shift;
shift += 7;
} while ((b & 0x80) != 0);
return result;
}
public byte[] getTile(int z, int x, int y) throws IOException {
long tileId = getTileId(z, x, y);
DirectoryEntry entry = directory.get(tileId);
if (entry == null) {
return null; // Tile not found
}
return readTileData(entry.offset, entry.length);
}
private byte[] readTileData(long offset, int length) throws IOException {
ByteBuffer buffer = ByteBuffer.allocate(length);
channel.read(buffer, header.tile_data_offset + offset);
byte[] data = buffer.array();
// Handle compression
if (header.tile_compression == 1) { // GZIP
return decompressGzip(data);
} else if (header.tile_compression == 2) { // Brotli
return decompressBrotli(data);
}
return data; // No compression
}
private byte[] decompressGzip(byte[] data) throws IOException {
// Implement GZIP decompression
java.util.zip.GZIPInputStream gis =
new java.util.zip.GZIPInputStream(new ByteArrayInputStream(data));
return gis.readAllBytes();
}
private byte[] decompressBrotli(byte[] data) throws IOException {
// Brotli decompression would require additional library
// For now, return as-is - implement with brotli-dec dependency
return data;
}
public static long getTileId(int z, int x, int y) {
// Convert Z/X/Y to Hilbert tile ID
return zxyToTileId(z, x, y);
}
private static long zxyToTileId(int z, int x, int y) {
// Implementation of Z/X/Y to Hilbert curve ID
// This is a simplified version - full implementation needed
long acc = 0;
long t = 0;
for (int i = 0; i < z; i++) {
acc += (1L << (2 * i));
}
for (int i = z - 1; i >= 0; i--) {
long mask = 1L << i;
long rx = (x & mask) != 0 ? 1 : 0;
long ry = (y & mask) != 0 ? 1 : 0;
t += (1L << (2 * i)) * ((3 * rx) ^ ry);
rotate(i, rx, ry, x, y);
}
return acc + t;
}
private static void rotate(int n, long rx, long ry, long x, long y) {
if (ry == 0) {
if (rx == 1) {
x = (1L << n) - 1 - x;
y = (1L << n) - 1 - y;
}
long t = x;
x = y;
y = t;
}
}
public Header getHeader() {
return header;
}
public Map<String, Object> getMetadata() throws IOException {
if (header.metadata != null && !header.metadata.isEmpty()) {
// Parse JSON metadata
// Using simple JSON parsing - consider using Jackson/Gson in production
return parseMetadata(header.metadata);
}
return Collections.emptyMap();
}
private Map<String, Object> parseMetadata(String metadataJson) {
// Simplified JSON parsing
Map<String, Object> metadata = new HashMap<>();
// Implement proper JSON parsing here
return metadata;
}
@Override
public void close() throws IOException {
if (channel != null) {
channel.close();
}
if (file != null) {
file.close();
}
}
}
PMTiles Writer Implementation
import java.io.*;
import java.n.ByteBuffer;
import java.nio.channels.FileChannel;
import java.util.*;
public class PMTilesWriter implements AutoCloseable {
private final RandomAccessFile file;
private final FileChannel channel;
private final List<TileEntry> tiles;
private final Map<String, Object> metadata;
private long currentOffset = 0;
public PMTilesWriter(String filePath) throws IOException {
this.file = new RandomAccessFile(filePath, "rw");
this.channel = file.getChannel();
this.tiles = new ArrayList<>();
this.metadata = new HashMap<>();
// Reserve space for header
writeHeaderPlaceholder();
}
public void addTile(int z, int x, int y, byte[] data) throws IOException {
long tileId = PMTilesReader.getTileId(z, x, y);
TileEntry entry = new TileEntry();
entry.tile_id = tileId;
entry.offset = currentOffset;
entry.length = data.length;
// Write tile data
channel.write(ByteBuffer.wrap(data), getTileDataOffset() + currentOffset);
currentOffset += data.length;
tiles.add(entry);
}
public void setMetadata(String key, Object value) {
metadata.put(key, value);
}
private void writeHeaderPlaceholder() throws IOException {
ByteBuffer buffer = ByteBuffer.allocate(127);
buffer.put("PMTiles".getBytes()); // Magic number
buffer.position(127); // Fill with zeros
buffer.flip();
channel.write(buffer, 0);
}
private void finalizeFile() throws IOException {
// Sort tiles by tile ID
tiles.sort(Comparator.comparingLong(entry -> entry.tile_id));
// Write directory
long directoryOffset = writeDirectory();
// Write tile contents index
long contentsOffset = writeTileContents();
// Update header with final positions
updateHeader(directoryOffset, contentsOffset);
}
private long writeDirectory() throws IOException {
long offset = channel.position();
ByteBuffer buffer = ByteBuffer.allocate(calculateDirectorySize());
for (TileEntry entry : tiles) {
writeVarint(buffer, entry.tile_id);
writeVarint(buffer, entry.offset);
writeVarint(buffer, entry.length);
}
buffer.flip();
channel.write(buffer, offset);
return offset;
}
private long writeTileContents() throws IOException {
long offset = channel.position();
// Write tile contents entries
// Implementation depends on specific PMTiles version
return offset;
}
private void updateHeader(long directoryOffset, long contentsOffset) throws IOException {
Header header = new Header();
header.tile_data_offset = 127; // After header
header.tile_data_length = currentOffset;
header.tile_entries_offset = directoryOffset;
header.tile_entries_length = calculateDirectorySize();
header.tile_contents_offset = contentsOffset;
// Set other header fields...
writeHeader(header);
}
private void writeHeader(Header header) throws IOException {
ByteBuffer buffer = ByteBuffer.allocate(127);
// Serialize header to buffer
buffer.flip();
channel.write(buffer, 0);
}
private void writeVarint(ByteBuffer buffer, long value) {
while (true) {
if ((value & ~0x7FL) == 0) {
buffer.put((byte) value);
return;
} else {
buffer.put((byte) ((value & 0x7F) | 0x80));
value >>>= 7;
}
}
}
private int calculateDirectorySize() {
// Calculate size needed for directory entries
return tiles.size() * 17; // Approximate size per entry
}
private long getTileDataOffset() {
return 127; // After 127-byte header
}
@Override
public void close() throws IOException {
finalizeFile();
if (channel != null) {
channel.close();
}
if (file != null) {
file.close();
}
}
}
Usage Examples
Reading PMTiles Files
public class PMTilesExample {
public static void main(String[] args) {
try (PMTilesReader reader = new PMTilesReader("map.pmtiles")) {
// Read metadata
Map<String, Object> metadata = reader.getMetadata();
System.out.println("Map name: " + metadata.get("name"));
// Get tile at specific coordinates
byte[] tileData = reader.getTile(14, 4824, 6158);
if (tileData != null) {
// Use tile data (PNG, JPEG, or vector tiles)
System.out.println("Tile size: " + tileData.length + " bytes");
}
// Get header information
Header header = reader.getHeader();
System.out.println("Zoom range: " + header.min_zoom + " to " + header.max_zoom);
} catch (IOException e) {
e.printStackTrace();
}
}
}
Creating PMTiles Files
public class PMTilesCreator {
public static void main(String[] args) {
try (PMTilesWriter writer = new PMTilesWriter("output.pmtiles")) {
// Set metadata
writer.setMetadata("name", "My Map");
writer.setMetadata("description", "Generated map tiles");
writer.setMetadata("format", "pbf");
// Add tiles
for (int z = 0; z <= 14; z++) {
for (int x = 0; x < (1 << z); x++) {
for (int y = 0; y < (1 << z); y++) {
byte[] tileData = generateTileData(z, x, y);
if (tileData != null) {
writer.addTile(z, x, y, tileData);
}
}
}
}
} catch (IOException e) {
e.printStackTrace();
}
}
private static byte[] generateTileData(int z, int x, int y) {
// Generate or load tile data
// This could be from a database, filesystem, or generated dynamically
return new byte[0]; // Implementation specific
}
}
Advanced Features
HTTP Range Request Support
public class PMTilesHttpServer {
private final PMTilesReader reader;
public PMTilesHttpServer(String filePath) throws IOException {
this.reader = new PMTilesReader(filePath);
}
public byte[] handleRangeRequest(String rangeHeader, String tileRequest) throws IOException {
// Parse range header: "bytes=start-end"
String[] rangeParts = rangeHeader.substring(6).split("-");
long start = Long.parseLong(rangeParts[0]);
long end = Long.parseLong(rangeParts[1]);
// Read specific range from PMTiles file
return readRange(start, end);
}
private byte[] readRange(long start, long end) throws IOException {
int length = (int) (end - start + 1);
ByteBuffer buffer = ByteBuffer.allocate(length);
// For actual implementation, you'd need low-level file access
// This is a simplified version
reader.getChannel().read(buffer, start);
return buffer.array();
}
}
Performance Considerations
- Caching: Implement tile caching for frequently accessed tiles
- Memory Mapping: Use memory-mapped files for better performance on large PMTiles files
- Connection Pooling: For HTTP serving, implement connection pooling
- Compression: Choose appropriate compression based on your data type
Conclusion
The PMTiles format offers significant advantages for distributing and serving map tiles in Java applications. This implementation provides a solid foundation for reading and writing PMTiles files, with support for the core features of the specification. For production use, consider adding error handling, caching, and integration with popular web frameworks like Spring Boot for serving tiles over HTTP.
The key benefits of using PMTiles in Java include simplified deployment, efficient storage, and excellent performance for spatial data serving, making it an ideal choice for modern web mapping applications.
Java Observability, Logging Intelligence & AI-Driven Monitoring (APM, Tracing, Logs & Anomaly Detection)
https://macronepal.com/blog/beyond-metrics-observing-serverless-and-traditional-java-applications-with-thundra-apm/
Explains using Thundra APM to observe both serverless and traditional Java applications by combining tracing, metrics, and logs into a unified observability platform for faster debugging and performance insights.
https://macronepal.com/blog/dynatrace-oneagent-in-java-2/
Explains Dynatrace OneAgent for Java, which automatically instruments JVM applications to capture metrics, traces, and logs, enabling full-stack monitoring and root-cause analysis with minimal configuration.
https://macronepal.com/blog/lightstep-java-sdk-distributed-tracing-and-observability-implementation/
Explains Lightstep Java SDK for distributed tracing, helping developers track requests across microservices and identify latency issues using OpenTelemetry-based observability.
https://macronepal.com/blog/honeycomb-io-beeline-for-java-complete-guide-2/
Explains Honeycomb Beeline for Java, which provides high-cardinality observability and deep query capabilities to understand complex system behavior and debug distributed systems efficiently.
https://macronepal.com/blog/lumigo-for-serverless-in-java-complete-distributed-tracing-guide-2/
Explains Lumigo for Java serverless applications, offering automatic distributed tracing, log correlation, and error tracking to simplify debugging in cloud-native environments. (Lumigo Docs)
https://macronepal.com/blog/from-noise-to-signals-implementing-log-anomaly-detection-in-java-applications/
Explains how to detect anomalies in Java logs using behavioral patterns and machine learning techniques to separate meaningful incidents from noisy log data and improve incident response.
https://macronepal.com/blog/ai-powered-log-analysis-in-java-from-reactive-debugging-to-proactive-insights/
Explains AI-driven log analysis for Java applications, shifting from manual debugging to predictive insights that identify issues early and improve system reliability using intelligent log processing.
https://macronepal.com/blog/titliel-java-logging-best-practices/
Explains best practices for Java logging, focusing on structured logs, proper log levels, performance optimization, and ensuring logs are useful for debugging and observability systems.
https://macronepal.com/blog/seeking-a-loguru-for-java-the-quest-for-elegant-and-simple-logging/
Explains the search for simpler, more elegant logging frameworks in Java, comparing modern logging approaches that aim to reduce complexity while improving readability and developer experience.