BeakGraph is an Apache Jena Graph implementation of RDF HDT technology pumped into a HDF5 file - or, the same store, a Zarr directory store - and extended to support a full RDF Dataset.
Documentation: ebremer.github.io/BeakGraph
Requires Java 25 and Maven 3.9, and no HDF5 install: every engine reads and writes HDF5 and Zarr in
pure Java, through Falcon (see
docs/INSTRUCTIONS.md §1-2). Falcon is not on Maven Central yet: install it
first, at the version the pom names as falcon.version (0.3.0 for BeakGraph 0.20.0):
git clone https://github.com/ebremer/falcon && cd falcon && mvn -pl hdf5,zarr -am install
Then:
mvn -Plib clean package # self-contained library jar: target/BeakGraph-<version>.jar
mvn -Pcmdlinejar clean package # the same jar, as the command-line tool
mvn -Pcmdlinenative clean package # GraalVM native binary: target/beakgraph
The native binary runs every engine (-method 0..5) and takes Substrate VM's default heap -
raise it per run with beakgraph -XX:MaxHeapSize=16g ... (the equivalent of java -Xmx16g).
The builder JVM needs a large heap (-Dnative.build.heap=32G by default).
Its reachability metadata under src/main/resources/META-INF/native-image is generated with the
tracing agent and must be regenerated after CLI or dependency changes (NativeImageMetadataDriftTest
fails the build when a command-line option is missing from it). Trace into a copy of
reachability-metadata.json in a directory of its own and copy the result back - with
resource-config.json beside it, the agent of GraalVM 25 moves every resource entry into that file:
mkdir meta && cp src/main/resources/META-INF/native-image/reachability-metadata.json meta/
java -agentlib:native-image-agent=config-merge-dir=meta \
-jar target/BeakGraph-<version>.jar <a representative run: -method 0..5, -merge, -export, -verify, -endpoint>
java -jar target/BeakGraph-<version>.jar -help lists every option; the full reference is
docs/INSTRUCTIONS.md. In short:
-src <file|dir> -dest <file|dir>converts RDF to.h5(-method 0..5selects the engine: in-memory 0/2/3, disk-based 1/4/5 for inputs larger than RAM;-mergefolds a tree into one store).-zarr, or a-destnamed*.zarr, writes the same store as a Zarr v3 directory store instead of an HDF5 file, and-zip, or a-destnamed*.zip, as that Zarr store in one ZIP file; every command below takes any of them.-src <store|dir> -export NT|NQ|TTL|TRIG|JSON-LDdumps stores back to RDF.-verify <store|dir> [-deep]integrity-checks stores before publishing them.-endpoint <store|dir> -port 8888serves one store, or a whole directory as W3C LWS storage, over SPARQL. The endpoint is unauthenticated and read-only by design (withSERVICEfederation on); authentication belongs in the proxy or application in front of it - see docs/INSTRUCTIONS.md, "Deployment and trust model".
JMH benchmarks for the read path live in benchmarks/ as a standalone module:
mvn -DskipTests install # install BeakGraph into the local repo
cd benchmarks && mvn package # build the benchmarks jar
java -jar target/benchmarks.jar # run (see benchmarks/README.md for options)
The source syntax is detected from the file name - Turtle, N-Triples, N-Quads,
TriG, RDF/XML and JSON-LD, each also as .gz or .zip - so named graphs can be
loaded from the quad-capable formats (TriG, N-Quads).
BG.getBGWriterBuilder()
.setSource(new File("mydata.ttl"))
.setDestination(new File("mydata.ttl.h5"))
.setSpatial(true) // only needed if GeoSPARQL spatial data is present
.setFeatures(false) // optional: derive 2D shape features for geometries
.build()
.write();A destination named *.zarr (new File("mydata.ttl.zarr")) is written as a Zarr v3 directory store
instead: the same groups, arrays and attributes, in a directory tree any Zarr reader can open. One named
*.zip (mydata.ttl.zarr.zip) is that Zarr store in one file, a ZIP archive with nothing compressed.
Both are read the same way - BG.getBeakGraph takes the directory or the archive - and
BG.getBeakGraph(URI.create("https://host/data/mydata.ttl.zarr")) queries a Zarr store where it lies on a
web server, by range requests, as it does an .h5 file.
File file = new File("mydata.ttl.h5");
try (BeakGraph bg = BG.getBeakGraph(file)) {
Dataset ds = bg.getDataset();
ds.getDefaultModel().write(System.out, "NTRIPLE");
}SPARQL runs with BeakGraph's own execution (index-driven joins, filter pushdown, spatial seeding) whether it is
issued against the dataset or against a Model over the graph (ds.getDefaultModel(),
ModelFactory.createModelForGraph(bg)).
A store on a web server or object store is queried through HTTP range requests, without downloading it (block-cached, retried on transient failures):
try (BeakGraph bg = BG.getBeakGraph(
new HTTPSeekableByteChannel(URI.create("https://example.org/data.h5")))) {
Dataset ds = bg.getDataset();
}Document-relative IRIs (<>, <sibling.png>) are stored unresolved and resolved against the
URL a store is served from; see docs/INSTRUCTIONS.md "Document-relative
IRIs" and SPECIFICATIONS.md §9.2.
- BeakGraph stores are read-only. The in-memory engines (
-method 0/2/3) hold the whole input on the heap; the disk-based engines (-method 1/4/5, spill space under-workdir) bound RAM by spilling and are the way to build stores larger than memory (docs/INSTRUCTIONS.md §6-7). - GeoSPARQL support covers
geof:sfIntersectsonly. It is fully functional: a recall-safe Hilbert cell-cover index produces candidate geometries and every candidate is verified with real JTS geometry, so results are exact. Other GeoSPARQL functions are not implemented. - Coordinate reference systems are not interpreted: the optional
<crs>prefix of ageo:wktLiteralis stripped on both the index and the verification side, every geometry in a store and every query constant is assumed to share one Cartesian CRS (the intended domain is slide/pixel coordinates), and no axis-order or datum transformation is performed. An unprefixed literal adopts the other operand's CRS; a comparison between two literals that both name a CRS and disagree is an evaluation error (the row is dropped) rather than a silent raw-coordinate comparison. .h5files written before the spatial-index redesign carry the old corner-based index entries, which the query side no longer reads - rebuild them from source for spatial queries (their spatial answers were unsound anyway; non-spatial queries are unaffected). Stores built with-spatialby 0.19.0 or earlier can hold repaired pyramid levels (hal:asWKT{s}) with fractional vertices; rebuild them for a consumer that reads integer coordinates only (CHANGELOG.md, 0.20.0).- Numeric literals typed
xsd:int,xsd:long,xsd:floatorxsd:doubleare stored by value and canonicalized at ingest:"01"^^xsd:intis stored - and matched - as"1"^^xsd:int. - A store carries exactly two quad indexes,
GSPOandGPOS(SPECIFICATIONS.md §8); there is no object-first index. A pattern that binds only the object (?s ?p <o>) is answered with oneGPOSprobe per predicate present in the graph - cheap for the usual handful of predicates, proportional to the predicate count otherwise. A pattern binding neither subject nor predicate nor object is a scan of the graph. Property paths are evaluated by Jena's generic path engine overGraph.find: each step is one index lookup (object-seeded steps such as?s :p+ <o>use the per-predicate probes above), but an ungrounded*or+path (?s :p* ?o) makes Jena collect every subject and object node of the graph in memory before the first row, and underurn:x-arq:UnionGrapheach step costs one lookup per named graph. Ground one end of such paths, or run them insideGRAPH <g>.GRAPH ?g { ... }likewise visits every named graph: in one walk on ids for patterns, also underFILTER,OPTIONALandUNION, and on several threads once the walk has produced enough rows. A block that holdsMINUS,BIND, a sub-select, a property path orEXISTS, or one that is more than a pattern and names?gitself, runs graph by graph through Jena's operator, which costs several times as much a graph. ACOUNTof every quad underGRAPH ?g, whole or grouped by graph or predicate, is read off the index without visiting a row. Any otherGROUP BY,COUNT,SUM,AVG,MINorMAXover patterns the engine walks - in one graph or underGRAPH ?g- is computed on ids by the threads that read the rows; a sum of
xsd:doubleorxsd:floatvalues is then added in another order than one thread adds it, and can differ in its last digits. A key that is an expression,SAMPLE,GROUP_CONCATandDISTINCTinside an aggregate other thanCOUNT(DISTINCT ?v)are grouped by Jena, on one thread.
- is computed on ids by the threads that read the rows; a sum of
- Dataset clauses (
FROM,FROM NAMED, and the SPARQL protocol'sdefault-graph-uri/named-graph-uri) keep the BeakGraph engine: a singleFROM <g>is the graph's own view and severalFROMgraphs are a set-union view de-duplicated on ids (the whole-graphDISTINCT ?p/COUNT(*)shortcuts apply to a single graph only). A dataset with a BeakGraph default graph that is not a BeakGraph dataset (a JenaModelover the graph) takes Jena's generic construction instead. - RDF 1.2 support is at the spec's full conformance level (since format v5):
base-direction literals (
"x"@en--ltr,rdf:dirLangString; format v4) and triple terms (<<( s p o )>>, including the reifier/annotation sugar and nesting; format v5) are stored and matched term-exactly across every writer engine, and SPARQL 1.2 triple-term patterns — embedded variables included — are answered from the index. Evidence: the vendored W3C RDF 1.2 suites (215 executed, 0 failures) and SPARQL 1.2 suites (266 executed, 0 failures) run in CI over real stores, on every writer engine (W3CRdf12SuiteTest,W3CSparql12SuiteTest; see RDF_1.2-compliance.md). Versions ≤ 0.17.0 silently stored base-direction literals as plain"x"@en— rebuild affected stores.-verifyprints each store's format version: anything below v4 must be rebuilt if its source held base-direction literals, below v5 if it held triple terms (docs/INSTRUCTIONS.md, "Format versions"). - SPARQL-CDT composite literals (
cdt:List/cdt:Map; the spec is an Unofficial Draft) are stored term-exactly and queryable with Jena'scdt:functions,FOLD, andUNFOLD— the spec's own 659-test suite runs against BeakGraph stores in CI (SparqlCdtSuiteTest). Caveats: the dictionary orders composite literals lexically, so stores built by ≤ 0.17.0 that contain them must be rebuilt; blank nodes inside composite literals are rejected at ingest (labels regenerate from rank, which would silently sever their co-reference); and composite elements are opaque to the index — filtering or joining inside a list/map parses the whole value in RAM, so model data as triples when you need to query it and as composite literals when you need compact, exact round-tripping.
- One store, two containers. A store is groups, named byte arrays and integer attributes
(SPECIFICATIONS.md §3-§4). It is kept in one HDF5 file, each array a contiguous dataset, or in a Zarr
v3 directory store, each array one uncompressed chunk: the same bytes either way, memory-mapped where
they lie, so a query reads both the same and at the same speed. HDF5 is one file to copy, publish and
serve over HTTP range requests; Zarr is a directory of some 130 small files that Zarr tools open, and
that can sit inside a larger Zarr hierarchy - or one ZIP archive of those files, uncompressed, which is
mapped the same way. Both are queried in place over HTTP by range requests (a Zarr store through one
block cache for all its arrays), and a storage directory serves either as a store. The code above the
container (
com.ebremer.beakgraph.store) does not know which it has. - One self-contained jar. The published library jar shades every dependency (Jena, Fuseki,
Falcon, JTS, aircompressor, log4j and its
log4j2.ymlat the classpath root), so a consumer gets one artifact that runs on an empty classpath. The trade-off: a consumer that also brings its own Jena or JTS sees duplicate, unrelocated classes, and the embedded logging configuration takes over that application's log4j setup (it writes a rollinglogs/beakgraph.log;-Dbeakgraph.log.dirmoves it).mvn -Dthinbuilds the plain jar. - Six engines, one format. Every
-methodtargets the same on-disk format (SPECIFICATIONS.md); the RAM-family engines produce byte-identical stores, the disk family isomorphic ones. Rules shared by all engines live in one helper each (RdfSources.parser,RelativeIris,Params.gridGraph,NodeComparator, andBlockedEncoderfor the format v6 id sequences), because six hand-mirrored copies of a rule were the recurring source of divergence. - Locale-independent code. Identifiers, protocol strings and file names are case-mapped
and formatted with
Locale.ROOT(never the JVM default:"trig".toUpperCase()isTRİGunder tr-TR,%dprints Arabic-Indic digits under ar-EG);LocaleRuleTestscans the sources for violations. - Format history and the RDF 1.2 design notes: CHANGELOG.md.
Apache License 2.0 (LICENSE). Source files without a header are BeakGraph's own,
copyright 2021-2026 Erich Bremer; the vendored zstd codec (io.airlift.compress.v3.zstdFFM)
and the Jena-derived GeoSPARQL vocabularies keep their original headers - see NOTICE.
The first iteration of BeakGraph was backed by Apache Arrow instead of HDF5. An Apache Arrow version will return. Reasons for this are varied with some of these reasons being just experimentation. The general idea of BeakGraph is a read-only, searchable, indexed set of binary succinct data structures to represent an RDF Dataset. What these succinct data structures are stored in, is somewhat immaterial, but the choice of container has its pros and cons. HDF5 treats multi-dimensional arrays as first class citizens, and has a free viewer for HDF5 files called HDFView. HDFView provides a nice way to debug the succinct data structures during development. There are other perks to HDF5 which will become apparent in time.
Spatial indexing based on GeoSPARQL is supported for geof:sfIntersects (see Limitations above).
The containers, done and under consideration:
- HDF5 - implemented
- Zarr - implemented, as a directory store
- Zip - implemented, as a Zarr store in one ZIP archive
- Apache Arrow
- DICOM
- LWS - a directory of stores is already served as LWS storage (
-endpoint <dir>)
The original BeakGraph was an Apache Jena Graph implementation backed by Apache Arrow wrapped in a Research Object Crate (RO-Crate) inspired by HDT.
Developed to power Halcyon. See Arxiv paper at http://arxiv.org/abs/2304.10612
