By Theofilos Ioannidis (tioannid [at] di [dot] uoa [dot] gr), created on , last updated on
Containers are not ideal for benchmarking purposes similar to the one GeoRDFBench Framework performs, because they do not allow clearing system caches. The reason for this is that:
Therefore, in the following example, although the user can verify that the experiments run properly and results are correctly calculated and reported, the COLD cache response times will not be accurate. However, for experiments that do not require COLD cache response time measurements, e.g., macro benchmark scenarios, response times should be accurate enough for drawing basic conclusions.
This example, features:
For this example, we will use a Windows 10 host machine, with 16 GB of memory and Intel i7-9700 CPU and a 1TB HDD 2.5" SATA III 7200rpm for data disk.
The docker image is stored in the Github registry under tioannid/geordfbench/multistore/scal10k_1m. We assume that the current directory is D:\TEMP. Then we issue the following commands, which pull the image from the registry, and verify its presence in the docker image list:
D:\TEMP>docker pull ghcr.io/tioannid/geordfbench/multistore/scal10k_1m:latest
latest: Pulling from tioannid/geordfbench/multistore/scal10k_1m
e1a89dea01a6: Pull complete
1162e3c920f1: Pull complete
216c5f626388: Pull complete
5ab937946237: Pull complete
69dff4e72af9: Pull complete
1811f555bb47: Pull complete
6258a85f4ce5: Pull complete
e158e3f4fef3: Pull complete
d0189f622cdd: Pull complete
Digest: sha256:8d6b66d718feed82da7c6c1352ace11f9b6272b09c515dc60c360447a5077389
Status: Downloaded newer image for ghcr.io/tioannid/geordfbench/multistore/scal10k_1m:latest
ghcr.io/tioannid/geordfbench/multistore/scal10k_1m:latest
D:\TEMP>docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ghcr.io/tioannid/geordfbench/multistore/scal10k_1m latest 7e79e5f5c2d6 3 hours ago 3.08GB
We start a container named mybench from this image with the following command:
D:\TEMP>docker run -d -e POSTGRES_PASSWORD=postgres -p 5432:5432/tcp --hostname NUC8i7BEH --cpus="4" --memory="11g" --memory-swap="11g" \
--mount "type=bind,src=%cd%,target=/src" --name multiscal10k1m ghcr.io/tioannid/geordfbench/multistore/scal10k_1m:latest
3bd7489259941bff8a4c8343d7554972fdd0f38fb42f129054b0d48e5e0c31cb
This command, launches the mybench container, while defining:
We start the experiments by connecting to the mybench running container with a terminal and issue the command:
D:\TEMP>docker exec -it multiscal10k1m /bin/bash
root@NUC8i7BEH:/data# ./startUpScript.sh
Git repo exists
Running script with syntax: source prepareRunEnvironment.sh NUC8I7BEH RDF4JSUT CreateRepo_Scalability10K_1M_RDF4J
Updated 1 path from the index
All of the following environment variables (geordfbench/scripts/prepareRunEnvironment.sh) are defined:
{DatasetBaseDir, RDF4JRepoBaseDir, JVM_Xmx, EnableLuceneSail, CompletionReportDaemonIP, CompletionReportDaemonPort, ScalabilityGenScriptName, ScalabilityGzipRefDSName}
Running script with syntax:
./createAllRDF4JRepos.sh false /data/Geographica2_Datasets /data/RDF4J_4.3.15_Repos/server -Xmx24g false 192.168.1.44 3333 /data/Geographica2_Datasets/Scalability/scalabilityDSGen.sh /data/Geographica2_Datasets/Scalability/scalability500MRefDS.nt.gz
Script start time: Fri May 16 10:20:07 UTC 2025
/data/RDF4J_4.3.15_Repos/server dir does not exist.
Creating it now ...
Checking/Creating scalability 10K dataset ...
Scalability 10K dataset already exists
Generating scalability 10K repository ...
The default terminal will act as a log window and after some time all experiments (3 stores * 3 workloads = 9) will end with:
...
232063 [main] INFO PostgreSQLRepSrc - Deferred mode for PostgreSQLRepSrc was enabled. 18 records were flushed
232066 [main] INFO GenericExprerimentResultsCollector - Export statistics in "/data/Results_Store/JenaGeoSPARQLSUT/2025-05-16_JenaGeoSPARQL_RunWL_Scal1M/Scalability/1M/JenaGeoSPARQLSUT-ExperimentWorkload"
232069 [main] INFO GenericExprerimentResultsCollector - Created non existing directory
232076 [main] INFO GenericExprerimentResultsCollector - Statistiscs printed: /data/Results_Store/JenaGeoSPARQLSUT/2025-05-16_JenaGeoSPARQL_RunWL_Scal1M/Scalability/1M/JenaGeoSPARQLSUT-ExperimentWorkload/00-SC1_Geometries_Intersects_GivenPolygon-cold
232081 [main] INFO GenericExprerimentResultsCollector - Statistiscs printed: /data/Results_Store/JenaGeoSPARQLSUT/2025-05-16_JenaGeoSPARQL_RunWL_Scal1M/Scalability/1M/JenaGeoSPARQLSUT-ExperimentWorkload/00-SC1_Geometries_Intersects_GivenPolygon-cold-long
232082 [main] INFO GenericExprerimentResultsCollector - Statistiscs printed: /data/Results_Store/JenaGeoSPARQLSUT/2025-05-16_JenaGeoSPARQL_RunWL_Scal1M/Scalability/1M/JenaGeoSPARQLSUT-ExperimentWorkload/01-SC2_Intensive_Geometries_Intersect_Geometries-cold
232082 [main] INFO GenericExprerimentResultsCollector - Statistiscs printed: /data/Results_Store/JenaGeoSPARQLSUT/2025-05-16_JenaGeoSPARQL_RunWL_Scal1M/Scalability/1M/JenaGeoSPARQLSUT-ExperimentWorkload/01-SC2_Intensive_Geometries_Intersect_Geometries-cold-long
232085 [main] INFO GenericExprerimentResultsCollector - Statistiscs printed: /data/Results_Store/JenaGeoSPARQLSUT/2025-05-16_JenaGeoSPARQL_RunWL_Scal1M/Scalability/1M/JenaGeoSPARQLSUT-ExperimentWorkload/02-SC3_Relaxed_Geometries_Intersect_Geometries-cold
232086 [main] INFO GenericExprerimentResultsCollector - Statistiscs printed: /data/Results_Store/JenaGeoSPARQLSUT/2025-05-16_JenaGeoSPARQL_RunWL_Scal1M/Scalability/1M/JenaGeoSPARQLSUT-ExperimentWorkload/02-SC3_Relaxed_Geometries_Intersect_Geometries-cold-long
232088 [main] INFO GenericExprerimentResultsCollector - Statistiscs printed: /data/Results_Store/JenaGeoSPARQLSUT/2025-05-16_JenaGeoSPARQL_RunWL_Scal1M/Scalability/1M/JenaGeoSPARQLSUT-ExperimentWorkload/00-SC1_Geometries_Intersects_GivenPolygon-warm
232090 [main] INFO GenericExprerimentResultsCollector - Statistiscs printed: /data/Results_Store/JenaGeoSPARQLSUT/2025-05-16_JenaGeoSPARQL_RunWL_Scal1M/Scalability/1M/JenaGeoSPARQLSUT-ExperimentWorkload/00-SC1_Geometries_Intersects_GivenPolygon-warm-long
232092 [main] INFO GenericExprerimentResultsCollector - Statistiscs printed: /data/Results_Store/JenaGeoSPARQLSUT/2025-05-16_JenaGeoSPARQL_RunWL_Scal1M/Scalability/1M/JenaGeoSPARQLSUT-ExperimentWorkload/01-SC2_Intensive_Geometries_Intersect_Geometries-warm
232092 [main] INFO GenericExprerimentResultsCollector - Statistiscs printed: /data/Results_Store/JenaGeoSPARQLSUT/2025-05-16_JenaGeoSPARQL_RunWL_Scal1M/Scalability/1M/JenaGeoSPARQLSUT-ExperimentWorkload/01-SC2_Intensive_Geometries_Intersect_Geometries-warm-long
232093 [main] INFO GenericExprerimentResultsCollector - Statistiscs printed: /data/Results_Store/JenaGeoSPARQLSUT/2025-05-16_JenaGeoSPARQL_RunWL_Scal1M/Scalability/1M/JenaGeoSPARQLSUT-ExperimentWorkload/02-SC3_Relaxed_Geometries_Intersect_Geometries-warm
232094 [main] INFO GenericExprerimentResultsCollector - Statistiscs printed: /data/Results_Store/JenaGeoSPARQLSUT/2025-05-16_JenaGeoSPARQL_RunWL_Scal1M/Scalability/1M/JenaGeoSPARQLSUT-ExperimentWorkload/02-SC3_Relaxed_Geometries_Intersect_Geometries-warm-long
232095 [main] INFO GenericExprerimentResultsCollector - Cache COLD
232095 [main] INFO GenericExprerimentResultsCollector - Query 0
232096 [main] INFO GenericExprerimentResultsCollector - Rep 0 <COMPLETED-NONE> 8617400 + 3906598700 = 3915216100 nsecs, 80500 results, 0 scan errors
232096 [main] INFO GenericExprerimentResultsCollector - Rep 1 <COMPLETED-NONE> 1294500 + 1773974200 = 1775268700 nsecs, 80500 results, 0 scan errors
232096 [main] INFO GenericExprerimentResultsCollector - Rep 2 <COMPLETED-NONE> 365400 + 1726653100 = 1727018500 nsecs, 80500 results, 0 scan errors
232096 [main] INFO GenericExprerimentResultsCollector - Query 1
232096 [main] INFO GenericExprerimentResultsCollector - Rep 0 <COMPLETED-NONE> 325800 + 5754457600 = 5754783400 nsecs, 813 results, 0 scan errors
232096 [main] INFO GenericExprerimentResultsCollector - Rep 1 <COMPLETED-NONE> 487800 + 5338568500 = 5339056300 nsecs, 813 results, 0 scan errors
232096 [main] INFO GenericExprerimentResultsCollector - Rep 2 <COMPLETED-NONE> 1032200 + 5459136200 = 5460168400 nsecs, 813 results, 0 scan errors
232096 [main] INFO GenericExprerimentResultsCollector - Query 2
232096 [main] INFO GenericExprerimentResultsCollector - Rep 0 <COMPLETED-NONE> 171300 + 5390920400 = 5391091700 nsecs, 239 results, 0 scan errors
232096 [main] INFO GenericExprerimentResultsCollector - Rep 1 <COMPLETED-NONE> 389600 + 5865371200 = 5865760800 nsecs, 239 results, 0 scan errors
232096 [main] INFO GenericExprerimentResultsCollector - Rep 2 <COMPLETED-NONE> 475200 + 5720907200 = 5721382400 nsecs, 239 results, 0 scan errors
232096 [main] INFO GenericExprerimentResultsCollector - Cache WARM
232096 [main] INFO GenericExprerimentResultsCollector - Query 0
232096 [main] INFO GenericExprerimentResultsCollector - Rep 0 <COMPLETED-NONE> 387300 + 450956800 = 451344100 nsecs, 80500 results, 0 scan errors
232096 [main] INFO GenericExprerimentResultsCollector - Rep 1 <COMPLETED-NONE> 310600 + 433903800 = 434214400 nsecs, 80500 results, 0 scan errors
232096 [main] INFO GenericExprerimentResultsCollector - Rep 2 <COMPLETED-NONE> 306500 + 445593400 = 445899900 nsecs, 80500 results, 0 scan errors
232096 [main] INFO GenericExprerimentResultsCollector - Query 1
232096 [main] INFO GenericExprerimentResultsCollector - Rep 0 <COMPLETED-NONE> 285900 + 4579062700 = 4579348600 nsecs, 813 results, 0 scan errors
232096 [main] INFO GenericExprerimentResultsCollector - Rep 1 <COMPLETED-NONE> 301200 + 4327118000 = 4327419200 nsecs, 813 results, 0 scan errors
232096 [main] INFO GenericExprerimentResultsCollector - Rep 2 <COMPLETED-NONE> 318700 + 4310809800 = 4311128500 nsecs, 813 results, 0 scan errors
232096 [main] INFO GenericExprerimentResultsCollector - Query 2
232096 [main] INFO GenericExprerimentResultsCollector - Rep 0 <COMPLETED-NONE> 338200 + 4720544600 = 4720882800 nsecs, 239 results, 0 scan errors
232096 [main] INFO GenericExprerimentResultsCollector - Rep 1 <COMPLETED-NONE> 333400 + 4624836200 = 4625169600 nsecs, 239 results, 0 scan errors
232096 [main] INFO GenericExprerimentResultsCollector - Rep 2 <COMPLETED-NONE> 541000 + 4649665000 = 4650206000 nsecs, 239 results, 0 scan errors
232097 [main] INFO RunJenaGeoSPARQLExperimentWorkload - End ScalabilityFunc
Start time = Fri May 16 11:21:12 UTC 2025
End time = Fri May 16 11:25:05 UTC 2025
root@NUC8i7BEH:/data# tree -L 1 RDF4J_4.3.15_Repos/server/repositories/
RDF4J_4.3.15_Repos/server/repositories/
|-- scalability_100K
|-- scalability_10K
`-- scalability_1M
root@NUC8i7BEH:/data# tree -L 1 graphdb-10.8.4/data/repositories/
graphdb-10.8.4/data/repositories/
|-- scalability_100K
|-- scalability_10K
`-- scalability_1M
root@NUC8i7BEH:/data# tree -L 1 JenaGeoSPARQL_4.10.0_Repos/
JenaGeoSPARQL_4.10.0_Repos/
|-- scalability_100K
|-- scalability_10K
`-- scalability_1M
root@NUC8i7BEH:/data# ls -lsa geordfbench/RDF4JSUT/scripts/CreateRepos/*.log
8 -rw-r--r-- 1 root root 6757 May 16 10:20 geordfbench/RDF4JSUT/scripts/CreateRepos/logCreateRepos_Scal10K_1M_RDF4J.log
root@NUC8i7BEH:/data# ls -lsa geordfbench/GraphDBSUT/scripts/CreateRepos/*.log
36 -rw-r--r-- 1 root root 35507 May 16 10:34 geordfbench/GraphDBSUT/scripts/CreateRepos/logCreateRepo_Scal10K_1M_GraphDB.log
root@NUC8i7BEH:/data# ls -lsa geordfbench/JenaGeoSPARQLSUT/scripts/CreateRepos/*.log
8 -rwxr-xr-x 1 root root 5512 May 16 11:16 geordfbench/JenaGeoSPARQLSUT/scripts/CreateRepos/logCreateRepo_Scal10K_1M_JenaGeoSPARQL.log
Experiment run logs may be quite long, therefore the user can click the links below to view the details of the execution.
root@NUC8i7BEH:/data# ls -lsa geordfbench/RDF4JSUT/scripts/RunTests3/*.log
76 -rw-r--r-- 1 root root 73058 May 16 10:27 geordfbench/RDF4JSUT/scripts/RunTests3/RunWLRDF4JExp_Scal100K.log
84 -rw-r--r-- 1 root root 79386 May 16 10:23 geordfbench/RDF4JSUT/scripts/RunTests3/RunWLRDF4JExp_Scal10K.log
76 -rw-r--r-- 1 root root 73111 May 16 10:33 geordfbench/RDF4JSUT/scripts/RunTests3/RunWLRDF4JExp_Scal1M.log
root@NUC8i7BEH:/data# ls -lsa geordfbench/GraphDBSUT/scripts/RunTests3/*.log
144 -rw-r--r-- 1 root root 141189 May 16 10:44 geordfbench/GraphDBSUT/scripts/RunTests3/RunWLGraphDBExp_Scal100K.log
140 -rw-r--r-- 1 root root 138303 May 16 10:37 geordfbench/GraphDBSUT/scripts/RunTests3/RunWLGraphDBExp_Scal10K.log
152 -rw-r--r-- 1 root root 148714 May 16 11:16 geordfbench/GraphDBSUT/scripts/RunTests3/RunWLGraphDBExp_Scal1M.log
root@NUC8i7BEH:/data# ls -lsa geordfbench/JenaGeoSPARQLSUT/scripts/RunTests3/*.log
80 -rw-r--r-- 1 root root 74261 May 16 11:21 geordfbench/JenaGeoSPARQLSUT/scripts/RunTests3/RunWLJenaGeoSPARQLExp_Scal100K.log
80 -rwxr-xr-x 1 root root 76617 May 16 11:18 geordfbench/JenaGeoSPARQLSUT/scripts/RunTests3/RunWLJenaGeoSPARQLExp_Scal10K.log
80 -rw-r--r-- 1 root root 74359 May 16 11:25 geordfbench/JenaGeoSPARQLSUT/scripts/RunTests3/RunWLJenaGeoSPARQLExp_Scal1M.log
Experiment results are stored by default in the file system. For this demonstration the base location for all results was /data/Results_Store. The output of the following command
has been modified to fully expand the last entry only, with the result of tree -L 4 Results_Store/RDF4JSUT/2025-05-16_RDF4JSUT_RunWL_Scal1M/ :
root@NUC8i7BEH:/data# tree -L 2 Results_Store/
Results_Store/
|-- GraphDBSUT
| |-- 2025-05-16_GraphDBSUT_RunWL_Scal100K
| |-- 2025-05-16_GraphDBSUT_RunWL_Scal10K
| `-- 2025-05-16_GraphDBSUT_RunWL_Scal1M
|-- JenaGeoSPARQLSUT
| |-- 2025-05-16_JenaGeoSPARQL_RunWL_Scal100K
| |-- 2025-05-16_JenaGeoSPARQL_RunWL_Scal10K
| `-- 2025-05-16_JenaGeoSPARQL_RunWL_Scal1M
`-- RDF4JSUT
|-- 2025-05-16_RDF4JSUT_RunWL_Scal100K
|-- 2025-05-16_RDF4JSUT_RunWL_Scal10K
`-- 2025-05-16_RDF4JSUT_RunWL_Scal1M
`-- Scalability
`-- 1M
`-- RDF4JSUT-ExperimentWorkload
|-- 00-SC1_Geometries_Intersects_GivenPolygon-cold
|-- 00-SC1_Geometries_Intersects_GivenPolygon-cold-long
|-- 00-SC1_Geometries_Intersects_GivenPolygon-warm
|-- 00-SC1_Geometries_Intersects_GivenPolygon-warm-long
|-- 01-SC2_Intensive_Geometries_Intersect_Geometries-cold
|-- 01-SC2_Intensive_Geometries_Intersect_Geometries-cold-long
|-- 01-SC2_Intensive_Geometries_Intersect_Geometries-warm
|-- 01-SC2_Intensive_Geometries_Intersect_Geometries-warm-long
|-- 02-SC3_Relaxed_Geometries_Intersect_Geometries-cold
|-- 02-SC3_Relaxed_Geometries_Intersect_Geometries-cold-long
|-- 02-SC3_Relaxed_Geometries_Intersect_Geometries-warm
`-- 02-SC3_Relaxed_Geometries_Intersect_Geometries-warm-long
root@NUC8i7BEH:/data# more Results_Store/RDF4JSUT/2025-05-16_RDF4JSUT_RunWL_Scal1M/Scalability/1M/RDF4JSUT-ExperimentWorkload/00-SC1_Geometries_Intersects_GivenPolygon-cold
80500 3398004600
root@NUC8i7BEH:/data# more Results_Store/RDF4JSUT/2025-05-16_RDF4JSUT_RunWL_Scal1M/Scalability/1M/RDF4JSUT-ExperimentWorkload/00-SC1_Geometries_Intersects_GivenPolygon-cold-long
80500 83046000 5377138700 5460184700
80500 4103200 3393901400 3398004600
80500 3973300 3304164100 3308137400
For each query and execution type (warm, cold) there are two files, a short and a long version. The long version has 4 columns (noOfResults, evaluationTime, scanTime, totalTime) and one row for each execution iteration performed. The short version has 2 columns (noOfResults, totalTime) and only one row which represents the average or median totalTime of the execution iterations from the long version. All times are in nano seconds.
Experiment results are also stored in a custom location, a PostgreSQL database. Each experiment details are recorded, with a unique ID, in a row of the EXPERIMENTS table. Each query execution iteration details are in a similar manner recorded in the QUERYEXECUTIONS table. A set of views can provide aggregation for the totalTime and calculation of the Average and Median value of totalTime for each query and execution type (warm, cold). All times are in milli seconds. In the figures below we can see the actual snapshots from the PgAdmin v4 interface.
An example bar chart with the results (median) of 3 experiments with Scalability-100K workload is shown below:
The above chart shows clearly that for the Scalability 100K workload, Jena GeoSPARQL performed the best and RDF4J followed. In this chart we can also see, that for RDF4J and GraphDB COLD and WARM caches are almost identical. This is attributed to the fact that clearing caches failed inside the container, as it was mentioned in the Caution section of this page, basically making all runs behave as WARM. JenaGeoSPARQL on the other hand is not affect by this because indexing and caching of spatial objects and relations is performed on-demand during query execution.
For the Scalability-10K workload we used a modified copy of the default generated JSON specification and created a Gold Standard. The differences between the two files are minimal and are shown in the following diff output:
root@NUC8i7BEH:/data/geordfbench/json_defs/workloads# diff scalabilityFunc10K_WLoriginal_GOLD_STANDARD.json scalabilityFunc10K_WLoriginal.json
45c45
< "expectedResults" : 554
---
> "expectedResults" : -1
51c51
< "expectedResults" : 2
---
> "expectedResults" : -1
57c57
< "expectedResults" : 2
---
> "expectedResults" : -1
79c79
< }
---
We replaced the -1 value with the expected number of results for each one of the queries for the specific dataset of the Scalability-10K workload. For example, we can search
for the ACCURATE token in the RDF4J experiment log file
/data/geordfbench/RDF4JSUT/scripts/RunTests3/RunWLRDF4JExp_Scal10K.log
in order to verify that the token is found 18 times (3 queries * 3 COLD repetitions + 3 queries * 3 WARM repetitions). The same holds for the corresponding GraphDB and JenaGeoSPARQL logs
for the Scalability-10K workload. For the non Gold Standard workloads, Scalability-{100K | 1M}, the response is different, since we receive ACCURACY NOT DETERMINED indicating
that no expected result value was provided for the queries in those experiments.
The more interested user, can look at the simple Bash script, /data/startUpScript.sh, which launched the experiments. The actions taken by this script are: