crush depth

Choosing Coffee

Now that jlink exists, it's become fashionable to bundle a JVM along with your Java bytecode. In applications that use the JPMS, the jlink tool can calculate the transitive closure of required modules and produce a tiny JVM that contains your application modules and only those platform modules required to run the application.

Because I'm using OSGi, I don't care about this particular aspect of jlink. What I am interested in is that, because of jlink, it has become a lot easier to bundle a JVM with an application. Projects such as AdoptOpenJDK produce production-ready builds of JVMs for a large range of platforms and architectures, including Hotspot and OpenJ9 VMs.

Now there is a downside to this: Java applications should, by their very nature, be completely platform-independent. If I give someone a jar file of Java 11 bytecode, it should run correctly on any JVM on any platform supporting Java 11+. If I instead start producing one distribution per platform and architecture including a bundled JVM, then by definition my application distributions are no longer platform-independent. Worse, even though jlink still bundles your application's bytecode along with the produced JVM, that bytecode is (by default) stored in an undocumented binary format along with the platform modules as one indivisible lump. I don't believe it's possible for the user to say "well, I don't want to use the bundled JVM, I'll just point my system JVM at the classes and use that".

What I would much rather do is still produce one distribution that contains a bundled JVM for each of the common platforms and architectures (let's say Linux x86_64, Windows x86_64, and MacOS x86_64) but also provide a basic shell script that can be used to start the application from a system JVM if the user so wishes (with the support caveat that if it doesn't work, they're on their own). The bytecode should still be stored exactly as it was in the original jar files (and in my case, would actually have to be stored like this because the jar files are OSGi bundles).

The ideal situation would be if I could have a Maven plugin that, given a range of platforms and architectures, automatically fetched one runtime suitable for each platform/architecture and then packaged up the whole application including all of the runtimes into one platform-independent distribution archive.

So, I've decided to write one!

The CoffeePick project provides an API and an interactive shell used to search for and download Java runtimes. The API works via a pluggable service provider model where new repositories of runtimes can be added at build-time or, if running on OSGi, at run-time.

Currently, arguably the two most popular places to get JVMs are:

The CoffeePick project currently contains providers for both of those repositories, and the AdoptOpenJDK module automatically finds new builds as the AdoptOpenJDK project publishes them.

There now follows small demo of the interactive shell.

First, we start up the shell:

$ java -jar com.io7m.coffeepick.shell/target/com.io7m.coffeepick.shell-0.0.1-main.jar
User directory: /home/rm/local/coffeepick
Loading repositories…
INFO com.io7m.coffeepick.client.vanilla.CoffeePickCatalog: setting up repository from provider net.java.jdk (urn:net.java.jdk)
INFO com.io7m.coffeepick.client.vanilla.CoffeePickCatalog: setting up repository from provider net.adoptopenjdk.raw (urn:net.adoptopenjdk.raw)

The shell uses jline3 internally and so has a full command-line history and limited tab-completion for command names.

Let's say we want to grab a Java 11 compatible runtime that will execute on Linux on the x86_64 architecture. The CoffeePick API refers to the current set of runtimes that we can download as the catalog. The catalog command, without any arguments, will list every available runtime:

[coffeepick]$ catalog
ID                                                                     | Arch     | Platform | Version  | VM       | Conf | Size       | Repository                   | Tags
8eee0aede947b804f9a5f49c8a38b52aace8a30a9ebd9383b7d06042fb5a237c       | aarch64  | linux    | 8        | hotspot  | jdk  | 75.38   MB | urn:net.adoptopenjdk.raw     | production
b0b3923a48243fa98b71c7fd8ff96def453e07b33156e90cb6851b7b7cacd4b1       | aarch64  | linux    | 8        | hotspot  | jre  | 39.53   MB | urn:net.adoptopenjdk.raw     | production
4f245d8bb2e6b84876edcc3573d9f8609e7596cb99102419b2174ed146aca642       | ppc64    | aix      | 8        | openj9   | jdk  | 87.50   MB | urn:net.adoptopenjdk.raw     | production
af2532a428fbeb6e6896279a874a3d4b574b74a6ebeae16e4771d7ad517f653a       | ppc64le  | linux    | 8        | openj9   | jdk  | 89.53   MB | urn:net.adoptopenjdk.raw     | production
e7fd779a6448259641565ca31f258d11052c9997e762bc63bbaa2b5776de98f1       | ppc64le  | linux    | 8        | openj9   | jre  | 48.62   MB | urn:net.adoptopenjdk.raw     | production
b092f4e6553a7c50872b0963f4de2b3a2bce78ce11d1bf1905a2ad5a6576924b       | s390x    | linux    | 8        | openj9   | jdk  | 90.37   MB | urn:net.adoptopenjdk.raw     | production
f0bc82fa49a2a02e881d403170cd8b273fc3d1585a4e65ff639666f056ce7362       | x64      | linux    | 8        | openj9   | jdk  | 90.12   MB | urn:net.adoptopenjdk.raw     | large-heap production
fd7035c60ad3060fccf7ab67c15b722302a1863ebddd3141dc3dd98bd0b25a68       | x64      | linux    | 8        | openj9   | jdk  | 90.24   MB | urn:net.adoptopenjdk.raw     | production
d1ee522053bb4135536b1d3173be8224bf54563d166e882b7ee2ccf609c2517f       | aarch64  | linux    | 9        | hotspot  | jdk  | 190.65  MB | urn:net.adoptopenjdk.raw     | production
36e4955c0fec6ff6494c5274e6950fdd45e73b11ebe97efdcb9768e07981f626       | aarch64  | linux    | 9        | hotspot  | jre  | 37.12   MB | urn:net.adoptopenjdk.raw     | production
d304cbfc9674ff1e1b0caf916b59c32ee9c477e5be6cfa071131b866cd54ec4c       | ppc64    | aix      | 9        | hotspot  | jdk  | 198.43  MB | urn:net.adoptopenjdk.raw     | production
359c65179f839c5ba0959710803c556c3ce2046dcb0762f2165595706b9b8272       | ppc64le  | linux    | 9        | hotspot  | jre  | 36.84   MB | urn:net.adoptopenjdk.raw     | production
bd4736feea702c859102bb78145fe3c7dbf6774ce1d66d00a9fc582d6bbeb564       | ppc64le  | linux    | 9        | hotspot  | jdk  | 190.36  MB | urn:net.adoptopenjdk.raw     | production
6382fe043c0ef1cf5b38cf3f70cc300bbdd66f16fa6e9b1ffbfe6a9cdda9011d       | ppc64le  | linux    | 9        | hotspot  | jdk  | 190.49  MB | urn:net.adoptopenjdk.raw     | production
84bbfca72cbabf04a2fd0e35d0faba03f91977573600edc23714b03aa1ec6fd9       | ppc64le  | linux    | 9        | hotspot  | jre  | 36.86   MB | urn:net.adoptopenjdk.raw     | production
0510af0db7531eb45bb8723810a10e146a80223cf5b72da077bc41e0e66724e3       | s390x    | linux    | 9        | hotspot  | jre  | 38.52   MB | urn:net.adoptopenjdk.raw     | production
7bfdc389c25820e08a85eb3a32c4193189f13945847619b14966f7fd45a5c345       | s390x    | linux    | 9        | hotspot  | jdk  | 193.60  MB | urn:net.adoptopenjdk.raw     | production
aa4fc8bda11741facaf3b8537258a4497c6e0046b9f4931e31f713d183b951f1       | x64      | linux    | 9        | hotspot  | jdk  | 207.46  MB | urn:net.adoptopenjdk.raw     | production
b58848cb7229a1dc2c135dbb85cce56cee33ab29849aa432e2ba26bf8f517dca       | x64      | linux    | 9        | hotspot  | jre  | 41.55   MB | urn:net.adoptopenjdk.raw     | production
b6a14f8dda60528bbb890877b62b7cf6e81bbbe19256bfd38a3ba4390c4eca5e       | x64      | linux    | 9        | hotspot  | jre  | 41.53   MB | urn:net.adoptopenjdk.raw     | production
ebe1eaad73584f70ab4e587ab9367fb15af820745979e68350fe80fadf615df8       | x64      | linux    | 9        | hotspot  | jdk  | 207.33  MB | urn:net.adoptopenjdk.raw     | production
bbe348baab3cca097b77f7fff6c8465858e642ad42c5851bde33a1553ca12653       | x64      | macos    | 9        | hotspot  | jdk  | 189.34  MB | urn:net.adoptopenjdk.raw     | production
f92931d258486d77dd986fa0fd2574075569e2f76624822d50bfae01ed72e7e7       | x64      | macos    | 9        | hotspot  | jre  | 35.70   MB | urn:net.adoptopenjdk.raw     | production
0373b5be04f98c0077c5b87c2020ecac2fedd36f68e75bbae38d433753506323       | x64      | windows  | 9        | hotspot  | jre  | 36.21   MB | urn:net.adoptopenjdk.raw     | production
b2de88155b2eaad123c2615f7eb3c0442d77545046bf85cfbcfe44fc97931dbd       | x64      | windows  | 9        | hotspot  | jdk  | 192.42  MB | urn:net.adoptopenjdk.raw     | production
3bb2faaba9f1a4f6e212ff34c5f2cfb4d5f083c1623cd2ad24108e0bb542e394       | x64      | windows  | 9        | hotspot  | jre  | 36.25   MB | urn:net.adoptopenjdk.raw     | production
709baaf217d24503dff3194fd66aa25ae5976b3f52ed3a564cdd270b6f5eef40       | x64      | windows  | 9        | hotspot  | jdk  | 192.27  MB | urn:net.adoptopenjdk.raw     | production
b66121b9a0c2e7176373e670a499b9d55344bcb326f67140ad6d0dc24d13d3e2       | aarch64  | linux    | 11       | hotspot  | jdk  | 193.39  MB | urn:net.adoptopenjdk.raw     | production
0f4edea90e57765562740721d052946408273e9744093266f65c02a7fb26800c       | ppc64    | aix      | 11       | openj9   | jre  | 46.93   MB | urn:net.adoptopenjdk.raw     | production
eaae79f3a89a6d71f34e9b41cbdfdb14f70988284d9e76427ed705514c614cc3       | ppc64    | aix      | 11       | openj9   | jdk  | 196.64  MB | urn:net.adoptopenjdk.raw     | production
39f97d76a0121f9a2b6e26bc201194921797fcc6c2181871086f09f8bce5862f       | ppc64    | aix      | 11       | openj9   | jdk  | 196.64  MB | urn:net.adoptopenjdk.raw     | production
001fd98ca5b5215eac616ed26341052ac1b426d7a55bbca7d28d48137fa9f295       | ppc64    | aix      | 11       | openj9   | jdk  | 196.65  MB | urn:net.adoptopenjdk.raw     | production
43f727ce6bca1ed826181f1a924c9f64a43728e3a222284fd2fbfbe9e4ef199c       | ppc64    | aix      | 11       | openj9   | jdk  | 196.65  MB | urn:net.adoptopenjdk.raw     | production
35e36d7a20a773e291b728de5d4977485a0d44a54f1433cdf6e6288c331557ea       | ppc64le  | linux    | 11       | openj9   | jdk  | 196.51  MB | urn:net.adoptopenjdk.raw     | production
4577350c35f4193430b3ae0094e54869b54aadbc76a50c1518069937a63d54d8       | ppc64le  | linux    | 11       | openj9   | jdk  | 196.52  MB | urn:net.adoptopenjdk.raw     | production
747c4613b36f6e86ee4a10848b1a7f009d1bde729acf31bb6350c3cc08a7475d       | ppc64le  | linux    | 11       | openj9   | jre  | 41.38   MB | urn:net.adoptopenjdk.raw     | production
fcc9a2b183a9238e1927d6088d7c99031f517d62026e8b33d02dfd883b1ca95e       | ppc64le  | linux    | 11       | openj9   | jdk  | 196.55  MB | urn:net.adoptopenjdk.raw     | production
c18364a778b1b990e8e62d094377af48b000f9f6a64ec21baff6a032af06386d       | ppc64le  | linux    | 11       | hotspot  | jdk  | 179.52  MB | urn:net.adoptopenjdk.raw     | production
fb3f4ddaf3e2a9490e34b385e00d76cf6142806d47bd619ef7c193c12acfbb01       | s390x    | linux    | 11       | openj9   | jdk  | 197.27  MB | urn:net.adoptopenjdk.raw     | production
249ad469735c42fa80fcf4123bc76f7e398ff3349ce6fb4bfa65a850fb70f6f8       | s390x    | linux    | 11       | openj9   | jdk  | 197.27  MB | urn:net.adoptopenjdk.raw     | production
119bd9d9faf1bb596b9ca74740f12cc23c20fe9331d312f1c2b59eb15a76625f       | x32      | windows  | 11       | hotspot  | jdk  | 174.24  MB | urn:net.adoptopenjdk.raw     | production
0b6050cc670eefd9465370ab19ae70401476430fca329e65f0dd636ca9cce9bd       | x64      | linux    | 11       | openj9   | jdk  | 195.41  MB | urn:net.adoptopenjdk.raw     | large-heap production
ef9bf07cba79082285a9d426ea4eb3e8df57561ce2afe07cc5f299a8fa203279       | x64      | linux    | 11       | openj9   | jdk  | 195.55  MB | urn:net.adoptopenjdk.raw     | production
10b2400add0070b64db91026f12b798708e59772a81492fd2b638638f71ed14f       | x64      | linux    | 11       | hotspot  | jdk  | 186.61  MB | urn:net.java.jdk             | early-access valhalla
cac1fe096ea161613d5afe8a04ce3294dc1b535f5f4ea1a4235ce425b2610678       | x64      | linux    | 11       | openj9   | jdk  | 195.36  MB | urn:net.adoptopenjdk.raw     | large-heap production
765947ab9457a29d2aa9d11460a4849611343c1e0ea3b33b9c08409cd4672251       | x64      | linux    | 11       | openj9   | jdk  | 195.50  MB | urn:net.adoptopenjdk.raw     | production
e0446e80eeb90ab97fd59f91ae094bdd658c354943d4e33e0ddf691714f9da4d       | x64      | linux    | 11       | openj9   | jdk  | 195.55  MB | urn:net.adoptopenjdk.raw     | production
518e9330f2aeb020bca31a07034a08c3a568823e703fc503e03206acdad9dae5       | x64      | linux    | 11       | openj9   | jre  | 40.91   MB | urn:net.adoptopenjdk.raw     | large-heap production
9306fa569d2d5abf982df0b4ffa5d30c652cf16139d7b2268db30d2886f3adbf       | x64      | linux    | 11       | openj9   | jdk  | 195.42  MB | urn:net.adoptopenjdk.raw     | large-heap production
7e9904e1bfa197e6e72570207cb2bd63dd7cc4bf5f97bc5454c1fc5a559a8bf1       | x64      | linux    | 11       | hotspot  | jdk  | 195.71  MB | urn:net.adoptopenjdk.raw     | production
a016413fd8415429b42e543fed7a1bee5010b1dbaf71d29a26e1c699f334c6ff       | x64      | linux    | 11       | openj9   | jre  | 40.96   MB | urn:net.adoptopenjdk.raw     | production
11cb48e3592f46f2c2dab095e3b9a4fec60b89199827f2b737a14b54dc5fd68d       | x64      | linux    | 11       | openj9   | jdk  | 195.36  MB | urn:net.adoptopenjdk.raw     | large-heap production
c5e9b588b4ac5b0bd5b4edd69d59265d1199bb98af7ca3270e119b264ffb6e3f       | x64      | macos    | 11       | openj9   | jdk  | 180.99  MB | urn:net.adoptopenjdk.raw     | production
0901dc5946fdf967f92f7b719ddfffdcdde5bd3fef86a83d7a3f2f39ddbef1f8       | x64      | macos    | 11       | openj9   | jre  | 38.63   MB | urn:net.adoptopenjdk.raw     | production
92494f79cc296625778a50d04816aabedca9cd9666b7bab8f1c3b96f06c82f97       | x64      | macos    | 11       | hotspot  | jdk  | 180.75  MB | urn:net.java.jdk             | early-access valhalla
c9a816d6a3f8aac9dc5b3b41c5a9e4e5460af433a06e003ae25d5a06dea8375f       | x64      | macos    | 11       | openj9   | jdk  | 180.95  MB | urn:net.adoptopenjdk.raw     | production
67785f46b264f7a91358cea41e4f7e0880685175a2a18f1f3890828a577fe067       | x64      | macos    | 11       | openj9   | jdk  | 180.95  MB | urn:net.adoptopenjdk.raw     | production
e219e7e2d586ed09ae65f4ec390fca5d5f0c37a61b47677648610194daf1aaa7       | x64      | macos    | 11       | hotspot  | jdk  | 189.91  MB | urn:net.adoptopenjdk.raw     | production
f34ced35547e53dd400c1ee99a63d8bf68e45370bc7b098e32f0b7107a60e00b       | x64      | macos    | 11       | openj9   | jdk  | 181.00  MB | urn:net.adoptopenjdk.raw     | production
b8960753a66190fa81982682357a2449b4183f3e23c20a5e3b4cf01e2989846b       | x64      | macos    | 11       | openj9   | jdk  | 181.00  MB | urn:net.adoptopenjdk.raw     | production
ef7c411c3b3e06f93874b15b598c36020b192a1a3096c5d9d5eae7799f8bcd49       | x64      | windows  | 11       | openj9   | jdk  | 196.96  MB | urn:net.adoptopenjdk.raw     | production
d104c7337ff3cd03fb37d048860192a3de5cbab4b158a681bc20cc521b6da954       | x64      | windows  | 11       | openj9   | jdk  | 196.90  MB | urn:net.adoptopenjdk.raw     | production
b1eb84f5bfce03058f266d7303ad02fcf4d6f2b05c989989ae7b0bd5a663a5a7       | x64      | windows  | 11       | hotspot  | jdk  | 192.97  MB | urn:net.adoptopenjdk.raw     | production
106c242e946a74ba71ff22b5a980c375b7cd13fd4550fb3698333e06a1f33243       | x64      | windows  | 11       | hotspot  | jdk  | 180.82  MB | urn:net.java.jdk             | early-access valhalla
d33551f15f8d4bc5f97471bca669872665e5db15cc8a5ac88f068e23f56bb129       | x64      | windows  | 11       | openj9   | jre  | 40.06   MB | urn:net.adoptopenjdk.raw     | production
7a6bb980b9c91c478421f865087ad2d69086a0583aeeb9e69204785e8e97dcfd       | x64      | linux    | 11.0.1   | hotspot  | jdk  | 187.60  MB | urn:net.java.jdk             | production
fa07eee08fa0f3de541ee1770de0cdca2ae3876f3bd78c329f27e85c287cd070       | x64      | macos    | 11.0.1   | hotspot  | jdk  | 182.09  MB | urn:net.java.jdk             | production
289dd06e06c2cbd5e191f2d227c9338e88b6963fd0c75bceb9be48f0394ede21       | x64      | windows  | 11.0.1   | hotspot  | jdk  | 187.42  MB | urn:net.java.jdk             | production
33a5f5b063dde3a66a98cf691bd00b21c012950ae1af253e19dcc68d74a75f65       | x64      | linux    | 12       | hotspot  | jdk  | 304.88  MB | urn:net.java.jdk             | early-access panama
95a033710ed71962c33b226afdbb222442bf0098ea3705448f0d14f0b3044bab       | x64      | macos    | 12       | hotspot  | jdk  | 233.21  MB | urn:net.java.jdk             | early-access panama

As can be seen, there are currently rather a lot of runtimes. We can start narrowing down the runtimes that we want by passing in search parameters. Searching is provided by the API directly and the API for this is believed to be stable, however the syntax for the shell is currently subject to change and may even be outdated by the time you read this. Let's limit ourselves to Linux runtimes:

[coffeepick]$ catalog platform:linux
ID                                                                     | Arch     | Platform | Version  | VM       | Conf | Size       | Repository                   | Tags
8eee0aede947b804f9a5f49c8a38b52aace8a30a9ebd9383b7d06042fb5a237c       | aarch64  | linux    | 8        | hotspot  | jdk  | 75.38   MB | urn:net.adoptopenjdk.raw     | production
b0b3923a48243fa98b71c7fd8ff96def453e07b33156e90cb6851b7b7cacd4b1       | aarch64  | linux    | 8        | hotspot  | jre  | 39.53   MB | urn:net.adoptopenjdk.raw     | production
af2532a428fbeb6e6896279a874a3d4b574b74a6ebeae16e4771d7ad517f653a       | ppc64le  | linux    | 8        | openj9   | jdk  | 89.53   MB | urn:net.adoptopenjdk.raw     | production
e7fd779a6448259641565ca31f258d11052c9997e762bc63bbaa2b5776de98f1       | ppc64le  | linux    | 8        | openj9   | jre  | 48.62   MB | urn:net.adoptopenjdk.raw     | production
b092f4e6553a7c50872b0963f4de2b3a2bce78ce11d1bf1905a2ad5a6576924b       | s390x    | linux    | 8        | openj9   | jdk  | 90.37   MB | urn:net.adoptopenjdk.raw     | production
fd7035c60ad3060fccf7ab67c15b722302a1863ebddd3141dc3dd98bd0b25a68       | x64      | linux    | 8        | openj9   | jdk  | 90.24   MB | urn:net.adoptopenjdk.raw     | production
f0bc82fa49a2a02e881d403170cd8b273fc3d1585a4e65ff639666f056ce7362       | x64      | linux    | 8        | openj9   | jdk  | 90.12   MB | urn:net.adoptopenjdk.raw     | large-heap production
d1ee522053bb4135536b1d3173be8224bf54563d166e882b7ee2ccf609c2517f       | aarch64  | linux    | 9        | hotspot  | jdk  | 190.65  MB | urn:net.adoptopenjdk.raw     | production
36e4955c0fec6ff6494c5274e6950fdd45e73b11ebe97efdcb9768e07981f626       | aarch64  | linux    | 9        | hotspot  | jre  | 37.12   MB | urn:net.adoptopenjdk.raw     | production
359c65179f839c5ba0959710803c556c3ce2046dcb0762f2165595706b9b8272       | ppc64le  | linux    | 9        | hotspot  | jre  | 36.84   MB | urn:net.adoptopenjdk.raw     | production
bd4736feea702c859102bb78145fe3c7dbf6774ce1d66d00a9fc582d6bbeb564       | ppc64le  | linux    | 9        | hotspot  | jdk  | 190.36  MB | urn:net.adoptopenjdk.raw     | production
6382fe043c0ef1cf5b38cf3f70cc300bbdd66f16fa6e9b1ffbfe6a9cdda9011d       | ppc64le  | linux    | 9        | hotspot  | jdk  | 190.49  MB | urn:net.adoptopenjdk.raw     | production
84bbfca72cbabf04a2fd0e35d0faba03f91977573600edc23714b03aa1ec6fd9       | ppc64le  | linux    | 9        | hotspot  | jre  | 36.86   MB | urn:net.adoptopenjdk.raw     | production
0510af0db7531eb45bb8723810a10e146a80223cf5b72da077bc41e0e66724e3       | s390x    | linux    | 9        | hotspot  | jre  | 38.52   MB | urn:net.adoptopenjdk.raw     | production
7bfdc389c25820e08a85eb3a32c4193189f13945847619b14966f7fd45a5c345       | s390x    | linux    | 9        | hotspot  | jdk  | 193.60  MB | urn:net.adoptopenjdk.raw     | production
aa4fc8bda11741facaf3b8537258a4497c6e0046b9f4931e31f713d183b951f1       | x64      | linux    | 9        | hotspot  | jdk  | 207.46  MB | urn:net.adoptopenjdk.raw     | production
ebe1eaad73584f70ab4e587ab9367fb15af820745979e68350fe80fadf615df8       | x64      | linux    | 9        | hotspot  | jdk  | 207.33  MB | urn:net.adoptopenjdk.raw     | production
b58848cb7229a1dc2c135dbb85cce56cee33ab29849aa432e2ba26bf8f517dca       | x64      | linux    | 9        | hotspot  | jre  | 41.55   MB | urn:net.adoptopenjdk.raw     | production
b6a14f8dda60528bbb890877b62b7cf6e81bbbe19256bfd38a3ba4390c4eca5e       | x64      | linux    | 9        | hotspot  | jre  | 41.53   MB | urn:net.adoptopenjdk.raw     | production
b66121b9a0c2e7176373e670a499b9d55344bcb326f67140ad6d0dc24d13d3e2       | aarch64  | linux    | 11       | hotspot  | jdk  | 193.39  MB | urn:net.adoptopenjdk.raw     | production
35e36d7a20a773e291b728de5d4977485a0d44a54f1433cdf6e6288c331557ea       | ppc64le  | linux    | 11       | openj9   | jdk  | 196.51  MB | urn:net.adoptopenjdk.raw     | production
4577350c35f4193430b3ae0094e54869b54aadbc76a50c1518069937a63d54d8       | ppc64le  | linux    | 11       | openj9   | jdk  | 196.52  MB | urn:net.adoptopenjdk.raw     | production
747c4613b36f6e86ee4a10848b1a7f009d1bde729acf31bb6350c3cc08a7475d       | ppc64le  | linux    | 11       | openj9   | jre  | 41.38   MB | urn:net.adoptopenjdk.raw     | production
fcc9a2b183a9238e1927d6088d7c99031f517d62026e8b33d02dfd883b1ca95e       | ppc64le  | linux    | 11       | openj9   | jdk  | 196.55  MB | urn:net.adoptopenjdk.raw     | production
c18364a778b1b990e8e62d094377af48b000f9f6a64ec21baff6a032af06386d       | ppc64le  | linux    | 11       | hotspot  | jdk  | 179.52  MB | urn:net.adoptopenjdk.raw     | production
fb3f4ddaf3e2a9490e34b385e00d76cf6142806d47bd619ef7c193c12acfbb01       | s390x    | linux    | 11       | openj9   | jdk  | 197.27  MB | urn:net.adoptopenjdk.raw     | production
249ad469735c42fa80fcf4123bc76f7e398ff3349ce6fb4bfa65a850fb70f6f8       | s390x    | linux    | 11       | openj9   | jdk  | 197.27  MB | urn:net.adoptopenjdk.raw     | production
e0446e80eeb90ab97fd59f91ae094bdd658c354943d4e33e0ddf691714f9da4d       | x64      | linux    | 11       | openj9   | jdk  | 195.55  MB | urn:net.adoptopenjdk.raw     | production
0b6050cc670eefd9465370ab19ae70401476430fca329e65f0dd636ca9cce9bd       | x64      | linux    | 11       | openj9   | jdk  | 195.41  MB | urn:net.adoptopenjdk.raw     | large-heap production
518e9330f2aeb020bca31a07034a08c3a568823e703fc503e03206acdad9dae5       | x64      | linux    | 11       | openj9   | jre  | 40.91   MB | urn:net.adoptopenjdk.raw     | large-heap production
9306fa569d2d5abf982df0b4ffa5d30c652cf16139d7b2268db30d2886f3adbf       | x64      | linux    | 11       | openj9   | jdk  | 195.42  MB | urn:net.adoptopenjdk.raw     | large-heap production
7e9904e1bfa197e6e72570207cb2bd63dd7cc4bf5f97bc5454c1fc5a559a8bf1       | x64      | linux    | 11       | hotspot  | jdk  | 195.71  MB | urn:net.adoptopenjdk.raw     | production
a016413fd8415429b42e543fed7a1bee5010b1dbaf71d29a26e1c699f334c6ff       | x64      | linux    | 11       | openj9   | jre  | 40.96   MB | urn:net.adoptopenjdk.raw     | production
ef9bf07cba79082285a9d426ea4eb3e8df57561ce2afe07cc5f299a8fa203279       | x64      | linux    | 11       | openj9   | jdk  | 195.55  MB | urn:net.adoptopenjdk.raw     | production
10b2400add0070b64db91026f12b798708e59772a81492fd2b638638f71ed14f       | x64      | linux    | 11       | hotspot  | jdk  | 186.61  MB | urn:net.java.jdk             | early-access valhalla
11cb48e3592f46f2c2dab095e3b9a4fec60b89199827f2b737a14b54dc5fd68d       | x64      | linux    | 11       | openj9   | jdk  | 195.36  MB | urn:net.adoptopenjdk.raw     | large-heap production
cac1fe096ea161613d5afe8a04ce3294dc1b535f5f4ea1a4235ce425b2610678       | x64      | linux    | 11       | openj9   | jdk  | 195.36  MB | urn:net.adoptopenjdk.raw     | large-heap production
765947ab9457a29d2aa9d11460a4849611343c1e0ea3b33b9c08409cd4672251       | x64      | linux    | 11       | openj9   | jdk  | 195.50  MB | urn:net.adoptopenjdk.raw     | production
7a6bb980b9c91c478421f865087ad2d69086a0583aeeb9e69204785e8e97dcfd       | x64      | linux    | 11.0.1   | hotspot  | jdk  | 187.60  MB | urn:net.java.jdk             | production
33a5f5b063dde3a66a98cf691bd00b21c012950ae1af253e19dcc68d74a75f65       | x64      | linux    | 12       | hotspot  | jdk  | 304.88  MB | urn:net.java.jdk             | early-access panama

That narrows things down a little, but there are still far too many matching runtimes. Let's specify that we want Linux, x86_64 (the CoffeePick API refers to 32-bit x86 as x32 and 64-bit x86 as x64 for uniformity), and let's specify a version range that says that we want any version of Java greater than or equal to 11 but less than 12:

[coffeepick]$ catalog platform:linux arch:x64 version:[11,12)
ID                                                                     | Arch     | Platform | Version  | VM       | Conf | Size       | Repository                   | Tags
e0446e80eeb90ab97fd59f91ae094bdd658c354943d4e33e0ddf691714f9da4d       | x64      | linux    | 11       | openj9   | jdk  | 195.55  MB | urn:net.adoptopenjdk.raw     | production
7e9904e1bfa197e6e72570207cb2bd63dd7cc4bf5f97bc5454c1fc5a559a8bf1       | x64      | linux    | 11       | hotspot  | jdk  | 195.71  MB | urn:net.adoptopenjdk.raw     | production
a016413fd8415429b42e543fed7a1bee5010b1dbaf71d29a26e1c699f334c6ff       | x64      | linux    | 11       | openj9   | jre  | 40.96   MB | urn:net.adoptopenjdk.raw     | production
ef9bf07cba79082285a9d426ea4eb3e8df57561ce2afe07cc5f299a8fa203279       | x64      | linux    | 11       | openj9   | jdk  | 195.55  MB | urn:net.adoptopenjdk.raw     | production
10b2400add0070b64db91026f12b798708e59772a81492fd2b638638f71ed14f       | x64      | linux    | 11       | hotspot  | jdk  | 186.61  MB | urn:net.java.jdk             | early-access valhalla
0b6050cc670eefd9465370ab19ae70401476430fca329e65f0dd636ca9cce9bd       | x64      | linux    | 11       | openj9   | jdk  | 195.41  MB | urn:net.adoptopenjdk.raw     | large-heap production
11cb48e3592f46f2c2dab095e3b9a4fec60b89199827f2b737a14b54dc5fd68d       | x64      | linux    | 11       | openj9   | jdk  | 195.36  MB | urn:net.adoptopenjdk.raw     | large-heap production
cac1fe096ea161613d5afe8a04ce3294dc1b535f5f4ea1a4235ce425b2610678       | x64      | linux    | 11       | openj9   | jdk  | 195.36  MB | urn:net.adoptopenjdk.raw     | large-heap production
518e9330f2aeb020bca31a07034a08c3a568823e703fc503e03206acdad9dae5       | x64      | linux    | 11       | openj9   | jre  | 40.91   MB | urn:net.adoptopenjdk.raw     | large-heap production
9306fa569d2d5abf982df0b4ffa5d30c652cf16139d7b2268db30d2886f3adbf       | x64      | linux    | 11       | openj9   | jdk  | 195.42  MB | urn:net.adoptopenjdk.raw     | large-heap production
765947ab9457a29d2aa9d11460a4849611343c1e0ea3b33b9c08409cd4672251       | x64      | linux    | 11       | openj9   | jdk  | 195.50  MB | urn:net.adoptopenjdk.raw     | production
7a6bb980b9c91c478421f865087ad2d69086a0583aeeb9e69204785e8e97dcfd       | x64      | linux    | 11.0.1   | hotspot  | jdk  | 187.60  MB | urn:net.java.jdk             | production

We are making progress. Let's assume, for the sake of this example, that we know our application isn't compatible with OpenJ9. We can require that the VM implementation be Hotspot:

[coffeepick]$ catalog platform:linux arch:x64 version:[11,12) vm:hotspot
ID                                                                     | Arch     | Platform | Version  | VM       | Conf | Size       | Repository                   | Tags
7e9904e1bfa197e6e72570207cb2bd63dd7cc4bf5f97bc5454c1fc5a559a8bf1       | x64      | linux    | 11       | hotspot  | jdk  | 195.71  MB | urn:net.adoptopenjdk.raw     | production
10b2400add0070b64db91026f12b798708e59772a81492fd2b638638f71ed14f       | x64      | linux    | 11       | hotspot  | jdk  | 186.61  MB | urn:net.java.jdk             | early-access valhalla
7a6bb980b9c91c478421f865087ad2d69086a0583aeeb9e69204785e8e97dcfd       | x64      | linux    | 11.0.1   | hotspot  | jdk  | 187.60  MB | urn:net.java.jdk             | production

Let's also assume that we're only interested in builds that are production-ready. The CoffeePick API allows for repositories to provide arbitrary tags to builds in order to further distinguish them beyond the standard metadata defined by the API. The convention is to tag production-ready builds with the production tag, so let's list builds that have that tag:

[coffeepick]$ catalog platform:linux arch:x64 version:[11,12) vm:hotspot require-tag:production
ID                                                                     | Arch     | Platform | Version  | VM       | Conf | Size       | Repository                   | Tags
7e9904e1bfa197e6e72570207cb2bd63dd7cc4bf5f97bc5454c1fc5a559a8bf1       | x64      | linux    | 11       | hotspot  | jdk  | 195.71  MB | urn:net.adoptopenjdk.raw     | production
7a6bb980b9c91c478421f865087ad2d69086a0583aeeb9e69204785e8e97dcfd       | x64      | linux    | 11.0.1   | hotspot  | jdk  | 187.60  MB | urn:net.java.jdk             | production

We've narrowed it down to two builds. We can see from the Repository column that one of those builds comes from the jdk.java.net repository and the other comes from AdoptOpenJDK. Let's arbitrarily decide that we want the one from jdk.java.net and download it:

[coffeepick]$ download 7a6bb980b9c91c478421f865087ad2d69086a0583aeeb9e69204785e8e97dcfd
[    2.10 MB /   187.60 MB] |+                                                           | 2.10MB/s
[    4.20 MB /   187.60 MB] |++                                                          | 2.10MB/s
[    7.72 MB /   187.60 MB] |+++                                                         | 3.52MB/s
[   10.49 MB /   187.60 MB] |++++                                                        | 2.77MB/s
[   12.60 MB /   187.60 MB] |+++++                                                       | 2.11MB/s

(Time passes...)

[  178.69 MB /   187.60 MB] |++++++++++++++++++++++++++++++++++++++++++++++++++++++++++  | 2.35MB/s
[  180.67 MB /   187.60 MB] |++++++++++++++++++++++++++++++++++++++++++++++++++++++++++  | 1.98MB/s
[  182.67 MB /   187.60 MB] |+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ | 2.00MB/s
[  184.81 MB /   187.60 MB] |++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++| 2.14MB/s
[  187.15 MB /   187.60 MB] |++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++| 2.34MB/s
Download finished
Runtime archive: /home/rm/local/coffeepick/inventory/7a6bb980b9c91c478421f865087ad2d69086a0583aeeb9e69204785e8e97dcfd/archive

For safety reasons, the CoffeePick API automatically verifies that the downloaded archive matches the published checksum. This should provide a minimum level of confidence that you actually downloaded the right runtime, and that the runtime made it onto the disk without malicious or accidental modification. It's possible to manually verify an already-downloaded runtime at any time in the future with the verify command:

[coffeepick]$ verify 7a6bb980b9c91c478421f865087ad2d69086a0583aeeb9e69204785e8e97dcfd
Verified:
  Algorithm: SHA-256
  Received:  7a6bb980b9c91c478421f865087ad2d69086a0583aeeb9e69204785e8e97dcfd

Now that the runtime has been downloaded, it will appear in the inventory. In the CoffeePick API, the inventory is the set of runtimes that you currently have downloaded. The inventory command lists the downloaded runtimes and supports all of the same search parameters as the catalog command:

[coffeepick]$ inventory
ID                                                                     | Arch     | Platform | Version  | VM       | Conf | Size       | Repository                   | Tags
7e9904e1bfa197e6e72570207cb2bd63dd7cc4bf5f97bc5454c1fc5a559a8bf1       | x64      | linux    | 11       | hotspot  | jdk  | 195.71  MB | urn:net.adoptopenjdk.raw     | production
a016413fd8415429b42e543fed7a1bee5010b1dbaf71d29a26e1c699f334c6ff       | x64      | linux    | 11       | openj9   | jre  | 40.96   MB | urn:net.adoptopenjdk.raw     | production
d33551f15f8d4bc5f97471bca669872665e5db15cc8a5ac88f068e23f56bb129       | x64      | windows  | 11       | openj9   | jre  | 40.06   MB | urn:net.adoptopenjdk.raw     | production
7a6bb980b9c91c478421f865087ad2d69086a0583aeeb9e69204785e8e97dcfd       | x64      | linux    | 11.0.1   | hotspot  | jdk  | 187.60  MB | urn:net.java.jdk             | production

[coffeepick]$ inventory vm:openj9
ID                                                                     | Arch     | Platform | Version  | VM       | Conf | Size       | Repository                   | Tags
a016413fd8415429b42e543fed7a1bee5010b1dbaf71d29a26e1c699f334c6ff       | x64      | linux    | 11       | openj9   | jre  | 40.96   MB | urn:net.adoptopenjdk.raw     | production
d33551f15f8d4bc5f97471bca669872665e5db15cc8a5ac88f068e23f56bb129       | x64      | windows  | 11       | openj9   | jre  | 40.06   MB | urn:net.adoptopenjdk.raw     | production

Of course, a runtime isn't much use sitting there in the on-disk storage area. The unpack command unpacks a downloaded runtime into a user-specified directory:

[coffeepick]$ unpack --output /tmp/jdk11-linux-x64 --strip-leading-directory 7e9904e1bfa197e6e72570207cb2bd63dd7cc4bf5f97bc5454c1fc5a559a8bf1
[coffeepick]$

Now, in an ordinary shell:

$ ls -alF /tmp/jdk11-linux-x64/
total 4
drwxr-xr-x  10 rm   rm    220 2018-11-18 14:23 ./
drwxrwxrwt 198 root root 4420 2018-11-18 14:23 ../
drwxr-xr-x   2 rm   rm    680 2018-11-18 14:23 bin/
drwxr-xr-x   4 rm   rm    140 2018-11-18 14:23 conf/
drwxr-xr-x   4 rm   rm    100 2018-11-18 14:23 demo/
drwxr-xr-x   3 rm   rm    180 2018-11-18 14:23 include/
drwxr-xr-x   2 rm   rm   1440 2018-11-18 14:23 jmods/
drwxr-xr-x  72 rm   rm   1440 2018-11-18 14:23 legal/
drwxr-xr-x   6 rm   rm   1040 2018-11-18 14:23 lib/
drwxr-xr-x   4 rm   rm    100 2018-11-18 14:23 man/
-rw-r--r--   1 rm   rm   1202 2018-11-18 14:23 release

$ cat /tmp/jdk11-linux-x64/release 
IMPLEMENTOR="AdoptOpenJDK"
IMPLEMENTOR_VERSION="AdoptOpenJDK"
JAVA_VERSION="11.0.1"
JAVA_VERSION_DATE="2018-10-16"
MODULES="java.base java.compiler java.datatransfer java.xml java.prefs java.desktop java.instrument java.logging java.management java.security.sasl java.naming java.rmi java.management.rmi java.net.http java.scripting java.security.jgss java.transaction.xa java.sql java.sql.rowset java.xml.crypto java.se java.smartcardio jdk.accessibility jdk.internal.vm.ci jdk.management jdk.unsupported jdk.internal.vm.compiler jdk.aot jdk.internal.jvmstat jdk.attach jdk.charsets jdk.compiler jdk.crypto.ec jdk.crypto.cryptoki jdk.dynalink jdk.internal.ed jdk.editpad jdk.hotspot.agent jdk.httpserver jdk.internal.le jdk.internal.opt jdk.internal.vm.compiler.management jdk.jartool jdk.javadoc jdk.jcmd jdk.management.agent jdk.jconsole jdk.jdeps jdk.jdwp.agent jdk.jdi jdk.jfr jdk.jlink jdk.jshell jdk.jsobject jdk.jstatd jdk.localedata jdk.management.jfr jdk.naming.dns jdk.naming.rmi jdk.net jdk.pack jdk.rmic jdk.scripting.nashorn jdk.scripting.nashorn.shell jdk.sctp jdk.security.auth jdk.security.jgss jdk.unsupported.desktop jdk.xml.dom jdk.zipfs"
OS_ARCH="x86_64"
OS_NAME="Linux"
SOURCE=""

Finally, it's possible tell repository providers to self-update. This is useful in the case of a repository such as AdoptOpenJDK where new runtime builds are published frequently.

$ repository-update urn:net.adoptopenjdk.raw
Update of urn:net.adoptopenjdk.raw started…
[ 0.00   % ] |                                                            |
[ 1.79   % ] |++                                                          |
[ 3.57   % ] |+++                                                         |
[ 5.36   % ] |++++                                                        |
[ 7.14   % ] |+++++                                                       |

(Time passes...)

[ 91.07  % ] |+++++++++++++++++++++++++++++++++++++++++++++++++++++++     |
[ 92.86  % ] |++++++++++++++++++++++++++++++++++++++++++++++++++++++++    |
[ 94.64  % ] |+++++++++++++++++++++++++++++++++++++++++++++++++++++++++   |
[ 96.43  % ] |++++++++++++++++++++++++++++++++++++++++++++++++++++++++++  |
[ 98.21  % ] |+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ |
Update of urn:net.adoptopenjdk.raw finished.

Subsequent executions of the catalog command will then show new releases (if any are available) in addition to those that were already available.

Future directions for the project:

  1. A Maven plugin

    I want to be able to do all of the above in an automated manner during builds. The API already allows for electively downloading or reusing cached runtimes, and runtimes can be specified directly by hash value for reproducibility in builds.

  2. Add shipilev.net builds

    Aleksey Shipilëv publishes various early-access builds so that people can try out new JDK features. I'd like to get his builds included into a repository provider (ideally with some way to find new builds without having to crawl over his entire site to discover them).

  3. A standard schema for repository metadata

    I already have a very strict XSD schema that describes a format for repository metadata. It would just be a matter of persuading others to use it, and allowing the API to consume XML metadata.

You Cannot Put That There

Maven 3.6.0 was released today. It adds the ability to add a child.inherit.append.path attribute to various places within a POM file to turn off Maven's automatic appending of child module names to various paths. This has been a long-standing complaint with Maven as it had essentially baked-in SVN path semantics to the model in a way that couldn't be turned off or worked around sanely.

Unfortunately, the Maven Central rules can't accept POM files that contain the new attributes:

[ERROR] Rule failure while trying to close staging repository with ID "comio7m-1402".
[ERROR] 
[ERROR] Nexus Staging Rules Failure Report
[ERROR] ==================================
[ERROR] 
[ERROR] Repository "comio7m-1402" failures
[ERROR]   Rule "sources-staging" failures
[ERROR]     * Illegal POM: /com/io7m/primogenitor/com.io7m.primogenitor/3.0.0-beta0018/com.io7m.primogenitor-3.0.0-beta0018.pom Unknown attribute 'child.inherit.append.path' for tag 'project' (position: START_TAG seen ...ache.org/xsd/maven-4.0.0.xsd\n  child.inherit.append.path=false>... @6:37) 
[ERROR]   Rule "javadoc-staging" failures
[ERROR]     * Illegal POM: /com/io7m/primogenitor/com.io7m.primogenitor/3.0.0-beta0018/com.io7m.primogenitor-3.0.0-beta0018.pom Unknown attribute 'child.inherit.append.path' for tag 'project' (position: START_TAG seen ...ache.org/xsd/maven-4.0.0.xsd\n  child.inherit.append.path=false>... @6:37) 
[ERROR]   Rule "pom-staging" failures
[ERROR]     * Invalid POM: /com/io7m/primogenitor/com.io7m.primogenitor/3.0.0-beta0018/com.io7m.primogenitor-3.0.0-beta0018.pom: Parsing Error: Unknown attribute 'child.inherit.append.path' for tag 'project' (position: START_TAG seen ...ache.org/xsd/maven-4.0.0.xsd\n  child.inherit.append.path=false>... @6:37) 
[ERROR] 
[ERROR] 
[ERROR] Cleaning up local stage directory after a Rule failure during close of staging repositories: [comio7m-1402]
[ERROR]  * Deleting context 2a74942ea74dc7.properties
[ERROR] Cleaning up remote stage repositories after a Rule failure during close of staging repositories: [comio7m-1402]
[ERROR]  * Dropping failed staging repository with ID "comio7m-1402" (Rule failure during close of staging repositories: [comio7m-1402]).

I've filed a bug.

Java 11 and Maven 3.6.0

Java 11 is out, and is the current LTS release (at least if you're using the Oracle JDK).

I'll be moving all of my code to start requiring JDK 11 at a minimum as of now.

Additionally, Maven 3.6.0 is currently in the process of being released, and it contains a fix for an issue that's important to me, so I suspect I'm also going to start requiring Maven 3.6.0 everywhere too.

No More OpenGL

I'm dropping OpenGL in favour of Vulkan.

No more broken drivers (probably). No more dealing with source code level management of shader code at runtime (no more sombrero). No more broken GLSL shader compilers, for that matter. No more weak typing (no more jcanephora). No more logarithmic depth buffer - Vulkan has reversed floating-point depth buffers accessible without extensions.

Planning to rewrite r2 using Vulkan, yielding r3. A few design issues that were imposed by the OpenGL 3.3 core requirement can be corrected.

Still need to give the Vulkan API a friendly face though, to make it feel like a Java API instead of a C++ API.

Maven Central Signing

A while ago I started having a problem with signature verification when trying to upload to Maven Central.

After three months of intense debugging and a rather long conversation with Sonatype, it turned out that the issue was actually with the signing key.

At the start of 2018, I'd switched to ed25519 signing keys. It turned out that the version of Nexus running on Maven Central didn't support ed25519. To work around this, I created a new RSA key solely intended for signing Maven packages. Bizarrely, this key didn't work either. Nobody could work out why the signatures were failing, and the problem was escalated twice to Sonatype's internal support people.

It turned out that the problem was an ed25519 signature on the new RSA key!

The moral of the story: If you want to deploy to Maven Central, use only RSA keys and make sure that the signatures on those RSA keys only come from other RSA keys. If you fail to do this, you won't get an actionable error message when you try to deploy packages, you'll just get a "Signature verification failed" message. Sonatype are updating their documentation to ensure that nobody else has to lose time to this.

Thanks to Joel Orlina for being patient during those three months and for handling the support teams.

Competing Module Systems

It's been about eight months since the release of Java 9.

I won't bore anyone with the details of everything it introduced as that information is available just about anywhere you care to look. The main thing it added, however, is the subject of this post: The Java Platform Module System.

The JPMS was introduced, and programmers reacted in the normal way that people working in a field that's supposed to demand a rational and critical mind react: They started frothing at the mouth, claiming that Oracle were trying to kill Java, and planning a mass exodus to other more hip languages. Kotlin or Rust, probably. Needless to say, Oracle weren't trying to kill Java. If Oracle wanted to kill Java, they could certainly find a way to do it that didn't require seven expensive years of intense, careful, and painful software engineering. So far, the exodus appears to have been quietly called off.

I'm guessing the people that complained the loudest are the sort of people that write a ton of fragile, error-prone, unsupported reflection hacks and then spew bile when some tiny aspect of the platform changes and their code breaks.

I have a ton of code, none of which I'd describe as legacy. I'm somewhat invested in OSGi for reasons I'll go into shortly.

Today: I'm conflicted and slightly nervous!

I decided when Java 9 came out that I was going to pursue full modularization for all of my projects. As I've said before, my code is already modular because I've been designing it around OSGi. However, in order for it to become modular in the JPMS sense, I'd have to write module descriptors for each project.

I'm developing a commercial (but open-source) game, where the game itself and third party modifications are handled by a well-specified, strongly-versioned module system. Consider something that (from the user's perspective) behaves a bit like the system in OpenTTD (except with the entire game delivered this way, not just third-party addons):

OpenTTD

I briefly considered stopping using OSGi and using the JPMS to power the whole system. That experiment fairly quickly ended, though. Let's look at some of the things that OSGi does or has that are important to me:

  1. OSGi has a full repository specification that gives applications the ability to download and install packages at run-time. You tell the system what packages you want, and it fetches those and all of their dependencies. Everything is standardized, from the API to the actual metadata served by repository implementations. Here's an example of a standard repository index (albeit styled with an XSL stylesheet).

  2. OSGi bundles have rich metadata associated with them, including very fine-grained and versioned dependency metadata. If you have a bundle, you can find out very easily what else you need in order to be able to use it. The metadata can also express things beyond simple package dependencies.

  3. OSGi can load and unload modules at run-time, and has standard APIs to do so (and standard APIs that application code can implement in order to react to modules being loaded and unloaded).

  4. OSGi has wide support; it's fairly rare these days to find a Java project that isn't also implicitly an OSGi project. Making something OSGi-compatible is often just a case of adding a single Maven plugin invocation (assuming that the code doesn't make a lot of unfounded assumptions about class loaders).

Let's see what the JPMS has (or hasn't), in comparison:

  1. The JPMS doesn't have anything like a repository specification. The closest analogy is probably fetching things off of Maven Central, but there's no standard API for doing this. Essentially, obtaining modules is Someone Else's Problem.

  2. JPMS modules don't have much metadata associated with them. A module says what packages it exports and upon which other modules it depends. However, it doesn't say anything about which versions of those modules are required. Essentially, you're expected to use a system like Maven. This is Someone Else's Problem.

  3. The JPMS can, in some sense, load modules at run-time via ModuleLayers. There's no support for explicitly unloading a module. There are no standard APIs that code can use to be notified that a module has been loaded. Essentially, you're expected to build your own ad-hoc OSGi-like system on top of module layers. This is Someone Else's Problem.

  4. Most projects haven't even begun to modularize their code. Even in a well-staffed and extremely fundamental project like Google Protobuf, I'm still waiting after six months for them to make a harmless one-line JAR manifest update. I cannot begin to imagine what it would take to push full modularization through a system that slow moving. If a project you depend on hasn't modularized, you're not going to be modularizing either.

The first three points mean that I'd have to invent my own OSGi-like system. I'd have to come up with a uniform, platform-agnostic metadata standard that could be inserted into JAR files in order to specify dependency information. Third party projects (at least those that aren't written specifically for my game engine) aren't going to be interested in adding metadata for the sake of one man's low budget module system. I can't trust JAR files to contain Maven pom.xml files (although many do), as there are plenty of other build systems in use that don't include that information. Even then, you can't just parse a Maven pom.xml, you need to evaluate it. This would mean pulling in Maven APIs as a dependency, and those neither work in a JPMS modular context (because they're not modularized and never will be) nor an OSGi context (because they use their own incompatible module system called Plexus).

Let's assume by some miracle that I get a good percentage of the Java ecosystem to include dependency metadata in a format I can consume. I now need to come up with an API and tools to fetch modules from a repository. That's not hard, but it's still more than not having to do it at all. I'd want version ranges, so I'd also need to write code to solve the versioning problem which is NP-complete. Tricky.

So let's assume by now that I can, given the name of a module, fetch the dependencies at run-time using my shiny new API and metadata. I still need to be able to load and unload that set of modules at run-time, and have code in other modules react properly to that occurring. There aren't any standard APIs to do this in the JPMS, and so I'd have to write my own. Right now, the JPMS is not actually capable of expressing the kind of things that are possible with OSGi bundles, so any system I built would be strictly less capable than the existing OSGi system. At least part of the problem appears to come down to missing APIs in the JPMS, so at least some of this might be less of an issue in the future. Still, it's work I'd have to do.

So let's assume that I can fetch, load, and unload modules at run-time, and code in modules can react to this happening. I still need modules to put in the system, and not many Java projects are modularized. I tried, for a few months, to use only projects that have either added Automatic-Module-Name entries to their JAR manifests, or have fully modularized. It was fairly painful. I'd be limiting myself to a tiny percentage of the vast Java ecosystem if I did this. This, however, is something that OSGi also went through in the early days and is no longer a problem. It's just a matter of time.

So why am I nervous and still conflicted?

The default effect suggests that, just because the JPMS is the default Java module system, the JPMS is the one that developers will turn to. Tool support for JPMS modules is already vastly better in Intellij IDEA than it is for OSGi. In Eclipse, the support is about equal.

The OSGi R7 JAR files will not contain Automatic-Module-Name entries which means that if you have any dependency on any part of OSGi (including parts that are not even used at run-time, like the annotations), your code cannot be fully modularized. If developers are forced to choose between supporting that old module system that nobody uses or that shiny new default module system that everyone's talking (or possibly frothing) about, which one do you think they'll pick?

I'm also aware that this kind of dynamic module loading and unloading is not something that most applications need. For years people were happy with a static (although extremely error-prone) class path mechanism, and the majority of applications are just recompiled and re-deployed when updated. The JPMS can support this kind of deployment sufficiently. Previously, if developers wanted any kind of safe module system at all, they had to turn to OSGi. They might only use it in the context of a static application that is not dynamically updated, but they'd still use it. Why would those same developers still turn to OSGi when the JPMS exists?

Finally, I pay attention to the various new developments to the Java language and JVM and, fairly often, new features are proposed that have subtle (or not-so-subtle) dependencies on the new module system. There are various optimization strategies, for example, that can be enabled if you know that instances of a given type are completely confined to a single module. Unless the VM can be magically persuaded that OSGi bundles are JPMS modules (this is unlikely to happen), then code running in OSGi is very much going to become a second-class citizen.

So, I'm nervous and conflicted. I don't want to build some sort of ad-hoc OSGi-lite system on the JPMS. I don't want to do the work, I don't want to maintain it, and I don't think the result would be very good anyway. I also, however, am unsure about continuing to base my code on a system that's going to have to work hard not to be considered irrelevant. I believe OSGi is the superior choice, but it's not the default choice, and I think that's going to matter more than it should.

I suspect I'm going to finish assisting any remaining projects that I've started helping to modularize, and not do any more. I had decided that I was going to push hard to move all of my projects to requiring Java 9 as part of the modularization effort, but unfortunately this would leave me unable to deploy code on FreeBSD as there's no JDK 9 there and likely won't be. With the new six-month release cycle (and 18-month long-term release cycle), porting efforts will likely be directed towards JDK 11. This means that I won't be able to deploy anything newer than Java 8 bytecode on FreeBSD for at least another six months.

IPv6 And Linux

I wrote a while back about issues with IPv6 on Linux. It turns out that most of the pain occurs for two reasons:

  1. Linux doesn't accept router advertisements by default. If you configure your router to tell everyone on the network that it has a nonstandard MTU, Linux will ignore the advertisements.

  2. Linux will act upon any Packet Too Large messages it receives and in fact will create a temporary route (visible from the ip route command) that has the correct reduced MTU but, for whatever reason, most programs won't use the new route without a restart. That is, if my mail client hangs due to a Packet Too Large message, it won't be able to send any mail until I restart the program.

The first point can be addressed by adding the following to /etc/sysctl.conf:

net.ipv6.conf.default.accept_ra=1
net.ipv6.conf.default.accept_ra_mtu=1

Usually, net.ipv6.conf.default.accept_ra_mtu is already set to 1, but it's worth being explicit about it.

I also add net.ipv6.conf.default.autoconf=0 because I statically assign addresses.

The second point can be addressed by restarting the program, as sad as that is.

Module Chasing

https://github.com/io7m/modulechaser

modulechaser

I'm trying to get to the point where all of my projects are fully modularized as JPMS modules. My code already follows a very modular architecture thanks to designing it around OSGi, but it's necessary to write module descriptors to complete the picture. To write module descriptors, the projects upon which a project depends must first be modularized. This can either mean writing module descriptors for those projects, or it can simply mean assigning an Automatic-Module-Name. Writing a full module descriptor is better, because this means that the project can be used in combination with jlink to produce tiny custom JVM distributions.

My problem is that I have rather a large number of dependencies across all of my projects, and I need to know the most efficient order in which to approach maintainers in order to get them to modularize their projects. If a project A depends on project B, then project A can't be modularized before project B so it's a waste of my time to go asking project A's maintainers before B is modularized.

I wrote a Maven plugin to assist with this problem. It produces reports like this.

The plugin tells me if the current version of a dependency is modularized, if the latest version of the dependency on Maven Central is modularized, and whether the dependency has been fully modularized or simply assigned a name. The table of dependencies is shown in reverse topological order: Start at the top of the table and work downwards, contacting each project maintainer as you go.

Some dependencies will, of course, never be modularized. Whomever published the javax.inject jar, for example, didn't even bother to create a manifest in the jar file. I'm not sure that even constitutes a valid jar file according to the specification. Some dependencies, like javaslang, were renamed (javaslang became vavr) and so code should move to using the newer names instead. Some projects can barely manage to publish to Maven Central (like Xerces) and still appear to use tools and processes from the previous century, so are unlikely to be modularizing any time soon.

Let's Encrypt For Woe And Loss

In a crippling bout of sinusitis, and after reading that Chrome is going to mark http sites as insecure, I decided to put good sense aside and deploy Let's Encrypt certificates on the io7m servers.

I've complained about the complexity of this before, so I started thinking about how to reduce the number of moving parts, and the number of protection boundary violations implied by the average ACME setup.

I decided the following invariants must hold:

  • The web server must not have write access to its own configuration, certificates, or logs. This is generally a given in any server setup. Logging is actually achieved by piping log messages to a log program such as svlogd which is running as a different user. In my case, I can actually go much further and state that the web server must not have write access to anything in the filesystem. This means that if the web server is compromised (by a buffer overflow in the TLS library, for example), it's not possible for an attacker to write to any data or code without first having to find a way to escalate privileges.

  • The web server must have read access to a challenge directory. This is the directory containing files created in response to a Let's Encrypt (LE) challenge.

  • The acme client must have read and write access to the certificates, and it must have write access to a challenge directory, but nothing else in the filesystem. Specifically, the client must not have write access to the LE account key or the directory that the web server is serving. This means that if the client is compromised, it can corrupt or reissue certificates but it can't substitute its own account key and request certificates on behalf of someone else.

  • The acme client must not have any kind of administrative access to the web server. I don't want the acme client helpfully restarting the web server because it thinks it's time to do so.

There are some contradictions here: The acme client must not be able to write to the directory that the web server is serving, and yet the web server must be able to serve challenge responses to the LE server. The acme client must not be able to restart the web server, and yet the web server must be restarted in order to pick up new certificates when they're issued.

I came up with the following:

  • An httpd-reloader service sends a SIGHUP signal to the web server every 30 minutes. This causes the web server to re-read its own configuration data and reload certificates, but does not kill any requests that are in the process of being served and specifically does not actually restart the web server.

  • The acme client writes to a private challenge directory, and a private certificates directory. It doesn't know anything about the web server and is prevented from accessing anything other than those directories via a combination of access controls and chrooting.

  • The web server reads from a read-only nullfs mount of a wwwdata directory, and the challenge directory above is placed in wwwdata/.well-known/acme-challenge via another read-only nullfs mount. The web server also reads from a read-only nullfs mount of the certificates directory above in order to access the certificates that the acme client creates.

  • The acme client is told to update certificates hourly, but the acme client itself decides if it's really necessary to update the certificates each time (based on the time remaining before the certificates expire).

  • The intrusion detection system has to be told that the web server's certificates are permitted to change. The account key is never permitted to change. I don't want notifications every ~90 days telling me the certificates have been modified.

Data flow

I set up all of the above and also took some time to harden the configuration by enabling various HTTP headers such as Strict-Transport-Security, Content-Security-Policy, Referrer-Policy, etc. I'm not going to require https to read io7m.com and I'm not going to automatically redirect traffic from the http site to the https site. As far as I'm concerned, it's up to the people reading the site to decide whether or not they want https. There are plenty of browser addons that can tell the browser to try https first, and I imagine Chrome is already doing this without addons.

The Qualys SSL Labs result:

Result

Now we can all sleep soundly in the knowledge that a third party that we have no reason whatsoever to trust is telling us that io7m.com is safe.

Checkstyle Rules

I'm going to start making all projects use a common set of Checkstyle rules rather than having each project carry its own rules around. I can't remember exactly why I avoided doing this in the beginning. I think it may've been that I wasn't confident that I could write one set of rules that would work everywhere. I've decided instead that I'll beat code with a shovel until it follows the rules, rather than beat the rules with a shovel until they follow the code.

https://github.com/io7m/checkstyle_rules

Chasing Modules

I've been moving all of the projects I still maintain to Java 9 modules. In order to do this, however, I've needed to assist third party projects upon which I have dependencies to either modularize their projects or publish Automatic-Module-Name entries in their jar manifests. If you try to specify a dependency on a project that either hasn't been modularized or hasn't published an Automatic-Module-Name entry, you'll see this when building with Maven:

[WARNING] ********************************************************************************************************************
[WARNING] * Required filename-based automodules detected. Please don't publish this project to a public artifact repository! *
[WARNING] ********************************************************************************************************************

The reasons for this are documented on Stephen Colebourne's blog.

Here's a table of all of the third party projects upon which I depend, and an indication of the current state of modularization (I'll try to keep this updated as projects are updated):

Project State Updated
fastutil Considering 2018-02-07
jcpp Considering 2018-02-08
ed25519-java Considering 2018-02-09
LWJGL Fully Modularized 2018-02-07
Dyn4j Fully Modularized 2018-02-07
protobuf Ignored? 2018-02-07
antlr In Progress 2018-02-11
autovalue In Progress 2018-02-07
rome In Progress 2018-02-07
JGraphT Automatic Module Names (Full modularization in progress) 2018-02-11
Vavr Automatic Module Names 2018-02-07
commonmark-java Automatic Module Names 2018-02-07
javapoet Automatic Module Names 2018-02-07
xom Unclear 2018-02-07

Generally, if a project isn't planning to either modularize or publish automatic module names, then I'm looking for a replacement for that project.

Signing Issues

I'm having trouble deploying packages to Maven Central. The repository claims that it cannot validate the signatures I provide. I've filed a bug but haven't gotten a response. I'm wondering if it's down to me switching to Curve25519 PGP keys...

New PGP Keys

New PGP keys have been published.

Fingerprint                                       | UID
-----------------------------------------------------------------------------------
B84E 1774 7616 C617 4C68 D5E5 5C1A 7B71 2812 CC05 | Mark Raynsford (2018 personal)
F8C3 C5B8 C86A 95F7 42B9 36D2 97E0 2011 0410 DFAF | io7m.com (2018 release-signing)
Verifying blog.io7m.com... Again

See the the previous blog post:

$ gpg --recv-keys 8168DAE22B15D3EDC722C23D0F15B7D06FA80CB8
$ wget -r http://blog.io7m.com
$ cd blog.io7m.com
$ gpg < checksum.asc | sha512sum --check

Better.

How To Verify blog.io7m.com

This blog was designed to be verifiable:

$ gpg --recv-keys 8168DAE22B15D3EDC722C23D0F15B7D06FA80CB8
$ wget -r http://blog.io7m.com
$ find blog.io7m.com -name '*.asc' -exec gpg --verify {} \;

Note that the 8168DAE22B15D3EDC722C23D0F15B7D06FA80CB8 key ID assumes you're reading this in 2017. By the time 2018 comes around, this blog will be signed with a new key (and a new one for each passing year).

Possible points of failure:

  1. A malicious actor gets the remote keyserver to serve a different key than the one with fingerprint 8168DAE22B15D3EDC722C23D0F15B7D06FA80CB8. Does gnupg verify that a received key had the fingerprint that the user specified on the command line? What if the user specified my name and key comment instead of a fingerprint? The actor at this point might be able to convince you that the signatures on files on blog.io7m.com are invalid. It might be able to convince you that its own key is mine.

  2. A malicious actor modifies the files and signatures when wget downloads them. The actor can't generate valid signatures for the key 8168DAE22B15D3EDC722C23D0F15B7D06FA80CB8 (unless it can break RSA), but it can try to convince you that its own key is actually my key and therefore have you trust that the data you're seeing is unmodified and was signed by me. If the actor managed to perform step 1 above, then you're completely compromised.

  3. A malicious actor removes some of the signatures. If you didn't know exactly how many pages there should be, you'd not notice if gpg didn't verify one of them.

Step 1 has no technical solution; you need to verify the keys you receive and check the signatures on those keys assuming they come from other keys that you trust. If you're lucky, there is a chain of signatures leading to a key that you do trust with certainty. I have yearly expiring keys, and I sign each new year's keys with the old keys. Unless my keys have been compromised yearly, there's a reasonable chance that the keys you see are mine!

Step 2 is partially mitigated by https, assuming that the actor doesn't have backdoor access to the CA issuing the certificate. The actor can have the CA issue a new certificate, redirect you to a server that the actor controls, decrypt the traffic, modify it, and then re-encrypt it. You'd never know anything was wrong.

Step 3 is manually mitigated by reading the posts by year page and checking that you have at least as many signatures as pages. I may start publishing a CHECKSUMS file that just contains signed SHA512 hashes of every file on the site.

I'm considering using keybase in addition to publishing PGP keys on the public keyservers. I don't do social media, so I'm not sure how much it'd apply to me. I do use GitHub heavily though.

New PGP Key
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

2017-12-17T14:19:49+0000

Pushed a new key for signing commits in Android work. I'll still be
releasing new personal and release signing keys in January (and I'll
be switching to ed25519 keys from RSA).

E134 3512 A805 115A C1A0 1D15 C578 C3C0 C9B1 2BFF
Mark Raynsford (2018 android commit signing)

-----BEGIN PGP SIGNATURE-----

iQJCBAEBCgAsFiEEgWja4isV0+3HIsI9DxW30G+oDLgFAlo2fa4OHG1hcmtAaW83
bS5jb20ACgkQDxW30G+oDLjKiA/9HRrc0F7d/4C4ybdKFpp8N3S/RT/NJLfrYOTV
XQvt9Nw+eJydsygY8IbaZiWo9hkxddI1DLuWtahcrXFWGQ/VpAUmZcIondaJzLna
42Ui5jFpkweOVH2VYmuuDTV5rpkfH7IkTml5m2OsnsVU4hO9V1DGoNL+/5p2xv0E
sLkopFX/9qaRtw0qJ7u8Bl5217kodlI2inEMfomI8QcMp+JarkTogEdkkzFBz7Qo
/XGevkdeIBMCQ0NpsvoQmclGbgOtu6js1LvoQjDaXoVtT1yyIM831FDgFxbKCMw8
gzxx0f/TZjoRizIE0fNIwPmLYG5HXLCt6wN7iT6MYhp7ijBABZB8tH7cffCF1sM5
QE+eg8bzOL4FT57XLPpp9eDXZmLqmy6EdrWedaWughPPjUofqaIw0Bar1iIp4pY0
lIkAUTsqtEI32sAdInJ0PPar8nYVM8COSoyZ08kdxImO3DHRGerI7DSi88JpTiRy
vX96LZ0UvwyUhFxUSJQuxXYd82bQgoBhdjsLWiurZZYdZC5EGCwA7m5zapLWF76m
SJloK1ogK628TygNrhnNNsUirrEsDJM2CaNnp8/viN1eFlWMb13dgyLOQijyFyYV
8Q56XeILyckUY3ERw+v5BPN5g3qdF4nL5O23L3zrSQKcn6yUGgcFa6gia9cc1/Hm
EiTv9cc=
=D3uo
-----END PGP SIGNATURE-----
Back To Java

I initially wrote jstructural as a set of XSLT stylesheets. That quickly became unmaintainable as the complexity of the stylesheets increased. I rewrote them in Java. I eventually got tired of writing documentation in XML (I didn't know that better XML editors existed) so I ended up wanting to add the ability to write documents as S-expressions.

The right way to do this was to redesign jstructural so that it defined a core AST type and added multiple parsers that all produced values of this core type. Converting between documents of different types would then be a matter of parsing using one syntax, and then serializing the AST using another.

The most natural way to express an AST is as a large algebraic data type. Java doesn't have direct support for these yet (although it's slowly getting them via Project Amber). I therefore looked at other JVM languages that did support them, and the choice was either Scala or Kotlin. I chose Kotlin, after multiple nasty experiences with Scala. kstructural was born.

Fast forward to the present day, and I'm utterly sick of working on Kotlin code. The tools are slow. Compilation is slow. The people in charge of the ecosystem think devolving to special snowflake imperative build tools like Gradle is a good idea.

I'm going to do a new Java implementation of the structural language, and I'm probably going to redesign the language to make it even easier to implement. Nowadays, XML editing is fast and painless thanks to various tools (with real-time validation errors). XOM is essentially unmaintained, so I'll replace the XML parsing code with an efficient SAX parser (and actually get lexical information in the parsed AST, unlike with XOM).

DKIM

Deployed DKIM on the mail server today. All mail that originates from io7m.com will be signed, and I've published DMARC and ADSP policies that tell other servers to be suspicious of any mail that isn't signed. Seems to be working well.

TLS

I've enabled TLS on all io7m.com domains.

I'm using certificates from CaCert.org. These aren't trusted by all browsers, but I don't care all that much. If trust becomes a serious issue at some point, I'll buy real TLS certs. I PGP sign all text on the server anyway, so anyone that really cares about checking to see if some nefarious third party has changed the data in transit can do so, albeit manually.

No, I'm not using Let's Encrypt. I've looked into it several times and I just can't get around the fact that it requires a huge number of moving parts and that the average ACME client requires a ridiculous level of privileges to work properly: If you want any possibility of security, this is what it takes to get it.

At a minimum:

  • The client has to be able to make connections to a remote server in order to download an extremely security-critical bit of data (the certificate). If this step fails for any reason (the remote side being unavailable, breakage in the client), the existing certificate expires and the https service is dead.

  • The client has to be intelligent enough to know when to try to get a new certificate. When is the right time? Who knows. Trying to request a new certificate a minute before the current one expires is suicidally reckless. Trying to do it the day before might be acceptable, but what if it goes wrong? Is a day long enough to try to fix a problem in something as gruesomely complex as the average ACME client? The availability of your https service essentially becomes tied to the availability of the ACME server. This wouldn't be so bad if the update was a yearly thing and could be planned for, but LE certificates are valid for 90 days.

  • The client has to be able to write to the directory being served by the http server in order to be able to respond to challenges. If the client is compromised, it has the ability to trash the served web site(s). I run my services in a highly compartmentalized manner, and having to allow this would be horrible. There are other challenge types, such as publishing a DNS record containing a response to a challenge, but those have the same problem of what should be an unprivileged program having to cross a security boundary and threatening the integrity of another service's data.

  • The client has to be able to write to the http server's configuration data (in order to publish the new certificate). Again, if the client is compromised, it can trash the http server's configuration. If the update fails here, the existing certificate expires and the https service is dead.

  • Assuming that the client actually does manage to respond to a challenge and get a certificate issued, and does actually manage to write that certificate to the filesystem, the problem then becomes getting the https server to use it. Most servers read certificates once on startup and don't reread them. Therefore, the client needs privileges to restart the server. This is totally unacceptable; no service should ever be able to forcefully restart another.

There are so many possible points of failure and every single one results in a broken https service at best, or a system compromise at worst. I'm vaguely surprised at how little in the way of criticism I've seen online of the complexity of Let's Encrypt given the usual reaction of information security experts to any new software system. To paraphrase Neal Stephenson, the usual reaction is to crouch in a fetal position under a blanket, screaming hoarsely that the new system is way too complex and is a security disaster. I have to wonder how many unpublished attacks against the ACME protocol there are out there.

Contrast this to a typical TLS certificate provider: I download a text file once a year and put it on the server. I then restart the https service. Done. No extra software running, no other points of failure.

Maven JavaDoc Plugin Fixed

The Maven JavaDoc plugin 3.0.0 is finally ready for release. This means that I can migrate 60+ projects to Java 9 and finally get the new versions pushed to Central.

Big thanks to Robert Scholte who worked hard to ensure that everything worked properly, and even got my rather unusual usage of the plugin (aggregating documentation into a single module) working as well.

Obstructing JavaDoc

I've been anxiously awaiting the 3.0.0 release of the maven-javadoc-plugin for weeks, and in an ironic twist of fate, I'm now responsible for delaying the release even further.

I found two rather nasty bugs in the version that was to become 3.0.0, but submitted a fix for the first and had it merged. The second problem seems like it's going to take rather more work to fix though, and my message asking for implementation advice to the javadoc-dev list is currently sitting in a moderation queue.

Expected But Got

Inference

What To Save And Throw Away

What to save and throw away?

pr?The last hour is on us both?mr.s?tuck this little kitty into the impenetrable brainpan?

pr?Contents under pressure?Do not expose to excessive heat, vacuum, blunt trauma, immersion in liquids, disintegration, reintegration, hypersleep, humiliation, sorrow or harsh language?

pr?When the time comes, whose life will flash before yours?

pr?A billion paths are here inside me? pr?yes, yes, yes, Bernhard, 110? pr?potential, jewels, jewels, yes, jewels? %

Decided to kill off some old packages. jnull and jfunctional in particular I've used in just about every project I've ever worked on. There's really very little reason for them to exist anymore though. Java 8 added Objects.requireNotNull which standardized terse null checking. Noone cares about the @NonNull annotations (including myself). The entire contents of jfunctional were made redundant by Java 8. Both of these packages served their purposes well back in the days of Java 6 when they were first designed, but now they're just a burden.

It's good to throw away code.

Maven Java 9 Bugs

I'm still waiting on a set of issues to be resolved in order to push modules to all of my projects.

  • MJAVADOC-489 causes JavaDoc generation to fail when one module requires another.

  • MDEP-559 causes the dependency:analyze goal to fail. I use this goal as part of all my builds in order to keep dependencies clean and correct. Getting this fixed depends on MSHARED-660.

I've also removed japicmp from all of my builds. I don't want to disparage the project at all; it's good at what it does, but using it would require using custom MAVEN_OPTS on JDK 9, and that's just not good enough. I'm in the process of writing a replacement for japicmp and will announce it within the next few weeks.

TCP MSS Clamping

Instead of using a non-default MTU on my network, I've instead implemented TCP MSS clamping.

Specifically, I reset all of the interfaces on my networks back to using an MTU of 1500 (including those on the router), and added the following pf rule:

scrub on $nic_ppp max-mss 1440

That rule clamps the maximum TCP segment length on the PPP interface to 1440. Why 1440? It's essentially down to the per-packet overhead of each protocol that's involved. Typically, that'll be 40 or so bytes for an IPv6 packet header, 8 bytes for PPPoE, and some loose change.

So far, nothing has broken with the new settings. No TLS handshake failures, no sudden broken pipes on SSH sessions, no issues sending mail.

IPv6 And MTU Woes

I've just recently deployed IPv6. Everything went well except for one painful issue that is still not really resolved my satisfaction. To recount the story requires covering quite a bit of ground and digging through a pile of acronyms. Hold on tight!

My ISP provides a native /48 network per customer. That means that I get a mere 1208925819614629174706176 public IP addresses spread over 65536 /64 networks to use as I please.

I want to use my existing FreeBSD router to do the routing for the individual networks. I want to do this for several reasons:

  1. The ISP-provided box is a standard consumer router and is fairly limited in what it can do. It's not actively harmful; it's a respectable brand and fairly powerful hardware, but it's still only a consumer-grade box with a web interface.

  2. I'd rather have the intricate configuration details of my network be stored in text configuration files on commodity hardware and on an operating system that I mostly trust. The ISP-provided box runs Linux on proprietary hardware and only provides shell access via an undocumented (authenticated) backdoor (side-door?).

  3. I trust myself to write safe pf rules.

  4. Exposing the FreeBSD machine directly to the WAN eliminates one routing hop.

However, in order to allow my FreeBSD machine to do the routing of the individual networks (as opposed to letting the entirely optional ISP-provided box do it), I had to get it to handle the PPP connection. The machine doesn't have a modem, so instead I have to run the ISP-provided modem/router in bridging mode and get the FreeBSD machine to send PPP commands using the PPPoE protocol. Encouragingly, my ISP suggested that yes, I should be using FreeBSD for this. It's a testament to the quality of IDNet: They are a serious technical ISP, they don't treat their customers like idiots, and they respect the freedom of choice of their customers to use whatever hardware and software they want.

For those that don't know, limitations in PPPoE mean that the MTU of the link is limited to at most 1492. For reference, most networks on the internet are using an MTU of 1500. In IPv4, if you send a packet that's larger than your router's MTU, the packet will be fragmented into separate pieces and then reassembled at the destination. This has, historically, turned out to be a rather nasty way to deal with oversized packets and therefore, in IPv6, packets that are larger than the MTU will be rejected by routers and will result in Packet Too Large ICMPv6 messages being returned to the sender.

In effect, this means that IPv6 networks are somewhat less tolerant of misconfigured MTU values than IPv4 networks. Various companies have written extensively about fragmentation issues.

So why am I mentioning this? Well, shortly after I'd enabled IPv6 for the network and all services, I suddenly ran into a problem where I couldn't send mail. The symptom was that my mail client would connect to the SMTP server, authenticate successfully, send an initial DATA command, and then sit there doing nothing. Eventually, the server would kick the client due to lack of activity. After asking on the mailing list for my mail client, Andrej Kacian pointed me at a thread that documented someone dealing with MTU issues. After some examination with Wireshark, I realized that my workstation was sending packets that were larger than the PPPoE link's MTU of 1492. My FreeBSD machine was dilligently responding with Packet Too Large errors, but for whatever reason, my Linux workstation was essentially ignoring them. Some conversations on the #ipv6 Freenode IRC channel have suggested that Linux handles this very badly. Worse, it seems that the MTU related issues are sporadic: Sometimes it works without issue, other times not.

The "solution" seems to be this: Set the MTUs of all interfaces on all machines in my network to 1492. If I, for example, set the MTU of my workstation's network interface to 1500 and set the FreeBSD router's interfaces to 1492, I can no longer SSH reliably into remote sites, and the majority of TLS handshakes fail. No Packet Too Large errors are generated, which seems counter to my understanding of how this stuff is supposed to work. I very much dislike having to use a non-default MTU on my network: It seems like I will inevitably forget to set it on one or more machines and will run into bizarre and intermittent network issues on that machine.

Some further conversation on the #ipv6 IRC channel suggests that I should not have to do this at all. However, I've so far spent roughly ten hours trying to debug the problem and am exhausted. Using a non-standard MTU in my LAN(s) works around the issue for now, and I'll re-examine the problem after my capacity for suffering has been replenished.

2018-02-23: Update: IPv6 And Linux

Maven Plugins Are Not Ripe Yet

I wanted to start moving all my projects to Java 9, but quickly discovered that a lot of Maven plugins I depend on aren't ready for Java 9 yet.

Chemriver

Chemriver/A on Vimeo

Chemriver/A

Chemriver/B on Vimeo

Chemriver/B

Sources available at GitHub.

Reproducible Builds

Considering moving to producing 100% reproducible builds for all of my packages.

It seems fairly easy. The following changes are required for the primogenitor:

  • Stop using maven.build.timestamp. The commit ID is enough!

  • Use the reproducible-build-maven-plugin to strip manifest headers such as Built-By, Build-JDK, etc, and repack jar files such that the timestamps of entries are set to known constant values and the entries are placed into the jar in a deterministic order.

  • Strip Bnd-LastModified and Tool headers from bundle manifests using the <_removeheaders> instruction in the maven-bundle-plugin configuration.

  • Stop using version ranges. This may be too painful.

Some early experiments show that this yields byte-for-byte identical jar files on each compile. This is pretty impressive.

The one open issue: Oracle (or OpenJDK's) javac appears to produce completely deterministic output; there aren't any embedded timestamps or other nonsense. However, someone building the packages from source isn't guaranteed to be using an Oracle JDK. I could use the Enforcer plugin to check that the user is using a known-deterministic JDK, but it would be pretty obnoxious to break builds if they aren't. Perhaps a warning message ("JDK is not known to produce deterministic output: Build may not be reproducible!") is enough.

Simulating Packet Loss And Damage

I'm currently working on some code that implements a simple reliable delivery protocol on top of UDP. UDP is used because latency must be minimized as much as possible.

In order to test that the protocol works properly in bad network conditions, I need a way to simulate bad network conditions. For example, I'd like to see how the protocol implementation copes when 50% of packets are lost, or when packets arrive out of order.

The Linux kernel contains various subsystems related to networking, and I found that a combination of network namespaces and network emulation was sufficient to achieve this.

The netem page states that you can use the tc command to set queuing disciplines on a network interface. For example, if your computer's primary network interface is called eth0, the following command would add a queueing discipline that would cause the eth0 interface to start dropping 50% of all traffic sent:

# tc qdisc add dev eth0 root netem loss 50%

This is fine, but it does create a bit of a problem; I want to use my network interface for other things during development, and imposing an unconditional 50% packet loss for my main development machine would be painful. Additionally, if I'm running a client and server on the same machine, the kernel will route network traffic over the loopback interface rather than sending packets to the network interface. Forcing the loopback interface to have severe packet loss and/or corruption would without a doubt break a lot of software I'd be using during development. A lot of software communicates with itself by sending messages over the loopback interface, and disrupting those messages would almost certainly lead to breakages.

Instead, it'd be nice if I could create some sort of virtual network interface, assign IP addresses to it, set various netem options on that interface, and then have my client and server programs use that interface. This would leave my primary network interface (and loopback interface) free of interference.

This turns out to be surprisingly easy to achieve using the Linux kernel's network namespaces feature.

First, it's necessary to create a new namespace. You can think of a namespace as being a named container for network interfaces. Any network interface placed into a namespace n can only see interfaces that are also in n. Interfaces outside of n cannot see the interfaces inside n. Additionally, each namespace is given its own private loopback interface. For the sake of example, I'll call the new namespace virtual_net0. The namespace can be created with the following command:

# ip netns add virtual_net0

The list of current network namespaces can be listed:

# ip netns show
virtual_net0

Then, in order to configure interfaces inside the created namespace, it's necessary to use the ip netns exec command. The exec command takes a namespace n and a command c (with optional arguments) as arguments, and executes c inside the namespace n. To see how this works, let's examine the output of the ip link show command when executed outside of a namespace:

# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether f0:de:f1:7d:2a:02 brd ff:ff:ff:ff:ff:ff

You can see that it shows the lo loopback interface, and my desktop machine's primary network interface enp3s0. If the same command is executed inside the virtual_net0 namespace:

# ip netns exec virtual_net0 ip link show
1: lo: <LOOPBACK> mtu 65536 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

The only interface inside the virtual_net0 is lo, and that lo is not the same lo from the previous list - remember that namespaces get their own private lo interface. One obvious indicator that this is not the same lo interface is that the lo outside of the main system is in the UP state (in other words, active and ready to send/receive traffic). This namespace-private lo is DOWN. In order to do useful work, it has to be brought up:

# ip netns exec virtual_net0 ip link set dev lo up
# ip netns exec virtual_net0 ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

We can then create virtual "dummy" interfaces inside the namespace. These look and behave (mostly) like real network interfaces. The following commands create a dummy interface virtual0 inside the virtual_net0 namespace, and assign it an IPv6 address fd38:73b9:8748:8f82::1/64:

# ip netns exec virtual_net0 ip link add name virtual0 type dummy
# ip netns exec virtual_net0 ip addr add fd38:73b9:8748:8f82::1/64 dev virtual0
# ip netns exec virtual_net0 ip link set dev virtual0 up

# ip netns exec virtual_net0 ip addr show virtual0
2: virtual0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether aa:5f:05:93:5c:1b brd ff:ff:ff:ff:ff:ff
    inet6 fd38:73b9:8748:8f82::1/64 scope global
       valid_lft forever preferred_lft forever

In my case, I also created a second virtual1 interface and assigned it a different IPv6 address. It's then possible to, for example, run a client and server program inside that network namespace:

# ip netns exec virtual_net0 ./server
server: bound to [fd38:73b9:8748:8f82::1]:9999

# ip netns exec virtual_net0 ./client
client: connected to [fd38:73b9:8748:8f82::1]:9999

The server and client programs will do all of their networking in the virtual_net0 namespace and, because the Linux kernel knows that the addresses of the network interfaces are both on the same machine, the actual traffic sent between them will travel over the virtual_net0 namespace's lo interface.

A program like Wireshark can be executed in the virtual_net0 namespace and used to observe the traffic between the client and server by capturing packets on the lo interface.

Now, we want to simulate packet loss, corruption, and reordering. Well, unsurprisingly, the tc command from netem can be executed in the virtual_net0 namespace, meaning that its effects are confined to interfaces within that namespace. For example, to lose half of the packets that are sent between the client and server:

# ip netns exec virtual_net0 tc qdisc add dev lo root netem loss 50%

Finally, all of the above can be cleaned up by simply deleting the namespace:

# ip netns del virtual_net0

This destroys all of the interfaces within the namespace.