crush depth

To Fastmail

I'm shutting down my mail server and moving to fastmail. If everything works correctly, nothing should appear to be any different to the outside world; all old email addresses will continue to work.

Batch Files Are Your Only Option

Edit: This is wrong.

Wasted a day getting aggravated at the options for producing per-platform distributions for Java applications.

I have lots of applications. None of them are platform-specific in any way; the code is very much "write once, run anywhere". Additionally, they are fully modularized (and, indeed, probably only work correctly if everything is on the module path). They do, however, have some dependencies that are only automatic modules (but that nevertheless work correctly when placed on the module path).

Historically, Java applications were distributed either as a single executable jar file containing the entire bytecode of the application, or as a set of jar files in a directory such that the program is executed by running something equivalent to:

$ java -cp lib/* com.io7m.example.main

This has an obvious issue: You don't know which version of Java you're going to get when you run java. If you're not running the above on a command-line, but via some kind of frontend script (or double-clicking a desktop shortcut), and the Java version you have isn't compatible, you're likely just going to silently fail and do nothing. Users will burn you at the stake. Aside from this glaring problem, things are otherwise perfect:

  • Your build system, theoretically, is completely platform-independent. If you've been careful about reproducible builds, your build will produce the exact same binary artifacts on any system no matter where or when it is run.
  • The artifacts produced by your build are completely platform-independent. You can distribute a single archive file that works on all platforms.

Unfortunately, since the death of runtime delivery platforms such as Java Web Start, the only remaining way to deal with the "we don't know what runtime we might have" problem is to make your application distributions platform-specific and distribute a Java runtime along with your application.

Thankfully, there are APIs such as Foojay that allow for automatically downloading Java runtimes for platforms. These APIs can be used with, for example, the JDKs Maven Plugin to efficiently fetch Java runtimes and package them up with your application.

You can, therefore, have a Linux-specific distribution that contains your application's jar files in a lib subdirectory, and some kind of shell script that runs your included Java runtime with all of the jars in lib placed on the module or class path as necessary. You can obviously have a similar Windows-specific distribution that has the same arrangement but with a .bat file that serves the same purpose.

This maintains the advantage of the "historical" distribution method in that your build system remains completely platform independent. It does, however, gain the disadvantage that your build system no longer produces platform-independent artifacts. Despite Java being a very consciously platform-independent language, we're back to having to have platform-specific application distributions. I've decided I can live with this.

Worse, though, people want things like .exe files that can be double clicked on Windows. Ideally, they want those .exe files to have nice icons and that show meaningful values when looking at the properties:

Properties

A .bat file won't give you that on Windows. Additionally, Windows has things like the Process Explorer that will show all of your applications as being java.exe with a generic Java icon. Great.

On Linux, the whole issue is somewhat less of a problem because at least one of the following will probably be true:

  • Your Linux desktop doesn't have icons. Noone cares.
  • Someone is going to repackage your application in a distro-specific manner and users are going to get the correct Java runtime anyway, so they probably won't have any use for your platform-specific application distribution.
  • You can use something like flatpak and set icons and metadata out-of-band.

So let's assume that I'm willing to do the following:

  • My main application project will continue to produce a completely platform-independent distribution; I'll put the jar files that make up the application into a zip file. There is a hard requirement on not using any platform-specific tools, for the build to produce byte-for-byte identical outputs on any platform, and the build must be possible to complete on a single machine. The code can be executed on multiple different platforms during the build for automated testing purposes, but one-and-exactly-one machine is responsible for producing build artifacts that will be deployed to a repository somewhere. Users can, if they want, use this distribution directly on their own systems by running it with their installed java commands. It's their responsibility to use the right version, and deal with the consequences of getting it wrong.

  • I'll maintain separate projects that take those platform-independent artifacts and repackage them as necessary using platform-specific tools. It must be possible for the platform-specific build for platform P to conclude in a single build, on a single machine running platform P. These platform-specific distributions will treat users as being functionally illiterate and must mindlessly work correctly via a single double-clickable entry point of some kind.

Why do I insist on having each build run to completion and produce something useful on a single machine? Because coordinating distributed systems is hard, and trying to guarantee any kind of atomicity with regards to releasing and deploying code over them is a fool's errand. At least if the platform-specific builds happen in single shots on independent systems, we can make the independent releases and deployment of code on individual platforms somewhat atomic. I may not release Linux versions on the same day that I release Windows versions. Fine.

Additionally, I want the platform-specific distributions to feel like they actually belong on the platform they're running on. I want my application to look like MyApplication.exe in the Windows Process Explorer; I don't want to see java.exe and a Duke icon.

So what are the options?

Well, the OpenJDK people have been going on about jlink for ages now. Unfortunately, jlink falls over at the very first hurdle: It can't work with automatic modules. This means that all of my dependencies have to be modularized, and I know that at least some of them never will be. There are tools like moditect that claim to be able to somewhat automatically modularize, but the issue is that this takes the resulting bytecode artifacts further and further from the code that was actually tested during the platform-independent build; any rewriting of code is a potential for bugs that occur where the test suite won't have the opportunity to find them.

This is unacceptable. Ultimately, using jlink means that either your application runs in class path mode (which mine will not), or your application and all of its transitive dependencies have to be fully modularized and all of the modules have to be statically compiled into the resulting runtime image as jmod files. I've tried, but this isn't workable.

Moving on, the OpenJDK project now has the jpackage tool that claims to be capable of producing platform-specific application packages.

Unfortunately, it fails in a number of ways. It suffers from the exact same failure modes as jlink with regards to requiring complete modularization. Additionally, on Windows, it requires the installation of elderly versions of wix that are awkward to get working correctly in CI systems due to requiring PATH mangling and other magic. These issues ultimately made jpackage a dead end. Annoyingly, it looked like jpackage would have gotten me there, as it does produce executables that have the correct names in Process Explorer, and does apparently allow for setting icon resources and the like.

Other systems exist, such as winrun4j and launch4j. Unfortunately, these are more or less unmaintained. Additionally, they don't know anything about Java modules as they pre-date the JPMS by many years. They ultimately demand your application run in class path mode. So, those are out too.

I toyed around with creating a kind of launcher.exe that simply executed a non-jlinked Java runtime included with the platform distribution with the right command line arguments to put everything onto the module path. This became a comedy of errors. I first attempted to write the launcher program in Java, and AOT compile it with GraalVM. This required the installation of Visual Studio, and after being unable to get it to actually find the Visual C++ compiler, I gave up. It became clear, anyway, that this wasn't going to be any use as the resulting executables don't allow for custom icons without using hacky closed-source third-part executable editor tools. There's a ticket about this that was closed without comment. Nice.

I then decided to try writing a launcher in C++ by simply downloading a portable development kit during the build and compiling a C++ program (with the correct resource script to include a nice icon and executable metadata). This didn't exactly fail, but ultimately didn't achieve what I wanted: The launcher can execute a Java runtime, but then we're back to the original problem of the program appearing as java.exe with a generic icon in process listings (because that's exactly what it is; it's just the java executable from the included runtime).

Ultimately, I gave up.

What can be done about this?

I'd really like some options for jpackage that work like this:

$ jpackage \
  --type app-image \
  --name MyApplication \
  --icon icon.png \
  --use-jdk-directly some-platform-specific-jdk \
  --use-jars-on-module-path lib

This would:

  • Copy the Java modules given in some-platform-specific-jdk without trying to minimize the modules included by using jlink to try to work out which are needed. Just give me all of them, I don't care.
  • Create an executable file called MyApplication with a nice icon given in icon.png
  • Take all of the jar files in lib, and include them in a directory in the resulting app image directly without touching them. The executable should be configured internally to give the same results as if I had run java with -p lib.

I feel like the documentation almost suggests that this is already possible, but I just couldn't get it to work. The tool always tried to analyze the modules of my application and then loudly complain about automatic modules and fail, rather than just shutting up and producing an executable that placed the jar files on an included module path instead.

This would, presumably, work exactly as well on Windows as on Linux. I don't care about Mac support. It would also be great if it didn't use obsolete tools to produce executables.

JDK 21

Like many people, I've been waiting for JDK 21 for a long time. Now that Eclipse Temurin has JDK 21 builds, I'm finally now starting the push to upgrade all of my projects to JDK 21.

Primarily, I'm interested in pattern matching for my existing code. I wrote an article back in 2016 investigating the various ways that people have implemented algebraic data types on the JVM.

Since then, sealed classes have been added to the JVM. Sealed classes allow for specifying that a group of classes form a closed set, and the compiler can then reason about the set as a whole. Unfortunately, up until JDK 21, there was effectively no way to safely pattern match on a value of a sealed class. The old way to do this was via the visitor pattern, which frankly noone would have done by choice if there'd been any other alternative.

Knowing that pattern matching was due to become a first-class language feature (but didn't know exactly when), I've been writing code that looks like this for the past couple of years:

    if (message instanceof final NPACommandType<?> command) {
      return toWireCommand(command);
    }
    if (message instanceof final NPAResponseType response) {
      return toWireResponse(response);
    }
    if (message instanceof final NPAEventType event) {
      return toWireEvent(event);
    }
    throw new IllegalStateException();

Why write this kind of obviously-fragile code? My reasoning was that there would be a more direct migration path from this kind of if-then-else over sealed classes than there would be had I made every single class I wanted to match on implement a visitor. I'm hoping that Intellij IDEA will offer to convert the above code into an exhaustive switch statement when I set the language level to JDK 21. Time will tell!

Journey To The ... Earth

I bought a Yamaha Pacifica PAC120H a while back. I wanted a guitar that I could use as a testbed for hardware experiments. Ideally, an inexpensive guitar, so that I wouldn't care too much about ruining it. I've played a lot of guitars over the years, so I was pretty shocked when this almost-bottom-of-the-range instrument turned out to be one the best sounding and best feeling guitars I'd ever played.

It did, however, have a slight grounding issue (the guitar would quietly hum until you touched any metal component). It was also fairly noisy when the coil split was engaged. This isn't exactly unexpected; when the coil split is engaged, one of the coils is disabled and therefore the hum cancelling property of the humbucker is also disabled.

I'd heard of people applying shielding to the interiors of guitars in order to reduce noise, so I decided to try it. Shielding essentially involves covering every surface of the guitar's internal cavities with conductive material. There are lots of options here, but the cheapest and simplest appears to be common garden copper tape (with conductive adhesive).

I bought a couple of rolls of this:

Kraftex Copper Tape

I tested it with a multimeter and it had roughly the expected resistance for copper of that thickness. I also checked that the adhesive was actually conductive, and it did appear to be.

I opened the guitar, and was presented with bad wiring. The Pacifica series has a slightly frustrating design where it's simultaneously front-routed and rear-routed. Most guitars pick one or the other, so that when you're doing work on the internals, you don't have to work on both sides of the guitar simultaneously. The wiring was chaotic, and wired in an order that meant desoldering many components just to move one wire:

Internals 1 Internals 2

The wiring also followed the horrific guitar convention of soldering every ground cable to the volume pot, resulting in a giant compost heap of solder:

Heap

I ended up ripping out all the components in the process of applying shielding to the cavity:

Shielded 1 Shielded 2

Unfortunately, the ground pin of the volume pot had been mutilated in the process of soldering the pin to the casing along with everything else. Clearly someone just grabbed the pin with a pair of needle-nose pliers and bent it around as hard as they could until it made contact with the casing. Rather than try to fix that mess, I replaced the volume pot with something with the same specifications.

I took off the scratchplate and shielded it and all the front cavities too, leaving plenty of excess so that the back of the scratchplate would be in contact with the internal shielding in as many places as possible.

Shielded 3

With a fresh volume pot in place, I didn't want to repeat the convention of soldering everything to it. I fabricated a little block with some screw-in terminals on it that could be mounted inside the back cavity that essentially provided a solderless connection to ground for all of the things that needed to be connected to ground.

Ground 0

Unfortunately, I constructed the board such that the screw terminals ended up mounted too high in the cavity. This meant that, with (or without) cables plugged into it, there would be no way to get the backplate on without squashing the cables somewhat.

Continuing the theme of overengineering, I decided the best and most compact way to get a good quality ground connection would be to have a small PCB made with eight side-mounted screw terminals on it. The small cavity makes it extremely difficult to get any kind of screwdriver into the cavity in order to loosen or tighten screws, so I decided that the most pleasant way to work would be to have something that could be attached and detached without tools that could host the connections. It would then no longer be necessary to try to jam a screwdriver into the cavity to work on anything.

I put together the most idiotic circuit ever designed in KiCad:

Grounding Schematic Grounding PCB

I uploaded the resulting Gerber files to pcbway and had some nice printed PCBs a week later. After soldering on the screw terminals the result was obviously far better than the original perfboard construction:

Ground 0

I didn't want to fix the PCB directly to the guitar. I instead designed and printed a small mounting plate that could be hooked onto a pair of wood screws inside the cavity.

Plate

Mounting the plate and PCB inside the cavity was painless, and the resulting solderless connections are solid. I subjected them to more-than-the-minimum amount of violence, and was unable to get the wires to come out of the connectors when screwed in.

Mounted 0 Mounted 1 Mounted 2

With the backplate back on, I strung the guitar with the cheapest, worst strings I could find (as I fully expected to have to take the strings off again to fix one or more accidental grounding problems). I plugged the guitar in and ... it worked!

So was it all worth it?

Well, I took a recording of the guitar before I added the shielding. Compared with a recording taken with the new shielding and wiring in place... Noise levels are lower by 1dB. At least the wiring is now actually maintainable.

beforeAfter

Join us again next week when we disassemble a washing machine to reduce the length of the rinse cycle by 3.2 seconds.

New PGP Keys

It's that time of year again.

Fingerprint                                       | Comment
---------------------------------------------------------------------------
A438 A737 C771 7871 95CF C166 F843 51F7 2C91 8476 | 2023 personal
62AB 091D 563E 51BE 9E54 B680 7E20 DC73 5505 FE84 | 2023 maven-rsa-key
567B 7EA4 703E D530 5B73 3FBC 10C3 9A85 438F 996F | 2023 jenkins-maven-rsa-key

Keys are published to the keyservers as usual.