crush depth

Primogenitor

The Maven POM files in io7m projects contain a fair amount of duplication with respect to each other. The (possibly not entirely rational) reason for this is that when I started moving projects to Maven about five years ago, I had an instinctive lack of trust for project inheritance after seeing what a disaster inheritance usually implies in object-oriented programming languages. Five years of experience, however, have taught me that in a non-Turing-complete description language such as Maven's POM, the problems usually caused by inheritance tend not to occur. I can't give any formal reasoning for this, it's purely anecdotal.

I've introduced inheritance in steps: The root POM of each project does most of the work, and then each of the POMs in the modules of the project specify the bare minimum extra information such as dependencies, plugin executions, etc. In practice, most modules just specify some dependencies and an OSGi manifest.

This is fine, but it does mean that the root POMs of all of the projects mostly contain the same few hundred lines of XML. If I make a change to the logic in one POM that I think would be useful in other projects, I have to make the same change in those projects too. Therefore, I'd like to complete the progression and move towards all projects inheriting from a common primogenitor POM. In addition I'd like to add in some extra information such as inserting the current Git revision into produced JAR files, and statically analyzing the bytecode of compiled files to ensure that semantic versioning rules are being followed. I had a fairly productive conversation with Curtis Rueden about some practical aspects of this (Curtis maintains a very large collection of projects that inherit from a common root POM) and I've made a first attempt at a new primogenitor POM. I'll try moving a few of the more recent leaf projects such as jregions to this new root and see how things work out.

Source Generation

Spent most of yesterday and a fairly decent amount of time today replacing the bulk of the hand-specialized jregions code with code generated from a template instead. I don't typically like templating as a rule: If I'm going to be producing text for a language (such as Java code, XML, etc) that's then going to be parsed and consumed by an external system (such as a compiler, an XHTML renderer, etc) then constructing the text step-by-step using an AST representation guarantees that the output will be well-formed. Templating systems don't provide any guarantees. In this case though, the output of the templating system is going to be immediately consumed by the Java compiler, which is then going to indicate any syntax and/or type errors before any other system has a chance to consume the code.

Of the templating systems I know about, only StringTemplate seems to have been developed with any kind of discipline: The primary concept is the strict separation of presentation and logic. A StringTemplate template does not contain any logic and simply defines formal parameters that must be supplied with values when the template is rendered. Note the must here: Failing to provide a value for a parameter causes an error to be raised at rendering time. Most template systems (notably Maven's resource filtering) tend to silently fail or insert garbage in this situation.

I used Kevin Birch's StringTemplate Maven plugin to generate sources as part of Maven's generate-sources phase. It appears to work well, but has some nasty failure modes when individual templates can't be found. Basically, if you tell the plugin to open a template file that doesn't exist, or if the name of the template doesn't match the name of the file within which it is defined, you'll get an unhelpful error like this:

[ERROR] Failed to execute goal com.webguys:string-template-maven-plugin:1.1:render (generate-D) on project com.io7m.jregions.core: Unable to execute template. -> [Help 1]

Even with Maven's -X switch (enables the display of exception stack traces and other debugging output) there was no useful information available. I ended up using the little-known mvnDebug executable (bundled with every Maven install but apparently undocumented) to step through the execution of the plugin and work out what was going wrong. For those that don't know, mvnDebug loads a Java agent into the JVM that causes Maven to wait until an external debugger (in my case, Intellij IDEA) connects before running the build. It turned out that the Maven plugin wasn't exactly at fault: StringTemplate's internal APIs indicate failure by returning null and maybe writing an error message to a provided mutable sequence. In my case, there were several mistakes:

  1. I specified the name of a template file including the suffix: Template.st. Internally, StringTemplate appends its own suffix, so the API was looking for Template.st.st, failing to find it, but not providing an error message.

  2. I specified the name of a template file that didn't exist at all. Same failure mode as the above.

  3. I specified the name of a template P. StringTemplate looked at the file P.st, found the file and parsed it, but it turned out that P.st actually contained a template called Q. Again, this resulted in StringTemplate being unable to find the template I'd named, but refusing to tell me about it.

When those mistakes were found and corrected, the rest of the error reporting was decent. Failing to provide template parameters, making syntax errors within templates, etc, all provided good error messages with line and column numbers.

The result of this is that a lot of the jregions source code (and test suite) is now generated from a common template. No more manual specialization. I also added float and BigDecimal specialized types to exercise the generation system during development. I suspect that I'm going to apply this same methodology to rewrite the jtensors codebase when Java gets value types.

Better Than 100%

Was amused by the pingdom test results for io7m.com:

io7m.com

io7m.com

blog.io7m.com

blog.io7m.com

The lesson is that a lot of engineering problems can be solved by refusing to do anything.

jregions

Have started unifying jareas and jboxes into a new project: jregions.

The original projects were written about three years apart and I'd not realized how much overlap there was between them until it was too late. This sort of code is a prime target for value types: There are four sets of specialized classes for int, long, double, and BigInteger coordinates because Java's generics don't allow for abstraction over primitive types without boxing. This is something that Brian Goetz has complained about frequently. To paraphrase, "you sometimes end up writing the same code eight times".

The jregions project is also a first attempt at moving to the OSGi conventions I mentioned previously. Thought I might as well use them for all new code and migrate the old code when JDK 9 appears.

The Question

The Question

Breaking compatibility in a patch release

Broke a pure-ftpd install this morning by recklessly failing to read the change log before upgrading. Missed this note for 1.0.44:

The Perl and Python wrappers are gone. The daemon can now use a configuration file without requiring external dependencies.

This meant that the s6 run script had to be updated:

#!/bin/sh
exec /usr/local/sbin/pure-config.pl /ftpd/pure-ftpd.conf 2>&1

Became:

#!/bin/sh
exec /usr/local/sbin/pure-ftpd /ftpd/pure-ftpd.conf 2>&1

The documentation was not updated. I had to work out how to get the server to consume the configuration file by guessing, and had to trace the executable with ktrace to make sure that it actually was reading the file.

I tend to forget that not all projects use semantic versioning and what I expected to be a simple bug-fix update from 1.0.43 to 1.0.45 turned out to be a service-disrupting change.

If you maintain software and you're reading this, please make your version numbers mean something!

pure-ftpd

Java Module Renaming

Right now, all io7m modules are consistently named. For version 1.0.0 of a given project p, the project usually has artifacts with coordinates such as the following:

com.io7m.p:io7m-p-core:1.0.0
com.io7m.p:io7m-p-documentation:1.0.0
com.io7m.p:io7m-p-tests:1.0.0

I follow the Maven conventions with the addition of an io7m- prefix on artifact names. This helps ensure uniqueness with respect to other Java projects when considering artifact names in isolation; people who aren't me are relatively unlikely to prefix their project names with io7m-.

However, the conventions used for OSGi projects typically look something like:

com.io7m.p:com.io7m.p.core:1.0.0
com.io7m.p:com.io7m.p.documentation:1.0.0
com.io7m.p:com.io7m.p.tests:1.0.0

Examples on Maven Central

I suspect that this naming convention is rooted in the way that OSGi implementations typically deploy bundles: Bundles are placed into a single directory which is polled frequently by the container, with new bundles being automatically deployed. With the old Maven convention, the artifact names can come into conflict when placed in a single directory:

com.acme.math:math:1.0.0    -> math-1.0.0.jar
org.example.math:math:1.0.0 -> math-1.0.0.jar

With the OSGi naming conventions, this would not occur:

com.acme.math:com.acme.math:1.0.0       -> com.acme.math-1.0.0.jar
org.example.math:org.example.math:1.0.0 -> org.example.math-1.0.0.jar

As I move all of my projects over to OSGi, I suspect that I'm going to make sweeping major-version-incrementing changes to all projects by changing their names to use the OSGi naming conventions. Personally, I find it more aesthetically pleasing anyway.

There is the possibility that changing the entire name of a project could be considered a non-compatibility-breaking change according to semantic versioning: If I change the name of the project, I can't be breaking anyone's code because there could be no code in existence that has been compiled against the new name. In order to signal a clean break, however, I'm going to treat it as one and increment the major version numbers everywhere. One minor issue is that JDK 9 is due out in a few months and when that happens, I'm going to move all projects to using Java 9 as a minimum requirement. That is, they're going to require JDK 9 to build and all produced artifacts will be Java 9 bytecode. This is without a doubt a compatibility-breaking change. It may be better to wait until JDK 9 is out and then do the project renames, module descriptors, and the bytecode version increment all in one go.

No One Loves Assembly

Deeply unimpressed by the complete lack of response to a report of a fairly serious bug in the Maven Assembly plugin. Polite requests for assistance on the developer mailing list went ignored. Hopefully this is just down to everyone being busy with the 3.5.0 release and not indicative of the Assembly plugin basically being abandonware.

Right now, if you want to create a distribution archive when using version ranges to refer to artifacts within the current reactor, you're basically out of luck.

2017-02-28: The title of this post has been modified in order to protect the pedantic.

2017-06-17: This turned out to be my fault. See: Assembly Redux

LWJGL OSGi

Huge respect for the LWJGL project with one of the best responses to a bug report I've ever had the pleasure to be involved with.

As a result, I'm now maintaining OSGi bundles for LWJGL. I also have working OSGi bundles for JogAmp but the JogAmp projects appear to be on life support at best.

Independent Module Versioning

Been experimenting with independent module versioning by developing an OSGi IRC bot. The idea is to expose any deficiencies in tools when modules within a single Maven project have different version numbers.

Initial indications are good!

One serious issue is that with independent versions, sooner or later there's going to be a release of the project where one or more modules haven't been updated and will therefore have the same version numbers as existing already-deployed modules. It's therefore going to be necessary to work out how to prevent Maven deploying bundles that already exist. Apparently, Charles Honton has a plugin for this.

Conceptually, moving to independently versioned modules means that a given project version now describes a set of module versions as opposed to simply defining a single version for all modules. I might rewrite changelog to better fit with this fundamental conceptual change.

Z

Z