crush depth

Been thinking about protocol versioning recently, and this post is more or less an attempt to organize my own thoughts on the matter.

Protocol Versioning is what it sounds like... Much like software has version numbers to track changes over time, protocols (such as HTTP) often have version numbers too. Semantic Versioning is an attempt to encode information about compatibility into software version numbers: "The major number changed, therefore if I depend on this software or API, I probably need to change my code". I don't think there has ever been anything analogous to semantic versioning when it comes to protocols, although many protocols seem to use two-part version numbers in a similar manner: When the major number changes, clients and servers will need to be updated and likely won't work with the new version of the protocol until this happens.

Semantic versioning usually concerns consumers of an API: An application using a library is a consumer of that library. There are, of course, APIs in the form of SPIs (service provider interfaces) that are intended to be implemented by providers as opposed to consumers, and those have different constraints with regards to semantic versioning, and aren't important here.

I believe that the semantic versioning of protocols primarily concerns readers of that protocol. That is, writers produce messages according to version M of a specification, and it's the responsibility of the party reading those messages to consult version M of the specification to work out how to read them properly. Writers can upgrade and produce messages according to version N (where N > M) at any time, and the burden is on the readers to update their code and understand the new protocol version N. The degree to which this is actually a burden is a function of the compatibility guarantees provided by that particular protocol. The designers of the protocol might, for example, try to follow a form of semantic versioning and design their protocol such that, for a server running a given protocol version M.P, any client that can speak version M.Q for any Q can speak to the server without issue. Concretely, a server speaking protocol 2.3 can be used by clients speaking protocols 2.0, 2.1, 2.2, 2.3, 2.4, and so on. Clients that only speak protocol 1.0 or 3.0 might not be able to speak to the server without an upgrade. Alternatively, protocol designers might provide no compatibility guarantees whatsoever, and in fact may even go further and say that only clients supporting exactly version M of a protocol can speak to a server supporting version M. This is quite common in computer game engines supporting online play.

Some protocols don't use versioning at all, and I believe this is very poor engineering practice for reasons I'll go into shortly.

Fundamentally, protocol versioning is about maintaining the correctness and interoperability of multiple independent implementations whilst also being able to evolve protocols over time. A detailed formal specification for a protocol is good; servers and clients can be demonstrated to be correct with respect to an abstract, written specification. This results in numerous benefits that I won't bother to list here - consult any book on formal verification techniques in software to learn more. A detailed formal specification that can either never be changed, or can change day by day without anyone actually being clear on which revision of the specification text they're targeting, is actively harmful. If the specification can't change, developers will introduce extensions to the protocol that harm interoperability. If the specification changes arbitrarily, clients and servers turn into spaghetti and nobody has any idea what "correct" even is.

There are numerous factors involved when designing a protocol: Designers generally want to find the optimum point between simplicity of implementation, ease of specification, provision of forwards or backwards compatibility guarantees, space efficiency of transmission encoding, ease of producing an optimal decoder (possibly in hardware), and so on. Some of those factors often end up being in opposition!

My impression, having worked extensively with systems such as Protobuf, is that the existing systems do protocol versioning extremely poorly. I think it stems from a basic lack of understanding that's very common to software developers at the time of writing: Developers think syntax is important, and don't understand semantics at all.

To explain this a little, take a look at how Protobuf handles protocol versioning. The short answer: They don't handle it at all. Protobuf "versioning" works like this: Users declare message types, which are conceptually record types except that the presence of every field is optional. Yes, you read that correctly, the presence of every field is optional. Despite every field being optional, removing a definition of a field from a message type is a backwards incompatible change. What does backwards incompatible mean in this context? Noone knows! The protocol defined by the set of your declared message types can be extended by adding fields onto the end of existing message types. The binary encoding of Protobuf is designed such that clients reading messages can skip missing fields, and can ignore fields that they don't understand. Protobuf claims that this provides forwards and backwards compatibility: Clients that only understand older "versions" of a protocol can ignore any new fields that didn't exist when the clients were written, and new clients that are presented with messages from older "versions" of a protocol can substitute default values for fields that are defined in the schema now but weren't back then. If you thought that would result in the definitions of message types growing indefinitely, you'd not be wrong. As a further side effect of specifying things such that readers of messages need to be able to identify unrecognized fields in order to skip them, the underlying binary encoding has to carry integer tag values along with every field in order to identify the fields. This isn't exactly a huge issue in the era of 100 Gigabit Ethernet but, amusingly, despite carrying all this extra structural information around, the underlying binary encoding is not self-describing. In other words, you actually need to have access to the schema that produced the code that produced the encoded data in order to read it. There's no versioning, as stated, so you can't know which schema you need but, hey, any schema is fine, right? Parsing can't fail! Worse, because it's so easy to add new fields to a message, it subtly hides the fact that every existing client that speaks the protocol as it was defined prior to the field addition is now ever-so-slightly wrong. I believe that a change that will affect protocol compatibility should look non-trivial in the schema language, so as to reflect the true cost of the change.

Anyone with a basic understanding of type theory and or semantics is likely feeling a certain amount of despair at this point. This is an excellent demonstration of developers thinking syntax is important and that semantics aren't: Clients can parse older or newer messages, because parsing in Protobuf almost never fails. Syntax. Can those clients actually understand those messages? What if I add a new field that has critical importance in a new version of the protocol? Older clients will simply ignore it. At that point, the two parties communicating with each other are speaking two different languages and neither of them really know it. There's no way to communicate to older clients that the field they ignored was actually very important. Semantics.

I'm picking on Protobuf as it seems to be the most poorly engineered out of all of the similarly-purposed systems such as Cap'n Proto, Thrift, Colfer, Kryo, MessagePack, and doubtless dozens of others of more or less cookie-cutter imitations of the other systems. I don't mean to insult any of the projects individually, but even the projects that appear to have corrected other awful aspects of Protobuf seem to have ended up using the same system as Protobuf for "versioning": Add new fields, don't remove old fields, skip fields you don't understand.

Can we do better than this?

I think we can.

Firstly, I think we need a solid theoretical framework to express the abstract types of messages before we even think about encoding or versioning or any other implementation details. Thankfully, we've had product and sum types for close to fifty years now, so let's start there. The definitions here are given in Haskell 2010.

A message type is either a record (product type) or a variant (sum type), and message types can be parameterized by other message types (parametric polymorphism). Message types are uniquely named.

type TypeName = String

data Message
  = Record [TypeParameterName] TypeName [Field]
  | Variant [TypeParameterName] TypeName [Case]

A record has zero or more type parameters, and zero or more fields. All fields are required (more on this shortly), and the order of fields is significant. A field is merely a name and a type expression:

type FieldName = String
data Field = Field FieldName TypeExpression

The names of fields are unique within a record.

A variant has zero or more type parameters, and zero or more cases. A case has a name unique to the enclosing variant type, zero or more fields and is morally an anonymous record type:

type CaseName = String
data Case = Case CaseName [Field]

A type expression is either the name of a message type, a reference to a type parameter, or an application of a named type to a series of type expressions:

type TypeParameterName = String

data TypeExpression
  = TypeName TypeName
  | TypeParameter TypeParameterName
  | TypeApplication TypeName [TypeExpression]

Both records and variants are type constructors; a type constructor applied to a series of type expressions yields a type. They are merely functions at the type level. We can support the partial application of type constructors if we want, but there are practical reasons not to do this (it's ever so slightly easier to translate the types to languages that don't support the partial application of generic types if partial application isn't supported in our schema language). We also don't support higher-kinded type parameters; type parameters always have kind * and type application expressions enforce this syntactically by naming concrete type constructors directly. We assume nominal typing: Two different record types with the same fields are not type-compatible.

This gives a basic and yet complete calculus to work with in order to define message types. Earlier, it was mentioned that all fields are required. That is, to construct a value of type T, all of the fields declared within T are required to be provided. How would a programmer define a field that is in some sense optional? Well, we can define the canonical Option type as:

option :: Message
option = Variant ["A"] "Option" [
  Case "None" [],
  Case "Some" [Field "value" (TypeParameter "A")]]

Developers can then declare fields of type Option Integer or similar, the values of which can either be None or Some x for some value x. We can define a set of opaque primitive types such as strings, machine integers, and so on, as with any standard programming or data description language. For the sake of explanation here, let's assume that strings are opaque UTF-8 character arrays, and integers are 32-bit unsigned machine integers. We can add whatever primitive types we like, but we don't need more than this for the sake of explaining this hypothetical protocol here.

Given that we can now describe message types, how might we encode these types in a binary form for transmission? Well, the most obvious and direct encoding assumes that both parties involved in exchanging messages have the original schema used to describe the messages. We assume that, much like in Protobuf, there'll be some kind of compiler that will take our schema and generate code for a programming language that can read and write values of the generated types to/from a stream. We can add an additional simplifying restriction that only message types with kind * can be serialized. In other words, we can't serialize a value of type ∀A. Option A on the wire (kind * → *), but we can serialize Option Integer (kind *) without issue.

Here's a simple encoding we could use for an output stream S:

• Integers are writen directly to S as four octets in big-endian order.
• Strings are encoded by writing an integer length to S, followed by N octets of UTF-8.
• Records are encoded by writing the values of the fields directly to S in the order in which the fields were declared in the schema.
• Variants are encoded such that the cases are numbered starting from 0, the index of the current case is written as an integer to S, and the values of the fields of the current case are written to S in the order in which the fields were declared in the schema.

As a concrete example, for our Option type above, a value of Some 23 would be encoded as 0x0 0x0 0x0 0x1 0x0 0x0 0x0 0x17. A value of None would be encoded as 0x0 0x0 0x0 0x0. The string "hello" would be encoded as 0x0 0x0 0x0 0x5 0x68 0x65 0x6c 0x6c 0x6f.

This is all fine so far. It's close to maximally efficient in terms of encoding and decoding time: Encoding and decoding an integer might be a single machine instruction on some hardware. It's not maximally efficient in terms of size - we could use variable-length integers if we're really concerned about counting every last bit, and as a result we could probably pack variant indices into 2-3 bits! I'm not personally convinced that this kind of bit-packing is warranted in 2020, however.

If we want to extend the model with sequences or lists, that's no problem. We could define a list in the traditional inductive form:

list :: Message
list = Variant ["A"] "List" [
  Case "Null" [],
  Case "Cons" [
    Field "value" (TypeParameter "A"),
    Field "rest" (TypeApplication "List" [TypeParameter "A"])]]

But for the sake of simplicity, and because sequences are so inherent in data description languages, and not least because it yields a nice encoding if we follow the spirit of our above encoding rules, we can just define lists as an opaque parameterized type. We can directly encode a list as an integer length, followed by the values of the list elements in order. The list [23, 100, 10] would be encoded as 0x0 0x0 0x0 0x3 0x0 0x0 0x0 0x17 0x0 0x0 0x0 0x64 0x0 0x0 0x0 0xa.

For maps, we can work similarly:

mapEntry :: Message
mapEntry =
  Record ["K", "V"] "MapEntry" [
    (Field "key" (TypeParameter "K")),
    (Field "value" (TypeParameter "V"))]

map :: Message
map = Record ["K", "V"] "Map" [Field "entries" (TypeApplication "MapEntry" [TypeParameter "K", TypeParameter "V"])]

A map is merely a list of key/value pairs. Whatever compiler generates the code to deserialize values of type Map can special-case this and generate something that behaves like a map in the target programming language.

Now, given that we've barely referenced the structure of encoded values in the binary form aside from an index value and a couple of length values, we've done pretty well. We can encode pretty much any data structure shape that we're likely to see inside the average message protocol. Because we know the schemas ahead of time, we can purposefully avoid including any details such as tags to identify fields or types. We can serialize fields in declaration order, because we require all fields to be present. Writing serializers and deserializers is trivial; they are simple recursive functions over a fixed set of possible data shapes. Typically, the messages that will be exchanged between the parties will be wrapped in a single variant type, and that variant type will be the one and only message type that's actually serialized directly. This allows for specifying protocols that may send any kind of message in any order; you only have to read the initial variant index from the stream to know which kind of message you need to deserialize. In all cases, if you get a variant index you don't recognize, someone has implemented the protocol incorrectly. Period!

So... What's the catch? Well, a pretty glaring omission is that we've not talked about versioning at all. We can't use the traditional Protobuf trick of appending new fields to the ends of record types, because we haven't prefixed messages with length values, and we haven't prefixed fields with integer tags. Anyway, as discussed earlier, appending new fields to the ends of record types that older clients are expected to ignore is not a solution: It gains we-can-always-parse-this syntax in the binary encoding at the expense of we-can-never-truly-understand-this semantics. We want a way for old clients to continue to work and to truly understand the semantics of our protocol whilst allowing us to freely update the protocol.

The solution is straightforward, and every codebase I've ever seen that used Protobuf or something similar to it has ended up having to implement one or more parts of this approach.

Firstly, forget the idea that your application's main domain types are going to be defined in someone else's schema language. The schema language is not going to give you the ability to express invariants as rich as your target programming language allows (if the chosen schema language even allows you to express them at all - Protobuf's does not). Define a set of types directly in your programming language of choice that represent the canonical set of messages your server and client accept, and add the necessary parameter validation code and preconditions directly to the constructors of those types. These types represent the idealized current set of messages, and can be changed at any time as will be explained shortly. These types are the only types your application will work with directly. We'll call this set of message types C. Bonus points for making these types immutable, so that any invariants you set up in the constructors are preserved for the lifetime of the values.

Secondly, make the simplifying assumption that a given conversation between a client and server will target exactly one known protocol version, agreed upon ahead of time, but allow the client and server to support any number of protocol versions concurrently. For each supported version V, there is a set of message types M(V) that belong to that version. Individual members of M(V) may appear in the sets of message types for other versions. After all, not every revision to every protocol changes every message.

Thirdly, for each supported version V, write a pair of translation functions f(m) → C and g(c) → M. f(m) translates each message type m in M(V) to the corresponding idealized message type in C, and g(c) is the inverse.

In the vast majority of cases, f(m) is just going to extract a few fields from m and call a constructor of one of the types in C. This is typically a one-liner in any language. As a happy side effect, because we added validation to the constructors of the types in C in step one, we get guaranteed validation for all messages in all versions of the protocol regardless of where those messages came from: They will all be translated to the application domain by going through the constructors of types in C and therefore we cannot fail to validate anywhere. Note that g(c) may become a partial function after we've updated our protocol a few times. More on this later. As another happy side effect, the actual application itself only ever has to deal with one protocol version (the types in C) because messages from other versions of the protocols are transformed into the version we want to use in this step. In most programming languages, we also get another happy side effect: Because we use sum types, we can statically guarantee that we handle all known message types. The compiler will simply not allow us to forget to handle one. Not least of all: We can trust that we are maintaining the semantics of the communication, because f(m) is responsible for transforming messages that mean one thing to messages that might mean another, and we always know exactly which message version we're talking about at all times. This also gives us the freedom to change the types in C at any time; the compiler will tell us about all of the instances of f(m) that are now nonsensical. We can update them and be confident that we've not broken things for any existing client. Additionally, having to update instances of f(m) and g(c) clearly shows us the exact cost of any particular change to C; you make a protocol change, and the compiler tells you exactly what that means for your clients. No more hiding behind "Oh, I can just add a field, it's free!".

Fourthly, define a trivial container protocol that provides version negotiation, and embed our application-specific protocol into it. This container protocol can be defined once and reused across an unlimited number of projects, because it specifically only needs to do one thing: Get two parties to agree which message protocol version they're using. In a similar manner to how Protobuf uses a standard set of encoding rules across all projects to to haphazardly mash together "versioning", we can use a simple, principled container protocol across all projects to unambiguously negotiate an accurate application protocol version that suits both parties.

Before we go into the details of the above, I should note that I have personally used this approach in production multiple times and have seamlessly communicated with clients using wildly incompatible versions of the same protocol. Jane Street have publicly stated that they use systems such as this, and they really care about correctness (any mistake in their line of work can cause financial devastation in minutes). Unfortunately, I'm unable to locate their original blog post that described the system, but they do appear to publish a very small version negotiation library that they presumably use as part of this.

Let's define our application's domain types in Java. We'll build a somewhat overengineered echo:

1. A client c connects to the server s, and s says nothing.
2. c then sends Hello, including a non-empty name n.
3. s sends back Hello with the same name n.
4. c then sends a series of Speak messages, and s sends them back.
5. c sends Goodbye and disconnects.

public sealed interface MessageType
{
  record Hello(String name) implements MessageType
  {
    public Hello
    {
      if (name.isBlank()) {
        throw new IllegalArgumentException("Names must be non-empty");
      }
    }
  }

  record Speak(String message) implements MessageType
  {

  }

  record Goodbye() implements MessageType
  {

  }
}

Now, let's define version 1 of our application protocol. Let's assume that we have a data description language compiler that accepts our sum and variant type declarations described earlier, and generates serialization code for us based on the encoding rules we described earlier.

[record Hello1
  [field name String]]

[record Speak1
  [field name String]]

[record Goodbye1]

[variant Messages1
  [case Hello   [field message Hello1]]
  [case Speak   [field message Speak1]]
  [case Goodbye [field message Goodbye1]]]

This would result in generated Java classes such as:

public sealed interface Messages1
{
  record Hello(String name) implements Messages1
  {

  }

  record Speak(String message) implements Messages1
  {

  }

  record Goodbye() implements Messages1
  {

  }
}

... along with code to serialize messages of those types to streams.

Now, the functions f(m) and g(c) to convert our version 1 protocol messages to instances of our application's domain types and back again:

  public static MessageType convertFrom1(
    final Messages1 message)
  {
    Objects.requireNonNull(message, "message");

    if (message instanceof Messages1.Hello h) {
      return new MessageType.Hello(h.name());
    }
    if (message instanceof Messages1.Speak s) {
      return new MessageType.Speak(s.message());
    }
    if (message instanceof Messages1.Goodbye g) {
      return new MessageType.Goodbye();
    }
    throw new UnreachableCodeException();
  }

  public static Messages1 convertTo1(
    final MessageType message)
  {
    Objects.requireNonNull(message, "message");

    if (message instanceof MessageType.Hello h) {
      return new Messages1.Hello(h.name());
    }
    if (message instanceof MessageType.Speak s) {
      return new Messages1.Speak(s.message());
    }
    if (message instanceof MessageType.Goodbye g) {
      return new Messages1.Goodbye();
    }
    throw new UnreachableCodeException();
  }

Sadly, although Java has sealed classes, it doesn't yet have pattern matching in switch expressions, so we have to rely on IDE warnings for inexhaustive instanceof expressions until pattern matching is implemented.

Now, let's introduce version 2 of our protocol. For clients using protocol version 2, the conversation is slightly different.

1. A client c connects to the server s, and s says ServerHello with name p.
2. c then sends ClientHello, including a non-empty name n.
3. s sends back ClientHello with the same name n.
4. c then sends a series of Speak messages, and s sends them back.
5. c sends Goodbye and disconnects.

We now change our original domain types to:

public sealed interface MessageType
{
  record ServerHello(String name) implements MessageType
  {
    public ServerHello
    {
      if (name.isBlank()) {
        throw new IllegalArgumentException("Names must be non-empty");
      }
    }
  }

  record ClientHello(String name) implements MessageType
  {
    public ClientHello
    {
      if (name.isBlank()) {
        throw new IllegalArgumentException("Names must be non-empty");
      }
    }
  }

  record Speak(String message) implements MessageType
  {

  }

  record Goodbye() implements MessageType
  {

  }
}

The compiler now tells us "Oh, your domain types have changed and your message conversion function now needs to be updated":

  public static MessageType convertFrom1(
    final Messages1 message)
  {
    Objects.requireNonNull(message, "message");

    if (message instanceof Messages1.Hello h) {
      return new MessageType.Hello(h.name());
                             ^^^^^^^^^^^^^^^
    }

So we now attempt to update the version 1 f(m) and g(c) functions, and run into an issue...

  public static MessageType convertFrom1(
    final Messages1 message)
  {
    Objects.requireNonNull(message, "message");

    if (message instanceof Messages1.Hello h) {
      return new MessageType.ClientHello(h.name());
    }
    if (message instanceof Messages1.Speak s) {
      return new MessageType.Speak(s.message());
    }
    if (message instanceof Messages1.Goodbye g) {
      return new MessageType.Goodbye();
    }
    throw new UnreachableCodeException();
  }

  public static Messages1 convertTo1(
    final MessageType message)
  {
    Objects.requireNonNull(message, "message");

    if (message instanceof MessageType.ServerHello h) {
      // ???

We've added a ServerHello type to the core application protocol, but there is no corresponding message type in the version 1 protocol! We can easily map incoming Hello messages to the core application protocol: We know that the ServerHello message did not exist in protocol version 1, so if we see a Hello message, it must mean that it maps to the ClientHello type. We just can't do the same for outgoing messages. This is where the g(c) function becomes partial as mentioned earlier. For now, we can just live with the partiality, and make the conversion functions return an optional value:

  public static MessageType convertFrom1(
    final Messages1 message)
  {
    Objects.requireNonNull(message, "message");

    if (message instanceof Messages1.Hello h) {
      return new MessageType.ClientHello(h.name());
    }
    if (message instanceof Messages1.Speak s) {
      return new MessageType.Speak(s.message());
    }
    if (message instanceof Messages1.Goodbye) {
      return new MessageType.Goodbye();
    }
    throw new UnreachableCodeException();
  }

  public static Optional<Messages1> convertTo1(
    final MessageType message)
  {
    Objects.requireNonNull(message, "message");

    if (message instanceof MessageType.ClientHello h) {
      return Optional.of(new Messages1.Hello(h.name()));
    }
    if (message instanceof MessageType.ServerHello h) {
      return Optional.empty();
    }
    if (message instanceof MessageType.Speak s) {
      return Optional.of(new Messages1.Speak(s.message()));
    }
    if (message instanceof MessageType.Goodbye) {
      return Optional.of(new Messages1.Goodbye());
    }
    throw new UnreachableCodeException();
  }

  public static MessageType convertFrom2(
    final Messages2 message)
  {
    Objects.requireNonNull(message, "message");

    if (message instanceof Messages2.ClientHello h) {
      return new MessageType.ClientHello(h.name());
    }
    if (message instanceof Messages2.ServerHello h) {
      return new MessageType.ServerHello(h.name());
    }
    if (message instanceof Messages2.Speak s) {
      return new MessageType.Speak(s.message());
    }
    if (message instanceof Messages2.Goodbye) {
      return new MessageType.Goodbye();
    }
    throw new UnreachableCodeException();
  }

  public static Optional<Messages2> convertTo2(
    final MessageType message)
  {
    Objects.requireNonNull(message, "message");

    if (message instanceof MessageType.ClientHello h) {
      return Optional.of(new Messages2.ClientHello(h.name()));
    }
    if (message instanceof MessageType.ServerHello h) {
      return Optional.of(new Messages2.ServerHello(h.name()));
    }
    if (message instanceof MessageType.Speak s) {
      return Optional.of(new Messages2.Speak(s.message()));
    }
    if (message instanceof MessageType.Goodbye) {
      return Optional.of(new Messages2.Goodbye());
    }
    throw new UnreachableCodeException();
  }

Now that our g(c) function returns an optional value, whatever code we're using to call the code that serializes values can simply do nothing if it encounters an Optional.empty() value. This allows us to keep the core application code ignorant of the changes in semantics. In real-life code, with more complex protocols that may be changing in more complicated ways, we'd likely want to do something a bit more advanced to handle this. Typically, each protocol version would have a versioned protocol handler class instantiated that makes the necessary calls to f(m) and g(c) and that does any version-specific massaging of messages. It's the gang of four adapter adapted (no pun intended) to protocol versioning. Load protocol handlers as services via OSGi or ServiceLoader and you effectively get a full plugin system for protocol versions. This kind of flexibility is enabled by the fact that we fully acknowledge versions exist, and take a disciplined and type-assisted approach to versioning. This obviously doesn't mean that you can make arbitrary changes to protocols at any time and expect not to have to pay for it in terms of software complexity somewhere, but it does clearly and unambiguously show you exactly where those costs are and where future costs will be. This is in stark contrast to software based on systems like Protobuf, where you always end up with massive, tangled, monolithic programs that essentially have to turn any arbitrary nonsense inside messages into something that might be workable.

So given all of the above, we now have a basic framework that can handle multiple protocol versions and that can be extended in a straightforward way more or less indefinitely. Protocol version-specific code is rigorously isolated and doesn't contribute much to the complexity of the overall application. The final missing piece is that we need a way for the clients and servers to agree on a protocol version. This is the easy part; we've been doing version negotiations in protocols such as TLS back when they were called things like SSL in 1994. Version negotiation protocols are not difficult.

When a client connects to the server, the server will immediately send the following message:

[record ContainerServerHello
  [field magic Integer]
  [field appProtocol Integer]
  [field supportedLow Integer]
  [field supportedHigh Integer]]

The magic field identifies our container protocol, because we want to define this protocol once and use it in all our protocols. This is a standard magic number scheme that allows connecting clients a chance to realize early that they can't speak to this particular server. The appProtocol field is a unique identifier for our application protocol. In real life, we'd probably want to use a much larger fixed-length identifier here such as a UUID or maybe even a variable-length humanly-readable string if we don't mind client implementations having to do a bit of extra work. We can assume that our application protocol is versioned using a single integer version number. We don't need major and minor versions, after all, because our careful approach to versioning means that we're not subjecting code that expects version M messages to be subjected to version N messages (for any M and N). We can therefore assume that version numbers increase monotonically, and can also therefore assume that we can express the set of supported versions as, for example, a half-open range rather than needing to provide a list of supported version numbers. Given that the above structure is fixed-length, this makes writing a client very easy: Connect to the server, read 16 octets, and then act accordingly.

The client sends back the following message in response:

[record ContainerClientHello
  [field version Integer]]

The version field indicates to the server which application protocol version it is going to use. At this point, depending on how far we want to go with the implementation, the server can send some kind of Ok or Error message depending on what the client said, and then all subsequent messages are conducted using the application protocol we defined above. It's the kind of thing that any developer could implement in their first week of network programming!

I'm currently working on implementing all of the above in a reusable package called Cedarbridge. At the time of writing, the compiler is complete, and the documentation and language specification is currently under construction. The versioning protocol has not yet been completed. Shouldn't take long.

To think we got all this just because we bothered to think about semantics...