Whilst working on smf, I ran into an issue when resampling 32-bit floating point mesh data to 16-bit floating point format. The issue turned out to be poor handling of subnormal values by my ieee754b16 package. I went looking for better implementations to borrow and found a nice paper by Jeroen van der Zijp called Fast Half Float Conversions. It uses precomputed lookup tables to perform conversions and appears to be drastically more accurate than my manual process (the mathematics of which I've almost entirely forgotten).
I decided to put together a simple C99 implementation in order to see
how the code worked but am having some strange issues with some very
specific values. My test suite basically tries to prove that packing a
double
value and then unpacking it should be an approximate identity
operation. Essentially, ∀x. unpack(pack(x)) ≈ x
. Unfortunately,
some very specific values are failing. For some reason, my
implementation yields these results:
unpack(pack(2048.0)) → 2048.0 unpack(pack(2047.0)) → -0.0 unpack(pack(2046.0)) → 2046.0
unpack(pack(16375.0)) → 16368.0 unpack(pack(16376.0)) → 0.0
All of the other values in the range [-32000, 32000]
appear to be
correct. The unusual 16375.0 → 16368.0
result is expected; the conversion is
necessarily a lossy procedure and 16368.0
is simply the nearest
representable value when converting down to 16-bits. However, the
0.0
values are utterly wrong. This suggests that there's an issue in
implementation that's almost certainly caused by a mistake generating
the conversion tables. It seems that packing is correct, but unpacking
isn't. I've gone over the code several times, even going so far as
to implement it twice in two different languages and have gotten
the same results every time. I've spoken to Jeroen and he showed me
some results from his own implementation and test suite that show
that the above isn't a problem with the algorithm. So, assuming that
I haven't managed to screw up the same implementation after some
five clean-room attempts, there may be a transcription mistake in
the paper. I'm waiting to hear more from Jeroen.