MD5 test suite version 1.8

The MD5 test suite 1.8 finally mentions RFC 6151, fixes a minor exit code bug in version 1.7, supports the web-safe base64 encoding specified in RFC 4648 as base64url alphabet, uses the USCYBERCOM easter egg as MD5 streaming test case, and covers the collision announced by Tao Xie and Dengguo Feng in December 2010 (2 of 512 bits modified). Read their PDF (one page) for details of the offered bounty for anybody who finds another collision of this type.

MD5 1.7: verified errata

All pending errata for the MD5 test suite are now verified by the IESG; thanks to Alexey Melnikov. The oldest erratum 749 about an MD5 example in RFC 2069 was submitted 2005-02-06 and verified 2010-07-11. Now I feel less bad about dropping out for two years.

For historical reasons the MD5 test suite is one of my REXX scripts still published as cmd-file instead of a rex-file. On an OS/2 box REXX is the default scripting language using file extension cmd. An ordinary OS/2 cmd.exe-shell script never starts with "/*"; any script starting with "/*" is interpreted as REXX.

Good old PC DOS 7 uses extension bat for command.com-shell scripts, and the text editor KEDIT uses kex for its macros; both also identify REXX by "/*" in line 1.

Please do not feed OS/2 REXX cmd-scripts to the Windows NT cmd.exe-shell; you would get numerous errors. For NT simply rename md5.cmd to md5.rex and let ooREXX interpret it.

MD5 1.6: POP3 and UUID

Version 1.6 of the MD5 test suite covers RFC 5034 (Digest-MD5 for POP3), RFC 4122 (UUID version 3), and an old RFC 1910 example. The POP3 example belongs to the MD5-sess series.

The UUID example is apparently wrong, it swaps the byte order unnecessarily. Examples published by python.org and ossp.org agree with that theory, I'll submit it as erratum.

MD5-sess and RFC 5090

RFC 5090 was published, it fixes some problems with the MD5 examples in RFC 4590. Version 1.5 of the MD5 test suite now contains the new RFC 5090 RADIUS examples.

The updated test suite contains a new procedure AUTHTTP for the RFC 2617 idea of MD5-sess noted in the errata. The old DIGEST procedure uses the RFC 2831 MD5-sess algorithm. Six new test cases cover MD5-sess examples published in RFC 2831 and 4643 using the RFC 2617 algorithm. Hopefully this will be documented in an RFC moving RFC 2831 to HISTORIC.

RFC errata

Various pending RFC errata have been published recently, among others for RFC 2069, 2822, and 4408. The RFC editor might soon offer a Web form for submissions as outlined in an Internet Draft.

• The 2069 erratum resulted in an editorial update of my MD5 test suite.
• The 2822 erratum was already covered in the 2822upd drafts.
• The 4408 erratum is actually a link to the OpenSPF errata page.

RFC 2617 vs. 2831 md5-sess

Good news first, RFC 4590bis (approved, still waiting for its number) will fix the Digest-MD5 examples in RFC 4590. I've updated the MD5 test suite using the fixed examples.

While I was at it I've also updated the RFC 3797 code to work for the NOMCOM 2007 case. The entropy limit 30 was too restrictive, 38 is good enough for MD5, 10^38 < 2^128.

Now the bad news, the issue with two md5-sess examples in draft smith-sipping-auth-examples might be in fact precisely what RFC 2617 says, as reported in a semi-official erratum. If that's correct the md5-sess in RFC 2831 would be different. Hopefully draft melnikov-digest-to-historic will shed some light on this before it moves RFC 2831 to historic. For more about this see the IETF SASL WG mailing list.

For now the MD5 test suite still uses only the binary x2c(HA1) form instead of the hex. HA1 form in its md5-sess calculation.

MD5 test suite 1.2

The MD5 test suite version 1.2 finally supports streaming and bit string input:

   hash = MD5( bytes )          ==> MD5 of an octet string
   ctxt = MD5( bytes, '' )      ==> init.  new MD5 context
   ctxt = MD5( bytes, ctxt )    ==> update old MD5 context
   hash = MD5( /**/ , ctxt )    ==> finalize   MD5 context
   hash = MD5( bytes, /**/, n ) ==> MD5 of n zero-fill bits
   ctxt = MD5( bytes, ''  , n ) ==> init.  MD5 bit context
   ctxt = MD5( bytes, ctxt, n ) ==> update MD5 bit context

Also added: APR1 can determine the hashed passwords used by BSD and Apache htpasswd. This is a function also offered by openssl passwd -1 and openssl passwd -apr1, for details see a manual of the openssl command line tool.

md5.cmd 1.1: Auth Digest + Digest-MD5

The IETF SASL WG recently decided to drop the RFC 2831bis draft from their agenda. Therefore I've removed the code handling <quoted-pair> (backslashes) from the MD5 test suite 1.0 (REXX script).

RFC 4590 contains four examples for Auth Digest. That's in essence the same as Digest-MD5 defined in RFC 2831, only based on the older RFC 2617. The examples were apparently copied as is to RFC 4590bis drafts. I've added the 2*2 (INVITE+rspauth, GET+rspauth) examples to md5.cmd (1.1).

The RFC 4590 examples still fail in my MD5 test suite, or rather my attempt to guess the used password failed. There's also an oddity in these examples not yet supported by the REXX script:

RFC 2617 states that a client sending any qop= parameter, for the RFC 4590 examples that's qop=auth, MUST also send a cnonce= (client nonce) together with a NC= (nonce counter). In the RFC 4590 examples the client doesn't do that, causing a trap in my REXX script.

There are two plausible ways to fix this, either use the RFC 2069 fallback algorithm, or simply omit the missing NC and CNONCE. In simplified REXX the second solution would be:

 return MD5( HA1 || ':' || NONCE || ':auth:' || MD5( XURL )) 

The first (2069) solution would use a colon : instead of :auth:. The "official" RFC 2617 string instead of :auth: is:

 ':' || NC || ':' || CNONCE || ':' || QOP || ':' 

Other variants of what RFC 4590 actually wants could be to use an empty CNONCE with a dummy NC in the direction of :00000001::auth:. As always Digest-MD5 is messy.

Related, an old 2069-erratum still rots in the pending errata mbox. I'm now confident that the 2069-code in md5.cmd works at least with the IETF tools server. I've not yet submitted an erratum for RFC 2983, three out of four 2983-examples are fine.