RFC 2617 vs. 2831 md5-sess

Good news first, RFC 4590bis (approved, still waiting for its number) will fix the Digest-MD5 examples in RFC 4590. I've updated the MD5 test suite using the fixed examples.

While I was at it I've also updated the RFC 3797 code to work for the NOMCOM 2007 case. The entropy limit 30 was too restrictive, 38 is good enough for MD5, 10^38 < 2^128.

Now the bad news, the issue with two md5-sess examples in draft smith-sipping-auth-examples might be in fact precisely what RFC 2617 says, as reported in a semi-official erratum. If that's correct the md5-sess in RFC 2831 would be different. Hopefully draft melnikov-digest-to-historic will shed some light on this before it moves RFC 2831 to historic. For more about this see the IETF SASL WG mailing list.

For now the MD5 test suite still uses only the binary x2c(HA1) form instead of the hex. HA1 form in its md5-sess calculation.

HTML PUBLIC "-//IETF//DTD HTML i18n//EN"

Admittedly RFC 2070 is old, and its status is historic. But it was the first HTML specification with I18N based on UNICODE, and the last HTML specification published by the IETF.

So far its DTD had to be extracted manually from the RFC, now IANA hosts an official master copy with the public identifier urn:ietf:params:xml:pi:-:IETF:DTD+HTML+i18N:EN. The urn:ietf:params:xml:pi registry was created by RFC 3688 for DTDs developed by the IETF. Of course HTML i18n is still SGML, not XML, but its DTD is now the first registered IETF DTD.

If you have old HTML i18n documents you can use an URL of this DTD as system identifier like this:

<DOCTYPE HTML PUBLIC "-//IETF//DTD HTML i18n//EN" "http://www.iana.org/assignments/xml-registry/publicid/html2070.dtd">

ftpsynch.rex

Many OS/2 SAAREXX programs work almost as is under WindowsNT ooRexx, the RexxUtil functions are similar, the WindowsNT CMD shell is similar, and the RxSock interface is almost identical. Some OS/2 RexxUtil functions are not yet or not more supported under WindowsNT, e.g. SysGetMessage, SysProcessType, and SysQueryProcessCodePage.

Scripts I've tested under W2K after renaming *.cmd to *.rex include popstop2.cmd, dir2html.cmd, and sitemap.cmd.

Unfortunately ooRexx doesn't come with RxFTP.dll, but it offers the RxFTP.cls class. I've created a new ftpsynch.rex based on ftpsynch.cmd for ooRexx, for details see ftpsynch.htm.

rxwhois 2.0.5

Version 2.0.5 of rxwhois.cmd now also works as WindowsNT ooRexx script, just rename it to rxwhois.rex. I've adopted additional local character sets from utf-8.cmd including codepage 923 (ISO 8859-15, Latin 9) and 878 (KOI8-R), but I've only tested 858 (pc-multilingual-850+euro) and 1252 (windows-1252).

As always some whois-servers for ccTLDs had to be updated, for details see the source. After a system crash of my OS/2 box in June I was unable to check anything beyond the whois servers already known by rxwhois.cmd, whois.iana.org, and whois-servers.net. Just for fun I've added the eleven IDN TLDs for the test beginning in September 2007:

1. Arabic xn--kgbechtv
2. Persian xn--hgbk6aj7f53bba
3. Chinese, simplified xn--0zwm56d
4. Chinese, traditional xn--g6w251d
5. Russian xn--80akhbyknj4f
6. Hindi xn--11b5bs3a9aj6g
7. Greek xn--jxalpdlp
8. Korean xn--9t4b11yi5a
9. Yiddish xn--deba0ad
10. Japanese xn--zckzah
11. Tamil xn--hlcj6aya9esc7a

Known issue: rxwhois.cmd expects UTF-8 as charset of whois servers, but whois.iana.org uses Latin-1 for at least one TLD ht (Haiti) entry. The IANA folks told me that they'll intend to use ASCII data for the eleven IDN test TLDs.

OpenSearch descriptions for Google CSEs

It's quite simple to create opensearch descriptions for any existing Google CSE. Here's an example using the mozillaZine KB CSE:

This CSE is identified by cx=003258325049489668794:ru2dpahviq8. The &cx=-parameter is used in links and anything else related to this CSE. The left hand side 003258325049489668794 is related to the Google account and the up to 5000 annotations (e.g. sites and URL patterns) associated with this account. The right hand side ru2dpahviq8 is related to the actual CSE context including details of its layout, references to the associated annotations also known as background labels, etc.

It's not my CSE, I can ignore most technical details only relevant for the CSE creator. One detail is probably important, this CSE uses FORID:1 unlike my own CSEs with FORID:0. The value is visible in the monstrous URL of search results, it's a part of the &cof= parameter.

Most other layout details noted in &cof= are set by Google on the fly based on the CSE definition a.k.a. context. For my own CSEs I force LP:0 and AH:center with &cof=FORID%3A0%3BLP%3A0%3BAH%3Acenter, but that's arguably pointless, opensearch only works with Firefox 2, IE7, or better, and these browsers have no issues with the default LP:1 logo position and AH:left aligned header on result pages.

CSEs refuse to return &output=xml or &output=xml-no-dtd results, therefore the opensearch description needs only one type="text/html" template. Just in case I added...

<SyndicationRight> limited </SyndicationRight>
<Attribution> Google CSE by Jason Kersey </Attribution>

...anyway, after all the search results are Google results. In this case results filtered and rearranged as defined by the CSE creator Jason Kersey. With up to three searched sites in a CSE Google allegedly also shows its supplemental results.

Putting it all together I arrived at this mozillazine opensearch description. I've no clue how and where Firefox or IE7 might use the Tags or Description, most likely these details are irrelevant for search results on ordinary type="text/html" pages. The validator wants a Query example as specified by opensearch.org, just for fun I picked about%3Aconfig.

One last detail, the icon, fortunately kb.mozillazine.org has a type="image/vnd.microsoft.icon" 16×16 favicon needing less than 10 KB, so this should work as is (http:-URL instead of data:-URL) for Firefox. It's tricky to get the icon right with *.googlepages.com, the Google Page Creator won't let you have your own favicon.ico. Just use another name.

One way to use opensearch descriptions is to add a link in the header of (X)HTML pages. The title in the link should match the ShortName in the description, otherwise browsers won't know if the corresponding search is already installed. I've done that here in my blogger-template:

<link rel="search" href="xyzzy.a9.xml" type="application/opensearchdescription+xml" title="xyzzy" />

Another way is the window.external.AddSearchProvider function, Firefox 2 users can then simply click on the link to copy an opensearch description to their browser. I haven't tested IE7, maybe it uses the same method, i.e. "copy description". Last step, test this OpenSearch description with Firefox 2 or better.

For another example see my googlets page.

Simple REXX mailto script

Yet another mailto command line tool, rxmailto.cmd v0.1 can so far send one text mail to one receiver via a Mail Submit Agent (MSA) at port 587 supporting SMTP AUTH with CRAM-MD5 and 8bitMIME. That's the minimum I could get away with after the spam flood finally drowned my old mailbox.

Various details are far from perfect, e.g. if a run of words with non-ASCII characters in the subject is longer than 56 octets the subject encoder will emit a folded line longer than 76 characters, and that's not permitted by RFC 2047. On the other hand the script won't break UTF-8 characters in the subject for platforms with UTF-8 as local charset. You get what you pay for, less than 30 KB. ;-)

MD5 test suite 1.2

The MD5 test suite version 1.2 finally supports streaming and bit string input:

   hash = MD5( bytes )          ==> MD5 of an octet string
   ctxt = MD5( bytes, '' )      ==> init.  new MD5 context
   ctxt = MD5( bytes, ctxt )    ==> update old MD5 context
   hash = MD5( /**/ , ctxt )    ==> finalize   MD5 context
   hash = MD5( bytes, /**/, n ) ==> MD5 of n zero-fill bits
   ctxt = MD5( bytes, ''  , n ) ==> init.  MD5 bit context
   ctxt = MD5( bytes, ctxt, n ) ==> update MD5 bit context

Also added: APR1 can determine the hashed passwords used by BSD and Apache htpasswd. This is a function also offered by openssl passwd -1 and openssl passwd -apr1, for details see a manual of the openssl command line tool.

sitemap.cmd 0.3

FWIW I've added the schema magic to sitemap.cmd (0.3), adjusting the documentation of the REXX ftpsynch wannabe-content management system.

Apart from being a bit longer and overwriting siteold.xml the new version now passes XML schema validation. Caveat, don't use its buggy text/html output at the moment.

Unrelated, Google's page creator rewrites an uploaded sitemap 0.90 automagically into a sitemap 0.84 removing all <lastmod> elements. Or rather it did that last week for e.g. this sitemap, maybe it's one of the experimental features.

md5.cmd 1.1: Auth Digest + Digest-MD5

The IETF SASL WG recently decided to drop the RFC 2831bis draft from their agenda. Therefore I've removed the code handling <quoted-pair> (backslashes) from the MD5 test suite 1.0 (REXX script).

RFC 4590 contains four examples for Auth Digest. That's in essence the same as Digest-MD5 defined in RFC 2831, only based on the older RFC 2617. The examples were apparently copied as is to RFC 4590bis drafts. I've added the 2*2 (INVITE+rspauth, GET+rspauth) examples to md5.cmd (1.1).

The RFC 4590 examples still fail in my MD5 test suite, or rather my attempt to guess the used password failed. There's also an oddity in these examples not yet supported by the REXX script:

RFC 2617 states that a client sending any qop= parameter, for the RFC 4590 examples that's qop=auth, MUST also send a cnonce= (client nonce) together with a NC= (nonce counter). In the RFC 4590 examples the client doesn't do that, causing a trap in my REXX script.

There are two plausible ways to fix this, either use the RFC 2069 fallback algorithm, or simply omit the missing NC and CNONCE. In simplified REXX the second solution would be:

 return MD5( HA1 || ':' || NONCE || ':auth:' || MD5( XURL )) 

The first (2069) solution would use a colon : instead of :auth:. The "official" RFC 2617 string instead of :auth: is:

 ':' || NC || ':' || CNONCE || ':' || QOP || ':' 

Other variants of what RFC 4590 actually wants could be to use an empty CNONCE with a dummy NC in the direction of :00000001::auth:. As always Digest-MD5 is messy.

Related, an old 2069-erratum still rots in the pending errata mbox. I'm now confident that the 2069-code in md5.cmd works at least with the IETF tools server. I've not yet submitted an erratum for RFC 2983, three out of four 2983-examples are fine.