FAT16 fun

Continuing the FAT fun saga with an unplanned sequel:  rexxfat used partition type 05 instead of 06 for BigFAT, i.e., FAT16 with 2**16 or more sectors.  I've fixed this stupid bug in the last rexxfat version.  The virtual hard disk image works fine in a Windows 2000 virtual machine:

The odd size 3759 MB matches what I have on a memory card.  It's rather tricky to copy the disk image to the memory card, either rawcopy doesn't support copies to a physical device, or it was not obvious for me how to manage this.  Using diskpart to unmount the one and only volume on this partitioned memory card wasn't good enough for write access, eventually the diskpart  delete volume magic did the trick.  Presumably that also explains my difficulties with other tools including dskprobe and HxD.

Looking for explanations I stumbled over SysInternals SDelete.  That article doesn't explain how Windows 7  manages volumes on memory cards, but answered another open question: Compressed, encrypted and sparse are managed by NTFS in 16-cluster blocks.  That matches my observation for rxsparse.rex, 16×4=64 KB, and if the SysInternals folks got that right Wikipedia also got it right.  Hint, it is very likely that SysInternals got it right, after all they were the first to offer a working DOS NTFS-driver permitting write access.

REXX FAT

After creating a somewhat dubious MBR in disk images formatted by rexxfat I've now also added a dummy VBR doing more than only INT 18h (hex. CD 18).  For an ordinary FAT12, FAT16, or FAT32 volume this VBR displays the OEM Id. (8 characters at offset 3 of the VBR), the file system type string (8 characters at offset -8 from the begin of the VBR code), the volume label (11 char.s, offset -19, default NO␠NAME␠␠␠␠), and the wannabe-unique volume Id. (4 bytes shown as 8 little endian hex. digits, offset -23).

Wikipedia claims that the file system type string and the volume label only exist for an extended boot signature introduced by 29h at offset -24 from the begin of the code.  In other words, for 28h only the volume id. can be expected to exist, and for anything else these 1+4+11+8 bytes won't exist at all, notably on media formatted with PC DOS 3.x or earlier.  The rexxfat dummy VBR code is designed to work for any 29h FAT layout with sector size 256, 512, or more.  Check out the script for the OpenWatcom WASM assembler MBR and VBR sources in two REXX comments.

Apparently Darwin VBRs expect that SI points to their partition table entry in the MBR, and just in case rexxfat now guarantees that when it boots an active primary partition.  Looking at a Darwin MBR source I think this is only used in its magic to boot a logical disk in an extended partition, i.e., the relative number of hidden sectors from the begin of the EBR to the logical disk VBR is updated on the fly to yield the absolute number of hidden sectors from MBR to VBR, and that's of course the LBA.  This won't help normal FAT VBR code expecting the number of hidden sectors in its own BPB, and the rexxfat MBR limits its efforts to initialize SI as expected for primary partitions.  Patching loaded FAT VBRs in an extended partition is left as an exercise for the EBR code — and so far rexxfat only formats one primary partition with a FAT.  More convoluted tasks are a job for GRUB, BCDEDIT, diskpart, parted, or what you have.

YAM (Yet Another MBR)

The Starman and Ray Knights disassembled various MBRs from ancient PC DOS to current Windows 7 versions, also covering LILO and GRUB MBRs.  The code used by TestDisk — when you are forced to fix a broken MBR — is a variant of the source published by Neil Turton.  The TestDisk variant tries to load the VBR for the active partition, and if that doesn't work the user can pick 1234A to load the VBR of partition 1…4 or floppy drive A:.  The original code has a timeout allowing to select 1234A, and in that variant the active partition is only the default.

A very simple MBR for the old INT 13h CHS interface is distributed with parted.  MBR CHS addressing works only for partitions starting in cylinder 0…1023 for any geometry, or in logical sector number (LBA) 0…16515070 in a (virtual) 1024×256×63 geometry.  Some interesting details found in these MBRs and related sources:

• All MBRs first define stack, data and extra segments, and then allow interrupts.  On entry SS:SP can be 0000:0400h at the end of the interrupt vector table, and picking something better is a good plan.  The new MBR code in rexxfat.rex simply uses 0000:0000h as stack top, i.e., the top of segment 0000h.
• The entry address for both MBRs and VBRs is 0000:7C00h.  To load any VBR at this address the MBR needs to move its own code and data to another address.  Allegedly some odd BIOS INT 19h code manages to start the MBR at the alias 07C0:0000h.  In other words, assume CS:nothing, or fix CS with a far jump.  The rexxfat MBR is designed for sector sizes from 256=100h up to 4096=1000h bytes.  Therefore the MBR copies itself to 0000:EC00h.  Adding 1000h yields a stack bottom at 0000:FC00h; and a stack with 1024 bytes should be enough for INT 13h LBA read operations.
• Allegedly some odd BIOS INT 19h code does not report the disk number for INT 13h in DL correctly.  Old MBR code apparently used the active partition flag 80h to populate DL, i.e., any non-zero flag instead of 80h could be used.  Variants of this scheme accept 80h…FFh as active, use DL as set by the BIOS, and reject 01h…7Fh as invalid  partition table.  The minimalistic rexxfat MBR accepts only 80h as active and does not look for trouble in the form of more than one boot partition.  Actually it scans the flags backwards checking its own 55AAh magic first.
• Most MBR variants support both CHS and LBA addressing.  The minimalistic rexxfat MBR always uses LBA to load six instead of only one VBR sector.  If something goes wrong, e.g., there is no active partition, or the loaded VBR has no 55AAh magic, an error is reported.  At that point the user can press space, 0…9, a (lower case "A"), or any other key.  0…9 tries to load the MBR for DL=80h…89h, and a tries DL=00h for the first floppy drive.  This won't work in emulation modes for say USB, where only 80h for this partitioned medium is supported.  Pressing space triggers INT 18h, a modern BIOS would then try to boot the next configured boot medium.  Any other key triggers INT 19h to boot the first configured boot medium.
• Various Windows MBRs have special code for the FAT32 backup boot sectors and partition types 0Bh or 0Ch.  Windows 7 MBRs support encrypted disks.  If you need these features use a hex. editor and copy the code in the first sector up to offset 1B8h.  The remaining 72 bytes are the disk ID (four bytes), two unclear nulls, 4×16 bytes for the four partitions, and the magic 55AAh.  Little endian whiners, it's 0xAA55 in your debugger ;-)

Allegedly MBRs and VBRs for sector sizes 1024, 2048, and 4096 are supposed to have their magic at the same offset as for sector size 512.  At the moment the rexxfat MBR expects that the 72 bytes from offset 1B8h to 1FFh contain the disk ID, partition table, and magic.  Flipping one byte in the code from 01h to 00h for the offset of the magic in theory allows sector size 256.  In practice VHD images only allow 512, real hard disks used 512, and El Torito or as the name says 512e emulate 512.

FAT32 fun

After NTFS and EXFAT now some FAT32 fun. The FORMAT and DISKPART tools on Windows NT have some limitations, therefore I created rexxfat.rex for experiments with disk image files and the FAT file system. In essence rexxfat can create a VFD image for a given number of sectors, example:

c:\etc\bin>rexxfat 4194304
1:   32264 * 2 FAT32 sectors;   7 +   0 SYS;   1 *   4129769 data:      4129769
2:   16257 * 2 FAT32 sectors;   8 +   0 SYS;   2 *   2080891 data:      4161782
3:    8161 * 2 FAT32 sectors;  10 +   0 SYS;   4 *   1044493 data:      4177972
4:    4089 * 2 FAT32 sectors;  14 +   0 SYS;   8 *    523264 data:      4186112
5:    2047 * 2 FAT32 sectors;  18 +   0 SYS;  16 *    261887 data:      4190192
6:    1024 * 2 FAT32 sectors;  32 +   0 SYS;  32 *    131007 data:      4192224
7:     128 * 2 FAT16 sectors;   1 + 127 SYS; 128 *     32765 data:      4193920
Pick 1..7 to create c:\etc\bin\rexxfat.vfd with 4194304 sectors

FAT16 cluster size 128 (hex. 80h) might be a bad idea for ancient tools. On the other hand FAT32 is not supported by ancient DOS versions up to MS DOS 6.22 or PC DOS 7.

FAT12 and FAT16 have a static root directory in the system area. For FAT32 the root directory is a dynamic part of the data area typically starting in the first data cluster. Consequently FAT12 and FAT16 never require unused sectors at the end of the data area, e.g., if a given cluster size cs=2**n yields u=1…cs-1 unused data sectors, then the data area can be reduced by u sectors, and the root directory can be increased by u sectors. rexxfat uses a default minimum of 6 sectors for the root directory, for sector size 512 that allows a minimum of 96 root directory entries.

For FAT32 any unused sectors are added to the reserved sectors at the begin of the system area. Normally FAT32 has 12 or more reserved sectors consisting of four sets:

• 00..02 Three boot sectors (01 is actually used as FSinfo)
• 03..05 Three unused sectors (nulls, no magic 55AAh)
• 06..08 Three backup boot sectors (the FSinfo pointer in 06 is 01, not 07)
• 09..11 Three or more unused sectors

Fortunately this madness can be trimmed, for starters zero or more unused reserved sectors in the last set are good enough. It is also possible to move the backup boot sectors from 06..08 to 03..05, simply modify the backup pointer in 00 and its backup.

There be dragons: If sector 00 is unreadable the backup pointer is also unreadable, and tools trying to fix or bypass this issue would assume that 06 is the backup. It is still possible to use 03..05 as official backup and keep 06 as unofficial third backup. For many purposes that should be good enough, with one notable exception: FAT32 boot code for MS DOS 7 (Windows 95, 98, ME) actually uses the third boot sector, and the Windows NT master boot code assumes that the backup boot sectors for FAT32 partitions start in relative sector 06. Obviously this cannot work as expected if MS DOS 7 backup boot code expected in sector 08 only exists in sector 05.

Fast forward, this is 2011, I've never used MS DOS 7, the dummy boot code in rexxfat consists of two bytes CD18h (INT 18h), and this works for all sector sizes in sectors 00, 03, or even 06. There is an option to enforce a classic layout with 9 or more reserved sectors.

For my purposes the odd layout with only 7 reserved sectors is better, adding one sector for the MBR yields 8, and removable media typically consists of 8×x sectors: No stupid gaps between the MBR and the FAT32 partition, and also no unused sectors after this partition or at its end. DISKPART and similar tools would skip 63-1 or 2048-1 sectors between MBR and partition, but for removable media with only one partition this is unnecessary: If some decent MBR code can load sector 63 or 2048 it can also load sector 1. There is no "second track" beginning in sector 63, and the new MS Vista alignment on MB boundaries (2048) is nice if you need to find a lost partition on a huge disk, but not required.

To create a VHD image instead of a VFD with rexxfat specify a negative total sector number as argument. One sector is used as MBR, the rest is used for the FAT partition. If the sector size is 512 a classic fixed VHD footer (511 bytes) is added. Sadly fixed VHDs cannot be created as NTFS sparse  files, and also do not allow other sector sizes (128, 256, 1024, 2048, or 4096). Here's the checkmbr output for a VHD with 66558 sectors:

...\bin\rexxfat.vhd (assuming geometry CHS 11093   1  6) id. [16F4-1407]
   MBR CHS     0   0  1 at          0, end     0   0  1, size          1
1: 0C: CHS     0   0  2 at          1, end 11092   0  6, size      66557 FAT32
2: 00: CHS     0   0  0 at          0, end     0   0  0, size          0 unused
3: 00: CHS     0   0  0 at          0, end     0   0  0, size          0 unused
4: 00: CHS     0   0  0 at          0, end     0   0  0, size          0 unused
                                                        total      66558

 FAT32 CHS     0   0  2 at          1, end 11092   0  6, size      66557
  boot CHS     0   0  2 at          1, end     0   0  4, size          3
backup CHS     0   0  5 at          4, end     1   0  1, size          3 boot
  rest CHS     1   0  2 at          7, end     1   0  2, size          1 boot
 FAT32 CHS     1   0  3 at          8, end    86   0  4, size        512 #1
 FAT32 CHS    86   0  5 at        520, end   171   0  6, size        512 #2
  data CHS   172   0  1 at       1032, end 11092   0  6, size      65526
[16F4-1408] (cluster size   1, number      65526)       total      66557

This is work in progress, the pseudo-geometry in the VHD footer fits into 28=16+4+8 bits for ATA, but an incomplete last cylinder in a geometry with 24=10+8+6 bits for INT 13h would be better for less than 16515072=1024×256×63 sectors. After all there are no cylinders in a VHD image, and the checkmbr output is designed for INT 13h CHS tuples in MBRs, not for ATA CHS tuples in VHDs.

EXFAT fun

For what's it worth checkmbr.rex can now handle EXFAT partitions. Partition type 07h is still tagged as NTFS, but the reported details now match EXFAT if applicable. Example:

       CHS     0   0  1 at          0, end     0 158 47, size      10000
hidden or total sectors 63 10000 do not match 0 10000
  boot CHS     0   0  1 at          0, end     0   0 12, size         12
backup CHS     0   0 13 at         12, end     0   0 24, size         12 boot
  rest CHS     0   0 25 at         24, end     0   2  2, size        104 boot
 EXFAT CHS     0   2  3 at        128, end     0   2 18, size         16 #1
unused CHS     0   2 19 at        144, end     0   4  4, size        112
  data CHS     0   4  5 at        256, end     0 158 46, size       9744
[6722-4357] (cluster size   8, number       1218)       total      10000

The Windows 7 FORMAT /FS:exFAT tool used an obscure number 63 for the hidden sectors in this unpartitioned VFD image also known as superfloppy, and checkmbr.rex dutifully reports that 63 is not 0. As long as you don't try to boot from a superfloppy or ordinary partition these hidden sectors are irrelevant.

It is interesting to see that FORMAT reserved 128=2×12+104 sectors for the 2×12 boot sectors. Most of the 12 boot sectors are already unused, and the boot checksum sector with 128 copies of the same 32bits checksum is hilarious. So what is the idea of the 104 additional sectors?

While at it Microsoft decided that two "FAT" copies are for cowards, and creates only one "FAT". It is not really a FAT, EXFAT uses a bit map for allocations, the "FAT" is only used for purposes where a bit is not good enough, i.e., bad clusters or fragmented cluster chains. And after saving 16 sectors for a second FAT there is another set of 112 apparently unused sectors, 128=16+112.

I'd get the idea if subsections of the system area are padded to max(4096/SS,CS) sectors: At some point in time we'll want to use 512e aligned to physical sector size 4096. But for that 56=2×12+2×16 instead of 256=2×128 would be good enough. For one FAT there are apparently 216=104+112 unused sectors, and instead of 1218 there could be 1235=1218+27 clusters (27=216/8).

SANS published a brilliant reverse engineering paper about EXFAT, but I'm not yet ready to outsmart FORMAT /FS:exFAT. Remotely related, checkmbr survived the forensic extended partition test case with two primary partitions in an extended partition. And I've fixed the output for zero FAT12 clusters. ToDo: checkmbr should report that 6 of the 16 FAT sectors are overkill for 1218 clusters, after all it does this already for FAT12/16/32.

NTFS fun

New REXX toy: rxsparse.rex converts a specified file on NTFS to a sparse file with FSUTIL SPARSE SETFLAG and FSUTIL SPARSE SETRANGE. Obviously you need fsutil.exe for this business, and it only works on NTFS.

NTFS files can be compressed and decompressed on the fly on Windows NT when the corresponding file attribute is set, and with inheritance this can be arranged for complete subdirectory trees. This feature replaced the obscure DOS double space approach, good riddance. Compression on the fly can be nice, but of course file accesses will be slower. Sparse files are an alternative, long runs of zero bytes in a file are not physically stored, but emulated. Unlike compression that's fast, only the position and length of sparse ranges has to be noted.

For NTFS a long run is not anything, but a multiple of the cluster size. As for FAT file systems the NTFS cluster size is 1, 2, 4, …, or 128 sectors. Wikipedia claims that a long run consists of 64 KB, and rxsparse can check this theory in its self test:

SPARSE SETFLAG C:\etc\bin\REXX\rxsparse.tmp
655360 = 5*131072 bytes with 8, 16, 32, 64, 128 "zero"-sectors at the end:
Bereich geringer Datendichte: [0] [655360]
SPARSE SETRANGE 126976 4096
SPARSE SETRANGE 253952 8192
SPARSE SETRANGE 376832 16384
SPARSE SETRANGE 491520 32768
SPARSE SETRANGE 589824 65536
Bereich geringer Datendichte: [0] [589824]

SPARSE SETFLAG C:\etc\bin\REXX\rxsparse.tmp
655360 = 5*131072 bytes with 8, 16, 32, 64, 128 "zero"-sectors at the begin:
Bereich geringer Datendichte: [0] [655360]
SPARSE SETRANGE 0 4096
SPARSE SETRANGE 131072 8192
SPARSE SETRANGE 262144 16384
SPARSE SETRANGE 393216 32768
SPARSE SETRANGE 524288 65536
Bereich geringer Datendichte: [0] [524288]
Bereich geringer Datendichte: [589824] [65536]

Test okay, maybe delete C:\etc\bin\REXX\rxsparse.tmp

The shown ranges [offset] [size] contain non-zero
bytes; check that there are 1..4 remaining ranges.
The smallest hidden zero-range size should be at
least 4096 (otherwise edit BLKLEN in the source).

The German gibberish is the output of FSUTIL SPARSE QUERYRANGE on a German Windows 7 x64 SP1. Oddly this is the opposite of SETRANGE, it shows ranges containing non-zero bytes. NTFS cluster size 4096=8×512 is perfectly normal, the definition of long run could still differ for other NTFS cluster and sector sizes.

Major caveat: When rxsparse found the end of what it considers as a long run of zero sectors it has to close the file before using FSUTIL SPARSE SETRANGE. If another process manages to write non-zero bytes in the critical range before FSUTIL gets write access these non-zero bytes are lost. Presumably REXX could somehow use a decent .NET API to avoid this potential race condition with the FSUTIL command line tool, but I didn't bother to figure it out.

Sadly my use case "FAT32 VHD image on NTFS" does not work as expected for the VHD variant known as fixed VHD (type 2). For NTFS VHD images it is also pointless, DISKPART can handle NTFS VHDs after defragmentation and precompact in a VPC VM guest OS supporting the precompact VM addition, notably Windows 2000 or better. Any "defrag" and wipe unused disk space tool for FAT file systems on another guest OS has presumably the same effect as precompact, but DISKPART supports only NTFS for its part of the VHD compactification magic. Not yet tested, maybe rxsparse makes sense for dynamic FAT VHD images.

Customised Search Engines

More than four years ago I explained the secrets of LP and AH in a Google CSE query parameter cof=FORID:0%3BAH:center%3BLP:0. The percent-encoded hex. %3B is a semicolon separating FORID:0, AH:center, and LP:0 in this example going back to Google's ancient free site search.

This undocumented cruft is now seriously dead, or rather, AH:center and LP:0 now have no visible effect. Maybe the cof= parameter was removed while some new CSE features were introduced since June, see an entry on the Custom Search Blog — I've no idea what the Element might be, and the officially deprecated features didn't mention any undocumented cruft, but there, it's dead, and the new layout for CSE search results hosted by Google works fine for the xyzzy CSE. Fortunately I modified it to work on the CSE layout test page some months ago, and as it happens that is now apparently good enough for new search result pages without AH:center and LP:0.

I'll remove any remaining obsolete cof= parameters where I find them, it might be hidden in obscure places such as the template for this blog, persistent URL (PURL) redirections, rel="search" link relations in the header of my web pages, and a tiny CSE googlet (PURL of XML source).

BTW, if you were always looking for an IETF-related search engine test the xyzzy CSE as shown near the bottom of all web pages for this blog. I still update this CSE when I find new promising sites, one of the last additions was a site tracking link relation registration requests. I added it after the registration of link relation rel="canonical" on the corresponding IETF expert review mailing list.

And if you were always looking for a REXX-related search engine check out my second (and last) REXX CSE — I rarely use it, but fix it when I stumble over broken links on my KEXX page. For a recipe to convert CSEs given by their cx= ID to OpenSearch Description Documents see another four years old entry on this blog, or just adopt one of the OSDDs on my googlet page.

MD5 test suite version 1.8

The MD5 test suite 1.8 finally mentions RFC 6151, fixes a minor exit code bug in version 1.7, supports the web-safe base64 encoding specified in RFC 4648 as base64url alphabet, uses the USCYBERCOM easter egg as MD5 streaming test case, and covers the collision announced by Tao Xie and Dengguo Feng in December 2010 (2 of 512 bits modified). Read their PDF (one page) for details of the offered bounty for anybody who finds another collision of this type.

checkMBR.rex

Some of my web pages are still messy and on their way to a new hoster, but there is a new version of checkmbr.rex (REXX script) maybe interesting for users of TestDisk. The script can analyze MBR disks on Windows platforms using \\.\PHYSICALDRIVEn for n=0..9, and identifies unused sectors caused by generous partitioning or by FAT and NTFS sizes not exactly fitting into their given partition. While at it the script creates base 64 backups of various boot sectors. Fixed bugs and new features:

• The max. cluster numbers for FAT12/16/32 were "off by one". The hardwired limits are now 4084, 65524, and 268435444.
• Added partition type 27h "WinRE" for a hidden windows recovery partition and file system NTFS.
• exFAT uses partition type 07h also used by NTFS. This is unsupported and should result in lots of "NTFS errors" for exFAT (untested).
• UEFI disks start with a protective MBR (partition type EEh). This is now reported, but checkMBR analyzes only MBR disks.
• VFD (virtual floppy disk) and fixed VHD (virtual hard disk) files can now be given instead of a physical drive number.

The new VHD feature would fail for ooREXX 3.x and VHDs greater than 2 GB, get ooREXX 4.x. Many identified partition types are untested or ambiguous, e.g., I've never seen an EFI FAT type EFh roughly corresponding to a Windows system NTFS partition on MBR disks, or the partition types allegedly used by the FreeDOS FDISK tool to hide FAT, NTFS, or extended partitions.

MD5 1.7: verified errata

All pending errata for the MD5 test suite are now verified by the IESG; thanks to Alexey Melnikov. The oldest erratum 749 about an MD5 example in RFC 2069 was submitted 2005-02-06 and verified 2010-07-11. Now I feel less bad about dropping out for two years.

For historical reasons the MD5 test suite is one of my REXX scripts still published as cmd-file instead of a rex-file. On an OS/2 box REXX is the default scripting language using file extension cmd. An ordinary OS/2 cmd.exe-shell script never starts with "/*"; any script starting with "/*" is interpreted as REXX.

Good old PC DOS 7 uses extension bat for command.com-shell scripts, and the text editor KEDIT uses kex for its macros; both also identify REXX by "/*" in line 1.

Please do not feed OS/2 REXX cmd-scripts to the Windows NT cmd.exe-shell; you would get numerous errors. For NT simply rename md5.cmd to md5.rex and let ooREXX interpret it.

RxWhois IETF 72 edition

Based on the latest DNSBL Internet-Draft rxwhois 2.1.2 now supports DNSBL queries for IPv6.

I've not yet found any black listed IPv6. The potential test entries ::FFFF:127.0.0.2 or ::2 are not listed on the nine pre-configured DNSBLs used by rxwhois.

The simple REXX code to transform an <IPv6address> into the reverse ip6.arpa format turned out to be more interesting than I thought; see procedure XIPV6 in the source… :-)

rxwhois 2.1.1

The modifications in version 2.1.1 of rxwhois.cmd include the addition of whois.nic.asia, whois.nic.bi, whois.adsib.gob.bo, whois.registry.gy, whois.nic.ht, whois.nic.im, whois.kcce.kp, whois.nic.mq, whois.nic.org.mt, whois.nic.pr, whois.co.ug, and vunic.vu from the usual sources: IANA, whois-servers.net, guesswork.

Lost in space: whois.nic.ki, whois.nic.tel, and whois.cctld.uz. Some domain grabbers in the TLDs CM, FM, GD, KN, and PN were ignored. Eleven other whois.whatever.tld hosts were added, but these are (yet) no whois servers. The rxwhois manual page contains the version history up to 2.1 over the last six years.

MD5 1.6: POP3 and UUID

Version 1.6 of the MD5 test suite covers RFC 5034 (Digest-MD5 for POP3), RFC 4122 (UUID version 3), and an old RFC 1910 example. The POP3 example belongs to the MD5-sess series.

The UUID example is apparently wrong, it swaps the byte order unnecessarily. Examples published by python.org and ossp.org agree with that theory, I'll submit it as erratum.

MD5-sess and RFC 5090

RFC 5090 was published, it fixes some problems with the MD5 examples in RFC 4590. Version 1.5 of the MD5 test suite now contains the new RFC 5090 RADIUS examples.

The updated test suite contains a new procedure AUTHTTP for the RFC 2617 idea of MD5-sess noted in the errata. The old DIGEST procedure uses the RFC 2831 MD5-sess algorithm. Six new test cases cover MD5-sess examples published in RFC 2831 and 4643 using the RFC 2617 algorithm. Hopefully this will be documented in an RFC moving RFC 2831 to HISTORIC.

RFC errata

Various pending RFC errata have been published recently, among others for RFC 2069, 2822, and 4408. The RFC editor might soon offer a Web form for submissions as outlined in an Internet Draft.

• The 2069 erratum resulted in an editorial update of my MD5 test suite.
• The 2822 erratum was already covered in the 2822upd drafts.
• The 4408 erratum is actually a link to the OpenSPF errata page.

New TLDs BL, MF, and TEL

Version 2.1 of rxwhois knows the new TLDs BL, MF, and TEL. The corresponding whois servers are not yet running, but whois.iana.org already supports these TLDs. The corresponding region codes BL and MF will be added to the IANA language subtag registry in the next weeks.

Now rxwhois also supports the eleven IDN test domains. The test started 2007-10-15, but it took me until yesterday to fix a stupid bug.

The worst issue so far from my POV is that popular XHTML validators like the W3C validator don't check the URL syntax in attributes like href="URI", see bug 4916. Many users will be misled to create invalid pages with "unencoded" IRIs in document types like HTML 4.01 and XHTML 1.0, where that's not allowed.

RFC 2617 vs. 2831 md5-sess

Good news first, RFC 4590bis (approved, still waiting for its number) will fix the Digest-MD5 examples in RFC 4590. I've updated the MD5 test suite using the fixed examples.

While I was at it I've also updated the RFC 3797 code to work for the NOMCOM 2007 case. The entropy limit 30 was too restrictive, 38 is good enough for MD5, 10^38 < 2^128.

Now the bad news, the issue with two md5-sess examples in draft smith-sipping-auth-examples might be in fact precisely what RFC 2617 says, as reported in a semi-official erratum. If that's correct the md5-sess in RFC 2831 would be different. Hopefully draft melnikov-digest-to-historic will shed some light on this before it moves RFC 2831 to historic. For more about this see the IETF SASL WG mailing list.

For now the MD5 test suite still uses only the binary x2c(HA1) form instead of the hex. HA1 form in its md5-sess calculation.

ftpsynch.rex

Many OS/2 SAAREXX programs work almost as is under WindowsNT ooRexx, the RexxUtil functions are similar, the WindowsNT CMD shell is similar, and the RxSock interface is almost identical. Some OS/2 RexxUtil functions are not yet or not more supported under WindowsNT, e.g. SysGetMessage, SysProcessType, and SysQueryProcessCodePage.

Scripts I've tested under W2K after renaming *.cmd to *.rex include popstop2.cmd, dir2html.cmd, and sitemap.cmd.

Unfortunately ooRexx doesn't come with RxFTP.dll, but it offers the RxFTP.cls class. I've created a new ftpsynch.rex based on ftpsynch.cmd for ooRexx, for details see ftpsynch.htm.

rxwhois 2.0.5

Version 2.0.5 of rxwhois.cmd now also works as WindowsNT ooRexx script, just rename it to rxwhois.rex. I've adopted additional local character sets from utf-8.cmd including codepage 923 (ISO 8859-15, Latin 9) and 878 (KOI8-R), but I've only tested 858 (pc-multilingual-850+euro) and 1252 (windows-1252).

As always some whois-servers for ccTLDs had to be updated, for details see the source. After a system crash of my OS/2 box in June I was unable to check anything beyond the whois servers already known by rxwhois.cmd, whois.iana.org, and whois-servers.net. Just for fun I've added the eleven IDN TLDs for the test beginning in September 2007:

1. Arabic xn--kgbechtv
2. Persian xn--hgbk6aj7f53bba
3. Chinese, simplified xn--0zwm56d
4. Chinese, traditional xn--g6w251d
5. Russian xn--80akhbyknj4f
6. Hindi xn--11b5bs3a9aj6g
7. Greek xn--jxalpdlp
8. Korean xn--9t4b11yi5a
9. Yiddish xn--deba0ad
10. Japanese xn--zckzah
11. Tamil xn--hlcj6aya9esc7a

Known issue: rxwhois.cmd expects UTF-8 as charset of whois servers, but whois.iana.org uses Latin-1 for at least one TLD ht (Haiti) entry. The IANA folks told me that they'll intend to use ASCII data for the eleven IDN test TLDs.

Simple REXX mailto script

Yet another mailto command line tool, rxmailto.cmd v0.1 can so far send one text mail to one receiver via a Mail Submit Agent (MSA) at port 587 supporting SMTP AUTH with CRAM-MD5 and 8bitMIME. That's the minimum I could get away with after the spam flood finally drowned my old mailbox.

Various details are far from perfect, e.g. if a run of words with non-ASCII characters in the subject is longer than 56 octets the subject encoder will emit a folded line longer than 76 characters, and that's not permitted by RFC 2047. On the other hand the script won't break UTF-8 characters in the subject for platforms with UTF-8 as local charset. You get what you pay for, less than 30 KB. ;-)