xyzzy

FAT16 fun

2011-11-28T01:34:00.007Z

Continuing the FAT fun saga with an unplanned sequel: rexxfat used partition type 05 instead of 06 for BigFAT, i.e., FAT16 with 2**16 or more sectors. I've fixed this stupid bug in the last rexxfat version. The virtual hard disk image works fine in a Windows 2000 virtual machine:

The odd size 3759 MB matches what I have on a memory card. It's rather tricky to copy the disk image to the memory card, either rawcopy doesn't support copies to a physical device, or it was not obvious for me how to manage this. Using diskpart to unmount the one and only volume on this partitioned memory card wasn't good enough for write access, eventually the diskpart delete volume magic did the trick. Presumably that also explains my difficulties with other tools including dskprobe and HxD.

Looking for explanations I stumbled over SysInternals SDelete. That article doesn't explain how Windows 7 manages volumes on memory cards, but answered another open question: Compressed, encrypted and sparse are managed by NTFS in 16-cluster blocks. That matches my observation for rxsparse.rex, 16×4=64 KB, and if the SysInternals folks got that right Wikipedia also got it right. Hint, it is very likely that SysInternals got it right, after all they were the first to offer a working DOS NTFS-driver permitting write access.

REXX FAT

2011-11-24T03:43:00.002Z

After creating a somewhat dubious MBR in disk images formatted by rexxfat I've now also added a dummy VBR doing more than only INT 18h (hex. CD 18). For an ordinary FAT12, FAT16, or FAT32 volume this VBR displays the OEM Id. (8 characters at offset 3 of the VBR), the file system type string (8 characters at offset -8 from the begin of the VBR code), the volume label (11 char.s, offset -19, default NO␠NAME␠␠␠␠), and the wannabe-unique volume Id. (4 bytes shown as 8 little endian hex. digits, offset -23).

Wikipedia claims that the file system type string and the volume label only exist for an extended boot signature introduced by 29h at offset -24 from the begin of the code. In other words, for 28h only the volume id. can be expected to exist, and for anything else these 1+4+11+8 bytes won't exist at all, notably on media formatted with PC DOS 3.x or earlier. The rexxfat dummy VBR code is designed to work for any 29h FAT layout with sector size 256, 512, or more. Check out the script for the OpenWatcom WASM assembler MBR and VBR sources in two REXX comments.

Apparently Darwin VBRs expect that SI points to their partition table entry in the MBR, and just in case rexxfat now guarantees that when it boots an active primary partition. Looking at a Darwin MBR source I think this is only used in its magic to boot a logical disk in an extended partition, i.e., the relative number of hidden sectors from the begin of the EBR to the logical disk VBR is updated on the fly to yield the absolute number of hidden sectors from MBR to VBR, and that's of course the LBA. This won't help normal FAT VBR code expecting the number of hidden sectors in its own BPB, and the rexxfat MBR limits its efforts to initialize SI as expected for primary partitions. Patching loaded FAT VBRs in an extended partition is left as an exercise for the EBR code — and so far rexxfat only formats one primary partition with a FAT. More convoluted tasks are a job for GRUB, BCDEDIT, diskpart, parted, or what you have.

YAM (Yet Another Mail)

2011-11-24T01:43:00.004Z

The ways of the IETF are completely bizarre, and the Tao of IETF does not really explain why. In a recent episode the YAM WG will be closed after moving two of four e-mail related RFCs to STDs:

STD 72 is RFC 6409 and covers "mail submission". That's what ordinary users like you and me do when they send e-mail using their ISP or another ESP such as Gmail, GMX, or Hotmail.
STD 71 is RFC 6152 and covers "8BITMIME". That's what everybody uses since about 1993 outside of pure US-ASCII 7bit e-mail.

Oddly YAM didn't finish the work on a 5321bis and a 5322bis. That leaves us with STD 10 (SMTP) and STD 11 (message format) in limbo, unless you still consider RFCs 821 and 822 (vintage 1982) as the ultimate truth in IETF e-mail standards.

For SPF it's quite interesting to look at the "reverse path" fine print of RFC 821, e.g.:

The first host in the <reverse-path> should be the host sending this command. (RFC 821 about MAIL FROM).
In any case, the SMTP adds its own identifier to the reverse-path. (RFC 821 about Relaying).

Sadly RFC 4408 specifying SPF also claims that this RFC 821 feature is archaic. Of course today nobody uses reverse and forward paths as designed about thirty years ago for e-mail routing, but for SPF the MAIL FROM address is still supposed to indicate the responsible party for technical problems with an e-mail, also known as bounce address. And e-mails forwarded to a third party need a new envelope sender address, otherwise SPF checks at this third party will fail, because IP-addresses authorized to send e-mail from the first party won't match IPs used by the forwarding second party. That's as simple as it can be, ignoring the minor detail that RFC 1123 broke it about twenty years ago.

By the way, the work on 4408bis is supposed to start really soon now, but my confidence in the IETF is somewhat limited.

Google ±

2011-11-24T00:54:00.003Z

Google+ wants me to join Google+, because I'm on Google+.

YAM (Yet Another MBR)

2011-11-08T16:53:00.003Z

The Starman and Ray Knights disassembled various MBRs from ancient PC DOS to current Windows 7 versions, also covering LILO and GRUB MBRs. The code used by TestDisk — when you are forced to fix a broken MBR — is a variant of the source published by Neil Turton. The TestDisk variant tries to load the VBR for the active partition, and if that doesn't work the user can pick 1234A to load the VBR of partition 1…4 or floppy drive A:. The original code has a timeout allowing to select 1234A, and in that variant the active partition is only the default.

A very simple MBR for the old INT 13h CHS interface is distributed with parted. MBR CHS addressing works only for partitions starting in cylinder 0…1023 for any geometry, or in logical sector number (LBA) 0…16515070 in a (virtual) 1024×256×63 geometry. Some interesting details found in these MBRs and related sources:

All MBRs first define stack, data and extra segments, and then allow interrupts. On entry SS:SP can be 0000:0400h at the end of the interrupt vector table, and picking something better is a good plan. The new MBR code in rexxfat.rex simply uses 0000:0000h as stack top, i.e., the top of segment 0000h.
The entry address for both MBRs and VBRs is 0000:7C00h. To load any VBR at this address the MBR needs to move its own code and data to another address. Allegedly some odd BIOS INT 19h code manages to start the MBR at the alias 07C0:0000h. In other words, assume CS:nothing, or fix CS with a far jump. The rexxfat MBR is designed for sector sizes from 256=100h up to 4096=1000h bytes. Therefore the MBR copies itself to 0000:EC00h. Adding 1000h yields a stack bottom at 0000:FC00h; and a stack with 1024 bytes should be enough for INT 13h LBA read operations.
Allegedly some odd BIOS INT 19h code does not report the disk number for INT 13h in DL correctly. Old MBR code apparently used the active partition flag 80h to populate DL, i.e., any non-zero flag instead of 80h could be used. Variants of this scheme accept 80h…FFh as active, use DL as set by the BIOS, and reject 01h…7Fh as invalid partition table. The minimalistic rexxfat MBR accepts only 80h as active and does not look for trouble in the form of more than one boot partition. Actually it scans the flags backwards checking its own 55AAh magic first.
Most MBR variants support both CHS and LBA addressing. The minimalistic rexxfat MBR always uses LBA to load six instead of only one VBR sector. If something goes wrong, e.g., there is no active partition, or the loaded VBR has no 55AAh magic, an error is reported. At that point the user can press space, 0…9, a (lower case "A"), or any other key. 0…9 tries to load the MBR for DL=80h…89h, and a tries DL=00h for the first floppy drive. This won't work in emulation modes for say USB, where only 80h for this partitioned medium is supported. Pressing space triggers INT 18h, a modern BIOS would then try to boot the next configured boot medium. Any other key triggers INT 19h to boot the first configured boot medium.
Various Windows MBRs have special code for the FAT32 backup boot sectors and partition types 0Bh or 0Ch. Windows 7 MBRs support encrypted disks. If you need these features use a hex. editor and copy the code in the first sector up to offset 1B8h. The remaining 72 bytes are the disk ID (four bytes), two unclear nulls, 4×16 bytes for the four partitions, and the magic 55AAh. Little endian whiners, it's 0xAA55 in your debugger ;-)

Allegedly MBRs and VBRs for sector sizes 1024, 2048, and 4096 are supposed to have their magic at the same offset as for sector size 512. At the moment the rexxfat MBR expects that the 72 bytes from offset 1B8h to 1FFh contain the disk ID, partition table, and magic. Flipping one byte in the code from 01h to 00h for the offset of the magic in theory allows sector size 256. In practice VHD images only allow 512, real hard disks used 512, and El Torito or as the name says 512e emulate 512.

FAT32 fun

2011-11-03T11:23:00.003Z

After NTFS and EXFAT now some FAT32 fun. The FORMAT and DISKPART tools on Windows NT have some limitations, therefore I created rexxfat.rex for experiments with disk image files and the FAT file system. In essence rexxfat can create a VFD image for a given number of sectors, example:

c:\etc\bin>rexxfat 4194304
1:   32264 * 2 FAT32 sectors;   7 +   0 SYS;   1 *   4129769 data:      4129769
2:   16257 * 2 FAT32 sectors;   8 +   0 SYS;   2 *   2080891 data:      4161782
3:    8161 * 2 FAT32 sectors;  10 +   0 SYS;   4 *   1044493 data:      4177972
4:    4089 * 2 FAT32 sectors;  14 +   0 SYS;   8 *    523264 data:      4186112
5:    2047 * 2 FAT32 sectors;  18 +   0 SYS;  16 *    261887 data:      4190192
6:    1024 * 2 FAT32 sectors;  32 +   0 SYS;  32 *    131007 data:      4192224
7:     128 * 2 FAT16 sectors;   1 + 127 SYS; 128 *     32765 data:      4193920
Pick 1..7 to create c:\etc\bin\rexxfat.vfd with 4194304 sectors

FAT16 cluster size 128 (hex. 80h) might be a bad idea for ancient tools. On the other hand FAT32 is not supported by ancient DOS versions up to MS DOS 6.22 or PC DOS 7.

FAT12 and FAT16 have a static root directory in the system area. For FAT32 the root directory is a dynamic part of the data area typically starting in the first data cluster. Consequently FAT12 and FAT16 never require unused sectors at the end of the data area, e.g., if a given cluster size cs=2**n yields u=1…cs-1 unused data sectors, then the data area can be reduced by u sectors, and the root directory can be increased by u sectors. rexxfat uses a default minimum of 6 sectors for the root directory, for sector size 512 that allows a minimum of 96 root directory entries.

For FAT32 any unused sectors are added to the reserved sectors at the begin of the system area. Normally FAT32 has 12 or more reserved sectors consisting of four sets:

00..02 Three boot sectors (01 is actually used as FSinfo)
03..05 Three unused sectors (nulls, no magic 55AAh)
06..08 Three backup boot sectors (the FSinfo pointer in 06 is 01, not 07)
09..11 Three or more unused sectors

Fortunately this madness can be trimmed, for starters zero or more unused reserved sectors in the last set are good enough. It is also possible to move the backup boot sectors from 06..08 to 03..05, simply modify the backup pointer in 00 and its backup.

There be dragons: If sector 00 is unreadable the backup pointer is also unreadable, and tools trying to fix or bypass this issue would assume that 06 is the backup. It is still possible to use 03..05 as official backup and keep 06 as unofficial third backup. For many purposes that should be good enough, with one notable exception: FAT32 boot code for MS DOS 7 (Windows 95, 98, ME) actually uses the third boot sector, and the Windows NT master boot code assumes that the backup boot sectors for FAT32 partitions start in relative sector 06. Obviously this cannot work as expected if MS DOS 7 backup boot code expected in sector 08 only exists in sector 05.

Fast forward, this is 2011, I've never used MS DOS 7, the dummy boot code in rexxfat consists of two bytes CD18h (INT 18h), and this works for all sector sizes in sectors 00, 03, or even 06. There is an option to enforce a classic layout with 9 or more reserved sectors.

For my purposes the odd layout with only 7 reserved sectors is better, adding one sector for the MBR yields 8, and removable media typically consists of 8×x sectors: No stupid gaps between the MBR and the FAT32 partition, and also no unused sectors after this partition or at its end. DISKPART and similar tools would skip 63-1 or 2048-1 sectors between MBR and partition, but for removable media with only one partition this is unnecessary: If some decent MBR code can load sector 63 or 2048 it can also load sector 1. There is no "second track" beginning in sector 63, and the new MS Vista alignment on MB boundaries (2048) is nice if you need to find a lost partition on a huge disk, but not required.

To create a VHD image instead of a VFD with rexxfat specify a negative total sector number as argument. One sector is used as MBR, the rest is used for the FAT partition. If the sector size is 512 a classic fixed VHD footer (511 bytes) is added. Sadly fixed VHDs cannot be created as NTFS sparse files, and also do not allow other sector sizes (128, 256, 1024, 2048, or 4096). Here's the checkmbr output for a VHD with 66558 sectors:

...\bin\rexxfat.vhd (assuming geometry CHS 11093   1  6) id. [16F4-1407]
   MBR CHS     0   0  1 at          0, end     0   0  1, size          1
1: 0C: CHS     0   0  2 at          1, end 11092   0  6, size      66557 FAT32
2: 00: CHS     0   0  0 at          0, end     0   0  0, size          0 unused
3: 00: CHS     0   0  0 at          0, end     0   0  0, size          0 unused
4: 00: CHS     0   0  0 at          0, end     0   0  0, size          0 unused
                                                        total      66558

 FAT32 CHS     0   0  2 at          1, end 11092   0  6, size      66557
  boot CHS     0   0  2 at          1, end     0   0  4, size          3
backup CHS     0   0  5 at          4, end     1   0  1, size          3 boot
  rest CHS     1   0  2 at          7, end     1   0  2, size          1 boot
 FAT32 CHS     1   0  3 at          8, end    86   0  4, size        512 #1
 FAT32 CHS    86   0  5 at        520, end   171   0  6, size        512 #2
  data CHS   172   0  1 at       1032, end 11092   0  6, size      65526
[16F4-1408] (cluster size   1, number      65526)       total      66557

This is work in progress, the pseudo-geometry in the VHD footer fits into 28=16+4+8 bits for ATA, but an incomplete last cylinder in a geometry with 24=10+8+6 bits for INT 13h would be better for less than 16515072=1024×256×63 sectors. After all there are no cylinders in a VHD image, and the checkmbr output is designed for INT 13h CHS tuples in MBRs, not for ATA CHS tuples in VHDs.

SideWiki shutdown

2011-11-02T09:54:00.012Z

Google SideWiki — good riddance. Just for fun I save my last (of five) entries below, they were all futile attempts to get in contact with a human being interested in fixing Google bugs. Next stop, let's kill Knol.

'Meta-searching' Google
The alleged "meta search" is an ordinary "cached" link offered in an ordinary Google search result, where the found Google groups article does not contain the searched `+"bigfat" fat16 -forum`, and I hoped that at least the cached result might be related to my query.
Instead I get a runabout ending in this "sidewiki" known to be yet another Dave Null at his support work. If you do not want cached Google groups search hits to be clicked just do not offer the links.
About Unusual traffic from your computer network - Web Search Help

Go, Duck, go

2011-11-02T08:46:00.004Z

Reader redesign: Terrible decision, or worst decision? and another article discuss the next step of what will be known as the demise of Google: Now they have crippled Google Reader. It even doesn't work anymore in Chrome.

What used to be a single click Like or Share in Reader is now gone, and the new Reader UI is a disaster, as one of the creators of the original UI states. In theory I could join the G+ beta tests, but as it happens I'm not interested to risk my Google account including Blogger and Gmail with an acknowledgement of the G+ beta test ToS.

Check out Duck Duck Go lite while it's still a free no-nonsense search engine. And do not waste time with Google APIs or services, sadly they will be gone or munged until no-good before you begin to grok what they were about.

EXFAT fun

2011-10-30T06:39:00.004Z

For what's it worth checkmbr.rex can now handle EXFAT partitions. Partition type 07h is still tagged as NTFS, but the reported details now match EXFAT if applicable. Example:

       CHS     0   0  1 at          0, end     0 158 47, size      10000
hidden or total sectors 63 10000 do not match 0 10000
  boot CHS     0   0  1 at          0, end     0   0 12, size         12
backup CHS     0   0 13 at         12, end     0   0 24, size         12 boot
  rest CHS     0   0 25 at         24, end     0   2  2, size        104 boot
 EXFAT CHS     0   2  3 at        128, end     0   2 18, size         16 #1
unused CHS     0   2 19 at        144, end     0   4  4, size        112
  data CHS     0   4  5 at        256, end     0 158 46, size       9744
[6722-4357] (cluster size   8, number       1218)       total      10000

The Windows 7 FORMAT /FS:exFAT tool used an obscure number 63 for the hidden sectors in this unpartitioned VFD image also known as superfloppy, and checkmbr.rex dutifully reports that 63 is not 0. As long as you don't try to boot from a superfloppy or ordinary partition these hidden sectors are irrelevant.

It is interesting to see that FORMAT reserved 128=2×12+104 sectors for the 2×12 boot sectors. Most of the 12 boot sectors are already unused, and the boot checksum sector with 128 copies of the same 32bits checksum is hilarious. So what is the idea of the 104 additional sectors?

While at it Microsoft decided that two "FAT" copies are for cowards, and creates only one "FAT". It is not really a FAT, EXFAT uses a bit map for allocations, the "FAT" is only used for purposes where a bit is not good enough, i.e., bad clusters or fragmented cluster chains. And after saving 16 sectors for a second FAT there is another set of 112 apparently unused sectors, 128=16+112.

I'd get the idea if subsections of the system area are padded to max(4096/SS,CS) sectors: At some point in time we'll want to use 512e aligned to physical sector size 4096. But for that 56=2×12+2×16 instead of 256=2×128 would be good enough. For one FAT there are apparently 216=104+112 unused sectors, and instead of 1218 there could be 1235=1218+27 clusters (27=216/8).

SANS published a brilliant reverse engineering paper about EXFAT, but I'm not yet ready to outsmart FORMAT /FS:exFAT. Remotely related, checkmbr survived the forensic extended partition test case with two primary partitions in an extended partition. And I've fixed the output for zero FAT12 clusters. ToDo: checkmbr should report that 6 of the 16 FAT sectors are overkill for 1218 clusters, after all it does this already for FAT12/16/32.

NTFS fun

2011-10-30T04:27:00.002Z

New REXX toy: rxsparse.rex converts a specified file on NTFS to a sparse file with FSUTIL SPARSE SETFLAG and FSUTIL SPARSE SETRANGE. Obviously you need fsutil.exe for this business, and it only works on NTFS.

NTFS files can be compressed and decompressed on the fly on Windows NT when the corresponding file attribute is set, and with inheritance this can be arranged for complete subdirectory trees. This feature replaced the obscure DOS double space approach, good riddance. Compression on the fly can be nice, but of course file accesses will be slower. Sparse files are an alternative, long runs of zero bytes in a file are not physically stored, but emulated. Unlike compression that's fast, only the position and length of sparse ranges has to be noted.

For NTFS a long run is not anything, but a multiple of the cluster size. As for FAT file systems the NTFS cluster size is 1, 2, 4, …, or 128 sectors. Wikipedia claims that a long run consists of 64 KB, and rxsparse can check this theory in its self test:

SPARSE SETFLAG C:\etc\bin\REXX\rxsparse.tmp
655360 = 5*131072 bytes with 8, 16, 32, 64, 128 "zero"-sectors at the end:
Bereich geringer Datendichte: [0] [655360]
SPARSE SETRANGE 126976 4096
SPARSE SETRANGE 253952 8192
SPARSE SETRANGE 376832 16384
SPARSE SETRANGE 491520 32768
SPARSE SETRANGE 589824 65536
Bereich geringer Datendichte: [0] [589824]

SPARSE SETFLAG C:\etc\bin\REXX\rxsparse.tmp
655360 = 5*131072 bytes with 8, 16, 32, 64, 128 "zero"-sectors at the begin:
Bereich geringer Datendichte: [0] [655360]
SPARSE SETRANGE 0 4096
SPARSE SETRANGE 131072 8192
SPARSE SETRANGE 262144 16384
SPARSE SETRANGE 393216 32768
SPARSE SETRANGE 524288 65536
Bereich geringer Datendichte: [0] [524288]
Bereich geringer Datendichte: [589824] [65536]

Test okay, maybe delete C:\etc\bin\REXX\rxsparse.tmp

The shown ranges [offset] [size] contain non-zero
bytes; check that there are 1..4 remaining ranges.
The smallest hidden zero-range size should be at
least 4096 (otherwise edit BLKLEN in the source).

The German gibberish is the output of FSUTIL SPARSE QUERYRANGE on a German Windows 7 x64 SP1. Oddly this is the opposite of SETRANGE, it shows ranges containing non-zero bytes. NTFS cluster size 4096=8×512 is perfectly normal, the definition of long run could still differ for other NTFS cluster and sector sizes.

Major caveat: When rxsparse found the end of what it considers as a long run of zero sectors it has to close the file before using FSUTIL SPARSE SETRANGE. If another process manages to write non-zero bytes in the critical range before FSUTIL gets write access these non-zero bytes are lost. Presumably REXX could somehow use a decent .NET API to avoid this potential race condition with the FSUTIL command line tool, but I didn't bother to figure it out.

Sadly my use case "FAT32 VHD image on NTFS" does not work as expected for the VHD variant known as fixed VHD (type 2). For NTFS VHD images it is also pointless, DISKPART can handle NTFS VHDs after defragmentation and precompact in a VPC VM guest OS supporting the precompact VM addition, notably Windows 2000 or better. Any "defrag" and wipe unused disk space tool for FAT file systems on another guest OS has presumably the same effect as precompact, but DISKPART supports only NTFS for its part of the VHD compactification magic. Not yet tested, maybe rxsparse makes sense for dynamic FAT VHD images.

Customised Search Engines

2011-08-12T06:50:00.008+01:00

More than four years ago I explained the secrets of LP and AH in a Google CSE query parameter cof=FORID:0%3BAH:center%3BLP:0. The percent-encoded hex. %3B is a semicolon separating FORID:0, AH:center, and LP:0 in this example going back to Google's ancient free site search.

This undocumented cruft is now seriously dead, or rather, AH:center and LP:0 now have no visible effect. Maybe the cof= parameter was removed while some new CSE features were introduced since June, see an entry on the Custom Search Blog — I've no idea what the Element might be, and the officially deprecated features didn't mention any undocumented cruft, but there, it's dead, and the new layout for CSE search results hosted by Google works fine for the xyzzy CSE. Fortunately I modified it to work on the CSE layout test page some months ago, and as it happens that is now apparently good enough for new search result pages without AH:center and LP:0.

I'll remove any remaining obsolete cof= parameters where I find them, it might be hidden in obscure places such as the template for this blog, persistent URL (PURL) redirections, rel="search" link relations in the header of my web pages, and a tiny CSE googlet (PURL of XML source).

BTW, if you were always looking for an IETF-related search engine test the xyzzy CSE as shown near the bottom of all web pages for this blog. I still update this CSE when I find new promising sites, one of the last additions was a site tracking link relation registration requests. I added it after the registration of link relation rel="canonical" on the corresponding IETF expert review mailing list.

And if you were always looking for a REXX-related search engine check out my second (and last) REXX CSE — I rarely use it, but fix it when I stumble over broken links on my KEXX page. For a recipe to convert CSEs given by their cx= ID to OpenSearch Description Documents see another four years old entry on this blog, or just adopt one of the OSDDs on my googlet page.

MD5 test suite version 1.8

2011-07-29T06:01:00.004+01:00

The MD5 test suite 1.8 finally mentions RFC 6151, fixes a minor exit code bug in version 1.7, supports the web-safe base64 encoding specified in RFC 4648 as base64url alphabet, uses the USCYBERCOM easter egg as MD5 streaming test case, and covers the collision announced by Tao Xie and Dengguo Feng in December 2010 (2 of 512 bits modified). Read their PDF (one page) for details of the offered bounty for anybody who finds another collision of this type.

checkMBR.rex

2011-07-14T22:52:00.006+01:00

Some of my web pages are still messy and on their way to a new hoster, but there is a new version of checkmbr.rex (REXX script) maybe interesting for users of TestDisk. The script can analyze MBR disks on Windows platforms using \\.\PHYSICALDRIVEn for n=0..9, and identifies unused sectors caused by generous partitioning or by FAT and NTFS sizes not exactly fitting into their given partition. While at it the script creates base 64 backups of various boot sectors. Fixed bugs and new features:

The max. cluster numbers for FAT12/16/32 were "off by one". The hardwired limits are now 4084, 65524, and 268435444.
Added partition type 27h "WinRE" for a hidden windows recovery partition and file system NTFS.
exFAT uses partition type 07h also used by NTFS. This is unsupported and should result in lots of "NTFS errors" for exFAT (untested).
UEFI disks start with a protective MBR (partition type EEh). This is now reported, but checkMBR analyzes only MBR disks.
VFD (virtual floppy disk) and fixed VHD (virtual hard disk) files can now be given instead of a physical drive number.

The new VHD feature would fail for ooREXX 3.x and VHDs greater than 2 GB, get ooREXX 4.x. Many identified partition types are untested or ambiguous, e.g., I've never seen an EFI FAT type EFh roughly corresponding to a Windows system NTFS partition on MBR disks, or the partition types allegedly used by the FreeDOS FDISK tool to hide FAT, NTFS, or extended partitions.

about: ftp:

2011-07-09T02:50:00.007+01:00

There are I-Ds for two URI schemes:

about: URIs tackle about:blank and similar beasts.
ftp: URIs are another missing piece in the quest to finally get rid of RFC 1738.

Please review these drafts and send any feedback to the relevant IETF mailing lists — subscriptions are cheap, they cost you a working e-mail address. ;-)

So far nobody volunteered to finish the existing file: URI I-D (2005). There are some dragons with respect to different browsers and operating systems, notably IE vs. Firefox and Windows vs. Linux, but really, almost any file: URI scheme RFC based on the work for other schemes would be better than RFC 1738 vintage 1994 today. And you are not forced to emulate some negative records created by me for RFC 5538. I think file: and ftp: URIs are the last missing pieces in this puzzle. Check out the mailserver: (RFC 6196) and tn3270: (RFC 6270) URI schemes for pieces added in 2011.

Completely unrelated, and not deserving a separate article: I've recently added a CC-BY-SA license for this blog. That is mostly an experiment to show my support for Creative Commons and Wikimedia Commons as explained on my commons page with a fascinating (but long) video of a CCC lecture by Lawrence Lessig in 2006. Clearly I can't revert this license to some more restrictive form for existing pages, but maybe I'll change it to public domain in the future if I feel like it, or if somebody asks for it with a sound reason (from my POV).

Firefox 5 in W2K under windows 7 x64

2011-06-25T18:50:00.008+01:00

Now that is really something new (for me) in Windows 7 x64:

What you see is the bottom of a 1152×768 virtual Windows 2000 SP4 desktop with quick links for Firefox 5, IE6, KeditW 1.6, etc. The host system is a Windows 7 x64 SP1 home premium Sony VAIO (1600×900, ATI mobile Radeon catalyst 11.6). Sadly XP mode is not supported for home premium, or rather, I couldn't test a manual XP activation, because I have no XP license key.

Some invisible details:

VPC does not really work for guests older than XP SP3, you have to install the virtual machine additions 2004.
Virtual PC guy posted a picture of the real network card virtualized in VPC for up to four networks. For my purposes "internal NAT" allows me to share an existing mobile broadband WWAN connection. This is not the VPC default, change it in the VM settings while the VM is not running — hibernating might be good enough.
Without the VPC integration features I found yet no simple way to exchange guest and host files (including simple things like the clipboard), WebDAV on a remote hoster clearly doesn't qualify as simple.
Using virtual floppies (VFD) is a major pain with Windows 7 VPC, but once you have attached a VFD to a VM it sticks. You can even format the VFD.
I didn't know how to "capture" (if that's the correct term) a W2K VHD from a genuine system, and used a public VHD. That beast required a lot of work (tons of missing updates, removal of obscure "bars" installed by the publisher, adding A/V-software, full scan with the latest MSRT, Firefox, Flash, Silverlight, 7Zip, XnView, DirectX, Secunia PSI 1.5x, etc.).
Just in case: Yes, WU still works for W2K — only up to June, 2010 for OS + IE6 updates, but still for any MS Office stuff. Sadly the still working monthly MSRT requires manual downloads. Avira announced that they'll stop to support Avira A/V Personal 10 under W2K in July 2011, which is kind of stupid: Anything better than W2K can and IMO should use MSE.
Somewhat unrelated, normally you cannot get ATI Catalyst 11.6 for Sony VAIO from AMD, but I found an unrestricted official download link in an obscure forum. Same procedure as for direct Flash AX downloads without Adobe's GetPlusPlus ~~mal~~Adware.
The latest Intel PIU fails to install under W2K, and an older version (working for W2K) died with an obscure error in VPC. FWIW installing DirectX 9 in the virtual W2K worked, but of course there isn't much to accelerate in a VPC. My real W2K is far slower than the virtual W2K: The real box has 256 MB RAM, the virtual box has 512 MB, and the windows 7 host has about 3950 MB.
Sometimes the virtual W2K hangs and needs a hard reset (= close VM). I'm not sure when this started, among my suspects are Firefox 5, Avira Personal 10, and Secunia PSI 1.5x. It's also possible that my (failed) attempts to install the VPC integration features, or my inconclusive attempts to install the VPC 2007 VMA screwed up the VHD. There is a suspicious unknown device in the device manager, claiming to be working, and using IRQ7 — is this some kind of time synchronization? To break out of a restart hanging VM loop modify the VM settings to "unconditional close" or "always ask". Normally I'd prefer "auto-hibernate on close", but that doesn't help if the VM hangs.
While a VHD is not running or hibernating it is relatively simple to mount it as a virtual disk in Windows 7. Unmounting can be slightly tricky, but it's necessary to start any VM using the VHD.
Having fun with VMs I stumbled over a simple VFD-driver and virtual CD control panel for Windows x86 platforms. Check out Elby Clonedrive for serious virtual CD applications. Apparently the MS XP virtual CD controlpanel 21 works also under W2K.
At some point in time I'll have to grok the diskpart manual — it would be nice to associate .vhd with a simple attach/detach (mount or unmount) script without going to a command line or the device manager.

Shortcut icon mysteries

2011-06-06T20:06:00.004+01:00

Tweak My Blogger proposes to use three link relations to replace the default Blogger favicon with a custom favicon. Historical background:

There can be various link relations in the <head> element of HTML or XHTML pages, e.g., <link rel="search" … /> for OpenSearch descriptions. For favicons IE originally used the old Windows .ico format. That's a kind of container for related Windows .bmp images in various sizes and with different numbers of colours. A favicon .ico should include an image with size 16×16 and up to 256 colours, and it is not required to offer other sizes. If compatibility with say IE5 is not your main problem 32×32 or 64×64, and more than 256 colours, might also work: Applications are supposed to pick the best image offered in an .ico for their needs, and scale it up or down if necessary. If an automatically scaled down image does not work for your icon the .ico format allows to include optimized smaller versions, notably 16×16.

Years ago there was no proper MIME type for these beasts, but type="image/x-icon" was widely supported. Later Microsoft registered type="image/vnd.microsoft.icon for this purpose, and modern browsers are supposed to know this type as far as they support the odd .ico format at all.

Web servers might be still configured to associate .ico with image/x-icon, but that does not affect a correct image/vnd.microsoft.icon in the <head> of pages. Without a link relation IE simply tried to fetch a file favicon.ico from the relevant directory. For various reasons that was a bad idea, and modern browsers rely on explicit link relations instead of default locations for favicons and other purposes.

Other browsers and other platforms were not eager to support the odd Microsoft .ico format, but liked the idea of shortcut icons. A much better format is .png, but some old browsers including IE did not support it, or had issues with certain PNG-features. Somehow these differences resulted in two link relations, rel="icon" and rel="shortcut", for in essence the same purpose. Only rel="link" is registered at the moment. It is possible to list more than one relation as in rel="shortcut icon", and today that should be good enough if the favicon is an .ico.

If you seriously hate this image format try rel="icon" type="image/png". GIF instead of PNG is also okay, of course excluding animated GIFs. Other formats are either unsuited for a favicon, e.g., JPEG and WebP, or still less widely supported, e.g., SVG, which will be an ideal solution for most kinds of icons.

So far for the theory of the favicon business. The practice on Blogger and other sites can be slightly more complex. IIRC you can't use the name favicon.ico, just pick something else, e.g., href="http://example.org/my-icon.ico" if this really is an .ico with MIME type="image/vnd.microsoft.icon". Whatever Blogger does, the type="icon/ico" recommended in Tweak My Blogger makes no sense, and at least in theory (not tested) more than one link relation is unnecessary. My Blogger template contains the following line immediately before the <title>:

<link href='http://purl.net/xyzzy/xyzzy.ico' rel='shortcut icon' type='image/vnd.microsoft.icon' />

Sadly Google reader still shows the Blogger icon for the feed, and unsurprisingly a Google profile still shows the Blogger icon for a Blogger profile, but otherwise it works as expected. Maybe create a 57×57 PNG for I-Phones and I-Pads, and use the (unregistered) link relation rel="apple-touch-icon", it can't get odder. I have no I‑Panything to test this.

URL fragment vs. query part

2011-05-01T16:59:00.001+01:00

Has Google finally reached the ultimate Microsoft state, "breaking standards for fun and for profit" ? In a recent entry on the Google Operating System blog (not affiliated with Google) two screen shots show #fragments instead of ?queries. My attempt to post a comment had no visible effect, maybe comments are moderated or the blogger template is broken or whatever the problem might be, zero feedback to users is just wrong. Google Chrome lost the content of the comment form before I could reproduce it verbatim here, in essence it said:

In URLs a hash mark "#" introduces the optional fragment at the end of the URL, and a question mark "?" introduces the optional query part — also at the end of the URL, but before any fragment.

Fragments are the local business of the client (browser) and depend on the document (MIME type). Queries are the business of the server, e.g., many servers expect name=value pairs separated by ampersands "&", other severs also permit semicolons ";" as separator, and so on.

The query syntax depends on the scheme, here http:, the details depend on the server, but in both cases the overall URL syntax would require to percent-encode "#" if it is a part of the query not intended to start the fragment.

Notably fragments do not participate in redirections from one scheme or server to another scheme or server, the target location never sees the source fragment — browsers are not expected to transmit a fragment in requests such as HTTP GET.

In other words, something is wrong in the googlesystem screen shots.

Google - you get what you paid for

2011-03-18T03:06:00.001Z

With a few exceptions Google services have the unacceptable feature to be associated with an account forever. So if you test say "feedburner" once, and then decide that this is yet another data kraken and never use it again (for years), it will still be listed in the Google accounts dashboard. In theory you could delete the complete account including blogs, mails, images, videos, docs, sites, etc. — in practice folks trying this often show up in the blogger help forum whining about their lost blog and images.

Episode 1: I sent a complaint about "google moderator" to Google's privacy officer. This is actually an obscure HTML form creating no confirmation mail and no ticket number. After weeks my privacy complaint still got no reply. I'll test it again when I get around to it, complete with screenshots of the HTML form, as this isssue might require an intervention by Hamburg's privacy officer.

Episode 2: My entry in "places" is associated with an off by two photo in "maps". After a considerable amount of time I figured out how to report this issue. This actually resulted in a timely mail answer, but the mail came from a noreply bounce address offering no way to answer. My report was that they swapped the photos for house X and X+2. Sh*t happens, this was incorrect, they systematically show X+2 for X for the given street. I have no way to fix my semi-erroneous report, this is stupid. Never send noreply mail if you are not a mail-bot, this infuriates customers and users. Why should they use valid and valuable mail-addresses, when the other side uses bounce-addresses?

Episode 3: The new Chrome "apps" store apparently records all installed apps, even after they were removed. I submitted a complaint about this, because I do not recall where or when I permitted this violation of my privacy, and I certainly do not know how to withdraw this permission.

mailto: URLs

2011-03-10T00:40:00.003Z

If you think that mailto: URLs are rather boring in comparison with http: you have a point, most mailto: URLs in the wild are straight forward and simple. But the syntax specified in RFC 2368 was based on

the RFC 1738 URL specification (1994),
the old STD 11 in RFC 822 (1982).

Updating the mailto: specification therefore had to replace the old (and tricky) RFC 822 syntax by the new RFC 5234 ABNF in STD 68 published 26 years later (2008). The actual content of RFC 822 is the Internet Message Format as used in e-mail and now specified in RFC 5322, the successor of RFC 2822. For URIs we are now at STD 66 in RFC 3986 (2005) with various subtle differences from the eleven years older RFC 1738. You won't often see these differences, but clearly there was no such thing as IPv6 in URIs back in 1994. UTF-8 was rarely used, supported 31bit Unicode points, and permitted "overlong" encodings — not exactly the STD 63 rules in RFC 3629 (2003) we use today.

The general syntax for URIs in STD 66 and the specific syntax of e-mail addresses in RFC 5322 are not directly compatible, some characters permitted in addresses are not permitted "as is" in URIs and have to be "percent-encoded" based on the hex. UTF-8 encoding. It's a miracle that the new mailto: URI specification in RFC 6068 managed to close these gaps in 2010, twelve years after RFC 2368. There are lots of interesting examples in RFC 6068, my favourite is a clever use of In-Reply-To e-mail header fields. The complicated examples are also fascinating. If you create a complicated e-mail address your chances that it works anywhere in a mailto: URLs are now better. Well, at least it is specified, implementing it at the interface of browsers and mail user agents is another story. Well done, one "like" from me to RFC 6068.

Service Status Feeds

2011-03-02T18:06:00.001Z

My favourite tool for 2009 and 2010 — while I was mostly offline — is Google Reader, the feed reader offered by Google for "popular browsers" (read: IE6, FF2, or better), because it dutifully kept all my subscriptions, and still works like a charme.

In a Gmail blog article Google reported a bug in the Gmail service, apparently my account belongs to the 99.98% not affected by this issue. The article mentions an Apps Status Dashboard showing the published issues for various Google services including Gmail. Some years ago I would have added this page to my bookmarks, and checked it when something went wrong. Today it is much simpler to subscribe to the relevant RSS feed — hopefully this results in no updates when there are no known issues.

Two other examples of status feeds are Blogger and DynDNS.

RFC 6055: IAB Thoughts on Encodings for Internationalized Domain Names

2011-02-22T22:43:00.002Z

The new RFC 6055 is quite interesting, it discusses issues of IDNA vs. UTF-8 encodings in the DNS and other namespaces.

By definition any valid IDNA A-label, i.e., an ASCII compatible label starting with xn-- and following the IDNA rules specified in RFC 5890 and RFC 5891 (among others), has a corresponding Unicode U-label, i.e., a label containing non-ASCII characters. The IDNA encoding mechanism punycode tries hard to create short A-labels, because there is an upper limit of 63 octets per label, and another upper limit of 255 octets per FQDN DNS query consisting of zero or more labels.

RFC 6055 does not propose to use redundant UTF-8 DNS entries for raw U-labels in addition to the IDNA A-labels, so I guess there are too many cases where this would anyway not work, e.g., when an otherwise valid U-label consists of more than 63 UTF-8 octets.

When I get "a round tuit" I'll look into a now long expired EAI SPF draft, where using raw UTF-8 labels was the only viable solution for an EAI extension of SPF. Even if nobody implements this draft and/or if EAI never really makes it the last "tombstone" version should reference RFC 5321 (SMTP), RFC 5890, RFC 5891, and RFC 6055.

John Klensin is the author or a co-author of all RFCs mentioned here. I should check what else he published between RFC 5321 and RFC 6055, but it is nice to see that for at least one person in the world "RFC" still works as a "Request For Comments". :-)

MD5 1.7: verified errata

2011-02-01T15:37:00.003Z

All pending errata for the MD5 test suite are now verified by the IESG; thanks to Alexey Melnikov. The oldest erratum 749 about an MD5 example in RFC 2069 was submitted 2005-02-06 and verified 2010-07-11. Now I feel less bad about dropping out for two years.

For historical reasons the MD5 test suite is one of my REXX scripts still published as cmd-file instead of a rex-file. On an OS/2 box REXX is the default scripting language using file extension cmd. An ordinary OS/2 cmd.exe-shell script never starts with "/*"; any script starting with "/*" is interpreted as REXX.

Good old PC DOS 7 uses extension bat for command.com-shell scripts, and the text editor KEDIT uses kex for its macros; both also identify REXX by "/*" in line 1.

Please do not feed OS/2 REXX cmd-scripts to the Windows NT cmd.exe-shell; you would get numerous errors. For NT simply rename md5.cmd to md5.rex and let ooREXX interpret it.

WebHop DDoS

2011-01-30T20:30:00.000Z

DynDNS reports a DDoS attack against their Webhop services. This also affects xyzzy.webhop.info and xn--80akhbyknj4f.boldlygoingnowhere.org. The first address used to be a HTTP redirection to my mirror site, at the moment it is a redundant redirection to xyzzy.

The second address is used in some IDNAbis experiments with IRIs; the actual content are now subpages of xyzzy/home/googlets.

videotex

2011-01-28T03:47:00.001Z

Instead of a thank you to the news.t-online.de team in RFC 5538 I added the historic videotex: URI scheme to the IANA registry.

When the WWW with http: and https: started to replace gopher: and various videotex: systems (BTX, minitel, Prestel) this URI scheme allowed links to videotex: resources for browsers supporting it. It was a nice system for limited clients including DOS boxes, but the software to create videotex: pages was very expensive.

Is it only me, or do we really invent the wheel again and again ? Now we have WAP and a .mobi TLD, but what was wrong with telnet: and gopher: ? At least ftp: is still alive and kicking. And maybe news: survives.

If you know a public specification for an unregistered URI scheme, please submit it to IANA following the rules in RFC 4395 — in a nutshell this is a public review on an expert mailing list.

RFC 1849

2011-01-26T21:05:00.001Z

The famous son-of-1036 Internet Draft about the Netnews article format was published as historic

RFC 1849: "Son of 1036": News Article Format and Transmission

Son-of-1036 (also known as 1036bis) was the first Internet Draft I've ever seen 15 years ago. Admittedly I didn't know the exact difference between draft and RFC at this time, let alone the various RFC series and status flags.

RFC 1849 was immediately obsoleted by the work of the IETF USEFOR WG:

RFC 5536: Netnews Article Format
RFC 5537: Netnews Architecture and Protocols

Folks interested in Usenet & Netnews can be rather stubborn, one of the reasons why this WG needed more than a decade to finish its work (presumably a new IETF record). Thanks to Charles Lindsey, Russ Allberry, Ken Murchinson, Henry Spencer, Alexey Melnikov, Harald Alvestrand, Lisa Dusseault, and many other contributors on the USEFOR mailing list.

Thanks also to anybody helping to publish…

RFC 5538: The 'news' and 'nntp' URI Schemes

… while I was offline for more than two years. That this RFC took so long was clearly my fault. Anonymous credits I coludn't add in this RFC: The news.t-online.de team triggered my interest in an Internet Draft by A. Gilman about news:, nntp:, and snews: URLs. Later Paul Hoffman started the work to obsolete all old URL schemes in RFC 1738 by new RFCs based on the Internet Standard for URLs (STD 66, RFC 3986).

There are still some schemes in RFC 1738 not yet obsoleted by fresh RFCs, notably file:. Maybe Martin Dürst could finish his work on the awfully complex mailto: scheme, I have not yet checked this. If you find no old URL scheme to work on try dict:.

Hostile to privacy

2008-08-19T21:22:00.009+01:00

For those who don't remember it, hostile to privacy is a code word, it stands for Google since they acquired doubleclick. To get over this negative image Google recently promoted privacy to one of the 28 words on their basic search form.

As user of several Google services I have vague ideas which host names have to be blocked at the network level for privacy, and what is anyway pointless, to some degree I trust them. Sadly, visitors of my Googlet pages will get a cookie. I have no idea why, and it is not obviously mentioned in Google's privacy policy. Likely I'm violating German privacy laws with this practise, and "but it is the server, not me" is a lame excuse. Having said this, if you don't trust Google, what do you expect here on blogger, or even on my pages dedicated to googlets ?

Another issue are third parties, just because I trust Google to some degree doesn't mean that I delegated this job to them, otherwise they could start a certificate authority in their quest to achieve world dominance. After a discussion with the HTML5 editor I feared that my trust in network.cookie.cookieBehavior 1 could be wrong, and switched this to "always ask". The effects are quite interesting, e.g., a Google search for network world tries to set a www.networkworld.com cookie. I didn't click or touch anything on the search results, but this third party tries to set a cookie.

So far still okay, switching to "always ask" is one of many ways to reset network.cookie.cookieBehavior 0, and Google offers an opt out cookie. Clearly opt out is a weasel word for net abuse today, and Google prefers to be a part of the problem.

When I set network.cookie.cookieBehavior 1 again the same site still offers its third party cookie on the search results. After some tests always restarting the browser my impression is that this setting has no effect in my BBFH. But "always ask" works, and it can be limited to "ask once" for any given site.

Secunia Psi RC3 and Psi IM v0.12

2008-08-14T23:51:00.006+01:00

Secunia PSI RC3 0.9.0.4 is a brilliant tool to find insecure cruft on a box; it just helped me to get the latest XnView 1.94.2. But Psi is definitely a tool for folks with a vague idea what they need, e.g., it didn't tell me that some of my Sysinternals tools are not more up to date. I read the blog and decided to skip some minor fixes.

More serious, Psi still doesn't tell me that QT Lite 1.1.2 is insecure and past its end of live for W2K. A Real Alternative 1.60 Lite and a K-Lite codec pack 3.6.5 on my box were also horribly obsolete, but not reported by Psi. These unofficial alternatives are quite popular for various compelling reasons, but on W2K it's time to say good bye to QT including its lite alternative.

More on the funny side, and also reported again as missing, Secunia Psi still does not identify Psi IM, a jabber client, now at version 0.12. The release notes don't mention security issues in 0.11, they also didn't tell me that running install works as an upgrade (fortunately it did), and I still have no clue how to convince Firefox and/or W2K that Psi can handle xmpp URIs. If that is the case, the documentation is rather inconclusive.

Google mail security

2008-08-14T21:49:00.004+01:00

Inspired by an Heisec article about the fine art of stealing insecure cookies I've upgraded the mailx googlet to use https for its Google mail mobile redirection. The settings allow to configure http instead of https.

A new entry in my googlet zoo is firxt.mobile for the firxt mobile OpenSearch portal. Often mobile also means accessible over slow connections and no nonsense. The googlet should work everywhere, check it out. Hint, what you see as "use this URL" within the googlet contains apparently a configuration number. Just in case this number can be stored in the googlet settings, maybe it helps when you delete your cookie.

Leader of the pack

2008-08-08T02:37:00.004+01:00

Keeping Internet Drafts "active" with two updates per year is a subversive tactics, but apparently already in use: Version 02 and 04 of the pack URI drafts are identical. This draft cannot be published as RFC, because its proposed URI scheme syntax is in conflict with the generic syntax in STD 66 [RFC 3986]. This is not permitted in BCP 115 [RFC 4395] about the registration of URI schemes, as noted by the URI review expert.

The pack scheme has to be fixed before it can get a permanent registry entry. The current entry is provisional, and the unmodified draft could be an attempt to prevent the reclassification of this wannabe-scheme as historical. The greater picture is that the Open Packaging Convention is a part of ECMA 376, the fast track input for ISO 29500 also known as ooXML.

SPF EAI and IDN TLDs

2008-08-05T16:29:00.001+01:00

SPF, EAI, and IDN TLDs are not directly related, but I ended up as author of two related Internet-Drafts: SPF EAI is now at version 03, and the IDN test TLDs are already at version 11.

The new 2606bis draft with the eleven IDN test TLDs now also covers the corresponding nine IDN example labels. Reserving the nine IDN example TLDs is one of those procedural stunts. The contributor tricking me into this adventure does this all the time, it's scaring.

For what's it worth, there is also a relatively new OpenSearch draft, most of it still TBD. It depends on a cleanup of the IANA Atom link-relation registry adding the mostly identical HTTP link-relations and other missing pieces.

No plans to transition management of the root zone file to ICANN

2008-08-03T09:54:00.007+01:00

In a letter to the ICANN Chair published 2008-07-30 the U.S. DoC notes:

First, it is important to note that the IANA functions contract was not part of the JPA mid-term review and the Department views these as two distinct instruments. As the community is aware, the IANA functions contract covers the performance of a series of currently interdependent technical functions that enable the continued efficient operation of the Internet, including processing requests to change the authoritative root zone file. Implementations of changes to the authoritative root zone file are performed by VeriSign under the terms of a separate agreement with the Department.
The Department believes that it is important to clarify that we are not in discussions with either party to change the respective roles of the Department, ICANN or VeriSign regarding the management of the authoritative root zone file, nor do we have any plans to undertake such discussions. Consistent with public statements made by the United States governement starting in 2000 and reinforced by the 2005 U.S. Principles on the Internet's Domain Name and Addressing System, the Department, while open to operational efficiency measures that address governments' legitimate public policy and sovereignty concerns with respect to the management of their ccTLD, has no plans to transition management of the authoritative root zone file to ICANN as suggested in the PSC documents.

In an article about the All important IANA the author wrote: An ICANN that is completely privatised and does not behave well afterward also might easily be stripped from the IANA contract and therefore its grip on the root zone.

RxWhois IETF 72 edition

2008-07-26T00:48:00.005+01:00

Based on the latest DNSBL Internet-Draft rxwhois 2.1.2 now supports DNSBL queries for IPv6.

I've not yet found any black listed IPv6. The potential test entries ::FFFF:127.0.0.2 or ::2 are not listed on the nine pre-configured DNSBLs used by rxwhois.

The simple REXX code to transform an <IPv6address> into the reverse ip6.arpa format turned out to be more interesting than I thought; see procedure XIPV6 in the source… :-)

The classic SPF FAQ

2008-07-24T11:35:00.007+01:00

After years on the SPF help list it is interesting that some folks still have difficulties with the very simple idea behind SPF. Modified question to protect the innocent:

my mail is MX relayed via anti.spam.example and then on to good.isp.example, and good.isp.example is reapplying SPF as if anti.spam.example were the originating MTA.

Answer: If anti.spam.example forwards mail to third parties it is the originating MTA as far as SPF checkers at these third parties are concerned.

can anyone say whether SPF would normally have any problems with mail relay?

Answer: No problems with mail relays on the side of the sender or on the side of the receiver. SPF is in essence for the one critical hop where two unknown strangers (sender and receiver) meet.

Now if you introduce a second critical hop in the form of "forwarding to third party" this will nowhere work "as is":

S -> F -> R instead of S -> R

The sender S defines S-IPs permitted to send MAIL FROM S. The sender S has no reason to permit sending IPs of any forwarder F or forger F, in fact S has no idea who F is.

The receiver R checks that MAIL FROM S comes from one of the S-IPs as defined in the sender policy of S. Without intervention sending IPs of forwarder F will FAIL, as for sending IPs of a forger F.

There is no difference between "forwarding to third party" and "forging" wrt SPF. Of course there are some things a legit F or R could optionally do:

R could white list F as legacy forwarder (legit forger).
F could rewrite MAIL FROM S to S@F (like mailing lists).
F could store mails, and let R use say POP3 to fetch it.

If F and R do nothing about this situation – and that is perfectly allowed – R sees a FAIL, and hopefully rejects the MAIL FROM S actually sent from F.

After that F is obliged to send an error report (bounce) back to S. The sender S can then bypass the forwarder F and send the mail directly to R (if the bounce contained the real address at R).

If you think that there is a problem, then your problem is almost certainly with forwarder F, not with S or R.

It is up to you how you solve your problem with F. In my opinion an entity using the name anti.spam.example should know that forwarding mail "as is" to third parties is a part of the problem, YMMV.

Frisian frs vs. stq

2008-07-19T21:46:00.005+01:00

Before 2006 there was only one language tag for the Frisian languages, fy. 2005-11-16 ISO 639.2 identified fy as Western Frisian, adding frr for Northern Frisian, and frs for Eastern Frisian. So far it is clear, ignoring the detail that Northern Frisian consists of several rather different dialects.

After the publication of RFC 4646 the IANA language subtag registry now contains fy, frr, and frs. But not fry, this is an alpha-3 alias of fy, and the RFC 4646 rules are roughly that shorter alpha-2 ISO 639 codes win.

It is not obvious what the Y, R, and S in these subtags stand for, one plausible theory is that S stands for Seeltersk, the Frisian language spoken in Saterland; see the S marker on the map.

Interestingly ISO 639-3 uses stq instead of frs for this language, this is also the code used for the stq.wikipedia. Two alpha-3 ISO 639 codes for one language are rather strange, and there is another theory to explain this situation: frs is the code of the Lower Saxon nds dialect spoken in Eastern Frisia, the territory including Aurich at the A marker on the map. In that case the S might stand for Saxon.

The ISO 639-2 rules make it fairly difficult to get codes for nonsense, there must be some evidence what they had in mind when frs was registered. OTOH with codes such as eur and tlh Occam's razor might miss the point.

rxwhois 2.1.1

2008-07-14T02:33:00.007+01:00

The modifications in version 2.1.1 of rxwhois.cmd include the addition of whois.nic.asia, whois.nic.bi, whois.adsib.gob.bo, whois.registry.gy, whois.nic.ht, whois.nic.im, whois.kcce.kp, whois.nic.mq, whois.nic.org.mt, whois.nic.pr, whois.co.ug, and vunic.vu from the usual sources: IANA, whois-servers.net, guesswork.

Lost in space: whois.nic.ki, whois.nic.tel, and whois.cctld.uz. Some domain grabbers in the TLDs CM, FM, GD, KN, and PN were ignored. Eleven other whois.whatever.tld hosts were added, but these are (yet) no whois servers. The rxwhois manual page contains the version history up to 2.1 over the last six years.

HTML5: asinine + selfish

2008-07-09T04:36:00.005+01:00

HTML5 "reuses the well-known URL moniker in the most asinine way, and blames the other standards (which he thankfully has no control over) for not supporting all of the possible crappy raw data that could be input in an HTML attribute."

"That's why this whole discussion about creating new identifiers and new protocols in HTML is a total waste of time — the rest of the world does not want it and will not allow it to be published as HTML5. Pound the sand all you like; the network standards will not change because they are designed to support everyone's needs, not just the selfish desires of a very small set of browser developers."

So far from a co-author of RFC 3986 (the Internet Standard STD 66 for URLs) and of RFC 2616 (the currently revised HTTP specification) in one of the various HTML5 flame wars.¹

Fast track IDN ccTLDs

2008-06-27T00:46:00.006+01:00

There will be hundreds, thousands, or even millions of new top-level domains, from the variety known as gTLDs. "ICANN is working towards accepting the first applications in the second quarter of 2009", as noted in their process FAQ.

This also affects the "fast track" introduction of internationalized top-level domain names for territories with a country code, also known as IDN ccTLDs. "The Board intends that the timing of the process for the introduction of IDN ccTLDs should be aligned with the process for the introduction of new gTLDs", stated in an ICANN Board resolution.

Enough time to fix the toplabel bug in RFC 1123, document the IDN test TLDs as reported here ten months ago, and get ready with IDNAbis.

Lost + found: 268,435,455 free IPs

2008-06-12T16:59:00.007+01:00

Three Internet Standards, one of them historic, RFC 3330, an RFC 3330 erratum, and three Internet Drafts entered the battle over 6.25% of all IPv4 addresses:

RFC 3330 lists special IPv4 addresses, among others 240.0.0.0/4 also known as class E. Allegedly "reserved for future use" on page 4 of the historic RFC 1700, formerly known as STD 2. The notation 240.0.0.0/4 stands for all IPv4 addresses starting with the four bits 1111, hex. F0 is decimal 240.

Page 4 of RFC 1700 actually belongs to an introduction with basic terms copied from RFC 1122, a part of STD 3. In fact RFC 1122 specifies 0.0.0.0/8 and 127.0.0.0/8 as listed in RFC 3330, but not 240.0.0.0/4.

RFC 1122 and the RFC 1700 intro only mention class E, but it is specified in RFC 1112, a part of STD 5. Apparently RFC 3330 got this wrong, RFC 1700 by itself did not register anything at all, it was a snapshot of various IANA registries with references to registrations of all "assigned numbers".

As the future of IPv4 is now before IPv4 is replaced by IPv6 the question is what to do with these 6.25% of all IPv4 addresses "reserved for future use" minus one 255.255.255.255 specified in RFC 1122. Three Internet-Drafts fight over it:

IANA proposes to keep class E as is, because RFC 1700 said so.
CISCO proposes to unreserve these 268,435,455 IPs reserved in RFC 3330.
A third draft suggested to reserve these IPs for private use updating RFC 3330.

Ignoring the expired third draft as bad idea it will be interesting how the IETF manages to free this huge pool of IPv4 addresses while the rest of the world is busy to upgrade their hard- and software to IPv6. Apparently RFC 1112 does not do anything with class E apart from reserving it, it is about class D 224.0.0.0/4 multicast addresses. Or as RFC 1812 puts it:

The Class D (IP Multicast) and Class E (Experimental) address spaces are preserved, although this is primarily an assignment policy.

Update 1, it's a bit more complex: RFC 988, a predecessor of RFC 1112, created the 224.0.0.0/4 multicast addresses as class D out of the in RFC 960 reserved 224.0.0.0/3 block, the remaining 240.0.0.0/4 block got the name class E in RFC 990 obsoleting RFC 960. After that RFC 997 updated RFC 990 and listed the "Internet numbers" including class E, while RFC 1010 obsoleted RFC 990 and listed the "assigned numbers". RFC 1010 was a predecessor of RFC 1700 over many steps, RFC 997 was the predecessor of RFC 1166. In other words, RFC 1112 specifies class D, and RFC 1166 explains the remaining class E. RFC 1166 is only "informational", that's good news from the procedural side.

Update 2, a fourth draft was published: T. Savolainen proposes to use class E for hosts indicating that they can handle this , e.g., for dial-up connections.

Blogger trackback form

2008-05-08T10:37:00.009+01:00

Encouraged by an immediately working backlink for the last post I added an experimental trackback form to the template of this blog, it allows manual trackback pings:

<!-- trackback form --><b:if cond='data:blog.pageType == &quot;item&quot;'>
 <b:if cond='data:blog.title != &quot;&quot;'><!-- trackback blog_name -->
 <b:if cond='data:blog.pageName != &quot;&quot;'> <!-- trackback title -->
 <form action='#requires.javascript' id='track' method='get' name='track'
       onsubmit='document.track.action=document.track.back.value;
                 document.track.method=&quot;post&quot;' target='_blank'>
 <input expr:value='data:blog.url' name='url' type='hidden'/>
 <input expr:value='data:blog.title' name='blog_name' type='hidden'/>
 <input expr:value='data:blog.pageName' name='title' type='hidden'/>
 <input name='back' size='50' type='text' value='http://delorie.com:81/'/>
 <input type='submit' value='ping'/></form>
</b:if></b:if></b:if><!-- end of trackback form -->

Apparently you can add this form to any type='HTML' widget after going to Layout: Edit HTML and checking Expand Widget Templates. I used #track:target { display: block; } in the CSS to show the form only when it's the target. Text mode browsers will see it anyway, but maybe they have no JavaScript.

I ✅ Unicode

2008-05-08T04:33:00.011+01:00

Some days ago Mark noted on the Official Google Blog that UTF-8 finally managed to be more popular than US-ASCII, I found it on the Q&A Weblog.

Actually it is not that impressive, it merely shows the Unicode conspiracy at work, UTF-8 is designed to replace US-ASCII in the next decades, one encoding to rule them all.

But it is fascinating to see the usual suspects working together on IDNAbis, another important step in the master plan, among others Vint as author of RFC 20, Mark as co-author of UTF-7 when UTF-8 was not yet ready for prime time, Harald as author of RFC 2277, and John who recently added two critical missing pieces to the puzzle:

RFC 5137 - ASCII Escaping of Unicode Characters (BCP 137)
RFC 5198 - Unicode Format for Network Interchange

Hamburg-Web

2008-05-02T13:57:00.008+01:00

An attempt to add this blog to HH-web.de finally worked, while I was at it I created a HH-web search googlet and learned some new CSS tricks.

For years I used hreflang="de" in links to German sites on English pages or vice versa, without visible effect, and likely no effect at all. Now I found a way to do something with it:

:link[hreflang]:hover:after {
     content: " " attr(hreflang); font-weight: 100;  
     vertical-align: super; font-size: xx-small;   }

This construct matches links with a hreflang, and displays the value as small as possible at the end of the link while the mouse hovers over it. Test it with the HH-web link above.

Another nice CSS feature is :target, not displaying the ugly blogger dashboard is one thing, but now I can also get it back when I need it with an invisible link in the upper left corner of this blog.

Googlet gallery

2008-05-02T12:52:00.004+01:00

After a year one of my googlets is apparently used by some folks, great, it encouraged me to work on the whole set, add some settings such as the help language on the PDA LEO googlets, and create a gallery.

New entries are googlets for the German and English Heise whois for older browsers without OpenSearch support. For a different preview see googlemodules. No, I've not given up on the command line rxwhois client, but allegedly some users don't like command lines... ;-)

Another new entry is a MetaWikiHelp googlet, the Wikipedia help often is not good enough when working on other MediaWikis with different features and rules.

Custom search googlet

2008-04-12T05:47:00.007+01:00

Transforming Google CSEs into OpenSearch descriptions is simple, even when Firefox needs some proprietary <moz:SearchForm> magic to get it right.

For older browsers not supporting OpenSearch each CSE has its own default gadget for uses on iGoogle, blogger, and elsewhere. I created a smaller configurable tiny CSE variant, to use it edit the cx-number, e.g., 001904119753490578822%25:zvsejdqm-pw for the xyzzy-CSE.

CSEs hosted by Google have one issue, creators cannot easily publish the source for users wanting to know what the CSE really does. Here's a snapshot of the sites currently covered by the xyzzy-CSE:

alvestrand.no, arkko.com, ecma-international.org, gmane.org/gmane.ietf.* (ditto org.w3c.*), iab.org, iana.org, icann.org, icir.org, iesg.org, ietf.org (excl. lists), imc.org, irtf.org, isc.org, isi.edu, isoc.org, itscj.ipsj.or.jp/ISO-IR, jabber.org, langtag.net, omniplex.blogspot.com (this blog), openspf.org, postel.org, rfc-editor.org, rfc-ignorant.org, rsa.com/rsalabs, standardstrack.com, unicode.org, w3.org, xml.resource.org, xmpp.org, and my own xyzzy mirrors (plus googlets).

MD5 1.6: POP3 and UUID

2008-03-08T09:04:00.004Z

Version 1.6 of the MD5 test suite covers RFC 5034 (Digest-MD5 for POP3), RFC 4122 (UUID version 3), and an old RFC 1910 example. The POP3 example belongs to the MD5-sess series.

The UUID example is apparently wrong, it swaps the byte order unnecessarily. Examples published by python.org and ossp.org agree with that theory, I'll submit it as erratum.

MD5-sess and RFC 5090

2008-02-22T09:08:00.004Z

RFC 5090 was published, it fixes some problems with the MD5 examples in RFC 4590. Version 1.5 of the MD5 test suite now contains the new RFC 5090 RADIUS examples.

The updated test suite contains a new procedure AUTHTTP for the RFC 2617 idea of MD5-sess noted in the errata. The old DIGEST procedure uses the RFC 2831 MD5-sess algorithm. Six new test cases cover MD5-sess examples published in RFC 2831 and 4643 using the RFC 2617 algorithm. Hopefully this will be documented in an RFC moving RFC 2831 to HISTORIC.

Hamburg's vote on ooXML

2008-02-18T02:58:00.007Z

Microsoft's quest to forge the one ring will soon reach a critical stage. In the BRM on OfficeOpen XML a so-called German delegation will have six votes, one of them the tax authorities of the city of Hamburg. In 2007 DIN supported ooXML in a controversial procedure, with Hamburg's fiscal authority under the yes votes.

Historical background, Hamburg was once the home of StarOffice, a name still used for Sun's OpenOffice version, closely related to what later became ODF, an ISO standard roughly covering what MS Office 2007 as so far only potential ooXML implementation offers. Of course MS Office isn't free like OpenOffice, and with about 6000 pages the ooXML draft is considerably more elaborated than ODF.

It is not obvious why Hamburg's fiscal authority supported a draft allegedly identifying 1900 as leap year, where dates before 1900 don't work. As Excel simplification that is funny or even acceptable, but Excel is a commercial product and no international standard. Just one of many ooXML issues, maybe addressed in the additional 2293 pages submitted for the BRM.

It fits that the citizens of Hamburg get a chance to elect a new parliament one day before the BRM. Related, Google doesn't like ooXML, their German HQ is in Hamburg, and they got no vote in the relevant DIN committee.

Patch that zeppelin

2008-02-13T18:49:00.006Z

XML is ten years old, good to know, waiting for the monthly W2K patch orgy to finish I used the time for my education. The worst update this month are 36 MB for Adobe Reader 8.1.2.

Also ten years old is RFC 2277, the IETF Policy on Character Sets and Languages. This is a nice piece of RFC number magic, RFC 2278 used to be the IANA Charset Registration Procedures, now 2978, and RFC 2279 used to be UTF-8, now 3629 (STD 63). The magic also works backwards, the predecessor of 2279 was 2044, and RFC 2045...2049 is MIME. The MIME type registration procedures used to be 1590 next to RFC 1591, i.e. what is today ICANN. After ten years they are now getting serious with I18N for domain names and other missing pieces in this puzzle.

RealPlayer BadWare

2008-02-01T16:35:00.004Z

RealPlayer 10.5 and 11 were identified as BadWare. The unofficial Real Alternative 1.75 claims to use 6.0.12.1662 components, apparently not affected by various 10.5 vulnerabilities.

STD 68: ABNF (RFC 5234)

2008-02-01T10:33:00.001Z

The IETF created the sixth Internet Standard (STD) in about 50 months, STD 68 specifies the ABNF used in many RFCs:

STD 68, RFC 5234, 2008-01: ABNF
STD 67, RFC 4506, 2006-06: XDR
STD 66, RFC 3986, 2005-01: URIs
STD 65, RFC 3551, 2003-07: RTP/AVP
STD 64, RFC 3550, 2003-07: RTP
STD 63, RFC 3629, 2003-11: UTF-8

The jump in the publication dates (2003-11 vs. 2003-07) is an effect of promoting RFCs 3550 and 3551 to STD without modification. STD 62 about SNMP version 2 published 2002-12 is a set of eight related RFCs.

QuickTime security flaws

2008-01-16T11:52:00.001Z

In a SANS Diary entry they forgot to mention that there is no secure version for W2K, affected users should uninstall QuickTime, well, quickly. Products depending on QuickTime and claiming to be W2K-compatible could be considered to be broken, IANAL.

Having said this it is already rather tricky to find the 2007-11-12 QT Lite 1.1.2 for W2K with the 7.2.0.245 components, the download for this link is a damaged file, and an apparently working file was removed from where I found it. QT Lite 1.1.1 and QuickTime Alternative 1.90 still use the 7.2.0.240 components, and all these unofficial variants likely inherited the security flaws of the corresponding official versions.

Hunting W2K tools can be a pain, I try to stay on Microsoft's sites for security reasons. Check out the Petri IT Knowledgebase if you're looking for say a W2K-version of NTrights.

ICANN Ombudsman

2008-01-10T06:32:00.000Z

After stumbling over a DomainTools Blog (1) entry discussing a new low in NetSol's (2) business practises, I tried to send an inquiry to the ICANN Ombudsman. That's a Web form with six required fields, but it won't let me submit it for no obvious reason. Fine, I can post it here:

Incident date	2008-01-08
Registrar	Network Solutions, LLC
Description of act, omission or decision	Please take a look into (1) and (2)

RFC 5064 Archived-At

2007-12-23T00:24:00.000Z

Great, RFC 5064 about the Archived-At mail and news message header field got its number, just in time to consider this as an early xmas gift ;-)

Fresh opensearch & google gadgets

2007-12-20T09:52:00.003Z

New Opensearch descriptions:

Google Code custom search
en×de translations by LEO
es×de translations by LEO
fr×de translations by LEO

New googlets:

About Flash googlet (version, links, search)
Atomic clock googlet (JAVA applet of the PTB)
Tiny map search googlet (local search)

The tiny map location can be set, default 53.55, 9.99.

RFC errata

2007-11-08T11:34:00.000Z

Various pending RFC errata have been published recently, among others for RFC 2069, 2822, and 4408. The RFC editor might soon offer a Web form for submissions as outlined in an Internet Draft.

The 2069 erratum resulted in an editorial update of my MD5 test suite.
The 2822 erratum was already covered in the 2822upd drafts.
The 4408 erratum is actually a link to the OpenSPF errata page.

Sorbian and Frisian use the Latin script

2007-11-07T11:14:00.000Z

The language subtag registry defined in BCP 47 was updated, it now contains the new region codes BL & MF and Suppress-Script: Latn for the Frisian languages frr, frs, and fy, for the Sorbian languages dsb and hsb, and for the Low and Swiss German languages nds and gsw.

The mis entry was updated some months ago in ISO 639, its description is now uncoded languages. I've created new experimental XML versions of the registry, for other formats check out the Language Tags site.

Lever dood as slaav ;-)

ABNF, Archived-At, News, and NNTP

2007-11-05T08:50:00.000Z

RFC 4234bis was approved, we'll soon see a new Internet Standard (STD) about ABNF, the syntax used in many RFCs. An RFC defining the Archived-At header field in e-mail and news was approved earlier while I was in essence incommunicado after a system crash... :-(

I've submitted new versions of the news and nntp URI draft, adding an appendix with a detailed example about the relations between Archived-At, Message-ID, Xref, news-, and nntp-URLs. Two typos in version 06 fixed, this draft reached a point where it's easier to spoil it than to improve it.

Broken validators

2007-11-02T14:32:00.004Z

Popular validators like the WDG and the W3C validator unfortunately still accept various kinds of broken URIs not limited to unencoded IRIs as "valid". For the W3C validator that's a known bug.

Admittedly it's almost impossible to fix this bug based on a DTD, renaming %URI; as in the related XML schema anyURI to %IRI; in the DTD has the same effect as renaming it to %FOO; for DTD-validators, the datatype is still CDATA, or in other words (almost) anything goes.

Hopefully even DTD-validators will be fixed really soon to check URIs. Broken URLs are abused for attacks, ironically that was a side effect of better URI tests, several applications failed to check the generic RFC 3986 syntax. All valid URIs match this generic syntax, scheme specific URIs are proper subsets of the generic syntax. URI "producers" including MediaWiki as well as URI "consumers" including validators have to get this right, otherwise bad things happen.

The folks at validome hope that they'll get this right soon, schema validators have an advantage. They already identify the IDNwiki and its E-mail test as invalid, big oops for accessibility tests (IANAL).

Update: The IDNwiki pages were fixed 2007-11-21.

New TLDs BL, MF, and TEL

2007-10-22T11:07:00.001+01:00

Version 2.1 of rxwhois knows the new TLDs BL, MF, and TEL. The corresponding whois servers are not yet running, but whois.iana.org already supports these TLDs. The corresponding region codes BL and MF will be added to the IANA language subtag registry in the next weeks.

Now rxwhois also supports the eleven IDN test domains. The test started 2007-10-15, but it took me until yesterday to fix a stupid bug.

The worst issue so far from my POV is that popular XHTML validators like the W3C validator don't check the URL syntax in attributes like href="URI", see bug 4916. Many users will be misled to create invalid pages with "unencoded" IRIs in document types like HTML 4.01 and XHTML 1.0, where that's not allowed.

RFC 2617 vs. 2831 md5-sess

2007-09-28T23:08:00.000+01:00

Good news first, RFC 4590bis (approved, still waiting for its number) will fix the Digest-MD5 examples in RFC 4590. I've updated the MD5 test suite using the fixed examples.

While I was at it I've also updated the RFC 3797 code to work for the NOMCOM 2007 case. The entropy limit 30 was too restrictive, 38 is good enough for MD5, 10^38 < 2^128.

Now the bad news, the issue with two md5-sess examples in draft smith-sipping-auth-examples might be in fact precisely what RFC 2617 says, as reported in a semi-official erratum. If that's correct the md5-sess in RFC 2831 would be different. Hopefully draft melnikov-digest-to-historic will shed some light on this before it moves RFC 2831 to historic. For more about this see the IETF SASL WG mailing list.

For now the MD5 test suite still uses only the binary x2c(HA1) form instead of the hex. HA1 form in its md5-sess calculation.

HTML PUBLIC "-//IETF//DTD HTML i18n//EN"

2007-09-23T13:56:00.000+01:00

Admittedly RFC 2070 is old, and its status is historic. But it was the first HTML specification with I18N based on UNICODE, and the last HTML specification published by the IETF.

So far its DTD had to be extracted manually from the RFC, now IANA hosts an official master copy with the public identifier urn:ietf:params:xml:pi:-:IETF:DTD+HTML+i18N:EN. The urn:ietf:params:xml:pi registry was created by RFC 3688 for DTDs developed by the IETF. Of course HTML i18n is still SGML, not XML, but its DTD is now the first registered IETF DTD.

If you have old HTML i18n documents you can use an URL of this DTD as system identifier like this:

<DOCTYPE HTML PUBLIC "-//IETF//DTD HTML i18n//EN" "http://www.iana.org/assignments/xml-registry/publicid/html2070.dtd">

ftpsynch.rex

2007-08-23T14:03:00.001+01:00

Many OS/2 SAAREXX programs work almost as is under WindowsNT ooRexx, the RexxUtil functions are similar, the WindowsNT CMD shell is similar, and the RxSock interface is almost identical. Some OS/2 RexxUtil functions are not yet or not more supported under WindowsNT, e.g. SysGetMessage, SysProcessType, and SysQueryProcessCodePage.

Scripts I've tested under W2K after renaming *.cmd to *.rex include popstop2.cmd, dir2html.cmd, and sitemap.cmd.

Unfortunately ooRexx doesn't come with RxFTP.dll, but it offers the RxFTP.cls class. I've created a new ftpsynch.rex based on ftpsynch.cmd for ooRexx, for details see ftpsynch.htm.

rxwhois 2.0.5

2007-08-21T13:43:00.000+01:00

Version 2.0.5 of rxwhois.cmd now also works as WindowsNT ooRexx script, just rename it to rxwhois.rex. I've adopted additional local character sets from utf-8.cmd including codepage 923 (ISO 8859-15, Latin 9) and 878 (KOI8-R), but I've only tested 858 (pc-multilingual-850+euro) and 1252 (windows-1252).

As always some whois-servers for ccTLDs had to be updated, for details see the source. After a system crash of my OS/2 box in June I was unable to check anything beyond the whois servers already known by rxwhois.cmd, whois.iana.org, and whois-servers.net. Just for fun I've added the eleven IDN TLDs for the test beginning in September 2007:

Arabic xn--kgbechtv
Persian xn--hgbk6aj7f53bba
Chinese, simplified xn--0zwm56d
Chinese, traditional xn--g6w251d
Russian xn--80akhbyknj4f
Hindi xn--11b5bs3a9aj6g
Greek xn--jxalpdlp
Korean xn--9t4b11yi5a
Yiddish xn--deba0ad
Japanese xn--zckzah
Tamil xn--hlcj6aya9esc7a

Known issue: rxwhois.cmd expects UTF-8 as charset of whois servers, but whois.iana.org uses Latin-1 for at least one TLD ht (Haiti) entry. The IANA folks told me that they'll intend to use ASCII data for the eleven IDN test TLDs.

OpenSearch descriptions for Google CSEs

2007-07-27T18:38:00.003+01:00

It's quite simple to create opensearch descriptions for any existing Google CSE. Here's an example using the mozillaZine KB CSE:

This CSE is identified by cx=003258325049489668794:ru2dpahviq8. The &cx=-parameter is used in links and anything else related to this CSE. The left hand side 003258325049489668794 is related to the Google account and the up to 5000 annotations (e.g. sites and URL patterns) associated with this account. The right hand side ru2dpahviq8 is related to the actual CSE context including details of its layout, references to the associated annotations also known as background labels, etc.

It's not my CSE, I can ignore most technical details only relevant for the CSE creator. One detail is probably important, this CSE uses FORID:1 unlike my own CSEs with FORID:0. The value is visible in the monstrous URL of search results, it's a part of the &cof= parameter.

Most other layout details noted in &cof= are set by Google on the fly based on the CSE definition a.k.a. context. For my own CSEs I force LP:0 and AH:center with &cof=FORID%3A0%3BLP%3A0%3BAH%3Acenter, but that's arguably pointless, opensearch only works with Firefox 2, IE7, or better, and these browsers have no issues with the default LP:1 logo position and AH:left aligned header on result pages.

CSEs refuse to return &output=xml or &output=xml-no-dtd results, therefore the opensearch description needs only one type="text/html" template. Just in case I added...

<SyndicationRight> limited </SyndicationRight> <Attribution> Google CSE by Jason Kersey </Attribution>

...anyway, after all the search results are Google results. In this case results filtered and rearranged as defined by the CSE creator Jason Kersey. With up to three searched sites in a CSE Google allegedly also shows its supplemental results.

Putting it all together I arrived at this mozillazine opensearch description. I've no clue how and where Firefox or IE7 might use the Tags or Description, most likely these details are irrelevant for search results on ordinary type="text/html" pages. The validator wants a Query example as specified by opensearch.org, just for fun I picked about%3Aconfig.

One last detail, the icon, fortunately kb.mozillazine.org has a type="image/vnd.microsoft.icon" 16×16 favicon needing less than 10 KB, so this should work as is (http:-URL instead of data:-URL) for Firefox. It's tricky to get the icon right with *.googlepages.com, the Google Page Creator won't let you have your own favicon.ico. Just use another name.

One way to use opensearch descriptions is to add a link in the header of (X)HTML pages. The title in the link should match the ShortName in the description, otherwise browsers won't know if the corresponding search is already installed. I've done that here in my blogger-template:

<link rel="search" href="xyzzy.a9.xml" type="application/opensearchdescription+xml" title="xyzzy" />

Another way is the window.external.AddSearchProvider function, Firefox 2 users can then simply click on the link to copy an opensearch description to their browser. I haven't tested IE7, maybe it uses the same method, i.e. "copy description". Last step, test this OpenSearch description with Firefox 2 or better.

For another example see my googlets page.

Simple REXX mailto script

2007-06-11T21:32:00.000+01:00

Yet another mailto command line tool, rxmailto.cmd v0.1 can so far send one text mail to one receiver via a Mail Submit Agent (MSA) at port 587 supporting SMTP AUTH with CRAM-MD5 and 8bitMIME. That's the minimum I could get away with after the spam flood finally drowned my old mailbox.

Various details are far from perfect, e.g. if a run of words with non-ASCII characters in the subject is longer than 56 octets the subject encoder will emit a folded line longer than 76 characters, and that's not permitted by RFC 2047. On the other hand the script won't break UTF-8 characters in the subject for platforms with UTF-8 as local charset. You get what you pay for, less than 30 KB. ;-)

MD5 test suite 1.2

2007-06-09T05:40:00.000+01:00

The MD5 test suite version 1.2 finally supports streaming and bit string input:

   hash = MD5( bytes )          ==> MD5 of an octet string
   ctxt = MD5( bytes, '' )      ==> init.  new MD5 context
   ctxt = MD5( bytes, ctxt )    ==> update old MD5 context
   hash = MD5( /**/ , ctxt )    ==> finalize   MD5 context
   hash = MD5( bytes, /**/, n ) ==> MD5 of n zero-fill bits
   ctxt = MD5( bytes, ''  , n ) ==> init.  MD5 bit context
   ctxt = MD5( bytes, ctxt, n ) ==> update MD5 bit context

Also added: APR1 can determine the hashed passwords used by BSD and Apache htpasswd. This is a function also offered by openssl passwd -1 and openssl passwd -apr1, for details see a manual of the openssl command line tool.

sitemap.cmd 0.3

2007-05-20T19:44:00.000+01:00

FWIW I've added the schema magic to sitemap.cmd (0.3), adjusting the documentation of the REXX ftpsynch wannabe-content management system.

Apart from being a bit longer and overwriting siteold.xml the new version now passes XML schema validation. Caveat, don't use its buggy text/html output at the moment.

Unrelated, Google's page creator rewrites an uploaded sitemap 0.90 automagically into a sitemap 0.84 removing all <lastmod> elements. Or rather it did that last week for e.g. this sitemap, maybe it's one of the experimental features.

md5.cmd 1.1: Auth Digest + Digest-MD5

2007-05-20T17:06:00.000+01:00

The IETF SASL WG recently decided to drop the RFC 2831bis draft from their agenda. Therefore I've removed the code handling <quoted-pair> (backslashes) from the MD5 test suite 1.0 (REXX script).

RFC 4590 contains four examples for Auth Digest. That's in essence the same as Digest-MD5 defined in RFC 2831, only based on the older RFC 2617. The examples were apparently copied as is to RFC 4590bis drafts. I've added the 2*2 (INVITE+rspauth, GET+rspauth) examples to md5.cmd (1.1).

The RFC 4590 examples still fail in my MD5 test suite, or rather my attempt to guess the used password failed. There's also an oddity in these examples not yet supported by the REXX script:

RFC 2617 states that a client sending any qop= parameter, for the RFC 4590 examples that's qop=auth, MUST also send a cnonce= (client nonce) together with a NC= (nonce counter). In the RFC 4590 examples the client doesn't do that, causing a trap in my REXX script.

There are two plausible ways to fix this, either use the RFC 2069 fallback algorithm, or simply omit the missing NC and CNONCE. In simplified REXX the second solution would be:

 return MD5( HA1 || ':' || NONCE || ':auth:' || MD5( XURL ))

The first (2069) solution would use a colon : instead of :auth:. The "official" RFC 2617 string instead of :auth: is:

 ':' || NC || ':' || CNONCE || ':' || QOP || ':'

Other variants of what RFC 4590 actually wants could be to use an empty CNONCE with a dummy NC in the direction of :00000001::auth:. As always Digest-MD5 is messy.

Related, an old 2069-erratum still rots in the pending errata mbox. I'm now confident that the 2069-code in md5.cmd works at least with the IETF tools server. I've not yet submitted an erratum for RFC 2983, three out of four 2983-examples are fine.

LP Logo Position, AH Align Header

2007-05-14T01:36:00.000+01:00

Some troubles with Google's custom search engines (CSE):

The watermark "branding" fails miserably with Javascript 1.1. The code doesn't check this old version resulting in garbage with browsers still using it. I always disable JS 1.1, but I can't ask visitors of pages using my "xyzzy" CSE to disable JS 1.1 before, they wouldn't know what it is. Now I use the ordinary "branding" with one of Google's six CSE logos.
The default position of the search form on the result page is "left" set by AH:left. Fans of the old free site search form know this "Align Header" parameter, it's (kind of) documented on my lab page. It's also straight forward to modify it, add ;AH:center to the cof=FORID:0 parameter, where 0 might be something else depending on the used form.
The free site search result pages show my logo immediately above the search form. The used parameters L: for the logo URL and S: for the site URL are still the same for CSEs, it's only not more necessary to specify all these odd values as part of the cof= parameter, Google inserts them on the fly. Unfortunately it also uses some CSS magic (style sheets) to get this right, failing miserably with browsers not supporting CSS. After some experiments I found the culprit: There's a new LP:1 "Logo Position" , this has to be disabled by LP:0 to get the desired effect.

Putting it all together I ended up with this form input:

 <input type="hidden" name="cof" value="FORID:0;AH:center;LP:0" />

For an example see this form, I'll update the other forms later. Of course I'll need my own "googlet" for this CSE. But the normal "add to Google" CSE gadget is anyway far too big for my taste.

Inline googlets

2007-05-04T17:44:00.002+01:00

I've moved leo-dict.xml and leo-gghp.xml to Web space hosted by Google. Both offer a simple form for a service provided by LEO, English to German (or vice versa) translations.

leo-dict.xml uses Content type html for Googlets with target="_top" as required by LEO, resulting in an <iframe> on an iGoogle-page or wherever it's used. Of course this only works with browsers and devices supporting <iframe>.

leo-gghp.xml uses Content type html-inline without target="_top", resulting in an ordinary search form on an iGoogle-page working with any browser, at least it works for me. There are various disadvantages of html-inline, e.g., users are asked if they really trust that this Googlet won't screw up the layout of their iGoogle-page (it could with some JavaScript magic). On the other hand html-inline could also work on PDAs.

Fictitious u+1E9E character endorsed by German Home Office

2007-05-04T15:19:00.001+01:00

A misrepresentation of some ß uses in upper case head lines, on books, and on tombstones resulted in a proposal to add an "upper case ß" to Unicode as code point u+1E9E (PDF).

I've got a warning that this PDF might crash some PDF viewers, but exceptionally AcroReader 3 survived this attack.

The next likely step could be demands to permit ß in I18N domain labels because it suddenly got an upper case variant.

Of course there's also the "minor" problem of upgrading fonts and software for a fictitious "upper case ß" worldwide, as far as they support German.

Googlet

2007-04-15T17:17:00.001+01:00

The Google Gadgets are a cute idea, they allow to encapsulate Web forms into pieces of XML, which then can be added to personal Google start pages and similar services.

A bit like the old favlet/bookmarklet concept based on javascript-URLs, but with more features like user preferences. I'm not yet sure where they store user preferences, on their servers or in cookies.

The name add.gif for the link icon isn't very intuitive, I renamed my copies to googlet.gif. My second experiment after the xyzzy "custom search engine" is a LEO dict search form, but so far Google claims that it's unavailable or empty when I try to add it.

sitemap.cmd

2007-04-15T16:42:00.000+01:00

For users of sitemap.cmd, a REXX script generating sitemaps, the improved sitemap specification doesn't require an update.

The location of the sitemap can now be given in a robots.txt file, see their protocol page, example:

 Sitemap: http://www.xyzzy.claranet.de/sitemap.xml

Any services above finding the sitemap like Google's webmaster tools still require some form of submission proving that the submitter has write access on the submitted site.

Syndic8

2007-04-15T16:06:00.000+01:00

History, this isn't my first attempt with a "mail2blog" service, for two years mailbucket.org did what I wanted.

Unfortunately guessing the submit address for a mailbucket feed was simple, and the spammers found it. Other services I tested (including blogger) had no "mail2blog" feature OR required a version of SSL not supported by my browser(s) OR required a version of JavaScript not supported by my browser(s). Therefore I was forced to drop my old (pseudo-) blog redirecting to an error page without proper HTTP error code.

Now after it works again I found that Syndic8 never gave up to poll the feed URL for its FeedID 62125.

Validation issues

2007-04-15T15:40:00.000+01:00

The Blogger help pages are messy, many links are broken, SSL login fails miserably with old browsers, the feedback link without login doesn't work, etc. If you're looking for a blog hoster try to find a better service. I want the "blog-by-mail" feature available here, so it's my own fault.

Known issue, the application/atom+xml media type for the atom feed apparently degenerates into text/html after the first access. This might be a cache issue, resulting in a warning by the W3C validator:

HTML test

2007-04-09T20:49:00.000+01:00

Here's my OpenSPF page, it offers a customized search engine provided by Google.