EXFAT fun

For what's it worth checkmbr.rex can now handle EXFAT partitions. Partition type 07h is still tagged as NTFS, but the reported details now match EXFAT if applicable. Example:

       CHS     0   0  1 at          0, end     0 158 47, size      10000
hidden or total sectors 63 10000 do not match 0 10000
  boot CHS     0   0  1 at          0, end     0   0 12, size         12
backup CHS     0   0 13 at         12, end     0   0 24, size         12 boot
  rest CHS     0   0 25 at         24, end     0   2  2, size        104 boot
 EXFAT CHS     0   2  3 at        128, end     0   2 18, size         16 #1
unused CHS     0   2 19 at        144, end     0   4  4, size        112
  data CHS     0   4  5 at        256, end     0 158 46, size       9744
[6722-4357] (cluster size   8, number       1218)       total      10000

The Windows 7 FORMAT /FS:exFAT tool used an obscure number 63 for the hidden sectors in this unpartitioned VFD image also known as superfloppy, and checkmbr.rex dutifully reports that 63 is not 0. As long as you don't try to boot from a superfloppy or ordinary partition these hidden sectors are irrelevant.

It is interesting to see that FORMAT reserved 128=2×12+104 sectors for the 2×12 boot sectors. Most of the 12 boot sectors are already unused, and the boot checksum sector with 128 copies of the same 32bits checksum is hilarious. So what is the idea of the 104 additional sectors?

While at it Microsoft decided that two "FAT" copies are for cowards, and creates only one "FAT". It is not really a FAT, EXFAT uses a bit map for allocations, the "FAT" is only used for purposes where a bit is not good enough, i.e., bad clusters or fragmented cluster chains. And after saving 16 sectors for a second FAT there is another set of 112 apparently unused sectors, 128=16+112.

I'd get the idea if subsections of the system area are padded to max(4096/SS,CS) sectors: At some point in time we'll want to use 512e aligned to physical sector size 4096. But for that 56=2×12+2×16 instead of 256=2×128 would be good enough. For one FAT there are apparently 216=104+112 unused sectors, and instead of 1218 there could be 1235=1218+27 clusters (27=216/8).

SANS published a brilliant reverse engineering paper about EXFAT, but I'm not yet ready to outsmart FORMAT /FS:exFAT. Remotely related, checkmbr survived the forensic extended partition test case with two primary partitions in an extended partition. And I've fixed the output for zero FAT12 clusters. ToDo: checkmbr should report that 6 of the 16 FAT sectors are overkill for 1218 clusters, after all it does this already for FAT12/16/32.

NTFS fun

New REXX toy: rxsparse.rex converts a specified file on NTFS to a sparse file with FSUTIL SPARSE SETFLAG and FSUTIL SPARSE SETRANGE. Obviously you need fsutil.exe for this business, and it only works on NTFS.

NTFS files can be compressed and decompressed on the fly on Windows NT when the corresponding file attribute is set, and with inheritance this can be arranged for complete subdirectory trees. This feature replaced the obscure DOS double space approach, good riddance. Compression on the fly can be nice, but of course file accesses will be slower. Sparse files are an alternative, long runs of zero bytes in a file are not physically stored, but emulated. Unlike compression that's fast, only the position and length of sparse ranges has to be noted.

For NTFS a long run is not anything, but a multiple of the cluster size. As for FAT file systems the NTFS cluster size is 1, 2, 4, …, or 128 sectors. Wikipedia claims that a long run consists of 64 KB, and rxsparse can check this theory in its self test:

SPARSE SETFLAG C:\etc\bin\REXX\rxsparse.tmp
655360 = 5*131072 bytes with 8, 16, 32, 64, 128 "zero"-sectors at the end:
Bereich geringer Datendichte: [0] [655360]
SPARSE SETRANGE 126976 4096
SPARSE SETRANGE 253952 8192
SPARSE SETRANGE 376832 16384
SPARSE SETRANGE 491520 32768
SPARSE SETRANGE 589824 65536
Bereich geringer Datendichte: [0] [589824]

SPARSE SETFLAG C:\etc\bin\REXX\rxsparse.tmp
655360 = 5*131072 bytes with 8, 16, 32, 64, 128 "zero"-sectors at the begin:
Bereich geringer Datendichte: [0] [655360]
SPARSE SETRANGE 0 4096
SPARSE SETRANGE 131072 8192
SPARSE SETRANGE 262144 16384
SPARSE SETRANGE 393216 32768
SPARSE SETRANGE 524288 65536
Bereich geringer Datendichte: [0] [524288]
Bereich geringer Datendichte: [589824] [65536]

Test okay, maybe delete C:\etc\bin\REXX\rxsparse.tmp

The shown ranges [offset] [size] contain non-zero
bytes; check that there are 1..4 remaining ranges.
The smallest hidden zero-range size should be at
least 4096 (otherwise edit BLKLEN in the source).

The German gibberish is the output of FSUTIL SPARSE QUERYRANGE on a German Windows 7 x64 SP1. Oddly this is the opposite of SETRANGE, it shows ranges containing non-zero bytes. NTFS cluster size 4096=8×512 is perfectly normal, the definition of long run could still differ for other NTFS cluster and sector sizes.

Major caveat: When rxsparse found the end of what it considers as a long run of zero sectors it has to close the file before using FSUTIL SPARSE SETRANGE. If another process manages to write non-zero bytes in the critical range before FSUTIL gets write access these non-zero bytes are lost. Presumably REXX could somehow use a decent .NET API to avoid this potential race condition with the FSUTIL command line tool, but I didn't bother to figure it out.

Sadly my use case "FAT32 VHD image on NTFS" does not work as expected for the VHD variant known as fixed VHD (type 2). For NTFS VHD images it is also pointless, DISKPART can handle NTFS VHDs after defragmentation and precompact in a VPC VM guest OS supporting the precompact VM addition, notably Windows 2000 or better. Any "defrag" and wipe unused disk space tool for FAT file systems on another guest OS has presumably the same effect as precompact, but DISKPART supports only NTFS for its part of the VHD compactification magic. Not yet tested, maybe rxsparse makes sense for dynamic FAT VHD images.

Customised Search Engines

More than four years ago I explained the secrets of LP and AH in a Google CSE query parameter cof=FORID:0%3BAH:center%3BLP:0. The percent-encoded hex. %3B is a semicolon separating FORID:0, AH:center, and LP:0 in this example going back to Google's ancient free site search.

This undocumented cruft is now seriously dead, or rather, AH:center and LP:0 now have no visible effect. Maybe the cof= parameter was removed while some new CSE features were introduced since June, see an entry on the Custom Search Blog — I've no idea what the Element might be, and the officially deprecated features didn't mention any undocumented cruft, but there, it's dead, and the new layout for CSE search results hosted by Google works fine for the xyzzy CSE. Fortunately I modified it to work on the CSE layout test page some months ago, and as it happens that is now apparently good enough for new search result pages without AH:center and LP:0.

I'll remove any remaining obsolete cof= parameters where I find them, it might be hidden in obscure places such as the template for this blog, persistent URL (PURL) redirections, rel="search" link relations in the header of my web pages, and a tiny CSE googlet (PURL of XML source).

BTW, if you were always looking for an IETF-related search engine test the xyzzy CSE as shown near the bottom of all web pages for this blog. I still update this CSE when I find new promising sites, one of the last additions was a site tracking link relation registration requests. I added it after the registration of link relation rel="canonical" on the corresponding IETF expert review mailing list.

And if you were always looking for a REXX-related search engine check out my second (and last) REXX CSE — I rarely use it, but fix it when I stumble over broken links on my KEXX page. For a recipe to convert CSEs given by their cx= ID to OpenSearch Description Documents see another four years old entry on this blog, or just adopt one of the OSDDs on my googlet page.

MD5 test suite version 1.8

The MD5 test suite 1.8 finally mentions RFC 6151, fixes a minor exit code bug in version 1.7, supports the web-safe base64 encoding specified in RFC 4648 as base64url alphabet, uses the USCYBERCOM easter egg as MD5 streaming test case, and covers the collision announced by Tao Xie and Dengguo Feng in December 2010 (2 of 512 bits modified). Read their PDF (one page) for details of the offered bounty for anybody who finds another collision of this type.

checkMBR.rex

Some of my web pages are still messy and on their way to a new hoster, but there is a new version of checkmbr.rex (REXX script) maybe interesting for users of TestDisk. The script can analyze MBR disks on Windows platforms using \\.\PHYSICALDRIVEn for n=0..9, and identifies unused sectors caused by generous partitioning or by FAT and NTFS sizes not exactly fitting into their given partition. While at it the script creates base 64 backups of various boot sectors. Fixed bugs and new features:

• The max. cluster numbers for FAT12/16/32 were "off by one". The hardwired limits are now 4084, 65524, and 268435444.
• Added partition type 27h "WinRE" for a hidden windows recovery partition and file system NTFS.
• exFAT uses partition type 07h also used by NTFS. This is unsupported and should result in lots of "NTFS errors" for exFAT (untested).
• UEFI disks start with a protective MBR (partition type EEh). This is now reported, but checkMBR analyzes only MBR disks.
• VFD (virtual floppy disk) and fixed VHD (virtual hard disk) files can now be given instead of a physical drive number.

The new VHD feature would fail for ooREXX 3.x and VHDs greater than 2 GB, get ooREXX 4.x. Many identified partition types are untested or ambiguous, e.g., I've never seen an EFI FAT type EFh roughly corresponding to a Windows system NTFS partition on MBR disks, or the partition types allegedly used by the FreeDOS FDISK tool to hide FAT, NTFS, or extended partitions.

about: ftp:

There are I-Ds for two URI schemes:

1. about: URIs tackle about:blank and similar beasts.
2. ftp: URIs are another missing piece in the quest to finally get rid of RFC 1738.

Please review these drafts and send any feedback to the relevant IETF mailing lists — subscriptions are cheap, they cost you a working e-mail address. ;-)

So far nobody volunteered to finish the existing file: URI I-D (2005).  There are some dragons with respect to different browsers and operating systems, notably IE vs. Firefox and Windows vs. Linux, but really, almost any file: URI scheme RFC based on the work for other schemes would be better than RFC 1738 vintage 1994 today.  And you are not forced to emulate some negative records created by me for RFC 5538.  I think file: and ftp: URIs are the last missing pieces in this puzzle.  Check out the mailserver: (RFC 6196) and tn3270: (RFC 6270) URI schemes for pieces added in 2011.

Completely unrelated, and not deserving a separate article:  I've recently added a CC-BY-SA license for this blog.  That is mostly an experiment to show my support for Creative Commons and Wikimedia Commons as explained on my commons page with a fascinating (but long) video of a CCC lecture by Lawrence Lessig in 2006.  Clearly I can't revert this license to some more restrictive form for existing pages, but maybe I'll change it to public domain in the future if I feel like it, or if somebody asks for it with a sound reason (from my POV).

Firefox 5 in W2K under windows 7 x64

Now that is really something new (for me) in Windows 7 x64:

What you see is the bottom of a 1152×768 virtual Windows 2000 SP4 desktop with quick links for Firefox 5, IE6, KeditW 1.6, etc.  The host system is a Windows 7 x64 SP1 home premium Sony VAIO (1600×900, ATI mobile Radeon catalyst 11.6).  Sadly XP mode is not supported for home premium, or rather, I couldn't test a manual XP activation, because I have no XP license key.

Some invisible details:

• VPC does not really work for guests older than XP SP3, you have to install the virtual machine additions 2004.
• Virtual PC guy posted a picture of the real network card virtualized in VPC for up to four networks.  For my purposes "internal NAT" allows me to share an existing mobile broadband WWAN connection.  This is not the VPC default, change it in the VM settings while the VM is not running — hibernating might be good enough.
• Without the VPC integration features I found yet no simple way to exchange guest and host files (including simple things like the clipboard), WebDAV on a remote hoster clearly doesn't qualify as simple.
• Using virtual floppies (VFD) is a major pain with Windows 7 VPC, but once you have attached a VFD to a VM it sticks.  You can even format the VFD.
• I didn't know how to "capture" (if that's the correct term) a W2K VHD from a genuine system, and used a public VHD.  That beast required a lot of work (tons of missing updates, removal of obscure "bars" installed by the publisher, adding A/V-software, full scan with the latest MSRT, Firefox, Flash, Silverlight, 7Zip, XnView, DirectX, Secunia PSI 1.5x, etc.).
• Just in case: Yes, WU still works for W2K — only up to June, 2010 for OS + IE6 updates, but still for any MS Office stuff.  Sadly the still working monthly MSRT requires manual downloads.  Avira announced that they'll stop to support Avira A/V Personal 10 under W2K in July 2011, which is kind of stupid: Anything better than W2K can and IMO should use MSE.
• Somewhat unrelated, normally you cannot get ATI Catalyst 11.6 for Sony VAIO from AMD, but I found an unrestricted official download link in an obscure forum.  Same procedure as for direct Flash AX downloads without Adobe's GetPlusPlus malAdware.
• The latest Intel PIU fails to install under W2K, and an older version (working for W2K) died with an obscure error in VPC.  FWIW installing DirectX 9 in the virtual W2K worked, but of course there isn't much to accelerate in a VPC.  My real W2K is far slower than the virtual W2K: The real box has 256 MB RAM, the virtual box has 512 MB, and the windows 7 host has about 3950 MB.
• Sometimes the virtual W2K hangs and needs a hard reset (= close VM).  I'm not sure when this started, among my suspects are Firefox 5, Avira Personal 10, and Secunia PSI 1.5x.  It's also possible that my (failed) attempts to install the VPC integration features, or my inconclusive attempts to install the VPC 2007 VMA screwed up the VHD.  There is a suspicious unknown device in the device manager, claiming to be working, and using IRQ7 — is this some kind of time synchronization?  To break out of a restart hanging VM loop modify the VM settings to "unconditional close" or "always ask".  Normally I'd prefer "auto-hibernate on close", but that doesn't help if the VM hangs.
• While a VHD is not running or hibernating it is relatively simple to mount it as a virtual disk in Windows 7.  Unmounting can be slightly tricky, but it's necessary to start any VM using the VHD.
• Having fun with VMs I stumbled over a simple VFD-driver and virtual CD control panel for Windows x86 platforms.  Check out Elby Clonedrive for serious virtual CD applications.  Apparently the MS XP virtual CD controlpanel 21 works also under W2K.
• At some point in time I'll have to grok the diskpart manual — it would be nice to associate .vhd with a simple attach/detach (mount or unmount) script without going to a command line or the device manager.

Shortcut icon mysteries

Tweak My Blogger proposes to use three link relations to replace the default Blogger favicon with a custom favicon. Historical background:

There can be various link relations in the <head> element of HTML or XHTML pages, e.g., <link rel="search" … /> for OpenSearch descriptions. For favicons IE originally used the old Windows .ico format. That's a kind of container for related Windows .bmp images in various sizes and with different numbers of colours. A favicon .ico should include an image with size 16×16 and up to 256 colours, and it is not required to offer other sizes. If compatibility with say IE5 is not your main problem 32×32 or 64×64, and more than 256 colours, might also work: Applications are supposed to pick the best image offered in an .ico for their needs, and scale it up or down if necessary. If an automatically scaled down image does not work for your icon the .ico format allows to include optimized smaller versions, notably 16×16.

Years ago there was no proper MIME type for these beasts, but type="image/x-icon" was widely supported. Later Microsoft registered type="image/vnd.microsoft.icon for this purpose, and modern browsers are supposed to know this type as far as they support the odd .ico format at all.

Web servers might be still configured to associate .ico with image/x-icon, but that does not affect a correct image/vnd.microsoft.icon in the <head> of pages. Without a link relation IE simply tried to fetch a file favicon.ico from the relevant directory. For various reasons that was a bad idea, and modern browsers rely on explicit link relations instead of default locations for favicons and other purposes.

Other browsers and other platforms were not eager to support the odd Microsoft .ico format, but liked the idea of shortcut icons. A much better format is .png, but some old browsers including IE did not support it, or had issues with certain PNG-features. Somehow these differences resulted in two link relations, rel="icon" and rel="shortcut", for in essence the same purpose. Only rel="link" is registered at the moment. It is possible to list more than one relation as in rel="shortcut icon", and today that should be good enough if the favicon is an .ico.

If you seriously hate this image format try rel="icon" type="image/png". GIF instead of PNG is also okay, of course excluding animated GIFs. Other formats are either unsuited for a favicon, e.g., JPEG and WebP, or still less widely supported, e.g., SVG, which will be an ideal solution for most kinds of icons.

So far for the theory of the favicon business. The practice on Blogger and other sites can be slightly more complex. IIRC you can't use the name favicon.ico, just pick something else, e.g., href="http://example.org/my-icon.ico" if this really is an .ico with MIME type="image/vnd.microsoft.icon". Whatever Blogger does, the type="icon/ico" recommended in Tweak My Blogger makes no sense, and at least in theory (not tested) more than one link relation is unnecessary. My Blogger template contains the following line immediately before the <title>:

<link href='http://purl.net/xyzzy/xyzzy.ico' rel='shortcut icon' type='image/vnd.microsoft.icon' />

Sadly Google reader still shows the Blogger icon for the feed, and unsurprisingly a Google profile still shows the Blogger icon for a Blogger profile, but otherwise it works as expected. Maybe create a 57×57 PNG for I-Phones and I-Pads, and use the (unregistered) link relation rel="apple-touch-icon", it can't get odder. I have no I‑Panything to test this.

URL fragment vs. query part

Has Google finally reached the ultimate Microsoft state, "breaking standards for fun and for profit" ? In a recent entry on the Google Operating System blog (not affiliated with Google) two screen shots show #fragments instead of ?queries. My attempt to post a comment had no visible effect, maybe comments are moderated or the blogger template is broken or whatever the problem might be, zero feedback to users is just wrong. Google Chrome lost the content of the comment form before I could reproduce it verbatim here, in essence it said:

In URLs a hash mark "#" introduces the optional fragment at the end of the URL, and a question mark "?" introduces the optional query part — also at the end of the URL, but before any fragment.

Fragments are the local business of the client (browser) and depend on the document (MIME type). Queries are the business of the server, e.g., many servers expect name=value pairs separated by ampersands "&", other severs also permit semicolons ";" as separator, and so on.

The query syntax depends on the scheme, here http:, the details depend on the server, but in both cases the overall URL syntax would require to percent-encode "#" if it is a part of the query not intended to start the fragment.

Notably fragments do not participate in redirections from one scheme or server to another scheme or server, the target location never sees the source fragment — browsers are not expected to transmit a fragment in requests such as HTTP GET.

In other words, something is wrong in the googlesystem screen shots.