FAT16 fun

Continuing the FAT fun saga with an unplanned sequel:  rexxfat used partition type 05 instead of 06 for BigFAT, i.e., FAT16 with 2**16 or more sectors.  I've fixed this stupid bug in the last rexxfat version.  The virtual hard disk image works fine in a Windows 2000 virtual machine:

The odd size 3759 MB matches what I have on a memory card.  It's rather tricky to copy the disk image to the memory card, either rawcopy doesn't support copies to a physical device, or it was not obvious for me how to manage this.  Using diskpart to unmount the one and only volume on this partitioned memory card wasn't good enough for write access, eventually the diskpart  delete volume magic did the trick.  Presumably that also explains my difficulties with other tools including dskprobe and HxD.

Looking for explanations I stumbled over SysInternals SDelete.  That article doesn't explain how Windows 7  manages volumes on memory cards, but answered another open question: Compressed, encrypted and sparse are managed by NTFS in 16-cluster blocks.  That matches my observation for rxsparse.rex, 16×4=64 KB, and if the SysInternals folks got that right Wikipedia also got it right.  Hint, it is very likely that SysInternals got it right, after all they were the first to offer a working DOS NTFS-driver permitting write access.

REXX FAT

After creating a somewhat dubious MBR in disk images formatted by rexxfat I've now also added a dummy VBR doing more than only INT 18h (hex. CD 18).  For an ordinary FAT12, FAT16, or FAT32 volume this VBR displays the OEM Id. (8 characters at offset 3 of the VBR), the file system type string (8 characters at offset -8 from the begin of the VBR code), the volume label (11 char.s, offset -19, default NO␠NAME␠␠␠␠), and the wannabe-unique volume Id. (4 bytes shown as 8 little endian hex. digits, offset -23).

Wikipedia claims that the file system type string and the volume label only exist for an extended boot signature introduced by 29h at offset -24 from the begin of the code.  In other words, for 28h only the volume id. can be expected to exist, and for anything else these 1+4+11+8 bytes won't exist at all, notably on media formatted with PC DOS 3.x or earlier.  The rexxfat dummy VBR code is designed to work for any 29h FAT layout with sector size 256, 512, or more.  Check out the script for the OpenWatcom WASM assembler MBR and VBR sources in two REXX comments.

Apparently Darwin VBRs expect that SI points to their partition table entry in the MBR, and just in case rexxfat now guarantees that when it boots an active primary partition.  Looking at a Darwin MBR source I think this is only used in its magic to boot a logical disk in an extended partition, i.e., the relative number of hidden sectors from the begin of the EBR to the logical disk VBR is updated on the fly to yield the absolute number of hidden sectors from MBR to VBR, and that's of course the LBA.  This won't help normal FAT VBR code expecting the number of hidden sectors in its own BPB, and the rexxfat MBR limits its efforts to initialize SI as expected for primary partitions.  Patching loaded FAT VBRs in an extended partition is left as an exercise for the EBR code — and so far rexxfat only formats one primary partition with a FAT.  More convoluted tasks are a job for GRUB, BCDEDIT, diskpart, parted, or what you have.

YAM (Yet Another Mail)

The ways of the IETF are completely bizarre, and the Tao of IETF does not really explain why.  In a recent episode the YAM WG will be closed after moving two of four e-mail related RFCs to STDs:

• STD 72 is RFC 6409 and covers "mail submission".  That's what ordinary users like you and me do when they send e-mail using their ISP or another ESP such as Gmail, GMX, or Hotmail.
• STD 71 is RFC 6152 and covers "8BITMIME".  That's what everybody uses since about 1993 outside of pure US-ASCII 7bit e-mail.

Oddly YAM didn't finish the work on a 5321bis and a 5322bis.  That leaves us with STD 10 (SMTP) and STD 11 (message format) in limbo, unless you still consider RFCs 821 and 822 (vintage 1982) as the ultimate truth in IETF e-mail standards.

For SPF it's quite interesting to look at the "reverse path" fine print of RFC 821, e.g.:

• The first host in the <reverse-path> should be the host sending this command. (RFC 821 about MAIL FROM).
• In any case, the SMTP adds its own identifier to the reverse-path. (RFC 821 about Relaying).

Sadly RFC 4408 specifying SPF also claims that this RFC 821 feature is archaic.  Of course today nobody uses reverse and forward paths as designed about thirty years ago for e-mail routing, but for SPF the MAIL FROM address is still supposed to indicate the responsible party for technical problems with an e-mail, also known as bounce address.  And e-mails forwarded to a third party need a new envelope sender address, otherwise SPF checks at this third party will fail, because IP-addresses authorized to send e-mail from the first party won't match IPs used by the forwarding second party.  That's as simple as it can be, ignoring the minor detail that RFC 1123 broke it about twenty years ago.

By the way, the work on 4408bis is supposed to start really soon now, but my confidence in the IETF is somewhat limited.

Google ±

Google+ wants me to join Google+, because I'm on Google+.

YAM (Yet Another MBR)

The Starman and Ray Knights disassembled various MBRs from ancient PC DOS to current Windows 7 versions, also covering LILO and GRUB MBRs.  The code used by TestDisk — when you are forced to fix a broken MBR — is a variant of the source published by Neil Turton.  The TestDisk variant tries to load the VBR for the active partition, and if that doesn't work the user can pick 1234A to load the VBR of partition 1…4 or floppy drive A:.  The original code has a timeout allowing to select 1234A, and in that variant the active partition is only the default.

A very simple MBR for the old INT 13h CHS interface is distributed with parted.  MBR CHS addressing works only for partitions starting in cylinder 0…1023 for any geometry, or in logical sector number (LBA) 0…16515070 in a (virtual) 1024×256×63 geometry.  Some interesting details found in these MBRs and related sources:

• All MBRs first define stack, data and extra segments, and then allow interrupts.  On entry SS:SP can be 0000:0400h at the end of the interrupt vector table, and picking something better is a good plan.  The new MBR code in rexxfat.rex simply uses 0000:0000h as stack top, i.e., the top of segment 0000h.
• The entry address for both MBRs and VBRs is 0000:7C00h.  To load any VBR at this address the MBR needs to move its own code and data to another address.  Allegedly some odd BIOS INT 19h code manages to start the MBR at the alias 07C0:0000h.  In other words, assume CS:nothing, or fix CS with a far jump.  The rexxfat MBR is designed for sector sizes from 256=100h up to 4096=1000h bytes.  Therefore the MBR copies itself to 0000:EC00h.  Adding 1000h yields a stack bottom at 0000:FC00h; and a stack with 1024 bytes should be enough for INT 13h LBA read operations.
• Allegedly some odd BIOS INT 19h code does not report the disk number for INT 13h in DL correctly.  Old MBR code apparently used the active partition flag 80h to populate DL, i.e., any non-zero flag instead of 80h could be used.  Variants of this scheme accept 80h…FFh as active, use DL as set by the BIOS, and reject 01h…7Fh as invalid  partition table.  The minimalistic rexxfat MBR accepts only 80h as active and does not look for trouble in the form of more than one boot partition.  Actually it scans the flags backwards checking its own 55AAh magic first.
• Most MBR variants support both CHS and LBA addressing.  The minimalistic rexxfat MBR always uses LBA to load six instead of only one VBR sector.  If something goes wrong, e.g., there is no active partition, or the loaded VBR has no 55AAh magic, an error is reported.  At that point the user can press space, 0…9, a (lower case "A"), or any other key.  0…9 tries to load the MBR for DL=80h…89h, and a tries DL=00h for the first floppy drive.  This won't work in emulation modes for say USB, where only 80h for this partitioned medium is supported.  Pressing space triggers INT 18h, a modern BIOS would then try to boot the next configured boot medium.  Any other key triggers INT 19h to boot the first configured boot medium.
• Various Windows MBRs have special code for the FAT32 backup boot sectors and partition types 0Bh or 0Ch.  Windows 7 MBRs support encrypted disks.  If you need these features use a hex. editor and copy the code in the first sector up to offset 1B8h.  The remaining 72 bytes are the disk ID (four bytes), two unclear nulls, 4×16 bytes for the four partitions, and the magic 55AAh.  Little endian whiners, it's 0xAA55 in your debugger ;-)

Allegedly MBRs and VBRs for sector sizes 1024, 2048, and 4096 are supposed to have their magic at the same offset as for sector size 512.  At the moment the rexxfat MBR expects that the 72 bytes from offset 1B8h to 1FFh contain the disk ID, partition table, and magic.  Flipping one byte in the code from 01h to 00h for the offset of the magic in theory allows sector size 256.  In practice VHD images only allow 512, real hard disks used 512, and El Torito or as the name says 512e emulate 512.

FAT32 fun

After NTFS and EXFAT now some FAT32 fun. The FORMAT and DISKPART tools on Windows NT have some limitations, therefore I created rexxfat.rex for experiments with disk image files and the FAT file system. In essence rexxfat can create a VFD image for a given number of sectors, example:

c:\etc\bin>rexxfat 4194304
1:   32264 * 2 FAT32 sectors;   7 +   0 SYS;   1 *   4129769 data:      4129769
2:   16257 * 2 FAT32 sectors;   8 +   0 SYS;   2 *   2080891 data:      4161782
3:    8161 * 2 FAT32 sectors;  10 +   0 SYS;   4 *   1044493 data:      4177972
4:    4089 * 2 FAT32 sectors;  14 +   0 SYS;   8 *    523264 data:      4186112
5:    2047 * 2 FAT32 sectors;  18 +   0 SYS;  16 *    261887 data:      4190192
6:    1024 * 2 FAT32 sectors;  32 +   0 SYS;  32 *    131007 data:      4192224
7:     128 * 2 FAT16 sectors;   1 + 127 SYS; 128 *     32765 data:      4193920
Pick 1..7 to create c:\etc\bin\rexxfat.vfd with 4194304 sectors

FAT16 cluster size 128 (hex. 80h) might be a bad idea for ancient tools. On the other hand FAT32 is not supported by ancient DOS versions up to MS DOS 6.22 or PC DOS 7.

FAT12 and FAT16 have a static root directory in the system area. For FAT32 the root directory is a dynamic part of the data area typically starting in the first data cluster. Consequently FAT12 and FAT16 never require unused sectors at the end of the data area, e.g., if a given cluster size cs=2**n yields u=1…cs-1 unused data sectors, then the data area can be reduced by u sectors, and the root directory can be increased by u sectors. rexxfat uses a default minimum of 6 sectors for the root directory, for sector size 512 that allows a minimum of 96 root directory entries.

For FAT32 any unused sectors are added to the reserved sectors at the begin of the system area. Normally FAT32 has 12 or more reserved sectors consisting of four sets:

• 00..02 Three boot sectors (01 is actually used as FSinfo)
• 03..05 Three unused sectors (nulls, no magic 55AAh)
• 06..08 Three backup boot sectors (the FSinfo pointer in 06 is 01, not 07)
• 09..11 Three or more unused sectors

Fortunately this madness can be trimmed, for starters zero or more unused reserved sectors in the last set are good enough. It is also possible to move the backup boot sectors from 06..08 to 03..05, simply modify the backup pointer in 00 and its backup.

There be dragons: If sector 00 is unreadable the backup pointer is also unreadable, and tools trying to fix or bypass this issue would assume that 06 is the backup. It is still possible to use 03..05 as official backup and keep 06 as unofficial third backup. For many purposes that should be good enough, with one notable exception: FAT32 boot code for MS DOS 7 (Windows 95, 98, ME) actually uses the third boot sector, and the Windows NT master boot code assumes that the backup boot sectors for FAT32 partitions start in relative sector 06. Obviously this cannot work as expected if MS DOS 7 backup boot code expected in sector 08 only exists in sector 05.

Fast forward, this is 2011, I've never used MS DOS 7, the dummy boot code in rexxfat consists of two bytes CD18h (INT 18h), and this works for all sector sizes in sectors 00, 03, or even 06. There is an option to enforce a classic layout with 9 or more reserved sectors.

For my purposes the odd layout with only 7 reserved sectors is better, adding one sector for the MBR yields 8, and removable media typically consists of 8×x sectors: No stupid gaps between the MBR and the FAT32 partition, and also no unused sectors after this partition or at its end. DISKPART and similar tools would skip 63-1 or 2048-1 sectors between MBR and partition, but for removable media with only one partition this is unnecessary: If some decent MBR code can load sector 63 or 2048 it can also load sector 1. There is no "second track" beginning in sector 63, and the new MS Vista alignment on MB boundaries (2048) is nice if you need to find a lost partition on a huge disk, but not required.

To create a VHD image instead of a VFD with rexxfat specify a negative total sector number as argument. One sector is used as MBR, the rest is used for the FAT partition. If the sector size is 512 a classic fixed VHD footer (511 bytes) is added. Sadly fixed VHDs cannot be created as NTFS sparse  files, and also do not allow other sector sizes (128, 256, 1024, 2048, or 4096). Here's the checkmbr output for a VHD with 66558 sectors:

...\bin\rexxfat.vhd (assuming geometry CHS 11093   1  6) id. [16F4-1407]
   MBR CHS     0   0  1 at          0, end     0   0  1, size          1
1: 0C: CHS     0   0  2 at          1, end 11092   0  6, size      66557 FAT32
2: 00: CHS     0   0  0 at          0, end     0   0  0, size          0 unused
3: 00: CHS     0   0  0 at          0, end     0   0  0, size          0 unused
4: 00: CHS     0   0  0 at          0, end     0   0  0, size          0 unused
                                                        total      66558

 FAT32 CHS     0   0  2 at          1, end 11092   0  6, size      66557
  boot CHS     0   0  2 at          1, end     0   0  4, size          3
backup CHS     0   0  5 at          4, end     1   0  1, size          3 boot
  rest CHS     1   0  2 at          7, end     1   0  2, size          1 boot
 FAT32 CHS     1   0  3 at          8, end    86   0  4, size        512 #1
 FAT32 CHS    86   0  5 at        520, end   171   0  6, size        512 #2
  data CHS   172   0  1 at       1032, end 11092   0  6, size      65526
[16F4-1408] (cluster size   1, number      65526)       total      66557

This is work in progress, the pseudo-geometry in the VHD footer fits into 28=16+4+8 bits for ATA, but an incomplete last cylinder in a geometry with 24=10+8+6 bits for INT 13h would be better for less than 16515072=1024×256×63 sectors. After all there are no cylinders in a VHD image, and the checkmbr output is designed for INT 13h CHS tuples in MBRs, not for ATA CHS tuples in VHDs.

SideWiki shutdown

Google SideWiki — good riddance. Just for fun I save my last (of five) entries below, they were all futile attempts to get in contact with a human being interested in fixing Google bugs. Next stop, let's kill Knol.

'Meta-searching' Google
The alleged "meta search" is an ordinary "cached" link offered in an ordinary Google search result, where the found Google groups article does not contain the searched +"bigfat" fat16 -forum, and I hoped that at least the cached result might be related to my query.
Instead I get a runabout ending in this "sidewiki" known to be yet another Dave Null at his support work. If you do not want cached Google groups search hits to be clicked just do not offer the links.
About Unusual traffic from your computer network - Web Search Help

Go, Duck, go

Reader redesign: Terrible decision, or worst decision? and another article discuss the next step of what will be known as the demise of Google: Now they have crippled Google Reader. It even doesn't work anymore in Chrome.

What used to be a single click Like or Share in Reader is now gone, and the new Reader UI is a disaster, as one of the creators of the original UI states. In theory I could join the G+ beta tests, but as it happens I'm not interested to risk my Google account including Blogger and Gmail with an acknowledgement of the G+ beta test ToS.

Check out Duck Duck Go lite while it's still a free no-nonsense search engine. And do not waste time with Google APIs or services, sadly they will be gone or munged until no-good before you begin to grok what they were about.