EXFAT fun

For what's it worth checkmbr.rex can now handle EXFAT partitions. Partition type 07h is still tagged as NTFS, but the reported details now match EXFAT if applicable. Example:

       CHS     0   0  1 at          0, end     0 158 47, size      10000
hidden or total sectors 63 10000 do not match 0 10000
  boot CHS     0   0  1 at          0, end     0   0 12, size         12
backup CHS     0   0 13 at         12, end     0   0 24, size         12 boot
  rest CHS     0   0 25 at         24, end     0   2  2, size        104 boot
 EXFAT CHS     0   2  3 at        128, end     0   2 18, size         16 #1
unused CHS     0   2 19 at        144, end     0   4  4, size        112
  data CHS     0   4  5 at        256, end     0 158 46, size       9744
[6722-4357] (cluster size   8, number       1218)       total      10000

The Windows 7 FORMAT /FS:exFAT tool used an obscure number 63 for the hidden sectors in this unpartitioned VFD image also known as superfloppy, and checkmbr.rex dutifully reports that 63 is not 0. As long as you don't try to boot from a superfloppy or ordinary partition these hidden sectors are irrelevant.

It is interesting to see that FORMAT reserved 128=2×12+104 sectors for the 2×12 boot sectors. Most of the 12 boot sectors are already unused, and the boot checksum sector with 128 copies of the same 32bits checksum is hilarious. So what is the idea of the 104 additional sectors?

While at it Microsoft decided that two "FAT" copies are for cowards, and creates only one "FAT". It is not really a FAT, EXFAT uses a bit map for allocations, the "FAT" is only used for purposes where a bit is not good enough, i.e., bad clusters or fragmented cluster chains. And after saving 16 sectors for a second FAT there is another set of 112 apparently unused sectors, 128=16+112.

I'd get the idea if subsections of the system area are padded to max(4096/SS,CS) sectors: At some point in time we'll want to use 512e aligned to physical sector size 4096. But for that 56=2×12+2×16 instead of 256=2×128 would be good enough. For one FAT there are apparently 216=104+112 unused sectors, and instead of 1218 there could be 1235=1218+27 clusters (27=216/8).

SANS published a brilliant reverse engineering paper about EXFAT, but I'm not yet ready to outsmart FORMAT /FS:exFAT. Remotely related, checkmbr survived the forensic extended partition test case with two primary partitions in an extended partition. And I've fixed the output for zero FAT12 clusters. ToDo: checkmbr should report that 6 of the 16 FAT sectors are overkill for 1218 clusters, after all it does this already for FAT12/16/32.

NTFS fun

New REXX toy: rxsparse.rex converts a specified file on NTFS to a sparse file with FSUTIL SPARSE SETFLAG and FSUTIL SPARSE SETRANGE. Obviously you need fsutil.exe for this business, and it only works on NTFS.

NTFS files can be compressed and decompressed on the fly on Windows NT when the corresponding file attribute is set, and with inheritance this can be arranged for complete subdirectory trees. This feature replaced the obscure DOS double space approach, good riddance. Compression on the fly can be nice, but of course file accesses will be slower. Sparse files are an alternative, long runs of zero bytes in a file are not physically stored, but emulated. Unlike compression that's fast, only the position and length of sparse ranges has to be noted.

For NTFS a long run is not anything, but a multiple of the cluster size. As for FAT file systems the NTFS cluster size is 1, 2, 4, …, or 128 sectors. Wikipedia claims that a long run consists of 64 KB, and rxsparse can check this theory in its self test:

SPARSE SETFLAG C:\etc\bin\REXX\rxsparse.tmp
655360 = 5*131072 bytes with 8, 16, 32, 64, 128 "zero"-sectors at the end:
Bereich geringer Datendichte: [0] [655360]
SPARSE SETRANGE 126976 4096
SPARSE SETRANGE 253952 8192
SPARSE SETRANGE 376832 16384
SPARSE SETRANGE 491520 32768
SPARSE SETRANGE 589824 65536
Bereich geringer Datendichte: [0] [589824]

SPARSE SETFLAG C:\etc\bin\REXX\rxsparse.tmp
655360 = 5*131072 bytes with 8, 16, 32, 64, 128 "zero"-sectors at the begin:
Bereich geringer Datendichte: [0] [655360]
SPARSE SETRANGE 0 4096
SPARSE SETRANGE 131072 8192
SPARSE SETRANGE 262144 16384
SPARSE SETRANGE 393216 32768
SPARSE SETRANGE 524288 65536
Bereich geringer Datendichte: [0] [524288]
Bereich geringer Datendichte: [589824] [65536]

Test okay, maybe delete C:\etc\bin\REXX\rxsparse.tmp

The shown ranges [offset] [size] contain non-zero
bytes; check that there are 1..4 remaining ranges.
The smallest hidden zero-range size should be at
least 4096 (otherwise edit BLKLEN in the source).

The German gibberish is the output of FSUTIL SPARSE QUERYRANGE on a German Windows 7 x64 SP1. Oddly this is the opposite of SETRANGE, it shows ranges containing non-zero bytes. NTFS cluster size 4096=8×512 is perfectly normal, the definition of long run could still differ for other NTFS cluster and sector sizes.

Major caveat: When rxsparse found the end of what it considers as a long run of zero sectors it has to close the file before using FSUTIL SPARSE SETRANGE. If another process manages to write non-zero bytes in the critical range before FSUTIL gets write access these non-zero bytes are lost. Presumably REXX could somehow use a decent .NET API to avoid this potential race condition with the FSUTIL command line tool, but I didn't bother to figure it out.

Sadly my use case "FAT32 VHD image on NTFS" does not work as expected for the VHD variant known as fixed VHD (type 2). For NTFS VHD images it is also pointless, DISKPART can handle NTFS VHDs after defragmentation and precompact in a VPC VM guest OS supporting the precompact VM addition, notably Windows 2000 or better. Any "defrag" and wipe unused disk space tool for FAT file systems on another guest OS has presumably the same effect as precompact, but DISKPART supports only NTFS for its part of the VHD compactification magic. Not yet tested, maybe rxsparse makes sense for dynamic FAT VHD images.