Ticket #54 (closed defect: fixed)

Opened 6 years ago

Last modified 5 years ago

FAT directory entries not cleared.

Reported by: Joel Owned by: Joel
Priority: blocker Milestone: Eraser 6.0
Component: Core Version: 6.0.4.875
Keywords: Cc: lowjoel@…; gtrant@…
Processor Architecture: Blocked By:
Blocking: Operating System:

Description

In DirectExecutor.EraseUnusedSpace, directory entries are not cleared. Look at Eraser.cpp:2348, r400.

Garrett: What do the equivalent functions in v5 do that are important? I see a lot of undocumented API use which is obviously dangerous because Windows undergo internal component rearrangement almost every service pack. I thought we wanted to remove the dependency on undocumented API calls?

Blocking

IdSummaryMilestone
#54FAT directory entries not cleared.Eraser 6.0

Blocked by

IdSummaryMilestone
#54FAT directory entries not cleared.Eraser 6.0

Attachments

FAT Directory Cleaning.patch (4.0 KB) - added by Joel 6 years ago.
Part 1 of 2. This is the patch to apply to the source tree.
DirectoryStream.cs (5.1 KB) - added by Joel 6 years ago.
Part 2 of 2. The DirectoryStream? class which allows client code to determine the number of clusters a directory occupies
2009-05-09.164209+0800MYT.htm (48.6 KB) - added by Joel 6 years ago.
Chat Log with Svante Seleborg on FAT erasures

Change History

comment:1 Changed 6 years ago by Joel

  • Priority changed from major to blocker

Quite important, this one. It has the potential to be a regression from v5.

comment:2 Changed 6 years ago by Overwriter

I am writing with limited knowledge here but is this related to ticket 23 ?

comment:3 Changed 6 years ago by Joel

  • Component set to Core

comment:4 Changed 6 years ago by Joel

Hmm, NTFS is safe, FAT isn't because we're reimplementing the data structures ourselves.

comment:5 Changed 6 years ago by Joel

I've looked at the FAT code again and I've come to the conclusion that techniques used in f5 to clean the FAT will not work in Vista. We'll need to find new ways of erasing the FAT in Vista. Garrett/Kaz? can you take charge of this?

I'll just need a document detailing what needs to be erased for FAT drives, the differences between FAT12/FAT16/FAT32 and how to do this without opening a Write handle to the raw disk sectors.

comment:6 Changed 6 years ago by Joel

  • Summary changed from Directory Entries not cleared to FAT directory entries not cleared.

Slightly more accurate title.

comment:7 Changed 6 years ago by Joel

  • Status changed from new to accepted
  • Owner changed from Garrett to Joel
  • Version set to 6.0.4.875
  • Type changed from task to defect

I was reading up about FAT and I've developed a method to erase the stuff, which requires verification by someone else, before I implement it.

FAT entries that require erasing are basically the directory tables, as documented in Wikipedia. After a directory is deleted, the corresponding clusters are marked as available, and the referencing Directory Table Entry has byte 0x00 reset to read 0xE5, which means that the entry has been removed.

So this leaves us with two things that need to be erased. The actual directory table, and the referencing directory table which refers to the new deleted directory. The erasure of the Directory Table structure itself is a non-issue to me because the entire structure's is now marked as available; an unused space erase will clear it.

The referring directory table structure, however, must be erased. The structure that must be erased is the parent directory of the directory that was deleted. As mentioned earlier, the old entry is merely marked as deleted through the setting of the byte at offset 0x00 to read 0xE5. The problem is further exacerbated by the presence of Long File Names, as outlined in Wikipedia as well. The long names are stored as "invalid" DOS file entries in the directory table. So deleting a folder or a file will leave multiple directory table entries marked as "deleted", with the old information still intact.

My proposed erasure method is to do what we do with NTFS now:

  1. Determine Directory Table Size.
  2. Determine the number of valid entries in the directory table. This will let us know how many directory entries must be erased.
  3. Erase the directory entries. This is a multi-step process. Repeat this for every entry that must be created.
    1. Create files inside the directory that needs entries to be erased with 8.3 names which fills up the first 11 bytes of the directory table entry structure.
    2. Set the Create, Access and Modified (CAM) date values to Jan 1, 1980
    3. Entry offset 0x14 and 0x1a should be reset to zero automatically, regardless of FAT variant
  4. Delete all the created files

That should erase the directory entry such that file names are no longer recoverable. Please comment if you have relevant experience and tell me if this works.

comment:8 Changed 6 years ago by Overwriter

Good grief !!

Well done Joel, stunning work !

I hope this fixes it, I will try to get help to verify it for you.

I think it will have to be Svante as this looks a bit complicated !

:o)

comment:9 Changed 6 years ago by Joel

Note to self: while I'm at it refactor all filesystem-specific code into helper classes/interfaces.

comment:10 Changed 6 years ago by Joel

Okay the note is no longer valid - I did that already.

I've now got a sample working: but it's hopelessly slow because we're practically forcing every directory file to fullness then clearing after ourselves. This leads to two bad effects:

  1. Every directory entry grows by one cluster at every wipe. This is statistically significant as a drive can have hundreds of directories and consuming 100's of clusters per wipe is unsustainable
  2. Because we're filling the directory file with filenames, this well, takes time. A lot of time.

Do we have alternative approaches?

P/s I've attached a patch created against r1032.

Changed 6 years ago by Joel

Part 1 of 2. This is the patch to apply to the source tree.

Changed 6 years ago by Joel

Part 2 of 2. The DirectoryStream? class which allows client code to determine the number of clusters a directory occupies

comment:11 Changed 6 years ago by Joel

I've had a chat with Svante Seleborg (of AxCrypt?) and I've attached our chat log.

Changed 6 years ago by Joel

Chat Log with Svante Seleborg on FAT erasures

comment:12 Changed 5 years ago by Joel

Okay I've spent the whole of today implementing FAT erasure (see r1206 r1224) and FAT32 erasure is now working, I'll implement FAT12 and FAT16 tomorrow.

comment:13 Changed 5 years ago by Joel

  • Status changed from accepted to closed
  • Resolution set to fixed

Completed as of r1235. Finally!

Note: See TracTickets for help on using tickets.