Eraser eats up all available RAM on Win7 x64

[pcworld]

First of all thanks for this great open-source effort!

(I'm a Linux user and just used Windows to try Eraser)

System: Windows 7 Home Premium 64-bit, 4 GB RAM
I told Eraser to wipe the free space of a FAT32-partition on an (external/USB) HDD. The partition's size is about 206GiB from which 84GiB are used.
At the end of the wiping process Eraser took about 2GB of memory, probably all it could allocate. (The OS was in a pretty unusable state)

At the end of the log I found:

Code: Select all

External component has thrown an exception

Just wanted to report this. If it doesn't affect anyone else, there's no need to fix it.

Another bug I've found:
When I select multiple entries in the log window, do right-click and choose "copy", all of the entries except for the last one are copied into the clipboard.

[DavidHB]

Assuming that you are using version 6.o.x, try clearing the log for the task; there will be loads of messages about non-erasure of cluster tips, which will take up RAM when Eraser is running. Then do a disk check on the drive (Windows will schedule this to take place when you reboot). Then see if you can run the task to completion.

The log copying bug is a known issue in .NET. Annoyingly, I don't think it has been fixed in .NET 4

David

[Joel]

If you're the reporter for ticket 445, thanks.

I think the key here is that you're using FAT32. FAT unused space erasures has got a component that will manually walk the directory tree of the drive and clear unused entries. My suspicion is that you've got a damaged directory structure, though I can't be certain.

It's also possible that the drive you've used was used with Linux which does implement FAT a little differently from Windows, which could throw off the Eraser code. This would require a bit of analysis to trace the bug, but let's see if I can do so.

[pcworld]

Assuming that you are using version 6.o.x, try clearing the log for the task; there will be loads of messages about non-erasure of cluster tips, which will take up RAM when Eraser is running.

In the log there are about 10 messages, so definitely not something taking up 2GB of RAM.

Then do a disk check on the drive (Windows will schedule this to take place when you reboot). Then see if you can run the task to completion.

I already ran dosfsck on Linux before using Eraser.

If you're the reporter for ticket 445, thanks.

No, I didn't even find that ticket.

I think the key here is that you're using FAT32. FAT unused space erasures has got a component that will manually walk the directory tree of the drive and clear unused entries.

That's the reason why I tried Eraser. I filled up all the free space with zeros, but that didn't stop file recovery programs to still find some folder names.
By the way, is there an option to just clear/overwrite unused entry names? That way I could still overwrite the files using my Linux method and just use Eraser to clear the inodes (or whatever you call them in FAT).

It's also possible that the drive you've used was used with Linux which does implement FAT a little differently from Windows, which could throw off the Eraser code.

You're right with this, the partition was mainly used in Linux.

Let me know if or how I could help you fixing this.

[DavidHB]

With this new information, I withdraw my suggestions until Joel is clear on the implications of the 'used with Linux' issue.

There isn't currently an option just to clear the MFT, but it is on the 'to do' list. I think that it's potentially quite useful, because it is quick and makes deleted file recovery significantly harder, though not impossible.

David

[Joel]

Oh, FAT doesn't have an MFT, so cleaning it is VERY easy, and already implemented in Eraser's code. It also will compact your directory indices for you.

Let me just confirm that we're both on the same page before I give suggestions: the memory usage shoots up ONLY when the progress of the erase is past the 90% mark? OR does the memory usage gradually increase until the 90% mark?

If you could just take some data points while doing an erase, that would be great. There's no need to get past 90% (and I'd advise against it -- crashing and modifying file system structures probably shouldn't exist in the same sentence) so hit cancel when that's at that stage. I just want to make sure it's the FAT code that's the problem.

As for debugging technique... do you have a spare drive of equal capacity? This could be asking for a lot here, but since you're familiar with Linux, I can do a few things with you that other users can't

[Joel]

Okay, I just did a static code analysis (i.e. the brain as the compiler, lol) and found out why memory's leaking like that. I've got a fix on my local machine that's untested, would you like to try it out? It should be in r2728 (so builds 2728 and later will have the fix)

I'll test it on a flash drive later.

[DavidHB]

I just did a static code analysis ... and found out why memory's leaking like that. I've got a fix on my local machine ...

This really intrigued me. I was under the impression that .NET did all the memory management, and so leaks should not arise as a result of program code, because .NET would clean up all the garbage.

I know that there are complex algorithms to determine what .NET regards as garbage, so I suppose that if program code does not behave in the way the algorithm assumes, memory requirements would grow. Is that how code can create leaks, or is there some other mechanism?

David

PS I subsequently saw in the change log that Eraser in fact uses unmanaged memory, and that is where the problem is. Which begs the question: why do you need to use unmanaged memory?

[Joel]

Because if I were to write the FAT code in C#, I'd probably foam at the mouth

FAT is so low-level that trying to express its intent in a high-level language like C# (where pointers and such are second-class citizens) is kinda committing Seppuku.

[DavidHB]

Ah. Well, we all have your health and well being at heart ...

David

[pcworld]

I'm sorry that I didn't reply sooner, somehow I didn't receive any e-mail notifications about the new posts.

Oh, FAT doesn't have an MFT, so cleaning it is VERY easy, and already implemented in Eraser's code. It also will compact your directory indices for you.

That's why I asked if there's an option to just clean the "inodes" instead of overwriting the whole partition.

Let me just confirm that we're both on the same page before I give suggestions: the memory usage shoots up ONLY when the progress of the erase is past the 90% mark? OR does the memory usage gradually increase until the 90% mark?

I'm not sure anymore. I only realized the high memory usage when the progress bar was near the end.

If you could just take some data points while doing an erase, that would be great.

I guess I should first test if r2728+ fixes this?

As for debugging technique... do you have a spare drive of equal capacity?

I fear I don't have one...

Okay, I just did a static code analysis (i.e. the brain as the compiler, lol) and found out why memory's leaking like that. I've got a fix on my local machine that's untested, would you like to try it out? It should be in r2728 (so builds 2728 and later will have the fix)

I hope I'll find some time to test it this weekend.

[Joel]

Have a go and let me know how it goes.

[pcworld]

Finally found time.
Tested with 6.1.0.2781 on the same hardware/OS/HDD as before
Here the results:

t: time the task started
t till t+90min: between 40 and 70 MiB (or whatever unit Windows' task manager uses)
t+92min till near the end of completion (t+105min): between 280 and 360 MiB

Definitely a lot less memory usage, I guess you fixed the bug!

Btw, I didn't receive an e-mail notification for your last post either (although I'm subscribed to this topic), is e-mail functionality broken in this board?

[Joel]

Great!

Email seems to be working for me, but I also had some email problems (on my own server) lately so I'm not sure...

[chuckles1066]

This really intrigued me. I was under the impression that .NET did all the memory management, and so leaks should not arise as a result of program code, because .NET would clean up all the garbage.

viewtopic.php?f=2&t=7540

I reported a similar problem a year ago, I seem to recall the response was along the lines "oh it's not Eraser it's some random third party bit of software"

Soooo......is it Eraser or not?

Incidentally, if you follow that thread you'll see that installing a new version of Eraser (same .NET installation) fixed the problem.......so I guessed deep down that Eraser was the culprit