24/7 working programs, like syncservers ?

Please post your support questions here.

Moderators: Eraser DevTeam, Eraser Moderators

24/7 working programs, like syncservers ?

Postby Ghost62 » Mon Apr 09, 2012 12:13 pm

Hi, i have a doubt ... if i plan a scheduled cleanup for the unused space (or also just a hand-launched one) on a machine that is running a permanent syncserver program, does this affect the files handled from the program ?

Basically, the syncserver is simply a program that run in background in permanence (like, as example, some P2P programs that work 24/7), and continuously check for the connection of a group of users, keeping always synchronized their workgroup files ... for do this, the program always read and write the content of the archive disk, saving files also on-the-fly, and due to the fact that the disk is NTFS this may also be done using "sparse" files handling and "reserved space" for files (when the connection is slow, and a file is too big for be transferred in a single step, or when the connection fail, so the program anticipately "reserve" the space for the file, that can also be transferred in multiple parts in different times)

I have not made that server, only host one of its datapoints, so i do not know exactly how it manage the files ... need to know if, for any reason, a "clean unused space" operation can delete or corrupt those parts of files or incomplete files, and if that operation can work in concomitance with the syncserver program, or the fact that the syncserver is continuously and randomly writing on the disk can cause problems.

Thanks for any informations, and best regards.
Ghost62
 
Posts: 14
Joined: Mon Apr 09, 2012 11:56 am

Re: 24/7 working programs, like syncservers ?

Postby DavidHB » Mon Apr 09, 2012 3:50 pm

In considering the answer to this question, you need to understand how erasing free space works. I think you will find that there is good news and bad news.

A free space erase is in fact, three separate operations carried out sequentially. Firstly (and optionally), cluster tips (the unused spaces between the end of the file data and the end of the final, partially used, block of space allocated to each file) are overwritten, for those files which the program has permission to access. Then randomly named files containing (in the default case) random data are written to the drive until it is full, and these files are deleted. Finally MFT entries marked as unused or belonging to deleted files are overwritten.

These operations, particularly the second, need, as you would expect, as much hard drive bandwidth as the machine can provide; free space erasing is a slow process and speed is limited primarily by hard drive performance, so the standard advice is to have as few programs as possible running in the background. In particular, for reasons which I hope are obvious, it is usually a good idea to pause antivirus programs while a free space erase is running.

The good news about Eraser from your point of view is that, apart from cluster tip erasing which can be disabled, free space erasing does not touch any space not marked as free. Also, Eraser knows about sparse files and leaves them well alone. If the process of reserving space for files is, in effect, itself a file creation process, Eraser would not touch any space allocated to the file.

The bad news is (1) erasing free space can (obviously) compromise disk performance for applications using the target drive and (2) results may be unpredictable if files are written to the drive at the point when the free space erase has filled or almost filled the target drive. This assumes that the target drive (including such things as RAID arrays that the OS sees as single drives) is where the sync server is storing its archive. Erasing free space on other drives on the same machine which are not used by the sync server would not, I guess, give the server any problems.

You will gather that Eraser was not designed for the use you describe. While I cannot be sure, I think that running a free space erase on a drive on which a sync server is actively running is likely to be problematic, and, in the worst case, could crash the machine with a full drive. If the the sync server is off line, doing a free space erase while it is not active should not create problems.

I hope this helps.

David
I am not an Eraser programmer, but a long-time user; my views may not be the same as those of the Eraser programming team.
Before posting, please read the top 4 topics in the Eraser FAQ, which already provide many of the answers users need.
DavidHB
Eraser Wizard
 
Posts: 2166
Joined: Sat Jan 23, 2010 8:10 pm
Location: Isle of Wight, UK

Re: 24/7 working programs, like syncservers ?

Postby Ghost62 » Mon Apr 09, 2012 5:59 pm

Thanks, i think i've understood all (hope :))

Anyway, the syncserver is not a true "full server" unit, is a program that is running on a PC connected permanently to the internet, and that use a separate disk for store the files of the workgroup and keep them synchronized each time one of them connect to the net, or modify and save part of the work (so all the ones that are connected can have the last versions of all the files) ... it's not full time writing, but that operation is discretely frequent ... more when is sending or receiving files, ofcourse ... so, seeing that what you say, is probably always better to stop the syncserver program, before doing a cleanup on its own disk.

I was hoping that it can be possible on-the-fly, like with the defrag program, cause our workgroup actually only have 2 machines with the syncserver installed and that can stay up 24/7, so all the times one of the 2 stop, the other get a bit overloaded and all slow down ... but if that cannot be done, well, patience.

And yes, i already used it on the other disks where the syncserver does not save its own files (system, storage, and backup disks), and none of them had problems performing the cleanup, nor gave any problem to the syncserver, except a little slow-down in saving new data (but i think this is normal ... being working to another disk, to slow down a bit the MB controller response, i mean), and the cleanup worked good too.
Ghost62
 
Posts: 14
Joined: Mon Apr 09, 2012 11:56 am

Re: 24/7 working programs, like syncservers ?

Postby DavidHB » Mon Apr 09, 2012 9:16 pm

I understand. Unless Joel thinks otherwise, my advice would be to use only one of the syncserver machines while you do a free space erase on the other. Expect the erase to run for a long time (typically many hours, but it will depend on the speed of the drive and the amount of free space to be erased). If the drive is more or less de-fragmented, that should help in my view.

David
I am not an Eraser programmer, but a long-time user; my views may not be the same as those of the Eraser programming team.
Before posting, please read the top 4 topics in the Eraser FAQ, which already provide many of the answers users need.
DavidHB
Eraser Wizard
 
Posts: 2166
Joined: Sat Jan 23, 2010 8:10 pm
Location: Isle of Wight, UK

Re: 24/7 working programs, like syncservers ?

Postby Joel » Tue Apr 10, 2012 12:09 am

The way you describe your server program is such that it will preemptively allocate extents on the disk ("preallocation.") How does it accomplish that? If you use the Windows APIs, then it should be fine. But if you're just keeping tabs within your program and nowhere else, you'll be running into problems (not just with Eraser)

My main concern here is data integrity -- slow downs should not be that big of a problem since you are sending the data over the internet, which is inherently slow.
Be sure to read the FAQ before posting. If you found this application useful, please contribute to Eraser's development.

I develop Eraser but I am not an employee of Heidi Computers Ltd. My views do not represent those of Heidi Computers Ltd.
Don't PM or Email me questions: they won't be answered any faster than on the forum and knowledge won't be accessible by all.
User avatar
Joel
Eraser DevTeam
 
Posts: 3688
Joined: Sat Aug 19, 2006 12:16 am
Location: Singapore

Re: 24/7 working programs, like syncservers ?

Postby Ghost62 » Tue Apr 10, 2012 7:47 am

Hi, and thanks again for the informations.

I don't know exactly how the program manage the reserved space, i have not built it personally (in the workgroup i'm not a programmer, my main parts are hardware tech, prototypes planning/building/testing, and sometimes 3D modeling ... my knowledge about programming languages is very low) ... I just have on my machine one of the 2 syncservers, cause i can keep it working and connected 24/7, and had space for add a 4th hard disk inside it.

Anyway, i asked the person that built it, and he said that, for save time, he used part of the "core" source of a P2P program, EmuleMorph, and modified it for convert it from a P2P application to a private synchronizer tool, adding encryptions and safety and autosynchronization features, but, he said, he have not changed the way in which it store the files and reserve the space, other than adding an automatic backup copy and crc checks (as he said me, the program now make 2 copies of each file on 2 different folders, and check their reciprocate CRC each time it need to send the file to an user, for more safety, but have not changed any of the codes that write the files and reserve "sparse" space for incomplete files, so writing and deletion of the files are performed in the usual windows ways ... whatever it exactly means :P ), and that these routines are probably already known from programmers cause the source is public ... can this help in knowing how it work ?
Ghost62
 
Posts: 14
Joined: Mon Apr 09, 2012 11:56 am

Re: 24/7 working programs, like syncservers ?

Postby Joel » Tue Apr 10, 2012 9:37 am

Then in all likelihood it should be fine. Test first, though.
Be sure to read the FAQ before posting. If you found this application useful, please contribute to Eraser's development.

I develop Eraser but I am not an employee of Heidi Computers Ltd. My views do not represent those of Heidi Computers Ltd.
Don't PM or Email me questions: they won't be answered any faster than on the forum and knowledge won't be accessible by all.
User avatar
Joel
Eraser DevTeam
 
Posts: 3688
Joined: Sat Aug 19, 2006 12:16 am
Location: Singapore

Re: 24/7 working programs, like syncservers ?

Postby Ghost62 » Tue Apr 10, 2012 2:34 pm

Thanks.

I checked with other users and can do it this way ... next sunday, i can ask the main programmer to disconnect its own server (for safety, cause they interact), make a backup copy of the files on another disk, and then start the syncserver, asking to some of the users to left their machines connected (also if they are not working, their syncserver clients check for updated files each 30 minutes, so my server will still receive requests), then launch the wipe free space cycle, and see what happens ...

If this gives any problem, or if it work good, i will let you know after this try.
Ghost62
 
Posts: 14
Joined: Mon Apr 09, 2012 11:56 am

Re: 24/7 working programs, like syncservers ?

Postby Ghost62 » Wed Apr 11, 2012 1:32 pm

Hi, sorry for double posting.

Today, when i tried eraser launched manually over some large files (Single pass by default, old toons, usually all over 300Mb, someone 700Mb or similars), it gave me always the same error for each file.

The file name disappear from the folder immediately, same as normal windows deletion, but the disk does not work and the message in the lower balloon say always "completed with errors" - then, when i open the eraser interface, i find all the operations in the "erase schedule" window, as not done, and when i right click on them and choose to do them immediately, they just disappear (no writing operations on the disk nor anything similar ... i tried on big files intentionally, so i can see if the disk works, cause for overwrite some 100s of MB it need time).

I tried to right click on them and show the logs too, and all of them are marked with the same error, it say that the file(s) "could not be erased because the file was either compressed, encrypted or a sparse file." ... btw, this til now happened with all the files (tried 20 of them, still have some for other tests anyway), regardless that i choose them one by one, or in groups.

Now, i Know that i have no compression enabled on my disks (only indexing from OS), and that the files was not encrypted, i was just testing the last version that i have installed (6.0.9.2343) over some old plain files that had to be deleted anyway for free space ... also, i'm using XP pro with all updates installed, and am logged as administrator ... can be that the files are saved as "sparse" from the OS itself ? ... but in this case, this mean that any file that the OS write as "sparse" cannot be deleted using Eraser, also launching the operation manually ?

Let me know if there are other tests that i can do for help with this.
Ghost62
 
Posts: 14
Joined: Mon Apr 09, 2012 11:56 am

Re: 24/7 working programs, like syncservers ?

Postby DavidHB » Wed Apr 11, 2012 1:45 pm

Quote from the Eraser manual:
Because encrypted files, compressed files and sparse behave differently when applying the standard erasure procedure, Eraser will not erase such files when they are encountered and will instead log an error.


David
I am not an Eraser programmer, but a long-time user; my views may not be the same as those of the Eraser programming team.
Before posting, please read the top 4 topics in the Eraser FAQ, which already provide many of the answers users need.
DavidHB
Eraser Wizard
 
Posts: 2166
Joined: Sat Jan 23, 2010 8:10 pm
Location: Isle of Wight, UK

Re: 24/7 working programs, like syncservers ?

Postby Ghost62 » Wed Apr 11, 2012 5:38 pm

Thanks.

Yes, i seen the exceptions in the user guide, but it does not refer to "sparse" files, only to encrypted and compressed ones ... so this happens also for sparse ones, and also for the direct erasure operations (i mean, when i select a file and say to the program to erase it directly).

Anyway, is strange that it see the files as "sparse" (i'm sure they are nor encrypted nor compressed), cause are just old files archived times ago in a folder, that i'm eliminating for make space ... is normal that the OS creates those files as "sparse" without different indications from the user ? ... if so, windows is working in a very strange way (ah, well, any windows user think that it work in strange ways, after all :P :D).

The only other possibility that come to me in mind, is the defrag program (third part one, not the windows one) ... maybe it stored the files in "sparse" mode ? ... really don't know.

By the way, this may cause problems to Eraser users ? ... i mean, the impossibility to erase, also intentionally, those files that the OS creates as "sparse" ? ... or, maybe, there is some settings for say to the program to ignore the fact that the files are "sparse" types, ad still overwrite them manually ? ... just an idea ...
Ghost62
 
Posts: 14
Joined: Mon Apr 09, 2012 11:56 am

Re: 24/7 working programs, like syncservers ?

Postby DavidHB » Wed Apr 11, 2012 6:56 pm

'Sparse' is a file attribute that programs can set. When it is set, the NTFS file system only allocates space for non-zero data, thereby (in effect) stripping out the zeroes from the stored data and restoring them only when the file is read. As many files have large blocks of zeroes, this obviously saves significant amounts of space. However, there is obviously also a degree of disconnect between what the MFT describes and what is present on the drive, so Eraser cannot safely handle sparse files and leaves them alone.

My guess is that whatever software created the large files marked them as sparse; with large files, it might very well make sense for it to do so. The only way to erase them is to delete them and then erase free space.

David
I am not an Eraser programmer, but a long-time user; my views may not be the same as those of the Eraser programming team.
Before posting, please read the top 4 topics in the Eraser FAQ, which already provide many of the answers users need.
DavidHB
Eraser Wizard
 
Posts: 2166
Joined: Sat Jan 23, 2010 8:10 pm
Location: Isle of Wight, UK

Re: 24/7 working programs, like syncservers ?

Postby Joel » Wed Apr 11, 2012 10:46 pm

From my observation, the most common program to make sparse files are BitTorrent clients - with good reason, since the stuff they are downloading off the network may not be immediately available, and it could be prudent to allocate space only when it is needed.

Unfortunately moving and copying the file around does not seem to get rid of the Sparse attribute.
Be sure to read the FAQ before posting. If you found this application useful, please contribute to Eraser's development.

I develop Eraser but I am not an employee of Heidi Computers Ltd. My views do not represent those of Heidi Computers Ltd.
Don't PM or Email me questions: they won't be answered any faster than on the forum and knowledge won't be accessible by all.
User avatar
Joel
Eraser DevTeam
 
Posts: 3688
Joined: Sat Aug 19, 2006 12:16 am
Location: Singapore

Re: 24/7 working programs, like syncservers ?

Postby Ghost62 » Thu Apr 12, 2012 12:18 pm

Thanks.

No, i'm not using bittorrent or other similar softwares ... also, the files are not on the syncserver dedicated disk (that use sparse files), but only on the other 3 disks ... i suppose it may be defrag program guilty, but i'm not sure about that, and also, i suppose that there is no ways for "manually" eliminate the "sparse" attribute, like as for "hidden" and similar (my XP Pro is set for show all the files and folders, but never seen "sparse" as attribute in any property windows, so i suppose only the OS can manage this ... and also searching the net, at least til now i've found no tools for manage this thing).

Anyway, if normally deleting them and then perform an unused space cleanup is enough, i think this is still ok ... probably, also better doing it after a defrag ( i mean, delete files > defrag disk > clean unused space, this may be also more efficent, right ? )

Well, anyway, if you need some deletion tests on "sparse" files, let me know ... i just discovered that both my data and backup disks are filled of "sparse" files that i can eliminate (around 800 files, aproximatively 350GB of total space, so the material for "evil" and "destructive" tests does not lack :D :P )
Ghost62
 
Posts: 14
Joined: Mon Apr 09, 2012 11:56 am

Re: 24/7 working programs, like syncservers ?

Postby Joel » Thu Apr 12, 2012 1:55 pm

No, defragmenters should be incapable of producing sparse files. It is an attribute that is set when the file is first created.
Be sure to read the FAQ before posting. If you found this application useful, please contribute to Eraser's development.

I develop Eraser but I am not an employee of Heidi Computers Ltd. My views do not represent those of Heidi Computers Ltd.
Don't PM or Email me questions: they won't be answered any faster than on the forum and knowledge won't be accessible by all.
User avatar
Joel
Eraser DevTeam
 
Posts: 3688
Joined: Sat Aug 19, 2006 12:16 am
Location: Singapore

Next

Return to Eraser Support

Who is online

Users browsing this forum: Yahoo [Bot] and 5 guests