Desktop

Ask something
Discussion Locked

Problem deleting large files (WinNT40)

0 helpful
11 replies

by time traveller 1 view

Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

I asked a similar question about a month ago, and received some answers, but none were satisfactory or consistent with what I'm seeing.
Here's the problem:
A user has a modelling program which produces both huge (1 to 20 Gigabyte) and numberous tiny (100 to 1000 bytes)ASCII text files. When the modelling run is completed he can have as many as 20 of the large files and literally hundreds of thousands of tiny files in a single folder.
The problem is that when he goes to delete the huge files(since they are transitional), the disk is locked to any further I/O for as long as 5-10 minutes. The drive light shows that the drive is active but Task Manager CPU useage counter shows the Idle process running at 99%. Also all perf mon counters indicate that no I/O acitivity is occuring on the disk...there are no reads, writes or anything else...yet the drive is unuseable for 5-10 minutes.
The reason this is an important problem is that this disk is connected to other computers which are doing similar modelling runs, which do disk I/O to this disk which becomes locked, causing the modelling to be aborted on the other computers whenever one of these large transition files is deleted. Note,however, that the problem still manifests when there is no connection to other computers.
Gory Details:;
System is a 1GHz PIII, with 512 Meg RAM, and the disk in question is a 80 MB Ultra100 disk.This disk also contains the system partition. It is NTFS. The OS is NT40 SP6a.

11 replies

by maxwell edison

Greetings,

According to Microsoft article Q184537, "Very Large Files Cause Performance Problems", The following may apply:

SYMPTOMS
Some user interface (UI) features, such as deleting files and viewing the properties of files, do not work. Also, the use of the WIN32 API, SetEndOfFile, generates inconsistent results.

CAUSE
The Windows NT user interface handles large files, for example, 200 to 300 GB, as buffered files rather than nonbuffered files. When a file is opened in a bufferedmanner, the system attempts to load the file into memory which it cannot do due to the size of the file.

RESOLUTION
To resolve this problem, obtain the latest service pack for Windows NT 4.0 or Windows NT Server 4.0, Terminal Server Edition. Foradditional information, please see the following article in the Microsoft Knowledge Base:

Q152734 How to Obtain the Latest Windows NT 4.0 Service Pack

STATUS
Microsoft has confirmed this to be a problem in Windows NT 4.0 and Windows NT Server 4.0, Terminal Server Edition. This problem was first corrected in Windows NT 4.0 Service Pack 4.0 and Windows NT Server 4.0, Terminal Server Edition Service Pack 4.

The article may be viewed at the following link:

http://support.microsoft.com/support/kb/articles/Q184/5/37.ASP

(REMOVE SPACES from the pasted URL if they snuck in.)

Maxwell

by maxwell edison

.
.
You could also try to use a third party software for the file management tools. Programs like Norton Commander, Singularity or Windows Commander.

Good luck,

Maxwell

by maxwell edison

.
.
Something else you might consider is this. Get a good disk optimization program (similar to but better than scan disk/disk defrag) that will optimize the drive AND put files in contiguous space. You could run the program nightly (when no one is accessing the system), and perhaps it would be less taxing on the system if it could find all the file segments on contiguous disk sectors instead of scattered around on the disk.

Maxwell

by maxwell edison

.
.
Perhaps a disk cache and/or ram disk software could be your solution. There is one called, "SuperSpeed" software for Windows NT environments. I'm sure there are other similar products as well, but the one I saw was: http://www.superspeed.com/
Maxwell

by maxwell edison

You rejected my answer while I was still posting comments, so I?m not sure if you saw my last two comments:

Something else you might consider is this. Get a good disk optimization program (similar to but better than scan disk/disk defrag) that will optimize the drive AND put files in contiguous space. You could run the program nightly (when no one is accessing the system), and perhaps it would be less taxing on the system if it could find all the file segments on contiguous disk sectors instead of scattered around on the disk.

OR

Perhaps a disk cache and/or ram disk software could be your solution. There is one called, "SuperSpeed" software for Windows NT environments. I'm sure there are other similar products as well, but the oneI saw was: http://www.superspeed.com/

I like my last suggestion best, the ?SuperSpeed software. Not only may it help solve your ?deleting? problem, but it may have other benefits as well. Here?s a link to their product page:

http://www.superspeed.com/superspeed/allproducts.html

I?m out of ideas.

Best of luck,

Maxwell

by curacao_dejavu

Like it or not, this will be a new investment.
You are doing a Server job on a workstation.
You have to go scsi, raid 5.
The very very least you can do is to to schedule a regular defragmentation of the hd (won't help too much).

by time traveller

Thanks for your response but....
Actually had the same problem on my server, where the problem was much worse...have a SCSI Hardware RAID 5 controller, and it brought the server to it's knees for 15 minutes...every time a file was deleted...which was pretty annoying to the other 200 people on the LAN. Here's the catcher...I can delete those big file no problem on my P333 MMX test machine with a garden variety ATA33 onboard IDE controller

by marccouture

Hi,

Seems to me that file locking mechanisms are the culprits here. These can be a local issue (where local processes are still accessing files, or you have a caching IDE controller or other similar hardware) or a networking related issue (whereupon a local or remote network process is accessing files, even if peer-to-peer or centralized network connections are not apparent).

One thing you might want to look at is having a shareware/freeware utility run on the machine at boot time to clean/delete certain files (the same as what you would do to securely erase index.dat files on your machine to cover your Internet surfing habits, for instance). One nice utility I have successfully used for this type of task is Necrofile (http://www.necrocosm.com/nfpro.html). Very useful, especially for secure disk or selective file wiping.

Good luck!

Regards,

Marc Couture

by time traveller

Thanks for the suggestion. Since your suggestion is a workaround rather than a solution, I'm going have to leave this question open. Really, I don't want to have to reboot to delete files, the goal is to delete them when they need deleting. I don't think these are file locking issues...usually you don't get the CPU pegged at 100% when this is going on, ..my experience is that you get a long timeout and then a message saying that the file can't be freed for exclusive use or something like that. In this case, the file does actually delete but it just takes forever.
It's interesting that Western Digital never responded to my request for help on this, as I thought it must be hardware issue, since "older slower smaller" hard drives did not have this as an issue.

This discussion is closed to new replies.