Discussion on:

NTFS allocation unit sizes for large volumes

How do you screw the focus on a screenshot?

Read the line below the Figure C image. Click to enlarge it. There's too much detail for it to be readable in the space allocated in the article.

I think they were simply taken and pasted in paint, then saved in .jpg with the worst converter ever made (the one in paint). When downsized they look blurry. Its not a matter of not being able to read them.

Because of MBR restrictions.

Can a large number of small files be made to share allocation units with compression?

For instance, we have large image library and avg file size is 50KB, so 64KB is the unit allocation, where as we also have documents that are on avg 10 to 12KB, so 16KB is the unit size. Unfortunately in storage, one size does NOT fit all. You have to put some work into it.

64Kb - 50Kb = 14Kb wasted space
16Kb - 12Kb = 4Kb wasted space

rather than
(13 * 4Kb) - 50Kb = 2Kb wasted space
(3 * 4Kb) - 12Kb = 0Kb wasted space

I use large allocation units for volumes that handle large files, such as video. Though I have not studied NTFS, my assumption is that each allocation unit has a checksum and is part of a link list or file table to track the units that are occupied by a file. I conducted a couple tests with large files which bore out that a volume with larger allocation units was faster, both reading and writing.

If your storing on volumes as large as this article is talking about, it may not really matter. The allocation unit is part of the equation of how large a volume can be, which is why the minimum size increases as the volume size passes the thresholds mentioned in the article.

Easily if the compression you use consolidate multiple files into a single file (such as using WinRar to create a .rar file). As for NTFS native compression, it will use the allocation unit size as the smallest unit. The size on disk will not decrease unless you compress multiple files into a single file or compress files from a multitude of units to a smaller number of units (ie large file compression.) It is never a good idea to compress a folder using NTFS compression if it is exclusively 1-2 unit size files. You will just be wasting resources and not getting any smaller data.

compressiion is a bandaid, for an active data store. The wasted processor cycles needed for compression/decompression, just off-loads the problem from storage to cpu.

It is "generally" cheaper to by larger, faster storage (without endangering production during peak access) than to upgrade servers.

I would think that a virtualized file allocation that uses the big allocation units to their fullest would be both doable and sellable... but I know jack about file systems, so I could be wrong.

Back in Novell NetWare 4, they came up with a good answer to this problem. You created larger blocks and when then if you had small files they suballocated some of the blocks. Let's say you use 64K blocks and then suballocate a 64K block into 64 1K blocks, this gives you the best of both worlds.

If you're running a SAN disk or a disk on any other sort of RAID it's a good idea to try and align the Allocation Unit with a stripe by either having them the same size for, say, a database or multiple if you're trying to waste less space. Before that's any good, though, you've got to align the start of the partition to the RAID stripe boundary. That may happen automatically with Windows 2008 but will be completely off if you create your patition with the GUI in windows 2003. Most people miss this one and can get a performance hit of as much as 30%.

How do you align the start of the partition to the RAID strip boundary?

My guess is that assuming 64K stripes, you would start the partition at the very beginning of the drive, or at a multiple of 64K.

Windows reserves and hides the first 31 or 63 sectors at the beginning of the first partition of a disk and puts the master boot record in there. It's not completely Microsofts fault but it's taken them a HELL of a long time to fix it! This results in misalignment with stripe units, disk controllers, and cache segment lines as the reserved sectors do not coincide with physical boundaries. The obvious result is that single clusters of user data are written across multiple stripe units of the RAID. Every nth operation is affected in that two reads are required to get one allocation unit. 'n', of course, depends on allocation unit size and stripe unit size but every part of the disk system is affected.

To get around this in Server 2003, partition alignment must be explicitly performed using Microsoft DISKPART. Now, you can get realy lairy and calculate the correct offset depending on disk sector size but the easiest way is to start the first partition on a 1MB boundary and that nails it for just about any combination. That's 2048 sectors.

Diskpart is a command line tool so...

Diskpart [enter]
List Disk [enter] will display the disks
Select Disk n [enter] (where n is the disk in which you wish to create the partition)
create partition primary align=1024 [enter]
exit [enter]

Then go on and format the parttiion in the GUI



This is absolutely mandatory for anything with a database running on RAID - Exchange, SQL, Oracle, whatever.

http://support.microsoft.com/kb/929491

Is the MBR on all drives, or only OS partitions?

Wondering if this affects my situation where the OS is on C, and databases are on D and E (all are different spindles/raids)...

or RAID LUN - that's the problem with the bloody things!

RAID database disks is where they really screw up the performance. Subsequent partitions on the disk, RAID or LUN take their offset from the end of the first one. Unfortunately, the only way to straighten your partitions out is to wipe-and-redo.

On volumes created with Windows 2008 it performs the alignment corrently...So I've read.
Thanks Neilb for your information. It has been very helpful an has given me some things to look at now. My concern is that alignment can't be done on Dynamic disks and that is how software raid for windows 2003/2000 is done in our environment. Most are RAID 1, I do have one RAID 5 array that is on an HBA, with 15k seagates.
Although I read that performance may be slower than expected, I haven't found any real numbers to say what amount are we talking about? When is performance affected most under reads/writes, all the time? If its less than a few percent, it might not be worth it to me to go through the changes here. Now combine that with the incorrect 4k cluster sizes they defined here and maybe I can see a 5% improvement. That would be nice.

There is a poor trade-off, if you are unaware of the nature of the data being stored.
The larger the unit sizes, the more potential waste - if the data does not require large extents.

For example, a file server, which contains mostly documents, text as opposed to a a DB server where you have predictable data sizes as well as predicatable access requirements ... mostly read and store vs. frequent modifications.

is that storage space is as cheap as chips. Performance should really be the only driver in current systems.

Neil

Doesn't mean you should. Yes, storage can be cheap. And I'd argue that you can always find a lesser expensive storage solution, but do you want to do that?

Also keep in mind that if you're going to use the Shadow Copies feature, you should use a 16K or larger cluster size. http://technet.microsoft.com/en-us/library/cc786349(WS.10).aspx

Unfortunately, I didn't follow best practices when I created a 2TB volume and eventually Shadow Copies started causing the volume to crash every day at the scheduled snapshot times. Took me a while to figure out it was Shadow Copy causing the problem.