Available Space Misreported by USB drive?, Part 2

A recent question raised a red flag in my mind.

I think most, if not all, of you read my 'brief' description of about what happened to the missing 32 MB on a 512 MB Thumb Drive:

Available Space Misreported by USB drive?

Allow me to explain in greater detail.

On a hard drive, thumb drive, or any other storage device in common use, the capacity displayed on the carton is the maximum 'raw' or usable number of bytes the device can hold.

However, a device that 'says' it can hold 512 MB of data will not display the number 512 MB after it has been formatted. As stated in the previous article, some space on the device will be taken up by sector marks and the file system table, which contains information about each sector on the drive.

But that isn't the whole story.

A sector is universally 512 bytes long. Sectors divide a drive up into tiny chunks that are used to store data. When the drive is formatted, each sector is given an address as it is marked so that the operating system can locate and use the sector.

The file system table then 'maintains' sector addresses such that the information stored inside the sector can be accessed by the operating system.

Futhermore, Sector Chaining (where sector 'A' is linked with sector 'B' and so on) is used to allow large files, bigger than 512 bytes, to be stored easily. The list of sector addresses in the chain is also kept in the table.

As you can probably see, the situation is getting complex. And yet, there's still more to know! ;-)

To keep things controllable and make it easier to handle, an Operating system will group sectors together into clusters. It is an attempt to keep a file's sectors together as much as possible. The term used is 'contiguous' drive space, where sector 'A' is physically adjacent to sector 'B', which is next to sector 'C' ,and so on, all of which makes retrieving the file quick.

If, however, if the file is 'fragmented', the sectors are not next to each other but scattered across the drive and the retrieval process is much longer since each sector has to be looked up, located, and read.

That's why Defragmenting your hard drive is necessary from time to time. It rearranges the file's sectors so they are 'next' to each other as much as possible.

There is one more piece to the storage puzzle we have to discuss, and that is sector clustering. Depending upon the size of the drive and operating system, sectors are grouped together in clusters to make the process of reading and writing files and data quicker and easier.

Since we are looking at Windows, there are three different file system protocols, FAT-16, FAT-32, and NTFS. I won't babble on about the history or reasons for these protocols; however, what's important to understand is that each protocol has a different default size for clusters. ;-)

In the case of the 512 MB Thumb Drive, each protocol considers that amount of storage to be 'large'. For FAT-16 the default cluster is 32 sectors or 16,384 bytes. For FAT-32, the cluster is 64 Sectors or 32 KB and NTFS uses 128 sectors or 64 KB.

This all helps keep things moving quickly and smoothly which is the reason why each protocol is used. The protocols are designed to effectively and efficiently support the average user so far as file size and type are concerned.

However, there are certain trade-offs.

If you have a file that is much smaller than the cluster, there is wasted space because the OS tends to try and place a single file in a cluster. In addition, there are times when the clusters do not all fit on the drive.

In our sample 512MB Thumb Drive, we would expect to see 1 Million sectors divided up into 8000 clusters on an NTFS file system.

Since space is lost due to the File Table I mentioned earlier, you actually lose a few sectors because the clusters do not completely fill the remaining left-over space. Therefore, you lose drive space because the OS will not write to or read from 'undefined' space, which is exactly what the left over room is. Since it does not fit in a full cluster, the space is ignored.

There are ways to improve the situation, but it requires a full drive format to change the cluster sizes or to change from NTFS to FAT-32 or FAT-32 to NTFS. I strongly advise you to NOT change things unless you know what you are doing and fully understand the process. Just accept the way things are and enjoy your computer.

HAVE FUN!