What's the deal with "unscannable space"?

[seanw]

This is actually one of the more commonly-asked questions, and the FAQ touches on it, but we may need to expand the FAQ to include it a little more exactingly. But because it's asked so often, I felt it was useful to make a sticky here for it.


What does "unscannable" mean?

When SpaceMonger encounters a folder in its analysis of your computer, it attempts to go inside that folder and see what's in there. Normally, it keeps going deeper and deeper in your folders until it's collected everything. However, sometimes, usually for security reasons, Windows will tell SpaceMonger, no, you can't look inside a folder. So SpaceMonger lists that folder as "unscannable" --- SpaceMonger knows the folder exists, but Windows isn't letting SpaceMonger learn anything more about the folder than the fact that it exists.

There is usually no good reason for this information to be blocked (and some software developers have been very critical on this issue), but Windows does it anyway, which means you and I have to deal with it.

In short, an unscannable folder is a folder that Windows isn't allowing any application --- including SpaceMonger --- to access.


What's typically marked "unscannable"?

The most common (large!) folder that is marked unscannable is "System Volume Information," which is where Windows stores System Restore Points and the Vista Indexing Service's data (for faster searches).

The second most common (large!) folder that is marked unscannable is called "RRBackups." This folder only appears on Lenovo computers, and it stores backup information in case you ever need to perform a Lenovo rescue-and-restore operation.

Other folders may be marked unscannable for various other reasons: Norton's Protected Recycle Bin, various folders from antivirus software, and various folders from virus software too. I've even seen the occasional device driver have an unscannable folder.


How do I fix it?

This is the hard part, because Windows makes it hard.

1. Applications with the SE_BACKUP privilege can often access protected folders, but in some cases, especially on Vista, this doesn't always work. Still, Schlitzer, one of our more enterprising forum members, has put together a nice little program that gives SpaceMonger SE_BACKUP privilege, which may allow you to scan the folders in question. Future versions of SpaceMonger will come with SE_BACKUP privilege built-in.
   
2. You can mark a single folder as readable manually. On Windows XP Pro and Vista, you can (sometimes!) right-click the folder, select "Properties," go to the (sometimes missing!) "Security" tab, and add yourself or "Administrator" as a user allowed to access that folder. (This thread discusses this in more detail.) If the "Security" tab is missing, you can fix this via a command at the command-line (The Elder Geek has a good article on this here); or you can try to force Windows to always display the "Security" tab, which is described in detail here.
   
3. If you're on a Lenovo or IBM computer and think there's a lot of data in a Lenovo "RRBackups" folder, the easiest answer may be to check your Rescue-and-Restore application and see how much space it's consuming. Many users have reported that it works to just tell the Rescue-and-Restore application to stop eating up half their space.

What other reasons could there be that my hard drive space doesn't add up?

1. Your hard drive's filesystem may be damaged and in need of repair. At a command line, use the "chkdsk /f" command to repair it and see if that fixes the problem. Microsoft has a good page on using chkdsk, so I won't repeat that information here.
   
2. In rare circumstances, some of your space may be in Windows' "Alternate Data Streams," which provide a way to store data in a file without it being visible to anything except the application that stored the data there. Some backup programs and antivirus programs make use of this "feature," and some obscure database programs do too. (I have yet to see a circumstance where this "feature" was more helpful than harmful.) If you've exhausted all other possibilities, you may want to try searching for alternate data streams, since SpaceMonger (and most other applications) can't see those. Some useful searching tools and information on this topic can be found here.
   
3. In other rare circumstances, you may have a rootkit installed, a piece of software designed specifically to alter Windows itself to hide other software. Some viruses/malware use rootkits, and Sony got into big trouble a couple of years ago for using one too. Microsoft/SysInternals has a good program called RootkitRevealer that will help you ferret out these nasty things. Microsoft's page includes both the program itself and a good description of what rootkits are and how they're located. (Unfortunately, if you have one, they're often incredibly difficult to remove, and may in some cases require a complete reinstall of Windows.)