Remember current session scans

[eradicator]

First up, great program!

Been using it for a little while, and the main thing that bugs me is the fact that if I scan say C drive, look at the results, then scan another drive, say D drive, then want to look at the drive C results again, the program needs to run the scan again to be able to display the results.

Would it be possible for the program to remember current session scan results?

Like atm, you double click a drive and it goes and re-scans it, maybe if you just single click a drive, it displays results of the scan you ran on it earlier.

[seanw]

First up, great program!

Been using it for a little while, and the main thing that bugs me is the fact that if I scan say C drive, look at the results, then scan another drive, say D drive, then want to look at the drive C results again, the program needs to run the scan again to be able to display the results.

Would it be possible for the program to remember current session scan results?

Like atm, you double click a drive and it goes and re-scans it, maybe if you just single click a drive, it displays results of the scan you ran on it earlier.

This is a difficult request, and you're not the first to make it.

Here's the problem: Sure, I can easily make it not drop the old information. But that comes with consequences, and some of them are pretty severe:

• If the old information is retained, that takes up memory. SM uses a lot of memory already to do what it does, and some people using SM literally have exhausted two gigabytes of RAM trying to scan multi-terabyte network drives. Some people have requested a 64-bit version so they can use four gigabytes or eight gigabytes or more of RAM to store the scan in. So let's say you scan a large disk like that, and then you scan another, and then SM runs out of memory to scan the second because it still has all its data on the first; that's not good.
  
  Even on an ordinary PC, many people are using an appreciable fraction of their computer's RAM to scan their computer's hard drive. Adding a feature that produces "out of memory" errors is not a good idea.
  
• The old information is, in a word, old. Normally, SM tries to watch your disk for changes, and updates the information accordingly (this will be a bit faster and more reliable in v2.2 when it comes out). So how should SM behave with a disk it's not actually watching? Is it safe to display files that may not actually exist anymore? Should it display a little notice that says that this display represents old data and may not be accurate? And if it does, the user's reaction will be to do a full rescan anyway, so how is that an improvement in usability?

I have some lingering plans for an "always-on scanner," designed for non-network environments. The "always-on scanner" would, somewhat like the "invisible" indexing services used by Vista and by Google Desktop Search, always be monitoring and recording the state of your disk(s) so that when you open up SM, a scan would be blindingly fast, since the information would always be known; however, implementing that kind of scanner is no small task (especially if it's to be implemented securely). That kind of service would obviate the need for "remembering" or "forgetting" previous scans, but it's a complicated piece of software to build, to put it mildly.

In the interim, the current solution works, even if it's not as straightforward as some people might like.

[jaqie]

Nowhere near as difficult as you make it out to be, seanw. just do a check for free RAM every now and then in the code (a good rule of thumb would be whenever something you are going to do will take more then 1% of the total physical RAM in the system, do a check to see if there is any 'old data' you can clean out), and if it's low dump all 'old' info, also have a checkbox in options to completely disable saving of old information as well for those that do not want it to save the old info ever.

Think of a FIFO type cache...

[seanw]

I know well how LRU caching and FIFO queues work: SpaceMonger already uses both those and some dynamic programming algorithms to give you the "instantaneous" treemap displays that you're used to.

The issue is that there's a lot more data stored in SM than just a list of files, and a lot of it is very interdependent. Most people who look at SM think the data inside it is a list of files, or maybe a tree; in reality, it's far closer to a gigantic spider-web of links between file data, ancillary data, dynamically-computed-and-cached data, selection lists, global statistics, and more. You can't simply take away part of a graph that's that complicated without the graph falling apart.

In a (distant) future version, I'd like the ability to invisibly archive old scans to disk in a high-speed database so that they can be quickly reloaded if you switch between drives, but what you're asking, a simple LRU cache, is nowhere near as feasable as it might initially seem.