Hi,
I think it's a good idea. In terms of large memory consumption, perhaps we could start caching when the size of the input test drops to some value (e.g. 10k)?
Or maybe an alternative would be to store the cached results as files on disk? Now that only every pass is cached it shouldn't be too much I hope. In this case a hash of the content could be used as key in the cache map, pointing to the location of the file (or rather the file name).
Another thought I had but which might be wrong: Take the position of the pass within an iteration -- currently $pass_num in the Perl code --, the pass name and argument, and a hash of the file contents before the pass has executed and store this information in a cache. If we now come to the point -- in a later iteration -- where this information is already in the cache when we want to store it, can we then abort the reduction immediately because apparently nothing has changed during a whole iteration?
Could that help to terminate the last round of the reduction when nothing changes earlier? Or have I forgot something that invalidates my logic?
Regards,
Moritz