RAM usage

GroupDocs.Search Product Family
,
1

Hello,

We are experimenting with “big data” and GroupDocs.Search.

Putting 10k files into the index works, putting 100k files into it works too (because of the recent ArithmeticOverflow fix), but indexing 1 million realistic files seems to be impossible for now.

We have extraction and adding bytes to the index + calling optimize separated. The extraction happens in a different process. What we experience is: When adding stuff to the index over hours, the memory usage of that process becomes higher and higher (passing 10 GB of RAM, then 20 GB)… And at some points, it exceeds the 32 GB of RAM that our system has and there is an OutOfMemoryException:

"System.OutOfMemoryException: Exception of type ‘System.OutOfMemoryException’ was thrown.\r\n at System.Runtime.CompilerServices.RuntimeHelpers.AllocateUninitializedClone(ObjectHandleOnStack objHandle)\r\n at System.Runtime.CompilerServices.RuntimeHelpers.AllocateUninitializedClone(ObjectHandleOnStack objHandle)\r\n at \u0005\u0010\u0002.\u0002()\r\n at \u0006\u0002\u001B.\u0002(\u0006\u0002\u001B \u0002)\r\n at \u0006\u0003\u0002.\u0002()\r\n at \u0006\u0003\u0002.\u0002(\u0006\u0019\u0003 \u0002, \u0006\u0015\u0003 \u0008, \u0006\u0018\u0002 \u0005, \u0002\u0002\u0017 \u0006)\r\n at \u0005\u0003\u001B.\u0002(Boolean \u0002, Int32 \u0008, \u0006\u0019\u0003 \u0005, \u0006\u0015\u0003 \u0006, \u0006\u0018\u0002 \u0003, OperationType \u000E)

Even when stopping the process and restarting it again… When the indexes are freshly loaded from disk, the private bytes of the process grow to 10 - 20 GB again within a short amount of time.

Questions:

  1. Is there a way to consume a GroupDocs index with less of it being loading into the RAM (e.g. a tradeoff option in the settings… Slower search, but less space…)
  2. Is the full stuff from disk (the index folder) loaded into the RAM in general?
  3. Is it even possible to put 1 million files into an index? Have you ever tried it?
  4. Is there an estimation how much RAM one needs for an index that is 25 GB on disk? Is it 1:1? A GB of RAM for each GB on disk…
  5. What could we do to not have to ask our customers to “just have 100 GB of RAM”? And what could be done from your side?