Hello,
We have observed in the past the indexes can get very large. Most people would probably expect the index to be 10% to maybe 30% of the total size of the original files. Unfortunately, we have observed values over 50% and you yourselves have observed that original files with a size of 466GB lead to an index of size 440GB (that’s over 94%) → RAM usage - #2 by yuriy.mazurchuk
| Source corpus | 986,262 files, 466 GB |
|---|---|
| Documents indexed | 969,288 |
| Extracted text | 91.2 GB |
| Resulting index | 440 GB across 740 segments |
| Machine | 32 GB RAM |
| Outcome | completed, no errors |
I know that having a smaller alphabet and many stop words can reduce it a bit, but could you please find a general solution for a more efficient index storage?
Best regards
Jam