Indexes with over 33%, sometimes even ~100% of the size of the original data

GroupDocs.Search Product Family
,
1

Hello,

We have observed in the past the indexes can get very large. Most people would probably expect the index to be 10% to maybe 30% of the total size of the original files. Unfortunately, we have observed values over 50% and you yourselves have observed that original files with a size of 466GB lead to an index of size 440GB (that’s over 94%) → RAM usage - #2 by yuriy.mazurchuk

Source corpus 986,262 files, 466 GB
Documents indexed 969,288
Extracted text 91.2 GB
Resulting index 440 GB across 740 segments
Machine 32 GB RAM
Outcome completed, no errors

I know that having a smaller alphabet and many stop words can reduce it a bit, but could you please find a general solution for a more efficient index storage?

Best regards
Jam

2

Hi @jamsharp

Let me share a brief analysis

Your attempt 1, 38.3 GB:

size share what it is
index*.body 24.08 GB 63% postings — the actual index
index*.head 12.20 GB 32% term offset tables
index1.info 1.86 GB 5% term dictionary
index.info 0.15 GB 0.4% doc metadata

A third of your index is bookkeeping, not index.

As mentioned before, one of our nearest plans is to optimise these values and the indexing process.
Here are few suggestion that probably you are aware of

  • One Optimize() at end of ingest (not every 300 MB). Merges segments, which collapses the .head overhead — that’s ~11 GB back on your current index, today, no new build needed.

  • Alphabet tuning. 238M unique terms isn’t vocabulary, it’s hex, base64, GUIDs and OCR noise. Making digits and +/= separators kills most of it, and shrinks the dictionary and the postings together.

  • UseStopWords = true. Stop words are 30–40% of tokens and you’re storing full positional data for all of them.

On the 10–33% expectation

Honest answer: for a positional full-text index that also stores case variants, 10% isn’t reachable — the postings genuinely are that big. Lucene-class engines land around 20–40% for the same feature set, and that’s the neighbourhood we should be in.

Next

I’m re-running the Corpora corpus with the size split captured properly, with and without a final Optimize(), so we have a real before/after instead of my one embarrassing number. I’ll post it here once new test repostory will be ready to verify with the metrics.

1 Like
3

Hi @jamsharp !

I double-checked multiple settings and the implementations; these are our points for improvement. There is exactly one thing worth changing on your side, and we measured the rest properly so we can prove the fixes land.

The one thing on your side - Call Optimize() once at the end of ingest, not every 300 MB.
That is it. Nothing else in your configuration is causing this.

What we measured

20 GB of mixed office documents, 29,031 files, one Optimize() at the end:

before Optimize() after
Index 18.56 GB 1.24 GB
index*.head — term offsets 16.71 GB 116 MB
index*.body — postings 1.60 GB 909 MB
Segments 365 1
Ratio to source 92.8% 6.2%

We reproduced your 94% on a different corpus,
then watched it collapse to 6.2% — inside the 10–33%
you expected.

The ratio turns out to be a function of segment count and nothing else:

Segments 1 81 229 365
Index / source 13.6% 52.7% 83.3% 99.3%

Each segment’s .head holds a 4-byte slot for every term in the whole index, present in that
segment or not. At 30.5M terms that is 116 MB per segment — 365 of them, one real and 364 padding.

So the fixes are ours:

  1. Automatic segment merging — the real one. Keeps segment count logarithmic so this stops

existing. No reindex, no configuration, no advice needed.

  1. Sparse .head tables — store only the terms a segment actually contains, removing the waste

at source instead of cleaning it up afterwards.

  1. Postings encoding, and a storage-diagnostics API so nobody has to paste a directory listing into

a forum again.

Two things not to bother with, since both look like levers and are not: **CompactIndex makes the

ratio worse** (postings shrink, .head does not, so overhead’s share rose to 84%), and alphabet or

stop-word tuning moves far less than the merge does.

How we will prove it

We built a profiler and published it on GitHub: Document Indexing Storage Lab.

It indexes a corpus under a size budget and reports where every byte went — the .head / .body / dictionary split, vocabulary size, ratios against both source bytes and extracted text, and memory including retained heap after a forced collection, before and after Optimize().

Every run is stored with the GroupDocs.Search version, runtime, GC mode and a hash of the exact file set, so the same corpus on a future release is provably comparable — that is the point. When auto-merge and sparse .head ship, we will re-run this identical 20 GB corpus and post the before/after rather than asking you to take our word for it. One number for the memory side: peak private was 26.15 GB on a 32 GB machine, and the single final Optimize() is the spike.
Auto-merge would do incrementally what one big merge currently does in a single very expensive step — which is why it fixes both threads at once.

Summary

There are multiple improvements we scheduled for the nest product releases. So give us some time to adjust and deliver them.

Thank you!

P.S. In the published tool you can see various sessions, and you can start new ones that will be saved. So this will be a good snapshot to compare with the other version.

4

Ok, thank you for analyzing it.

I note on our side that calling Optimize seldom is a good idea for now and that you work on a fix for this.

Best regards,
jam

1 Like
5

Hi @jamsharp!

Let me ask you a few questions that could potentially help us understand your needs better.

Can you please share the approximate index size for your customer(s)?
I am asking from the perspective of implementing a new index structure/format and the time of its conversion.

Q2. How many hours do you have for a safety production shutdown?
This is about the available time window in case of a long conversion, to avoid adding new documents during this process.

Q3. Are all indexed documents in the production environment still accessible?
When converting the index, it will require all documents to be accessible to optimize the search procedure as well besides index structure.

Thank you!

6

Hello,

Q1: It’s hard to answer that question, because our customers had problems like OverflowException: Arithmetic operation resulted in an overflow - #13 by jamsharp that lead to failed Optimize calls and therefore huge indices.

For me, it’d be easier to answer the question based on original files. If I’d had to tell you the realistic amount of documents our customers could have, I’d say under the assumption that a document is 1 MB in average:

  • 1k files (1 GB total original file size) is realistic.
  • 10k files (10 GB total original file size) is realistic.
  • 100k files (100 GB total original file size) is realistic.
  • 1m files (1 TB total original file size) is realistic.
  • 10m files (10 TB total original file size) is probably not that realistic anymore (more like an edge case). In that case, if you’d suggest splitting the index into multiple indexes, I’d probably agree.

So, a good amount for orientation for an index is probably a 7-digit number.

We of course hope that you find a good middle-way to have a small index size (< 33% of the original file size, the smaller, the better, but of course still with good search performance… We aim to have <= 30 seconds to get the first result from the index and are currently enabling the chunk search you offer.)

Q2: Because of the last index format migration, we already had to deal with this a bit. Of course, we’d wish that it’s as small as possible. More than an hour is maybe hard to explain to customers. It’s even hard when it takes multiple hours.

Q3: In case of a large drive, it’s very realistic that a few (maybe around 1-5% do not exist anymore). But in our case, it’s more difficult: We have a default user that deals with index management (adding bytes to index, allowing to search the index, …), but the user that extracts the files (in a different process) can be totally different (it’s the one that is guaranteed to have access to the documents). So, the user that manages the index is not guaranteed to have the rights to access the documents!

BTW: Thanks for asking these questions in advance! We appreciate this.

Best regards
jam

7

Hi @jamsharp !

That’s very helpful for understanding the real scenarios, so we can adapt our test cases to meet these conditions as well.
We will continue working on the September product release.
Thank you!