Hi, @jamsharp!
Thanks for the detailed report. We did a deep dive into the investigation, so let me share all aspects.
The stack trace was enough to locate the exact call site, so let me go through your questions one by one.
The short version of what your stack trace says
RunIndexing(...) // the Add/Update entry point
-> StartIndexing(...)
-> FinishIndexing() // the commit, after all documents are processed
-> IndexInfoContainer.CopyFrom(IndexInfoContainer)
-> MultiArrayTree.Clone()
-> array clone -> OutOfMemoryException
So you did not run out of memory while extracting or while adding documents. You ran out at the commit of an indexing operation, while the index metadata and the term dictionary are being deep-copied. That is important, because it means the fix on your side is mostly about how often you commit, not about how much text you push.
We reproduced it
Before answering, we built a small GitHub benchmark repo solution that indexes the same documents three different ways and reports what each costs. We used the Digital Corpora document set to run this.
Sixty mixed-format documents (HTML, PDF, TXT, DOC/DOCX, XLS/XLSX, PPT/PPTX), GroupDocs.Search 26.8.0, .NET 8, Release build, x64, workstation GC, Threads = 1. All three runs indexed a byte-identical document set (verified by hash) and returned identical search results, so nothing is traded away:
|
one Add() per document |
one Add() per batch of 60 |
3 shards of 20 |
Add() calls |
60 |
1 |
3 |
| Total allocated |
91.22 GB |
3.74 GB |
6.84 GB |
| Allocated per document |
1.52 GB |
63.9 MB |
116.8 MB |
| Peak private bytes |
8.77 GB |
1.80 GB |
4.28 GB |
| Retained managed heap |
68.0 MB |
68.0 MB |
46.3 MB |
| Retained LOH |
126.7 MB |
1.43 GB |
174.5 MB |
How to read the table
There are four memory rows because “memory” means four different things here, and they move independently. Conflating them is what makes this problem so confusing to diagnose:
| Row |
What it measures |
Why it is here |
| Total allocated |
Every byte the process ever asked the allocator for, cumulative (GC.GetTotalAllocatedBytes). Memory that was freed still counts. |
The only way to see a 1.4 GB buffer that is allocated and collected inside a single Add() call. It never appears in a heap snapshot, yet it is what drives GC pressure and LOH churn. |
| Allocated per document |
Total allocated ÷ 60. |
Makes the three approaches directly comparable, since all three index the same 60 files. |
| Peak private |
High-water mark of Process.PrivateMemorySize64, sampled by a background thread every 25 ms. |
This is what your monitoring graphs and what hit 32 GB on your box. Sampling it on a timer rather than at loop boundaries matters — we got that wrong in our first run, and it flattered the batched result. |
| Retained managed |
Managed heap still alive after a forced blocking collection at the end of the run. |
What an open index genuinely holds. Reading the heap without forcing a collection just measures uncollected garbage. |
| Retained LOH |
Large Object Heap size after that same forced collection. |
The LOH is not compacted by default and is not decommitted eagerly, so this is the part of your footprint that fragments and does not come back. |
All figures are build-phase only — index construction and Add() calls, nothing else. Sample 03’s repository load and cross-shard search are measured separately and excluded, because samples 01 and 02 have no equivalent step.
What the three columns tell you
Column 1 reproduces your symptom with sixty documents. The per-document run peaked at 8.77 GB of private bytes while retaining only 68 MB of managed heap once collected — a factor of 129 between what the process held from the operating system and what it was genuinely using. That gap is LOH churn and fragmentation, and it is exactly the “private bytes keep climbing while the application isn’t really holding anything” behaviour you described. It took 60 files, not a million.
Column 2 is the fix, and it is almost embarrassingly cheap. Same documents, same results, one Add() call instead of sixty: 24× less allocation and 4.9× lower peak private bytes. The only code change is passing a bigger array.
Column 3 shows what batching does not fix. Note that retained managed heap is identical between columns 1 and 2 — 68.0 MB either way. Batching does nothing for what an open index holds. Only closing indexes does, and the effect shows up in the LOH row rather than the managed heap row.
One thing we want to be straight about: our first pass at this table read the managed heap without forcing a collection, and we briefly concluded that sharding cut resident memory 7.7×. It does not — that was uncollected garbage, not retention. The corrected figures are above, and the real sharding win turned out to be in the LOH instead. We mention it because you should treat any memory number you take from us, or measure yourself, with that same suspicion.
Fitting the first two runs gives a model that predicts the third within 2.0%:
total allocated ~ 1.48 GB x (number of Add() calls) + ~39 MB x (documents)
That first term is the whole story. In the per-document run, 97.6% of all allocation was fixed per-call overhead that had nothing to do with the documents. And 1.48 GB measured lines up within 6% with the 1.43 GB we calculate from the source for the term-collector buffers - so the model isn’t a curve fit, it’s the mechanism.
The practical consequence is a table you can read your batch size off directly:
documents per Add() call |
fixed overhead per document |
| 1 |
1518 MB |
| 20 |
76 MB |
| 100 |
15 MB |
| 1 000 |
1.5 MB |
| 10 000 |
0.15 MB |
And there is a second lever, in the Large Object Heap. Look at the last row. The single 60-document batch leaves 1.43 GB of LOH standing after a forced collection, because the term-collector buffers are LOH-sized and the runtime does not decommit them eagerly. Building the same documents as three disposable shards holds that to 174.5 MB - an 8.4x difference. Since the LOH is exactly what fragments and pushes a long-running process’s private bytes up, that is the number that decides whether your indexing service survives a multi-hour run. Batching fixes your allocation rate; disposing indexes fixes your LOH footprint. You need both.
Run it yourself
The benchmark is a small .NET 8 solution — three console projects sharing one library — published as document-indexing-memory-optimization-net. Point it at a folder of your own documents, set your licence path in an environment variable, and it prints the same table:
export GROUPDOCS_LIC_PATH=/path/to/your.lic
export INDEXING_LAB_STORAGE_ROOT=/path/to/your/documents
export INDEXING_LAB_MAX_DOCUMENTS=60
dotnet run -c Release --project samples/Sample01.PerDocument
dotnet run -c Release --project samples/Sample02.Batched
INDEXING_LAB_DOCS_PER_SHARD=20 dotnet run -c Release --project samples/Sample03.Sharded
Each run writes a CSV and a JSON manifest recording the document-set hash, licence state, GC mode, build configuration and search results, so two runs can be compared rather than merely asserted to be comparable. The raw data behind the table above ships in the repository under results/published/.
Following your questions
1. Is there a way to consume an index with less of it loaded into RAM (a trade-off setting — slower search, less memory)?
Not today. There is no such switch, and I’d rather tell you that straight than point you at an option that won’t help.
Here is what the current design does, so you can see where the ceiling comes from:
- Postings (the inverted index itself) — the
index{N}.body / index{N}.head files — are streamed from disk. They are never fully loaded. This is the bulk of your 25 GB, and it is not your problem.
- The term dictionary (
index1.info) is loaded completely into RAM at index open. On disk it is Huffman-encoded; in memory it is expanded into raw 16 MB byte arrays at 7 bytes per trie node. Expect roughly 2–4× the file size in RAM.
- Per-document metadata (
index.info) is loaded completely into RAM as ~10 hash tables: path → id, id → path, document → elements, element → (document, field), term counts, encodings, container items, attributes. This scales linearly with document count and has no disk fallback.
- Stored text (if you enabled text storage for highlighting) stays on disk. In-memory text storage is only used for in-memory indexes.
So the answer to “can I trade speed for memory” is: the parts that could be traded are already on disk; the parts that are in RAM are in RAM unconditionally. Making the metadata and the term dictionary memory-mappable is exactly the change we need to make, and your report is the concrete case for prioritising it. I’ll come back to that at the end.
2. Is the full contents of the index folder loaded into RAM?
No — but the distinction matters less than you’d hope.
| File |
Loaded at open? |
Notes |
index{N}.body, index{N}.head |
No — streamed |
Usually >90% of the folder size |
index1.info (term dictionary) |
Yes, fully |
Expands 2–4× over the file size |
index.info (document metadata) |
Yes, fully |
~10 hash tables, linear in document count |
index0.info (image hashes) |
Yes, fully |
Small unless you index images |
| Text archive |
No — streamed |
Only if text storage is enabled |
If you want to know your own split without guessing: post the sizes of index.info, index1.info and index0.info. Those three numbers tell us your resident footprint almost exactly.
3. Is it even possible to put 1 million files into one index? Have you tried it?
Yes, even on 32 GB, the same as your machine. We have now built it, in a single index, using the Digital Corpora dataset:
|
|
| Source corpus |
986,262 files, 466 GB |
| Documents indexed |
969,288 |
| Extracted text |
91.2 GB |
| Resulting index |
440 GB across 740 segments |
| Machine |
32 GB RAM |
| Outcome |
completed, no errors |
The approach that made it work was batching. Nothing exotic: no sharding, no separate processes, no special GC configuration — one index, documents fed in batches rather than one at a time.
So the honest answer to your question has changed since we started looking at this: the ceiling is not where you are hitting it. A million files in one index is fine on 32 GB. What is not fine is calling Add() once per document, because that pays the ~1.48 GB fixed cost 969,288 times instead of a few hundred.
Three details worth having:
-
969,288 of 986,262 were indexed — about 1.7% were skipped as corrupt, encrypted, or an unsupported format. That is normal for a corpus assembled from the wild, and it is not a memory problem. Count your own skips before assuming something went wrong.
-
**Our index is 440 GB ** Ours is roughly seventeen times larger and still built inside 32 GB.
-
740 segments. Segment count grows with the number of indexing operations, and every search consults every segment. Keep an eye on it and run Optimize() periodically — but do it as a deliberate, occasional maintenance step, not after every batch, since merging takes the same expensive deep-copy path as indexing.
This also revises something we said earlier in this thread: we had guessed that resident metadata would make one large index impractical and that sharding was effectively mandatory. The run above shows the resident structures for ~1M documents sit comfortably within 32 GB. Sharding is still worth doing — it keeps the Large Object Heap small, which matters for a service that indexes continuously for days — but it is an optimization, not a prerequisite. You do not need to redesign around it to get to a million files.
4. Is there an estimate of how much RAM an index needs? Is it 1:1 with disk?
It is not proportional to total disk size — total disk size is dominated by the postings, which are not resident. Two different numbers matter here, and conflating them is what makes this confusing.
Resident memory — what an open index holds for as long as it is open:
steady-state RAM ~ (documents + container items) x ~0.8 KB // metadata hash tables
+ 2..4 x sizeof(index1.info) // decoded term dictionary
Allocation during an indexing or optimize operation — measured, not estimated:
allocated ~ 1.48 GB x (number of Add() calls) + ~37 MB x (documents)
plus, at the moment of commit, a transient peak of roughly 3 x metadata + 2 x term dictionary — the deep copies and the tree clone from your stack trace. That transient is negligible on a small index and dominant on yours, because it scales with the index you already have rather than with the batch you are adding. At 1M documents it is gigabytes, which is why the clone is where you actually died.
The 1.48 GB fixed term is the term collector, which preallocates fixed-size buffers up front regardless of how many documents you are adding. Here is what we calculate from the source — and the total agrees with the measured fixed cost within 6%:
| Buffer |
NormalIndex |
CompactIndex |
| positions |
1114 MB |
572 MB |
| term dictionary (pre-sized) |
173 MB |
788 MB |
| other index arrays + index data |
~143 MB |
~569 MB |
| total, per indexing thread |
~1.43 GB |
~1.93 GB |
Adding one document allocates the full ~1.43 GB. Adding 10,000 documents in a single call allocates the same ~1.43 GB. With IndexingOptions.Threads = 4 you get one of these per thread, so ~5.7 GB.
Two consequences worth internalising:
- Small batches are catastrophically expensive. The cost is per
Add(...) call, not per document. If you are adding documents one at a time or in batches of 100, you are paying 1.43 GB of allocation plus two full deep copies of your entire index metadata per batch.
CompactIndex costs more RAM, not less. It saves disk space, but its collector is ~500 MB larger. If you picked it hoping to save memory, switch back.
And this is why the process never gives the memory back: essentially all of these allocations are ≥ 85 KB, so they land on the Large Object Heap. The LOH is not compacted by default, so thousands of allocate/free cycles over several hours fragment it badly. Private bytes climb, the managed heap looks fine, and eventually a 16 MB contiguous request fails on a machine that still reports free memory. That is exactly the failure mode in your trace.
5. What can you do so you don’t have to tell customers “just buy 100 GB of RAM” — and what will we do?
What you can do today, in rough order of impact
-
Batch much harder. Use index.Add(ExtractedData[], IndexingOptions) or index.Add(Document[], IndexingOptions) with 5,000–20,000 documents per call. This is by far the biggest lever: in our benchmark it cut total allocation 24.4× and peak private bytes 4.9×, with identical search results. Read your target off the amortisation table above — at 10,000 documents per call the fixed overhead falls to 0.15 MB per document, from 1518 MB at one document per call.
-
Keep IndexingOptions.Threads = 1 (that is the default — verify you haven’t raised it). Each extra thread is another ~1.48 GB. Since you already extract in a separate process, indexing threads buy you very little here.
-
Build in shards you dispose, and query them with IndexRepository. Build indexes of ~100k documents each, closing each one before starting the next, then add them to a repository and search across all of them. In our benchmark this held the retained Large Object Heap to 174.5 MB against 1.43 GB for a single open index — an 8.4× difference. Because the LOH is what fragments and drives private bytes upward over hours, this is the lever that decides whether a long-running indexing service stays healthy.
Two honest caveats. First, sharding does not reduce allocation — our 3-shard run allocated more than the single batch (6.84 GB vs 3.74 GB), because each shard re-pays the same 1.48 GB fixed cost. Twenty documents per shard was deliberately extreme to force three shards out of sixty; at 100k per shard that cost is noise. Second, the managed heap barely differs (68 MB with one index open vs 46 MB sharded) — the win is specifically in the LOH, not in ordinary object retention. Batch for allocation rate, dispose indexes for LOH footprint, and do both.
-
Tune the GC for LOH fragmentation in the indexing process. On .NET Framework:
<configuration>
<runtime>
<gcServer enabled="true"/>
<gcAllowVeryLargeObjects enabled="true"/>
</runtime>
</configuration>
On .NET (Core) 6+: <ServerGarbageCollection>true</ServerGarbageCollection>, <ConcurrentGarbageCollection>false</ConcurrentGarbageCollection>. And between batches:
GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
GC.Collect();
This won’t fix the root cause, but it will stop private bytes from ratcheting upward across a long run.
-
Run on .NET 8+ rather than .NET Framework if you have the option — noticeably better behaviour for this allocation pattern.
-
Set IndexSettings.MaxIndexingReportCount = 0 (and MaxSearchReportCount = 0). By default the last 5 indexing reports are retained, and each one holds the full list of document paths from its batch. With 10k-document batches that’s 50k retained path strings for no benefit.
-
Don’t switch to CompactIndex for memory reasons — see the table above.
-
Call Optimize() sparingly and never concurrently with adds — it takes the same deep-copy path as indexing.
If you apply (1), (2), (4) and (6), I would expect the growth curve to flatten substantially even before we ship anything.
What we scheduled for the next releases
Scoped internally, in priority order:
- Rework and optimize the index format and creation.. This will decrease disk allocation and potentially RAM usage potentially as well
- Grow the term-collector buffers on demand. Currently preallocated up front per
Add() per thread, regardless of batch size. This is the 1.48 GB fixed term — 97.6% of all allocation in the per-document run. Takes small batches from GB to MB and removes most of the LOH churn with it.
- Replace the commit-side deep copy with an ownership transfer. The working state is discarded on the next line, so cloning it is pure waste: one term-dictionary clone plus one full metadata copy per operation. Scale-dependent — invisible on a small index, gigabytes per commit at 1M documents. This is the frame in your stack trace.
- Fix metadata sizing and duplication. Document paths held as two separate string instances; two internal dictionaries pre-sized 3–5× too large. ~200–300 MB per million documents.
- Document the cost model and the scale figure. The per-
Add() formula and the 969,288-document result belong in the docs, not in a forum thread.
Later, and no date on it: memory-map the metadata and term dictionary behind a bounded cache. That is what would turn question 1 into a real setting rather than an answer of “no”.
May I ask you to share a few details from your side
A few of these change the diagnosis materially:
- What is your batch size — how many documents per
Add(...) call? If it’s 1–100, that alone accounts for most of what you’re seeing.
- Which overload are you calling —
Add(ExtractedData[], ...), Add(Document[], ...), or the path-based ones? And what is IndexingOptions.Threads set to?
- What are your
IndexSettings? Specifically, IndexType (Normal / Compact / Metadata), whether TextStorageSettings is set, UseStopWords, and whether OCR or image indexing is enabled.
- Runtime details: .NET Framework or .NET 8+? Server GC or workstation GC? 64-bit process (I assume yes, given the numbers)?
- Index shape: Are the 1M “files” plain documents, or do they include archives / PST / mail that expand into many container items? Container items count as documents for every structure listed above. And roughly how many fields per document — the element tables scale with documents × fields.
- File sizes: please post the sizes of
index.info, index1.info and index0.info next to the 25 GB total. That converts my estimate in question 4 into your actual number.
- How often do you call
Optimize(), and how many segments accumulate between optimizations?
- Does memory climb during
Add, during Optimize, or during search? Your trace points at Add, but confirming rules out a second issue.
- Do you keep one
Index instance open for the whole run, or open and dispose one per batch?
If you can share your main code snippet of the main indexing process implementation and optionally with a memory profile snapshot (dotMemory / PerfView) taken mid-run, that would help us for investigation.