Memory consumption and Heap space issue during load test for GroupDocs.Conversion for Java (21.10)

ChourasiyaRajesh · March 11, 2022, 6:53am

Dear Team,

We did a load testing on our application using GroupDocs.Conversion for Java (21.10) to converst different files to PDF formats.
Unfortunately, we see that the system is unable to handle >2 requests at at time at times fails for 2 requests as well. It throws insuficient heap space error.
We tried these tests by converting just 1MB docx file to pdf. There can be files > 20MB of size.

For 1MB docx file, it can handle multiple requests one by one but that cannot be the ideal scenario.

We need some information related to average memory consumtion (for different formats) of Groupdocs.Conversion for Java while conversion to pdf.
This will help us in setting the correct JVM Heap size so that it should handle simultaneous requests.

Let us know if you need more information.

Below are the deployment environment information:

Linux Environment:
NAME=“Red Hat Enterprise Linux Server”
VERSION=“7.9 (Maipo)”
ID=“rhel”
ID_LIKE=“fedora”
VARIANT=“Server”
VARIANT_ID=“server”
VERSION_ID=“7.9”

openjdk version “1.8.0_311”
OpenJDK Runtime Environment (Zulu 8.57.0.14-SA-linux64) (build 1.8.0_311-b02)
OpenJDK 64-Bit Server VM (Zulu 8.57.0.14-SA-linux64) (build 25.311-b02, mixed mode)

VM settings:
Max. Heap Size: 1.00G

atir.tahir · March 11, 2022, 11:07am

@ChourasiyaRajesh

Actually, we do not have such recommendations as the memory/resources or the time consumption totally depends on multiple factors including:

Type of the files processed
File’s content
Count files processed simultaneously

Did you try increasing the heap space? Locally, at our end we allocate 2-3GB memory to the process. Moreover, we’d recommend you to use latest version of the API that is 22.3.
After increasing the heap space and using latest version, if issue persists, please share following details:

Sample code you wrote for multiprocessing
Problematic file(s)
A short screencast/video explaining the steps to reproduce the issue

ChourasiyaRajesh · March 11, 2022, 4:23pm

Hi @Atir_Tahir

Thanks for your reply. Definitely we will try increasing the heap memory size and get in touch with you if the issue persist. Right now with current configurations we are performing load test with help of JMeter.

Also the latest version of Groupdocs Conversion for Java 22.3 jar size is quite high, near double the size of previous 21.10 (ticket: CONVERSIONJAVA-1599)

We will try to update the heap size and let you know the details accordingly.

Thank you
Rajesh Chourasiya

atir.tahir · March 11, 2022, 6:06pm

@ChourasiyaRajesh

Yes, let us know if issue persists.

ChourasiyaRajesh · May 26, 2022, 6:25am

Hi @Atir_Tahir,

Continuing on this topic… we deployed our application in PROD environment including groupdocs conversion (java, v. 21.10) and observed that with increasing number of user requests we encountered a gradual increase in number of Out Of Memory events (OOM) on our running pods.

The OOM is caused by groupdocs conversion is mostly when we try to converts a large xlsx file (4MB) to pdf.

This is now resulting slowdown in running other services in the environment and causing different issues like Broken-Pipe, Timeouts and ultimately Service Not Available.

We have tried increasing each Pod’s Heap Size memory to 3GB however, seems this is also not sufficient for a 4MB xlsx file. It seems if we further increase the memory size, it could not be sufficient for say 6MB file. As a reference, we have attached a sample file which we have tested in our Test environment which was causing the same issues.

filterchoosecriteriacolumns.zip (3.6 MB)

Since we cannot show the complete logs, below mentioned are the log trace fragment:

org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space

            at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1055)

            at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)

            at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)

            at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)

            at javax.servlet.http.HttpServlet.service(HttpServlet.java:626)

            .

.

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

            at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

            at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.OutOfMemoryError: Java heap space

            at com.groupdocs.conversion.internal.c.a.c.b.a.d.h.b(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.b.a.d.h.c(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.b.a.d.h.a_(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.cw.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.cw.b(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.cn.c(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.ca.b(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.ca.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.dC.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.aD.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.dz.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.aD.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.dy.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.bJ.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.bY.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.bY.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.cd.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.ac.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.aD.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.ac.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.aD.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.ac.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.aD.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.ac.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.aD.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.dx.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.zbf.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.a.b.cd.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.zcjt.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.zcjt.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.zcjt.a(Unknown Source)

            at com.groupdocs.conversion.internal.c.a.c.zkb.a(Unknown Source)

Request if you could check the same at your end with the sample attached file. Please take this on priority basis as this is now impacting at our end. Furthermore, it would be great if you can confirm there is no memory leaks within the file conversion process.

Thank you
Rajesh Chourasiya

atir.tahir · May 26, 2022, 9:50am

@ChourasiyaRajesh

Could you please share the conversion code (Spreadsheet to PDF)? We’d appreciate if you share a short video of this issue (from starting the conversion to the exception).

ChourasiyaRajesh · May 27, 2022, 7:26am

Hi @Atir_Tahir,
I have shared the details on email.

Rajesh Chourasiya

atir.tahir · May 27, 2022, 1:08pm

@ChourasiyaRajesh

Thanks for the details. This issue is now under investigation with ticket ID CONVERSIONJAVA-1675. You’ll be notified in case of any update.

ChourasiyaRajesh · June 1, 2022, 6:41am

Hi @Atir_Tahir
Any updates on the below issue? The OOM incidents are increasing and impacting the performance of our application.
Please let us know.

Thank you
Rajesh Chourasiya

atir.tahir · June 1, 2022, 11:19am

@ChourasiyaRajesh

We’re still investigation this ticket. We’ll inform you in case of any progress update.

ChourasiyaRajesh · June 9, 2022, 1:12pm

Hi Atir Tahir,

It seems that large XLS(X) file, for example, with > 50 columns and > 20000 columns are creating issue while rendering as from the documentation of SpreadSheetOptions:

setOnePagePerSheet (by default as true) sets all the 50 X 20000 data in a single pdf page.

If we set this attribute to false, the data gets generated in “n” number of pdf pages (nearly thousands as expected to show all rows) but the columns are cut down to a limited number. Seems in this case it fits only print size columns only, for example just 8-10.

Can’t it be like to show all the columns with “n” number of pdf pages?
I don’t see any other options to achieve this. Any inputs are appreciated.

Thank you
Rajesh Chourasiya

atir.tahir · June 9, 2022, 8:19pm

@ChourasiyaRajesh

We will investigate this scenario as well. You’ll be notified about the outcomes.

ChourasiyaRajesh · June 17, 2022, 12:37pm

Hi Atir Tahir,

Can we have an ETA to the frequent OOM issues or at least the alternative temporary fix. We had purchased the license with paid support for this product and even after 1 month of raising the issue, we haven’t heard of any updates on this case from groupdocs end.
It feels disappointed when we just receive an intimation that issue is under investigation after when we enquire about it.

atir.tahir · June 17, 2022, 6:02pm

@ChourasiyaRajesh

Sorry for the inconvenience. The XLSX (filterchoosecriteriacolumns) has more than 2000 rows in “FoodSales” sheet and that is creating the issue. We are working on some improvements of memory usage which will be released as a hot fix in the beginning of the next week.

ChourasiyaRajesh · June 20, 2022, 10:10am

Thanks Atir for your response.
However, please let us know if the hotfix would be able to handle large XLS(X) file. Since we have data which are almost around 30K rows and thus we had provided similar sample file with such records.

Thank you
Rajesh Chourasiya

atir.tahir · June 20, 2022, 10:52am

@ChourasiyaRajesh

We are already investigating this ticket keeping in mind the large Excel files.

atir.tahir · June 20, 2022, 11:49am

@ChourasiyaRajesh

Could you please also attach/share your pom.xml?
Secondly, what is your maven.compiler.source, maven.compiler.target, and Java version in maven-compiler-plugin config. It seems that you are using Java 7.

ChourasiyaRajesh · June 21, 2022, 5:05am

Hi @Atir_Tahir, Since we don’t have upload rights outside of network, I am sharing below the maven-compiler-plugin as configured and java version properties as navigated.

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>${compiler-plugin.version}</version>
            <configuration>
                <source>${java.version}</source>
                <target>${java.version}</target>
                <encoding>${project.build.sourceEncoding}</encoding>
            </configuration>
        </plugin>

and

    <properties>
        <timestamp>${maven.build.timestamp}</timestamp>
        <maven.build.timestamp.format>yyyy-MM-dd HH:mm:ss</maven.build.timestamp.format>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <java.version>1.8</java.version>

…

atir.tahir · June 21, 2022, 10:35am

@ChourasiyaRajesh

We’ll continue the investigation and will let you know if any further details are required.

atir.tahir · June 22, 2022, 9:30pm

@ChourasiyaRajesh

We don’t get OutOfMemory exception using the large Excel file that you shared and Java 8. Please create and share test maven project where the issue is reproduced.