Problem with uploading a file larger than 512 MB .......
Thursday, 17 August 2023 · 2 messages · last reply 21 Aug 2023
DG
Dmitry Gorskiy17 Aug 2023 19:36 #
Dear Sirs,
What could be the problem?
Big file upload problem.
Below is the listing from the console.
My computer:
-------------------
CPU: 4 cores
RAM: 8G
OS: Linux
BaseX: 107 and BaseX108-20230816.163639
Java: Adoptium idk-17.0.8+7
I'm creating files:
dd if=/dev/random of=file_512M bs=1M count=512
…………………………file_530M
…………………………file_1G
…………………………file_2G
and etc.
Launching “basexhttp" ( default ...-Xmx2g … )
Let's go in … http://192.168.1.120:8080/dba/files <http://192.168.1.120:8080/dba/files> Downloading files - no problem.
Uploading: file 512MB size - no problem.
Uploading file 530MB and more - PROBLEM ( looking console output below )
OK, relaunching basexhttp whith -Xmx4G …..
Uploading file 512MB - ok
Uploading file 1GB - ok
More size - crashes
What could be the problem?
Thank you very much.
Dmitriy Gorskiy.
HTTP ERROR 500 java.lang.OutOfMemoryError: Java heap space
URI: /dba/file-upload
STATUS: 500
MESSAGE: java.lang.OutOfMemoryError: Java heap space
SERVLET: RESTXQ
CAUSED BY: java.lang.OutOfMemoryError: Java heap space
Caused by:
java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Arrays.java:3537)
at org.basex.util.list.ByteList.add(ByteList.java:43)
at org.basex.io.in.BufferInput.content(BufferInput.java:216)
at org.basex.http.RequestContext.body(RequestContext.java:119)
at org.basex.http.RequestContext.addMultipart(RequestContext.java:137)
at org.basex.http.RequestContext.formValues(RequestContext.java:103)
at org.basex.http.restxq.RestXqFunction.bind(RestXqFunction.java:223)
at org.basex.http.restxq.RestXqResponse.init(RestXqResponse.java:60)
at org.basex.http.web.WebResponse.create(WebResponse.java:54)
at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:72)
at org.basex.http.BaseXServlet.service(BaseXServlet.java:69)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel$$Lambda$140/0x00007f5974196000.dispatch(Unknown Source)
Powered by Jetty:// 9.4.51.v20230217 <https://eclipse.org/jetty>CG
Christian Grün21 Aug 2023 08:08 #
Hi Dimitry,
Do you get the same error if you increase the Xmx value?
The DBA code is based on RESTXQ. The binary contents of uploaded files are bound to a main-memory variable, and the underlying byte array is of limited size. We should probably limit the file size in the client before it is actually uploaded.
Best,
Christian
Quoted text · Dmitry Gorskiy · 108 lines
Dmitry Gorskiy <dv.gorskiy at gmail.com> schrieb am Do., 17. Aug. 2023, 19:37: > Dear Sirs, > > What could be the problem? > > Big file upload problem. > Below is the listing from the console. > > My computer: > ------------------- > CPU: 4 cores > RAM: 8G > OS: Linux > BaseX: 107 and BaseX108-20230816.163639 > Java: Adoptium idk-17.0.8+7 > > I'm creating files: > dd if=/dev/random of=file_512M bs=1M count=512 > …………………………file_530M > …………………………file_1G > …………………………file_2G > and etc. > > Launching “basexhttp" ( default ...-Xmx2g … ) > Let's go in … http://192.168.1.120:8080/dba/files > Downloading files - no problem. > > Uploading: file 512MB size - no problem. > Uploading file 530MB and more - PROBLEM ( looking console output below ) > > OK, relaunching basexhttp whith -Xmx4G ….. > Uploading file 512MB - ok > Uploading file 1GB - ok > More size - crashes > > What could be the problem? > > Thank you very much. > > Dmitriy Gorskiy. > > > HTTP ERROR 500 java.lang.OutOfMemoryError: Java heap space > URI: /dba/file-upload > STATUS: 500 > MESSAGE: java.lang.OutOfMemoryError: Java heap space > SERVLET: RESTXQ > CAUSED BY: java.lang.OutOfMemoryError: Java heap space Caused by: > > java.lang.OutOfMemoryError: Java heap space > at java.base/java.util.Arrays.copyOf(Arrays.java:3537) > at org.basex.util.list.ByteList.add(ByteList.java:43) > at org.basex.io.in.BufferInput.content(BufferInput.java:216) > at org.basex.http.RequestContext.body(RequestContext.java:119) > at org.basex.http.RequestContext.addMultipart(RequestContext.java:137) > at org.basex.http.RequestContext.formValues(RequestContext.java:103) > at org.basex.http.restxq.RestXqFunction.bind(RestXqFunction.java:223) > at org.basex.http.restxq.RestXqResponse.init(RestXqResponse.java:60) > at org.basex.http.web.WebResponse.create(WebResponse.java:54) > at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:72) > at org.basex.http.BaseXServlet.service(BaseXServlet.java:69) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450) > at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) > at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554) > at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) > at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) > at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) > at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) > at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) > at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) > at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) > at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) > at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) > at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) > at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > at org.eclipse.jetty.server.Server.handle(Server.java:516) > at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) > at org.eclipse.jetty.server.HttpChannel$$Lambda$140/0x00007f5974196000.dispatch(Unknown Source) > > ------------------------------ > Powered by Jetty:// 9.4.51.v20230217 <https://eclipse.org/jetty> > ------------------------------ > > > > > > > > > > > > > > > > > > > > > >
Written for BaseX 10.7. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery