Main Page » Mailing List » June 2012

Hanging when inserting attribute

Monday, 11 June 2012 · 5 messages · last reply 13 Jun 2012
AM
Alex Markin11 Jun 2012 23:34 #
Hello all. I've tried to execute query
for $i in collection('stars')/csv/record
return insert node (attribute {'id'} {generate-id()} ) into $i
with this <http://www.onlinedisk.ru/file/889304/> file (1000000 strings with 3 integer in any in csv format). I had a hang of basex and onece got the following:
Exception in thread "Thread-526" java.lang.OutOfMemoryError: Java heap space
Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC
overhead limit exceeded
        at java.awt.EventQueue.postEvent(EventQueue.java:261)
        at java.awt.EventQueue.postEventPrivate(EventQueue.java:215)
Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: GC overhead
limit exceeded
Exception in thread "SIGTERM handler" java.lang.OutOfMemoryError: GC
overhead limit exceeded
and again - the folowing:
Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: Java heap space
        at java.util.logging.LogRecord.<init>(LogRecord.java:151)
        at java.util.logging.Logger.log(Logger.java:607)
        at sun.awt.X11.XToolkit.processException(XToolkit.java:535)
        at sun.awt.X11.XToolkit.run(XToolkit.java:614)
        at sun.awt.X11.XToolkit.run(XToolkit.java:543)
        at java.lang.Thread.run(Thread.java:679)
Hang was even when I changed generate-id() to "1". Seems like a bug. Markin Alex.
CG
Christian Grün12 Jun 2012 04:16 #
Hi Alex,
Quoted text · Alex Markin · 2 lines
> for $i in collection('stars')/csv/record > return insert node (attribute {'id'} {generate-id()} ) into $i
before any update is performed, all operations first need to be put on the pending update list [1]. It looks as if this list requires too much memory. How much memory did you assign to the JRE? Christian [1] http://docs.basex.org/wiki/XQuery_Update#Pending_Update_List
Quoted text · Alex Markin · 25 lines
> with this file (1000000 strings with 3 integer in any in csv format). I had > a hang of basex and onece got the following: > > Exception in thread "Thread-526" java.lang.OutOfMemoryError: Java heap space > Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC > overhead limit exceeded > at java.awt.EventQueue.postEvent(EventQueue.java:261) > at java.awt.EventQueue.postEventPrivate(EventQueue.java:215) > Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: GC overhead limit > exceeded > Exception in thread "SIGTERM handler" java.lang.OutOfMemoryError: GC > overhead limit exceeded > > and again - the folowing: > > Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: Java heap space > at java.util.logging.LogRecord.<init>(LogRecord.java:151) > at java.util.logging.Logger.log(Logger.java:607) > at sun.awt.X11.XToolkit.processException(XToolkit.java:535) > at sun.awt.X11.XToolkit.run(XToolkit.java:614) > at sun.awt.X11.XToolkit.run(XToolkit.java:543) > at java.lang.Thread.run(Thread.java:679) > > Hang was even when I changed generate-id() to "1". Seems like a bug. > Markin Alex.
CG
Christian Grün12 Jun 2012 18:18 #
Hi Alex, I've tried to further reduce memory consumption of XQuery Update operations (however, this won't change the general behavior, as the XQUF specification will always expect update operations to be cached before they can be evaluated). Feel free to try the latest snapshot [1]. Christian [1] http://files.basex.org/releases/latest/
Quoted text · Christian Grün · 37 lines
>> for $i in collection('stars')/csv/record >> return insert node (attribute {'id'} {generate-id()} ) into $i > > before any update is performed, all operations first need to be put on > the pending update list [1]. It looks as if this list requires too > much memory. How much memory did you assign to the JRE? > > Christian > > [1] http://docs.basex.org/wiki/XQuery_Update#Pending_Update_List > __________________________________________________ > >> with this file (1000000 strings with 3 integer in any in csv format). I had >> a hang of basex and onece got the following: >> >> Exception in thread "Thread-526" java.lang.OutOfMemoryError: Java heap space >> Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC >> overhead limit exceeded >> at java.awt.EventQueue.postEvent(EventQueue.java:261) >> at java.awt.EventQueue.postEventPrivate(EventQueue.java:215) >> Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: GC overhead limit >> exceeded >> Exception in thread "SIGTERM handler" java.lang.OutOfMemoryError: GC >> overhead limit exceeded >> >> and again - the folowing: >> >> Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: Java heap space >> at java.util.logging.LogRecord.<init>(LogRecord.java:151) >> at java.util.logging.Logger.log(Logger.java:607) >> at sun.awt.X11.XToolkit.processException(XToolkit.java:535) >> at sun.awt.X11.XToolkit.run(XToolkit.java:614) >> at sun.awt.X11.XToolkit.run(XToolkit.java:543) >> at java.lang.Thread.run(Thread.java:679) >> >> Hang was even when I changed generate-id() to "1". Seems like a bug. >> Markin Alex.
AM
Alex Markin12 Jun 2012 22:46 #
I didn't find where the default value of jre memory allocation is set, but tried to call basex (BaseX73-20120612.205120) with the following command: java -cp "$CP" $VM "${vm_args[@]}" -Xms1024m -Xmx3072m org.basex.BaseXGUI "${general_args[@]}" and it still hanging. Even GUI does not response. Markin Alex. 2012/6/12 Christian Grün <christian.gruen at gmail.com>
Quoted text · Christian Grün · 55 lines
> Hi Alex, > > I've tried to further reduce memory consumption of XQuery Update > operations (however, this won't change the general behavior, as the > XQUF specification will always expect update operations to be cached > before they can be evaluated). Feel free to try the latest snapshot > [1]. > > Christian > > [1] http://files.basex.org/releases/latest/ > ___________________________ > > >> for $i in collection('stars')/csv/record > >> return insert node (attribute {'id'} {generate-id()} ) into $i > > > > before any update is performed, all operations first need to be put on > > the pending update list [1]. It looks as if this list requires too > > much memory. How much memory did you assign to the JRE? > > > > Christian > > > > [1] http://docs.basex.org/wiki/XQuery_Update#Pending_Update_List > > __________________________________________________ > > > >> with this file (1000000 strings with 3 integer in any in csv format). I > had > >> a hang of basex and onece got the following: > >> > >> Exception in thread "Thread-526" java.lang.OutOfMemoryError: Java heap > space > >> Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC > >> overhead limit exceeded > >> at java.awt.EventQueue.postEvent(EventQueue.java:261) > >> at java.awt.EventQueue.postEventPrivate(EventQueue.java:215) > >> Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: GC overhead > limit > >> exceeded > >> Exception in thread "SIGTERM handler" java.lang.OutOfMemoryError: GC > >> overhead limit exceeded > >> > >> and again - the folowing: > >> > >> Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: Java heap > space > >> at java.util.logging.LogRecord.<init>(LogRecord.java:151) > >> at java.util.logging.Logger.log(Logger.java:607) > >> at sun.awt.X11.XToolkit.processException(XToolkit.java:535) > >> at sun.awt.X11.XToolkit.run(XToolkit.java:614) > >> at sun.awt.X11.XToolkit.run(XToolkit.java:543) > >> at java.lang.Thread.run(Thread.java:679) > >> > >> Hang was even when I changed generate-id() to "1". Seems like a bug. > >> Markin Alex. >
CG
Christian Grün13 Jun 2012 16:10 #
Hi Alex, it looks as if the sheer number of updates to be performed is simply too large. This may look surprising, regarding the general simplicity of your use case (..add 1 mio. attributes to a document..), but the general problem of the update standard is that all update operations (and the accompanying meta-data) have to be cached first. I have a number of optimizations in mind that could further reduce memory consumption (such as introducing hashes for identical QNames and node instances), but this will take a little bit longer. Christian
Quoted text · Alex Markin · 69 lines
> I didn't find where the default value of jre memory allocation is set, but > tried to call basex (BaseX73-20120612.205120) with the following command: > > java -cp "$CP" $VM "${vm_args[@]}" -Xms1024m -Xmx3072m org.basex.BaseXGUI > "${general_args[@]}" > > and it still hanging. Even GUI does not response. > > Markin Alex. > > > 2012/6/12 Christian Grün <christian.gruen at gmail.com> >> >> Hi Alex, >> >> I've tried to further reduce memory consumption of XQuery Update >> operations (however, this won't change the general behavior, as the >> XQUF specification will always expect update operations to be cached >> before they can be evaluated). Feel free to try the latest snapshot >> [1]. >> >> Christian >> >> [1] http://files.basex.org/releases/latest/ >> ___________________________ >> >> >> for $i in collection('stars')/csv/record >> >> return insert node (attribute {'id'} {generate-id()} ) into $i >> > >> > before any update is performed, all operations first need to be put on >> > the pending update list [1]. It looks as if this list requires too >> > much memory. How much memory did you assign to the JRE? >> > >> > Christian >> > >> > [1] http://docs.basex.org/wiki/XQuery_Update#Pending_Update_List >> > __________________________________________________ >> > >> >> with this file (1000000 strings with 3 integer in any in csv format). I >> >> had >> >> a hang of basex and onece got the following: >> >> >> >> Exception in thread "Thread-526" java.lang.OutOfMemoryError: Java heap >> >> space >> >> Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC >> >> overhead limit exceeded >> >> at java.awt.EventQueue.postEvent(EventQueue.java:261) >> >> at java.awt.EventQueue.postEventPrivate(EventQueue.java:215) >> >> Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: GC overhead >> >> limit >> >> exceeded >> >> Exception in thread "SIGTERM handler" java.lang.OutOfMemoryError: GC >> >> overhead limit exceeded >> >> >> >> and again - the folowing: >> >> >> >> Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: Java heap >> >> space >> >> at java.util.logging.LogRecord.<init>(LogRecord.java:151) >> >> at java.util.logging.Logger.log(Logger.java:607) >> >> at sun.awt.X11.XToolkit.processException(XToolkit.java:535) >> >> at sun.awt.X11.XToolkit.run(XToolkit.java:614) >> >> at sun.awt.X11.XToolkit.run(XToolkit.java:543) >> >> at java.lang.Thread.run(Thread.java:679) >> >> >> >> Hang was even when I changed generate-id() to "1". Seems like a bug. >> >> Markin Alex. > >

Written for BaseX 7.2. BaseX may have changed since: please check the current documentation.

« fn:trace() does not printReport »
⚡Generated with XQuery