database creation baseX.8.6.7
Monday, 23 April 2018 · 7 messages
GC
FE
Fabrice ETANCHAUD23 Apr 2018 15:23 #
Hi Giuseppe,
I think it is because you are using xquery and your backfile is huge with respect to your java memory settings. Db:create function is an updating function, so all your inserted data has to go to the pending update list (see [1] for explanation) before database creation, causing out of memory. If you want to automate database creation/update you could have a look at the xml command files, a way to create a batch of BaseX commands ([2]). In my use cases, I found it easier to write XQuery queries, which generate XML Command files, to get rid of memory concerns.
Hoping it helps,
Best regards,
[1] : http://docs.basex.org/wiki/XQuery_Update#Pending_Update_List
[2] : http://docs.basex.org/wiki/Commands
-----Message d'origine-----
Quoted text · Giuseppe Celano · 13 lines
De : basex-talk-bounces at mailman.uni-konstanz.de [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de Giuseppe Celano Envoyé : lundi 23 avril 2018 15:03 À : BaseX Objet : [basex-talk] database creation baseX.8.6.7 Hi All, I can create a database via the GUI, but if I use db:create [1] I get the message "out of main memory": why? Thanks! db:create("myDB", "sourceDirectory", "destinationDirectory", map{"ftindex": true(), "language": false()} ) Best, Giuseppe
CG
Hi Giuseppe,
Apart rom Fabrice’s helpful hints, you could try to enable the
ADDCACHE option [1].
Cheers,
Christian
[1] http://docs.basex.org/wiki/Database_Module#db:createQuoted text · Giuseppe Celano · 15 lines
On Mon, Apr 23, 2018 at 3:03 PM, Giuseppe Celano <celano at informatik.uni-leipzig.de> wrote: > Hi All, > > I can create a database via the GUI, but if I use db:create [1] I get the message "out of main memory": why? Thanks! > > db:create("myDB", > "sourceDirectory", > "destinationDirectory", > map{"ftindex": true(), "language": false()} > ) > > Best, > Giuseppe >
GC
Giuseppe Celano23 Apr 2018 16:52 #
Yes, with "addcache" it works! Thanks to both of you.
Quoted text · Christian Grün · 28 lines
> On Apr 23, 2018, at 3:56 PM, Christian Grün <christian.gruen at gmail.com> wrote: > > Hi Giuseppe, > > Apart rom Fabrice’s helpful hints, you could try to enable the > ADDCACHE option [1]. > > Cheers, > Christian > > [1] http://docs.basex.org/wiki/Database_Module#db:create > > On Mon, Apr 23, 2018 at 3:03 PM, Giuseppe Celano > <celano at informatik.uni-leipzig.de> wrote: >> Hi All, >> >> I can create a database via the GUI, but if I use db:create [1] I get the message "out of main memory": why? Thanks! >> >> db:create("myDB", >> "sourceDirectory", >> "destinationDirectory", >> map{"ftindex": true(), "language": false()} >> ) >> >> Best, >> Giuseppe >> >
FE
Fabrice ETANCHAUD23 Apr 2018 16:55 #
Dear Christian,
I am a curious man,
Is the Pending Update List bypassed in some way when using addcache ?
Best regards,
Fabrice
-----Message d'origine-----
Quoted text · Giuseppe Celano · 34 lines
De : basex-talk-bounces at mailman.uni-konstanz.de [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de Giuseppe Celano Envoyé : lundi 23 avril 2018 16:53 À : Christian Grün Cc : BaseX Objet : Re: [basex-talk] database creation baseX.8.6.7 Yes, with "addcache" it works! Thanks to both of you. > On Apr 23, 2018, at 3:56 PM, Christian Grün <christian.gruen at gmail.com> wrote: > > Hi Giuseppe, > > Apart rom Fabrice’s helpful hints, you could try to enable the > ADDCACHE option [1]. > > Cheers, > Christian > > [1] http://docs.basex.org/wiki/Database_Module#db:create > > On Mon, Apr 23, 2018 at 3:03 PM, Giuseppe Celano > <celano at informatik.uni-leipzig.de> wrote: >> Hi All, >> >> I can create a database via the GUI, but if I use db:create [1] I get the message "out of main memory": why? Thanks! >> >> db:create("myDB", >> "sourceDirectory", >> "destinationDirectory", >> map{"ftindex": true(), "language": false()} >> ) >> >> Best, >> Giuseppe >> >
CG
Christian Grün23 Apr 2018 17:03 #
Hi Fabrice,
Quoted text · Fabrice ETANCHAUD · 1 line
> I am a curious man,
…a good thing ;)
Quoted text · Fabrice ETANCHAUD · 1 line
> Is the Pending Update List bypassed in some way when using addcache ?
If addcache is set, the nodes to be added will be written to a temporary database on disk. This way you can circumvent the main memory limitations, but it takes a bit longer (this is is why it’s an optional feature).
Cheers,
Christian
Quoted text · Fabrice ETANCHAUD · 39 lines
> -----Message d'origine----- > De : basex-talk-bounces at mailman.uni-konstanz.de [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de Giuseppe Celano > Envoyé : lundi 23 avril 2018 16:53 > À : Christian Grün > Cc : BaseX > Objet : Re: [basex-talk] database creation baseX.8.6.7 > > Yes, with "addcache" it works! Thanks to both of you. > > >> On Apr 23, 2018, at 3:56 PM, Christian Grün <christian.gruen at gmail.com> wrote: >> >> Hi Giuseppe, >> >> Apart rom Fabrice’s helpful hints, you could try to enable the >> ADDCACHE option [1]. >> >> Cheers, >> Christian >> >> [1] http://docs.basex.org/wiki/Database_Module#db:create >> >> On Mon, Apr 23, 2018 at 3:03 PM, Giuseppe Celano >> <celano at informatik.uni-leipzig.de> wrote: >>> Hi All, >>> >>> I can create a database via the GUI, but if I use db:create [1] I get the message "out of main memory": why? Thanks! >>> >>> db:create("myDB", >>> "sourceDirectory", >>> "destinationDirectory", >>> map{"ftindex": true(), "language": false()} >>> ) >>> >>> Best, >>> Giuseppe >>> >> >
GC
Giuseppe Celano23 Apr 2018 17:10 #
Yes, I confirm it is a bit longer than using the command via the GUI.
Quoted text · Christian Grün · 60 lines
> On Apr 23, 2018, at 5:03 PM, Christian Grün <christian.gruen at gmail.com> wrote: > > Hi Fabrice, > >> I am a curious man, > > …a good thing ;) > >> Is the Pending Update List bypassed in some way when using addcache ? > > If addcache is set, the nodes to be added will be written to a > temporary database on disk. This way you can circumvent the main > memory limitations, but it takes a bit longer (this is is why it’s an > optional feature). > > Cheers, > Christian > > > >> -----Message d'origine----- >> De : basex-talk-bounces at mailman.uni-konstanz.de [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de Giuseppe Celano >> Envoyé : lundi 23 avril 2018 16:53 >> À : Christian Grün >> Cc : BaseX >> Objet : Re: [basex-talk] database creation baseX.8.6.7 >> >> Yes, with "addcache" it works! Thanks to both of you. >> >> >>> On Apr 23, 2018, at 3:56 PM, Christian Grün <christian.gruen at gmail.com> wrote: >>> >>> Hi Giuseppe, >>> >>> Apart rom Fabrice’s helpful hints, you could try to enable the >>> ADDCACHE option [1]. >>> >>> Cheers, >>> Christian >>> >>> [1] http://docs.basex.org/wiki/Database_Module#db:create >>> >>> On Mon, Apr 23, 2018 at 3:03 PM, Giuseppe Celano >>> <celano at informatik.uni-leipzig.de> wrote: >>>> Hi All, >>>> >>>> I can create a database via the GUI, but if I use db:create [1] I get the message "out of main memory": why? Thanks! >>>> >>>> db:create("myDB", >>>> "sourceDirectory", >>>> "destinationDirectory", >>>> map{"ftindex": true(), "language": false()} >>>> ) >>>> >>>> Best, >>>> Giuseppe >>>> >>> >> >
Written for BaseX 9.0. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery