Xquery returns 'Out of Main Memory'

Monday, 3 February 2014 · 2 messages
CG
Christian Grünin reply to DAIANE ROBERTA CANDIDA, 3 Feb 2014, 14:593 Feb 2014 14:13 #
Hi Daiane, due to some nifty details in the XQFT spec., it is hardly possible to make "ftand" really efficent. But there is one way out: you can use the "all" keyword. Please find more details here: http://docs.basex.org/wiki/Full-Text#Combining_Results Hope this helps, Christian
Quoted text · DAIANE ROBERTA CANDIDA · 185 lines
On Mon, Feb 3, 2014 at 1:59 PM, DAIANE ROBERTA CANDIDA <dairc_8 at hotmail.com> wrote: > Hello, > > The query below returns 'Out of Main Memory': > > XQUERY > collection("/home/arwen/daiane_oracle/loading_inex2009/xml_pages/")//article[.//text() > contains text "Nobel" ftand "prize"] > > > I am using basexclient. > > I have a collection of xml files that have 14GB size. > -------------------------------------------------------------------- >> LIST > name Resources Size Input Path > ---------------------------------------------------------------------- > bd_inex2009 730283 16263937902 > --------------------------------------------------------------------- > > I dumped those files by the commands below: > > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles1/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles2/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles3/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles4/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles5/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles6/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles7/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles8/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles9/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles10/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles11/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles12/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles13/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles14/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles15/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles16/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles17/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles18/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles19/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles20/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles21/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles22/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles23/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles24/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles25/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles26/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles27/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles28/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles29/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles30/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles31/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles32/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles33/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles34/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles35/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles36/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles37/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles38/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles39/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles40/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles41/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles42/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles43/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles44/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles45/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles46/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles47/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles48/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles49/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles50/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles51/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles52/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles53/; > ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles54/; > > > ---------------------------------------------------------------------------------------- > > The command INFO returns: > INFO > General Information > Version: 7.7.2 > Used Memory: 79 MB > > Main Options > DBPATH: /home/arwen/BaseXData > DEBUG: false > EVENTPORT: 1985 > GLOBALLOCK: false > HOST: localhost > HTTPLOCAL: false > KEEPALIVE: 0 > LANG: English > LANGKEYS: false > LOG: true > LOGMSGMAXLEN: 1000 > NONPROXYHOSTS: > PARALLEL: 8 > PASSWORD: > PORT: 1984 > PROXYHOST: > PROXYPORT: 80 > REPOPATH: /home/arwen/BaseXRepo > RESTXQPATH: > SERVERHOST: > SERVERPORT: 1984 > STOPPORT: 8985 > TIMEOUT: 0 > USER: > WEBPATH: /home/arwen/BaseXWeb > > Options > ADDARCHIVES: true > ADDCACHE: false > ADDRAW: false > ATTRINDEX: true > AUTOFLUSH: true > BINDINGS: > CACHEQUERY: false > CASESENS: false > CATFILE: > CHECKSTRINGS: true > CHOP: true > COMPPLAN: true > CREATEFILTER: *.xml > CREATEONLY: false > DIACRITICS: false > DOTCOMPACT: false > DOTDISPLAY: true > DOTPLAN: false > DOTTY: dotty > DTD: false > EXPORTER: > FORCECREATE: false > FTINDEX: false > FTINDEXSPLITSIZE: 0 > HTMLOPT: > html=false,omit-xml-declaration=false,method=xml,nons=false,nobogons=false,nodefaults=false,nocolons=false,norestart=false,ignorable=false,emptybogons=false,any=false,norootbogons=false,nocdata=false,lexical=false,encoding=utf-8 > INDEXSPLITSIZE: 0 > INLINELIMIT: 0 > INTPARSE: true > LANGUAGE: en > LSERROR: 0 > MAINMEM: false > MAXCATS: 100 > MAXHITS: -1 > MAXLEN: 96 > MAXSTAT: 30 > PARSER: xml > PARSEROPT: > encoding=UTF-8,flat=false,header=false,jsonml=false,lines=true,separator=comma > QUERYINFO: false > QUERYPATH: > RUNS: 1 > SERIALIZE: true > SERIALIZER: > SKIPCORRUPT: false > STEMMING: false > STOPWORDS: > STRIPNS: false > TAILCALLS: 256 > TEXTINDEX: true > UPDINDEX: false > WRITEBACK: false > XMLPLAN: false > XQUERY3: true > > ----------------------------------------------------------- > > How can I run XQuery in that collection? > > Regards, > > Daiane > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
DC
DAIANE ROBERTA CANDIDA3 Feb 2014 14:59 #
Hello, The query below returns 'Out of Main Memory': XQUERY collection("/home/arwen/daiane_oracle/loading_inex2009/xml_pages/")//article[.//text() contains text "Nobel" ftand "prize"] I am using basexclient. I have a collection of xml files that have 14GB size. --------------------------------------------------------------------
Quoted text · 1 line
> LIST
name Resources Size Input Path ---------------------------------------------------------------------- bd_inex2009 730283 16263937902 --------------------------------------------------------------------- I dumped those files by the commands below:
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles1/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles2/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles3/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles4/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles5/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles6/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles7/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles8/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles9/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles10/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles11/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles12/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles13/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles14/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles15/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles16/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles17/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles18/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles19/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles20/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles21/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles22/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles23/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles24/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles25/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles26/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles27/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles28/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles29/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles30/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles31/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles32/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles33/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles34/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles35/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles36/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles37/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles38/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles39/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles40/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles41/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles42/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles43/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles44/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles45/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles46/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles47/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles48/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles49/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles50/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles51/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles52/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles53/;
ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles54/;
---------------------------------------------------------------------------------------- The command INFO returns: INFO General Information Version: 7.7.2 Used Memory: 79 MB Main Options DBPATH: /home/arwen/BaseXData DEBUG: false EVENTPORT: 1985 GLOBALLOCK: false HOST: localhost HTTPLOCAL: false KEEPALIVE: 0 LANG: English LANGKEYS: false LOG: true LOGMSGMAXLEN: 1000 NONPROXYHOSTS: PARALLEL: 8 PASSWORD: PORT: 1984 PROXYHOST: PROXYPORT: 80 REPOPATH: /home/arwen/BaseXRepo RESTXQPATH: SERVERHOST: SERVERPORT: 1984 STOPPORT: 8985 TIMEOUT: 0 USER: WEBPATH: /home/arwen/BaseXWeb Options ADDARCHIVES: true ADDCACHE: false ADDRAW: false ATTRINDEX: true AUTOFLUSH: true BINDINGS: CACHEQUERY: false CASESENS: false CATFILE: CHECKSTRINGS: true CHOP: true COMPPLAN: true CREATEFILTER: *.xml CREATEONLY: false DIACRITICS: false DOTCOMPACT: false DOTDISPLAY: true DOTPLAN: false DOTTY: dotty DTD: false EXPORTER: FORCECREATE: false FTINDEX: false FTINDEXSPLITSIZE: 0 HTMLOPT: html=false,omit-xml-declaration=false,method=xml,nons=false,nobogons=false,nodefaults=false,nocolons=false,norestart=false,ignorable=false,emptybogons=false,any=false,norootbogons=false,nocdata=false,lexical=false,encoding=utf-8 INDEXSPLITSIZE: 0 INLINELIMIT: 0 INTPARSE: true LANGUAGE: en LSERROR: 0 MAINMEM: false MAXCATS: 100 MAXHITS: -1 MAXLEN: 96 MAXSTAT: 30 PARSER: xml PARSEROPT: encoding=UTF-8,flat=false,header=false,jsonml=false,lines=true,separator=comma QUERYINFO: false QUERYPATH: RUNS: 1 SERIALIZE: true SERIALIZER: SKIPCORRUPT: false STEMMING: false STOPWORDS: STRIPNS: false TAILCALLS: 256 TEXTINDEX: true UPDINDEX: false WRITEBACK: false XMLPLAN: false XQUERY3: true ----------------------------------------------------------- How can I run XQuery in that collection? Regards, Daiane

⚡Generated with XQuery