JSON output

> Is there a way Basex server can send us JSON output instead of xml?

> On Wed, May 11, 2011 at 10:28 AM, Erol Akarsu <eakarsu at gmail.com> wrote: >> >> I am filtering one big xml db that 800Mb stored into basex db already. >> >> When I write filtered data into a file, I am getting out of memory. >> >> I know basex is generating result filtered document in memory before >> writing and facing memory error. >> >> Can basex write results block by block? >> >> Thanks >> >> >> On Wed, May 4, 2011 at 9:38 AM, Erol Akarsu <eakarsu at gmail.com> wrote: >>> >>> I tried to delete some nodes with "delete nodes .." xquery update >>> command. It actually did what was requested. >>> >>> I checked the database size that is still same as previous one. I know >>> the nodes deleted has a lot of data and I think database indexes would be >>> adjusted accordingly. >>> >>> Then, I exported database as xml file and recreated db. I can see it has >>> real size. >>> >>> My question is why DB is not adjusting indexes when nodes deleted? >>> >>> Thanks >>> >>> Erol Akarsu >>> >>> >>> On Fri, Apr 29, 2011 at 9:23 AM, Erol Akarsu <eakarsu at gmail.com> wrote: >>>> >>>> Hi, >>>> >>>> Have we thought on clustering basex servers so we can partition xml >>>> documents? >>>> >>>> Here, I am only interested in a global partitioning: >>>> >>>> Let's say if we have xml document like this, I would like to partition >>>> RECORDS content so that each host will have equal number of RECORD elements. >>>> Then, we need to aggregate results of individual hosts. >>>> Can we implement this simple clustering framework with Basex? >>>> >>>> >>>> <RECORDS> >>>> <RECORD> >>>> ..... >>>> </RECORD> >>>> <RECORD> >>>> ..... >>>> </RECORD> >>>> </RECORDS> >>>> >>>> On Mon, Apr 11, 2011 at 12:35 PM, Erol Akarsu <eakarsu at gmail.com> wrote: >>>>> >>>>> Ok, >>>>> >>>>> I was able to run full text search with another XML Database. >>>>> >>>>> I am primarily interested in how Basex will play with big XML file >>>>> Wikipedia. >>>>> >>>>> Actually, Database create of wikipedia is fine. But when I add full tet >>>>> search and indexes for it, it always throws out of memory exception error. >>>>> >>>>> I have changed -Xmx with 6GB that is still not enough to generate >>>>> indexes for Wikipedia. >>>>> >>>>> Can you help me on how to generate indexes with a machine that case >>>>> 6GB for Basex process? >>>>> >>>>> Thanks >>>>> >>>>> Erol Akarsu >>>>> >>>>> >>>>> >>>>> On Sun, Apr 10, 2011 at 4:07 AM, Andreas Weiler >>>>> <andreas.weiler at uni-konstanz.de> wrote: >>>>>> >>>>>> Hi, >>>>>> the following query could work for you: >>>>>> declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >>>>>> for $i in doc("enwiki-latest-pages-articles")//w:sitename >>>>>> return $i[. contains text "Wikipedia"]/.. >>>>>> -- Andreas >>>>>> Am 09.04.2011 um 20:43 schrieb Erol Akarsu: >>>>>> >>>>>> Hi >>>>>> >>>>>> I am having difficulty in running full text operators. This script >>>>>> gives siteinfo below >>>>>> declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >>>>>> let $d := fn:doc ("enwiki-latest-pages-articles") >>>>>> return ($d//w:siteinfo)[1] >>>>>> >>>>>> But return $d//w:siteinfo[w:sitename contains text 'Wikipedia'] does >>>>>> NOT give same node >>>>>> Why "contains" ft operator behave incorrectly? I remember it was >>>>>> working fine. I just dropped and recreated database and turn all indexes. >>>>>> Can you help me? >>>>>> Query info is here: >>>>>> >>>>>> Query: declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >>>>>> Compiling: >>>>>> - pre-evaluating fn:doc("enwiki-latest-pages-articles") >>>>>> - adding text() step >>>>>> - optimizing descendant-or-self step(s) >>>>>> - removing path with no index results >>>>>> - pre-evaluating (())[1] >>>>>> - binding static variable $res >>>>>> - pre-evaluating fn:doc("enwiki-latest-pages-articles") >>>>>> - binding static variable $d >>>>>> - adding text() step >>>>>> - optimizing descendant-or-self step(s) >>>>>> - removing path with no index results >>>>>> - simplifying flwor expression >>>>>> Result: () >>>>>> Timing: >>>>>> - Parsing: 0.46 ms >>>>>> - Compiling: 0.42 ms >>>>>> - Evaluating: 0.17 ms >>>>>> - Printing: 0.1 ms >>>>>> - Total Time: 1.15 ms >>>>>> Query plan: >>>>>> <sequence size="0"/> >>>>>> >>>>>> >>>>>> >>>>>> <siteinfo xmlns="http://www.mediawiki.org/xml/export-0.5/" >>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> >>>>>> <sitename>Wikipedia</sitename> >>>>>> <base>http://en.wikipedia.org/wiki/Main_Page</base> >>>>>> <generator>MediaWiki 1.17wmf1</generator> >>>>>> <case>first-letter</case> >>>>>> <namespaces> >>>>>> <namespace key="-2" case="first-letter">Media</namespace> >>>>>> <namespace key="-1" case="first-letter">Special</namespace> >>>>>> <namespace key="0" case="first-letter"/> >>>>>> <namespace key="1" case="first-letter">Talk</namespace> >>>>>> <namespace key="2" case="first-letter">User</namespace> >>>>>> <namespace key="3" case="first-letter">User talk</namespace> >>>>>> <namespace key="4" case="first-letter">Wikipedia</namespace> >>>>>> <namespace key="5" case="first-letter">Wikipedia talk</namespace> >>>>>> <namespace key="6" case="first-letter">File</namespace> >>>>>> <namespace key="7" case="first-letter">File talk</namespace> >>>>>> <namespace key="8" case="first-letter">MediaWiki</namespace> >>>>>> <namespace key="9" case="first-letter">MediaWiki talk</namespace> >>>>>> <namespace key="10" case="first-letter">Template</namespace> >>>>>> <namespace key="11" case="first-letter">Template talk</namespace> >>>>>> <namespace key="12" case="first-letter">Help</namespace> >>>>>> <namespace key="13" case="first-letter">Help talk</namespace> >>>>>> <namespace key="14" case="first-letter">Category</namespace> >>>>>> <namespace key="15" case="first-letter">Category talk</namespace> >>>>>> <namespace key="100" case="first-letter">Portal</namespace> >>>>>> <namespace key="101" case="first-letter">Portal talk</namespace> >>>>>> <namespace key="108" case="first-letter">Book</namespace> >>>>>> <namespace key="109" case="first-letter">Book talk</namespace> >>>>>> </namespaces> >>>>>> </siteinfo> >>>>>> >>>>>> On Mon, Apr 4, 2011 at 7:31 AM, Erol Akarsu <eakarsu at gmail.com> wrote: >>>>>>> >>>>>>> I imported wikipedia xml into basex and tried to search it. >>>>>>> >>>>>>> But searching it takes longer. >>>>>>> >>>>>>> I tried to search one element that is first child of whole document >>>>>>> and it took 52 sec. >>>>>>> I know the XML file is very big 31GB. How can I optimize the search? >>>>>>> >>>>>>> declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >>>>>>> >>>>>>> let $d := fn:doc ("enwiki-latest-pages-articles")//w:siteinfo >>>>>>> return $d >>>>>>> >>>>>>> Database info: >>>>>>> >>>>>>> > open enwiki-latest-pages-articles >>>>>>> Database 'enwiki-latest-pages-articles' opened in 778.49 ms. >>>>>>> > info database >>>>>>> Database Properties >>>>>>> Name: enwiki-latest-pages-articles >>>>>>> Size: 23356 MB >>>>>>> Nodes: 228090153 >>>>>>> Height: 6 >>>>>>> >>>>>>> Database Creation >>>>>>> Path: /mnt/hgfs/C/tmp/enwiki-latest-pages-articles.xml >>>>>>> Time Stamp: 03.04.2011 12:29:15 >>>>>>> Input Size: 30025 MB >>>>>>> Encoding: UTF-8 >>>>>>> Documents: 1 >>>>>>> Whitespace Chopping: ON >>>>>>> Entity Parsing: OFF >>>>>>> >>>>>>> Indexes >>>>>>> Up-to-date: true >>>>>>> Path Summary: ON >>>>>>> Text Index: ON >>>>>>> Attribute Index: ON >>>>>>> Full-Text Index: OFF >>>>>>> > >>>>>>> >>>>>>> >>>>>>> Timing info: >>>>>>> >>>>>>> Query: declare namespace >>>>>>> w="http://www.mediawiki.org/xml/export-0.5/"; >>>>>>> Compiling: >>>>>>> - pre-evaluating fn:doc("enwiki-latest-pages-articles") >>>>>>> - optimizing descendant-or-self step(s) >>>>>>> - binding static variable $d >>>>>>> - removing variable $d >>>>>>> - simplifying flwor expression >>>>>>> Result: element siteinfo { ... } >>>>>>> Timing: >>>>>>> - Parsing: 1.4 ms >>>>>>> - Compiling: 52599.0 ms >>>>>>> - Evaluating: 0.28 ms >>>>>>> - Printing: 0.62 ms >>>>>>> - Total Time: 52601.32 ms >>>>>>> Query plan: >>>>>>> <DBNode name="enwiki-latest-pages-articles" pre="5"/> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Result of query: >>>>>>> >>>>>>> <siteinfo xmlns="http://www.mediawiki.org/xml/export-0.5/" >>>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> >>>>>>> <sitename>Wikipedia</sitename> >>>>>>> <base>http://en.wikipedia.org/wiki/Main_Page</base> >>>>>>> <generator>MediaWiki 1.17wmf1</generator> >>>>>>> <case>first-letter</case> >>>>>>> <namespaces> >>>>>>> <namespace key="-2" case="first-letter">Media</namespace> >>>>>>> <namespace key="-1" case="first-letter">Special</namespace> >>>>>>> <namespace key="0" case="first-letter"/> >>>>>>> <namespace key="1" case="first-letter">Talk</namespace> >>>>>>> <namespace key="2" case="first-letter">User</namespace> >>>>>>> <namespace key="3" case="first-letter">User talk</namespace> >>>>>>> <namespace key="4" case="first-letter">Wikipedia</namespace> >>>>>>> <namespace key="5" case="first-letter">Wikipedia talk</namespace> >>>>>>> <namespace key="6" case="first-letter">File</namespace> >>>>>>> <namespace key="7" case="first-letter">File talk</namespace> >>>>>>> <namespace key="8" case="first-letter">MediaWiki</namespace> >>>>>>> <namespace key="9" case="first-letter">MediaWiki talk</namespace> >>>>>>> <namespace key="10" case="first-letter">Template</namespace> >>>>>>> <namespace key="11" case="first-letter">Template talk</namespace> >>>>>>> <namespace key="12" case="first-letter">Help</namespace> >>>>>>> <namespace key="13" case="first-letter">Help talk</namespace> >>>>>>> <namespace key="14" case="first-letter">Category</namespace> >>>>>>> <namespace key="15" case="first-letter">Category talk</namespace> >>>>>>> <namespace key="100" case="first-letter">Portal</namespace> >>>>>>> <namespace key="101" case="first-letter">Portal talk</namespace> >>>>>>> <namespace key="108" case="first-letter">Book</namespace> >>>>>>> <namespace key="109" case="first-letter">Book talk</namespace> >>>>>>> </namespaces> >>>>>>> </siteinfo> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Erol Akarsu >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> BaseX-Talk mailing list >>>>>> BaseX-Talk at mailman.uni-konstanz.de >>>>>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>>>>> >>>>> >>>> >>> >> > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >

> Is there a way Basex server can send us JSON output instead of xml? > >

Am 13.05.2011 um 15:33 schrieb Jan Vlčinský (CAD): > ching archive or finding your own - there used to be some instruction how to use functions from jar archives as XQuery ones, but I have always problems finding this).

> Jan, > > Thanks for sharing your experience. > > I am thinking to use Basex as database for product/services catalog and > show them in html document. I think jquery will be more comfortable with > json input that is also lightweight. > > I did not compare the complexity of parsing xml and json. Do you have any > experience? > > Thanks > > Erol Akarsu > > > > On Fri, May 13, 2011 at 9:33 AM, Jan Vlčinský (CAD) < > jan.vlcinsky at cad-programs.com> wrote: > >> Hi Erol >> I had exactly the same question some time ago. >> As far as I know, BaseX does not have "native" JSON output supported. I >> was advised to use either some generic XSLT transformation, which converts >> XML into JSON, or use some Java library, which is supporting this and can be >> optionally called as imported function (I do not remembter which one, try >> searching archive or finding your own - there used to be some instruction >> how to use functions from jar archives as XQuery ones, but I have always >> problems finding this). >> >> My real life project ended up with writting my own XQuery, generating text >> output. Any approach is making some assumptions and you easily find, that >> some parts of your system do not fit. My task was to generate json output, >> being usable for SIMILE framework and I had to follow the JSON structure, >> which SIMILE was expecting. >> >> Good luck >> >> Jan >> >> *Ing. Jan Vlčinský* >> CAD programy >> Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic >> tel: +420-597 602 024; mob: +420-608 979 040 >> skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com >> http://cz.linkedin.com/in/vlcinsky >> >> >> >> 2011/5/13 Erol Akarsu <eakarsu at gmail.com> >> >>> Is there a way Basex server can send us JSON output instead of xml? >>> >>> >

> And thanks to Michael for pointing to JSON implementation for Java.

On Wed, May 11, 2011 at 10:28 AM, Erol Akarsu <eakarsu at gmail.com> wrote: > I am filtering one big xml db that 800Mb stored into basex db already. > > When I write filtered data into a file, I am getting out of memory. > > I know basex is generating result filtered document in memory before > writing and facing memory error. > > Can basex write results block by block? > > Thanks > > > On Wed, May 4, 2011 at 9:38 AM, Erol Akarsu <eakarsu at gmail.com> wrote: > >> I tried to delete some nodes with "delete nodes .." xquery update command. >> It actually did what was requested. >> >> I checked the database size that is still same as previous one. I know the >> nodes deleted has a lot of data and I think database indexes would be >> adjusted accordingly. >> >> Then, I exported database as xml file and recreated db. I can see it has >> real size. >> >> My question is why DB is not adjusting indexes when nodes deleted? >> >> Thanks >> >> Erol Akarsu >> >> >> On Fri, Apr 29, 2011 at 9:23 AM, Erol Akarsu <eakarsu at gmail.com> wrote: >> >>> Hi, >>> >>> Have we thought on clustering basex servers so we can partition xml >>> documents? >>> >>> Here, I am only interested in a global partitioning: >>> >>> Let's say if we have xml document like this, I would like to partition >>> RECORDS content so that each host will have equal number of RECORD elements. >>> Then, we need to aggregate results of individual hosts. >>> Can we implement this simple clustering framework with Basex? >>> >>> >>> <RECORDS> >>> <RECORD> >>> ..... >>> </RECORD> >>> <RECORD> >>> ..... >>> </RECORD> >>> </RECORDS> >>> >>> On Mon, Apr 11, 2011 at 12:35 PM, Erol Akarsu <eakarsu at gmail.com> wrote: >>> >>>> Ok, >>>> >>>> I was able to run full text search with another XML Database. >>>> >>>> I am primarily interested in how Basex will play with big XML file >>>> Wikipedia. >>>> >>>> Actually, Database create of wikipedia is fine. But when I add full tet >>>> search and indexes for it, it always throws out of memory exception error. >>>> >>>> I have changed -Xmx with 6GB that is still not enough to generate >>>> indexes for Wikipedia. >>>> >>>> Can you help me on how to generate indexes with a machine that case 6GB >>>> for Basex process? >>>> >>>> Thanks >>>> >>>> Erol Akarsu >>>> >>>> >>>> >>>> On Sun, Apr 10, 2011 at 4:07 AM, Andreas Weiler < >>>> andreas.weiler at uni-konstanz.de> wrote: >>>> >>>>> Hi, >>>>> >>>>> the following query could work for you: >>>>> >>>>> declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >>>>> for $i in doc("enwiki-latest-pages-articles")//w:sitename >>>>> return $i[. contains text "Wikipedia"]/.. >>>>> >>>>> -- Andreas >>>>> >>>>> Am 09.04.2011 um 20:43 schrieb Erol Akarsu: >>>>> >>>>> Hi >>>>> >>>>> I am having difficulty in running full text operators. This script >>>>> gives siteinfo below >>>>> declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >>>>> let $d := fn:doc ("enwiki-latest-pages-articles") >>>>> return ($d//w:siteinfo)[1] >>>>> >>>>> But return $d//w:siteinfo[w:sitename contains text 'Wikipedia'] does >>>>> NOT give same node >>>>> Why "contains" ft operator behave incorrectly? I remember it was >>>>> working fine. I just dropped and recreated database and turn all indexes. >>>>> Can you help me? >>>>> Query info is here: >>>>> >>>>> Query: declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >>>>> Compiling: >>>>> - pre-evaluating fn:doc("enwiki-latest-pages-articles") >>>>> - adding text() step >>>>> - optimizing descendant-or-self step(s) >>>>> - removing path with no index results >>>>> - pre-evaluating (())[1] >>>>> - binding static variable $res >>>>> - pre-evaluating fn:doc("enwiki-latest-pages-articles") >>>>> - binding static variable $d >>>>> - adding text() step >>>>> - optimizing descendant-or-self step(s) >>>>> - removing path with no index results >>>>> - simplifying flwor expression >>>>> Result: () >>>>> Timing: >>>>> - Parsing: 0.46 ms >>>>> - Compiling: 0.42 ms >>>>> - Evaluating: 0.17 ms >>>>> - Printing: 0.1 ms >>>>> - Total Time: 1.15 ms >>>>> Query plan: >>>>> <sequence size="0"/> >>>>> >>>>> >>>>> >>>>> <siteinfo xmlns="http://www.mediawiki.org/xml/export-0.5/" xmlns:xsi=" >>>>> http://www.w3.org/2001/XMLSchema-instance"> >>>>> <sitename>Wikipedia</sitename> >>>>> <base>http://en.wikipedia.org/wiki/Main_Page</base> >>>>> <generator>MediaWiki 1.17wmf1</generator> >>>>> <case>first-letter</case> >>>>> <namespaces> >>>>> <namespace key="-2" case="first-letter">Media</namespace> >>>>> <namespace key="-1" case="first-letter">Special</namespace> >>>>> <namespace key="0" case="first-letter"/> >>>>> <namespace key="1" case="first-letter">Talk</namespace> >>>>> <namespace key="2" case="first-letter">User</namespace> >>>>> <namespace key="3" case="first-letter">User talk</namespace> >>>>> <namespace key="4" case="first-letter">Wikipedia</namespace> >>>>> <namespace key="5" case="first-letter">Wikipedia talk</namespace> >>>>> <namespace key="6" case="first-letter">File</namespace> >>>>> <namespace key="7" case="first-letter">File talk</namespace> >>>>> <namespace key="8" case="first-letter">MediaWiki</namespace> >>>>> <namespace key="9" case="first-letter">MediaWiki talk</namespace> >>>>> <namespace key="10" case="first-letter">Template</namespace> >>>>> <namespace key="11" case="first-letter">Template talk</namespace> >>>>> <namespace key="12" case="first-letter">Help</namespace> >>>>> <namespace key="13" case="first-letter">Help talk</namespace> >>>>> <namespace key="14" case="first-letter">Category</namespace> >>>>> <namespace key="15" case="first-letter">Category talk</namespace> >>>>> <namespace key="100" case="first-letter">Portal</namespace> >>>>> <namespace key="101" case="first-letter">Portal talk</namespace> >>>>> <namespace key="108" case="first-letter">Book</namespace> >>>>> <namespace key="109" case="first-letter">Book talk</namespace> >>>>> </namespaces> >>>>> </siteinfo> >>>>> >>>>> On Mon, Apr 4, 2011 at 7:31 AM, Erol Akarsu <eakarsu at gmail.com> wrote: >>>>> >>>>>> I imported wikipedia xml into basex and tried to search it. >>>>>> >>>>>> But searching it takes longer. >>>>>> >>>>>> I tried to search one element that is first child of whole document >>>>>> and it took 52 sec. >>>>>> I know the XML file is very big 31GB. How can I optimize the search? >>>>>> >>>>>> declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >>>>>> >>>>>> let $d := fn:doc ("enwiki-latest-pages-articles")//w:siteinfo >>>>>> return $d >>>>>> >>>>>> Database info: >>>>>> >>>>>> > open enwiki-latest-pages-articles >>>>>> Database 'enwiki-latest-pages-articles' opened in 778.49 ms. >>>>>> > info database >>>>>> Database Properties >>>>>> Name: enwiki-latest-pages-articles >>>>>> Size: 23356 MB >>>>>> Nodes: 228090153 >>>>>> Height: 6 >>>>>> >>>>>> Database Creation >>>>>> Path: /mnt/hgfs/C/tmp/enwiki-latest-pages-articles.xml >>>>>> Time Stamp: 03.04.2011 12:29:15 >>>>>> Input Size: 30025 MB >>>>>> Encoding: UTF-8 >>>>>> Documents: 1 >>>>>> Whitespace Chopping: ON >>>>>> Entity Parsing: OFF >>>>>> >>>>>> Indexes >>>>>> Up-to-date: true >>>>>> Path Summary: ON >>>>>> Text Index: ON >>>>>> Attribute Index: ON >>>>>> Full-Text Index: OFF >>>>>> > >>>>>> >>>>>> >>>>>> Timing info: >>>>>> >>>>>> Query: declare namespace w="http://www.mediawiki.org/xml/export-0.5/ >>>>>> "; >>>>>> Compiling: >>>>>> - pre-evaluating fn:doc("enwiki-latest-pages-articles") >>>>>> - optimizing descendant-or-self step(s) >>>>>> - binding static variable $d >>>>>> - removing variable $d >>>>>> - simplifying flwor expression >>>>>> Result: element siteinfo { ... } >>>>>> Timing: >>>>>> - Parsing: 1.4 ms >>>>>> - Compiling: 52599.0 ms >>>>>> - Evaluating: 0.28 ms >>>>>> - Printing: 0.62 ms >>>>>> - Total Time: 52601.32 ms >>>>>> Query plan: >>>>>> <DBNode name="enwiki-latest-pages-articles" pre="5"/> >>>>>> >>>>>> >>>>>> >>>>>> Result of query: >>>>>> >>>>>> <siteinfo xmlns="http://www.mediawiki.org/xml/export-0.5/" >>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> >>>>>> <sitename>Wikipedia</sitename> >>>>>> <base>http://en.wikipedia.org/wiki/Main_Page</base> >>>>>> <generator>MediaWiki 1.17wmf1</generator> >>>>>> <case>first-letter</case> >>>>>> <namespaces> >>>>>> <namespace key="-2" case="first-letter">Media</namespace> >>>>>> <namespace key="-1" case="first-letter">Special</namespace> >>>>>> <namespace key="0" case="first-letter"/> >>>>>> <namespace key="1" case="first-letter">Talk</namespace> >>>>>> <namespace key="2" case="first-letter">User</namespace> >>>>>> <namespace key="3" case="first-letter">User talk</namespace> >>>>>> <namespace key="4" case="first-letter">Wikipedia</namespace> >>>>>> <namespace key="5" case="first-letter">Wikipedia talk</namespace> >>>>>> <namespace key="6" case="first-letter">File</namespace> >>>>>> <namespace key="7" case="first-letter">File talk</namespace> >>>>>> <namespace key="8" case="first-letter">MediaWiki</namespace> >>>>>> <namespace key="9" case="first-letter">MediaWiki talk</namespace> >>>>>> <namespace key="10" case="first-letter">Template</namespace> >>>>>> <namespace key="11" case="first-letter">Template talk</namespace> >>>>>> <namespace key="12" case="first-letter">Help</namespace> >>>>>> <namespace key="13" case="first-letter">Help talk</namespace> >>>>>> <namespace key="14" case="first-letter">Category</namespace> >>>>>> <namespace key="15" case="first-letter">Category talk</namespace> >>>>>> <namespace key="100" case="first-letter">Portal</namespace> >>>>>> <namespace key="101" case="first-letter">Portal talk</namespace> >>>>>> <namespace key="108" case="first-letter">Book</namespace> >>>>>> <namespace key="109" case="first-letter">Book talk</namespace> >>>>>> </namespaces> >>>>>> </siteinfo> >>>>>> >>>>>> >>>>>> >>>>>> Thanks >>>>>> >>>>>> Erol Akarsu >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> BaseX-Talk mailing list >>>>> BaseX-Talk at mailman.uni-konstanz.de >>>>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>>>> >>>>> >>>>> >>>> >>> >> >

On Fri, May 13, 2011 at 9:33 AM, Jan Vlčinský (CAD) < jan.vlcinsky at cad-programs.com> wrote: > Hi Erol > I had exactly the same question some time ago. > As far as I know, BaseX does not have "native" JSON output supported. I > was advised to use either some generic XSLT transformation, which converts > XML into JSON, or use some Java library, which is supporting this and can be > optionally called as imported function (I do not remembter which one, try > searching archive or finding your own - there used to be some instruction > how to use functions from jar archives as XQuery ones, but I have always > problems finding this). > > My real life project ended up with writting my own XQuery, generating text > output. Any approach is making some assumptions and you easily find, that > some parts of your system do not fit. My task was to generate json output, > being usable for SIMILE framework and I had to follow the JSON structure, > which SIMILE was expecting. > > Good luck > > Jan > > *Ing. Jan Vlčinský* > CAD programy > Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic > tel: +420-597 602 024; mob: +420-608 979 040 > skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com > http://cz.linkedin.com/in/vlcinsky > > > > 2011/5/13 Erol Akarsu <eakarsu at gmail.com> > >> Is there a way Basex server can send us JSON output instead of xml? >> >>

> > Is there a way Basex server can send us JSON output instead of xml? > > Good point; it has been added to our issue list (feel free to extend it): > > https://github.com/BaseXdb/basex/issues/14 > > it's currently discussed by the W3C if the official standard will > include support for JSON. > > Christian > > > > On Wed, May 11, 2011 at 10:28 AM, Erol Akarsu <eakarsu at gmail.com> wrote: > >> > >> I am filtering one big xml db that 800Mb stored into basex db already. > >> > >> When I write filtered data into a file, I am getting out of memory. > >> > >> I know basex is generating result filtered document in memory before > >> writing and facing memory error. > >> > >> Can basex write results block by block? > >> > >> Thanks > >> > >> > >> On Wed, May 4, 2011 at 9:38 AM, Erol Akarsu <eakarsu at gmail.com> wrote: > >>> > >>> I tried to delete some nodes with "delete nodes .." xquery update > >>> command. It actually did what was requested. > >>> > >>> I checked the database size that is still same as previous one. I know > >>> the nodes deleted has a lot of data and I think database indexes would > be > >>> adjusted accordingly. > >>> > >>> Then, I exported database as xml file and recreated db. I can see it > has > >>> real size. > >>> > >>> My question is why DB is not adjusting indexes when nodes deleted? > >>> > >>> Thanks > >>> > >>> Erol Akarsu > >>> > >>> > >>> On Fri, Apr 29, 2011 at 9:23 AM, Erol Akarsu <eakarsu at gmail.com> > wrote: > >>>> > >>>> Hi, > >>>> > >>>> Have we thought on clustering basex servers so we can partition xml > >>>> documents? > >>>> > >>>> Here, I am only interested in a global partitioning: > >>>> > >>>> Let's say if we have xml document like this, I would like to partition > >>>> RECORDS content so that each host will have equal number of RECORD > elements. > >>>> Then, we need to aggregate results of individual hosts. > >>>> Can we implement this simple clustering framework with Basex? > >>>> > >>>> > >>>> <RECORDS> > >>>> <RECORD> > >>>> ..... > >>>> </RECORD> > >>>> <RECORD> > >>>> ..... > >>>> </RECORD> > >>>> </RECORDS> > >>>> > >>>> On Mon, Apr 11, 2011 at 12:35 PM, Erol Akarsu <eakarsu at gmail.com> > wrote: > >>>>> > >>>>> Ok, > >>>>> > >>>>> I was able to run full text search with another XML Database. > >>>>> > >>>>> I am primarily interested in how Basex will play with big XML file > >>>>> Wikipedia. > >>>>> > >>>>> Actually, Database create of wikipedia is fine. But when I add full > tet > >>>>> search and indexes for it, it always throws out of memory exception > error. > >>>>> > >>>>> I have changed -Xmx with 6GB that is still not enough to generate > >>>>> indexes for Wikipedia. > >>>>> > >>>>> Can you help me on how to generate indexes with a machine that case > >>>>> 6GB for Basex process? > >>>>> > >>>>> Thanks > >>>>> > >>>>> Erol Akarsu > >>>>> > >>>>> > >>>>> > >>>>> On Sun, Apr 10, 2011 at 4:07 AM, Andreas Weiler > >>>>> <andreas.weiler at uni-konstanz.de> wrote: > >>>>>> > >>>>>> Hi, > >>>>>> the following query could work for you: > >>>>>> declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; > >>>>>> for $i in doc("enwiki-latest-pages-articles")//w:sitename > >>>>>> return $i[. contains text "Wikipedia"]/.. > >>>>>> -- Andreas > >>>>>> Am 09.04.2011 um 20:43 schrieb Erol Akarsu: > >>>>>> > >>>>>> Hi > >>>>>> > >>>>>> I am having difficulty in running full text operators. This script > >>>>>> gives siteinfo below > >>>>>> declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; > >>>>>> let $d := fn:doc ("enwiki-latest-pages-articles") > >>>>>> return ($d//w:siteinfo)[1] > >>>>>> > >>>>>> But return $d//w:siteinfo[w:sitename contains text 'Wikipedia'] > does > >>>>>> NOT give same node > >>>>>> Why "contains" ft operator behave incorrectly? I remember it was > >>>>>> working fine. I just dropped and recreated database and turn all > indexes. > >>>>>> Can you help me? > >>>>>> Query info is here: > >>>>>> > >>>>>> Query: declare namespace w=" > http://www.mediawiki.org/xml/export-0.5/"; > >>>>>> Compiling: > >>>>>> - pre-evaluating fn:doc("enwiki-latest-pages-articles") > >>>>>> - adding text() step > >>>>>> - optimizing descendant-or-self step(s) > >>>>>> - removing path with no index results > >>>>>> - pre-evaluating (())[1] > >>>>>> - binding static variable $res > >>>>>> - pre-evaluating fn:doc("enwiki-latest-pages-articles") > >>>>>> - binding static variable $d > >>>>>> - adding text() step > >>>>>> - optimizing descendant-or-self step(s) > >>>>>> - removing path with no index results > >>>>>> - simplifying flwor expression > >>>>>> Result: () > >>>>>> Timing: > >>>>>> - Parsing: 0.46 ms > >>>>>> - Compiling: 0.42 ms > >>>>>> - Evaluating: 0.17 ms > >>>>>> - Printing: 0.1 ms > >>>>>> - Total Time: 1.15 ms > >>>>>> Query plan: > >>>>>> <sequence size="0"/> > >>>>>> > >>>>>> > >>>>>> > >>>>>> <siteinfo xmlns="http://www.mediawiki.org/xml/export-0.5/" > >>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> > >>>>>> <sitename>Wikipedia</sitename> > >>>>>> <base>http://en.wikipedia.org/wiki/Main_Page</base> > >>>>>> <generator>MediaWiki 1.17wmf1</generator> > >>>>>> <case>first-letter</case> > >>>>>> <namespaces> > >>>>>> <namespace key="-2" case="first-letter">Media</namespace> > >>>>>> <namespace key="-1" case="first-letter">Special</namespace> > >>>>>> <namespace key="0" case="first-letter"/> > >>>>>> <namespace key="1" case="first-letter">Talk</namespace> > >>>>>> <namespace key="2" case="first-letter">User</namespace> > >>>>>> <namespace key="3" case="first-letter">User talk</namespace> > >>>>>> <namespace key="4" case="first-letter">Wikipedia</namespace> > >>>>>> <namespace key="5" case="first-letter">Wikipedia > talk</namespace> > >>>>>> <namespace key="6" case="first-letter">File</namespace> > >>>>>> <namespace key="7" case="first-letter">File talk</namespace> > >>>>>> <namespace key="8" case="first-letter">MediaWiki</namespace> > >>>>>> <namespace key="9" case="first-letter">MediaWiki > talk</namespace> > >>>>>> <namespace key="10" case="first-letter">Template</namespace> > >>>>>> <namespace key="11" case="first-letter">Template > talk</namespace> > >>>>>> <namespace key="12" case="first-letter">Help</namespace> > >>>>>> <namespace key="13" case="first-letter">Help talk</namespace> > >>>>>> <namespace key="14" case="first-letter">Category</namespace> > >>>>>> <namespace key="15" case="first-letter">Category > talk</namespace> > >>>>>> <namespace key="100" case="first-letter">Portal</namespace> > >>>>>> <namespace key="101" case="first-letter">Portal talk</namespace> > >>>>>> <namespace key="108" case="first-letter">Book</namespace> > >>>>>> <namespace key="109" case="first-letter">Book talk</namespace> > >>>>>> </namespaces> > >>>>>> </siteinfo> > >>>>>> > >>>>>> On Mon, Apr 4, 2011 at 7:31 AM, Erol Akarsu <eakarsu at gmail.com> > wrote: > >>>>>>> > >>>>>>> I imported wikipedia xml into basex and tried to search it. > >>>>>>> > >>>>>>> But searching it takes longer. > >>>>>>> > >>>>>>> I tried to search one element that is first child of whole document > >>>>>>> and it took 52 sec. > >>>>>>> I know the XML file is very big 31GB. How can I optimize the > search? > >>>>>>> > >>>>>>> declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; > >>>>>>> > >>>>>>> let $d := fn:doc ("enwiki-latest-pages-articles")//w:siteinfo > >>>>>>> return $d > >>>>>>> > >>>>>>> Database info: > >>>>>>> > >>>>>>> > open enwiki-latest-pages-articles > >>>>>>> Database 'enwiki-latest-pages-articles' opened in 778.49 ms. > >>>>>>> > info database > >>>>>>> Database Properties > >>>>>>> Name: enwiki-latest-pages-articles > >>>>>>> Size: 23356 MB > >>>>>>> Nodes: 228090153 > >>>>>>> Height: 6 > >>>>>>> > >>>>>>> Database Creation > >>>>>>> Path: /mnt/hgfs/C/tmp/enwiki-latest-pages-articles.xml > >>>>>>> Time Stamp: 03.04.2011 12:29:15 > >>>>>>> Input Size: 30025 MB > >>>>>>> Encoding: UTF-8 > >>>>>>> Documents: 1 > >>>>>>> Whitespace Chopping: ON > >>>>>>> Entity Parsing: OFF > >>>>>>> > >>>>>>> Indexes > >>>>>>> Up-to-date: true > >>>>>>> Path Summary: ON > >>>>>>> Text Index: ON > >>>>>>> Attribute Index: ON > >>>>>>> Full-Text Index: OFF > >>>>>>> > > >>>>>>> > >>>>>>> > >>>>>>> Timing info: > >>>>>>> > >>>>>>> Query: declare namespace > >>>>>>> w="http://www.mediawiki.org/xml/export-0.5/"; > >>>>>>> Compiling: > >>>>>>> - pre-evaluating fn:doc("enwiki-latest-pages-articles") > >>>>>>> - optimizing descendant-or-self step(s) > >>>>>>> - binding static variable $d > >>>>>>> - removing variable $d > >>>>>>> - simplifying flwor expression > >>>>>>> Result: element siteinfo { ... } > >>>>>>> Timing: > >>>>>>> - Parsing: 1.4 ms > >>>>>>> - Compiling: 52599.0 ms > >>>>>>> - Evaluating: 0.28 ms > >>>>>>> - Printing: 0.62 ms > >>>>>>> - Total Time: 52601.32 ms > >>>>>>> Query plan: > >>>>>>> <DBNode name="enwiki-latest-pages-articles" pre="5"/> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Result of query: > >>>>>>> > >>>>>>> <siteinfo xmlns="http://www.mediawiki.org/xml/export-0.5/" > >>>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> > >>>>>>> <sitename>Wikipedia</sitename> > >>>>>>> <base>http://en.wikipedia.org/wiki/Main_Page</base> > >>>>>>> <generator>MediaWiki 1.17wmf1</generator> > >>>>>>> <case>first-letter</case> > >>>>>>> <namespaces> > >>>>>>> <namespace key="-2" case="first-letter">Media</namespace> > >>>>>>> <namespace key="-1" case="first-letter">Special</namespace> > >>>>>>> <namespace key="0" case="first-letter"/> > >>>>>>> <namespace key="1" case="first-letter">Talk</namespace> > >>>>>>> <namespace key="2" case="first-letter">User</namespace> > >>>>>>> <namespace key="3" case="first-letter">User talk</namespace> > >>>>>>> <namespace key="4" case="first-letter">Wikipedia</namespace> > >>>>>>> <namespace key="5" case="first-letter">Wikipedia > talk</namespace> > >>>>>>> <namespace key="6" case="first-letter">File</namespace> > >>>>>>> <namespace key="7" case="first-letter">File talk</namespace> > >>>>>>> <namespace key="8" case="first-letter">MediaWiki</namespace> > >>>>>>> <namespace key="9" case="first-letter">MediaWiki > talk</namespace> > >>>>>>> <namespace key="10" case="first-letter">Template</namespace> > >>>>>>> <namespace key="11" case="first-letter">Template > talk</namespace> > >>>>>>> <namespace key="12" case="first-letter">Help</namespace> > >>>>>>> <namespace key="13" case="first-letter">Help talk</namespace> > >>>>>>> <namespace key="14" case="first-letter">Category</namespace> > >>>>>>> <namespace key="15" case="first-letter">Category > talk</namespace> > >>>>>>> <namespace key="100" case="first-letter">Portal</namespace> > >>>>>>> <namespace key="101" case="first-letter">Portal > talk</namespace> > >>>>>>> <namespace key="108" case="first-letter">Book</namespace> > >>>>>>> <namespace key="109" case="first-letter">Book talk</namespace> > >>>>>>> </namespaces> > >>>>>>> </siteinfo> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thanks > >>>>>>> > >>>>>>> Erol Akarsu > >>>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> BaseX-Talk mailing list > >>>>>> BaseX-Talk at mailman.uni-konstanz.de > >>>>>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >>>>>> > >>>>> > >>>> > >>> > >> > > > > > > _______________________________________________ > > BaseX-Talk mailing list > > BaseX-Talk at mailman.uni-konstanz.de > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > >

> Do we have a timeline when JSON output can be added to Basex server?

> On Fri, May 13, 2011 at 9:32 AM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> > Is there a way Basex server can send us JSON output instead of xml? >> >> Good point; it has been added to our issue list (feel free to extend it): >> >> https://github.com/BaseXdb/basex/issues/14 >> >> it's currently discussed by the W3C if the official standard will >> include support for JSON. >> >> Christian >> >> >> > On Wed, May 11, 2011 at 10:28 AM, Erol Akarsu <eakarsu at gmail.com> wrote: >> >> >> >> I am filtering one big xml db that 800Mb stored into basex db already. >> >> >> >> When I write filtered data into a file, I am getting out of memory. >> >> >> >> I know basex is generating result filtered document in memory before >> >> writing and facing memory error. >> >> >> >> Can basex write results block by block? >> >> >> >> Thanks >> >> >> >> >> >> On Wed, May 4, 2011 at 9:38 AM, Erol Akarsu <eakarsu at gmail.com> wrote: >> >>> >> >>> I tried to delete some nodes with "delete nodes .." xquery update >> >>> command. It actually did what was requested. >> >>> >> >>> I checked the database size that is still same as previous one. I know >> >>> the nodes deleted has a lot of data and I think database indexes would >> >>> be >> >>> adjusted accordingly. >> >>> >> >>> Then, I exported database as xml file and recreated db. I can see it >> >>> has >> >>> real size. >> >>> >> >>> My question is why DB is not adjusting indexes when nodes deleted? >> >>> >> >>> Thanks >> >>> >> >>> Erol Akarsu >> >>> >> >>> >> >>> On Fri, Apr 29, 2011 at 9:23 AM, Erol Akarsu <eakarsu at gmail.com> >> >>> wrote: >> >>>> >> >>>> Hi, >> >>>> >> >>>> Have we thought on clustering basex servers so we can partition xml >> >>>> documents? >> >>>> >> >>>> Here, I am only interested in a global partitioning: >> >>>> >> >>>> Let's say if we have xml document like this, I would like to >> >>>> partition >> >>>> RECORDS content so that each host will have equal number of RECORD >> >>>> elements. >> >>>> Then, we need to aggregate results of individual hosts. >> >>>> Can we implement this simple clustering framework with Basex? >> >>>> >> >>>> >> >>>> <RECORDS> >> >>>> <RECORD> >> >>>> ..... >> >>>> </RECORD> >> >>>> <RECORD> >> >>>> ..... >> >>>> </RECORD> >> >>>> </RECORDS> >> >>>> >> >>>> On Mon, Apr 11, 2011 at 12:35 PM, Erol Akarsu <eakarsu at gmail.com> >> >>>> wrote: >> >>>>> >> >>>>> Ok, >> >>>>> >> >>>>> I was able to run full text search with another XML Database. >> >>>>> >> >>>>> I am primarily interested in how Basex will play with big XML file >> >>>>> Wikipedia. >> >>>>> >> >>>>> Actually, Database create of wikipedia is fine. But when I add full >> >>>>> tet >> >>>>> search and indexes for it, it always throws out of memory exception >> >>>>> error. >> >>>>> >> >>>>> I have changed -Xmx with 6GB that is still not enough to generate >> >>>>> indexes for Wikipedia. >> >>>>> >> >>>>> Can you help me on how to generate indexes with a machine that case >> >>>>> 6GB for Basex process? >> >>>>> >> >>>>> Thanks >> >>>>> >> >>>>> Erol Akarsu >> >>>>> >> >>>>> >> >>>>> >> >>>>> On Sun, Apr 10, 2011 at 4:07 AM, Andreas Weiler >> >>>>> <andreas.weiler at uni-konstanz.de> wrote: >> >>>>>> >> >>>>>> Hi, >> >>>>>> the following query could work for you: >> >>>>>> declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >> >>>>>> for $i in doc("enwiki-latest-pages-articles")//w:sitename >> >>>>>> return $i[. contains text "Wikipedia"]/.. >> >>>>>> -- Andreas >> >>>>>> Am 09.04.2011 um 20:43 schrieb Erol Akarsu: >> >>>>>> >> >>>>>> Hi >> >>>>>> >> >>>>>> I am having difficulty in running full text operators. This script >> >>>>>> gives siteinfo below >> >>>>>> declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >> >>>>>> let $d := fn:doc ("enwiki-latest-pages-articles") >> >>>>>> return ($d//w:siteinfo)[1] >> >>>>>> >> >>>>>> But return $d//w:siteinfo[w:sitename contains text 'Wikipedia'] >> >>>>>> does >> >>>>>> NOT give same node >> >>>>>> Why "contains" ft operator behave incorrectly? I remember it was >> >>>>>> working fine. I just dropped and recreated database and turn all >> >>>>>> indexes. >> >>>>>> Can you help me? >> >>>>>> Query info is here: >> >>>>>> >> >>>>>> Query: declare namespace >> >>>>>> w="http://www.mediawiki.org/xml/export-0.5/"; >> >>>>>> Compiling: >> >>>>>> - pre-evaluating fn:doc("enwiki-latest-pages-articles") >> >>>>>> - adding text() step >> >>>>>> - optimizing descendant-or-self step(s) >> >>>>>> - removing path with no index results >> >>>>>> - pre-evaluating (())[1] >> >>>>>> - binding static variable $res >> >>>>>> - pre-evaluating fn:doc("enwiki-latest-pages-articles") >> >>>>>> - binding static variable $d >> >>>>>> - adding text() step >> >>>>>> - optimizing descendant-or-self step(s) >> >>>>>> - removing path with no index results >> >>>>>> - simplifying flwor expression >> >>>>>> Result: () >> >>>>>> Timing: >> >>>>>> - Parsing: 0.46 ms >> >>>>>> - Compiling: 0.42 ms >> >>>>>> - Evaluating: 0.17 ms >> >>>>>> - Printing: 0.1 ms >> >>>>>> - Total Time: 1.15 ms >> >>>>>> Query plan: >> >>>>>> <sequence size="0"/> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> <siteinfo xmlns="http://www.mediawiki.org/xml/export-0.5/" >> >>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> >> >>>>>> <sitename>Wikipedia</sitename> >> >>>>>> <base>http://en.wikipedia.org/wiki/Main_Page</base> >> >>>>>> <generator>MediaWiki 1.17wmf1</generator> >> >>>>>> <case>first-letter</case> >> >>>>>> <namespaces> >> >>>>>> <namespace key="-2" case="first-letter">Media</namespace> >> >>>>>> <namespace key="-1" case="first-letter">Special</namespace> >> >>>>>> <namespace key="0" case="first-letter"/> >> >>>>>> <namespace key="1" case="first-letter">Talk</namespace> >> >>>>>> <namespace key="2" case="first-letter">User</namespace> >> >>>>>> <namespace key="3" case="first-letter">User talk</namespace> >> >>>>>> <namespace key="4" case="first-letter">Wikipedia</namespace> >> >>>>>> <namespace key="5" case="first-letter">Wikipedia >> >>>>>> talk</namespace> >> >>>>>> <namespace key="6" case="first-letter">File</namespace> >> >>>>>> <namespace key="7" case="first-letter">File talk</namespace> >> >>>>>> <namespace key="8" case="first-letter">MediaWiki</namespace> >> >>>>>> <namespace key="9" case="first-letter">MediaWiki >> >>>>>> talk</namespace> >> >>>>>> <namespace key="10" case="first-letter">Template</namespace> >> >>>>>> <namespace key="11" case="first-letter">Template >> >>>>>> talk</namespace> >> >>>>>> <namespace key="12" case="first-letter">Help</namespace> >> >>>>>> <namespace key="13" case="first-letter">Help talk</namespace> >> >>>>>> <namespace key="14" case="first-letter">Category</namespace> >> >>>>>> <namespace key="15" case="first-letter">Category >> >>>>>> talk</namespace> >> >>>>>> <namespace key="100" case="first-letter">Portal</namespace> >> >>>>>> <namespace key="101" case="first-letter">Portal >> >>>>>> talk</namespace> >> >>>>>> <namespace key="108" case="first-letter">Book</namespace> >> >>>>>> <namespace key="109" case="first-letter">Book talk</namespace> >> >>>>>> </namespaces> >> >>>>>> </siteinfo> >> >>>>>> >> >>>>>> On Mon, Apr 4, 2011 at 7:31 AM, Erol Akarsu <eakarsu at gmail.com> >> >>>>>> wrote: >> >>>>>>> >> >>>>>>> I imported wikipedia xml into basex and tried to search it. >> >>>>>>> >> >>>>>>> But searching it takes longer. >> >>>>>>> >> >>>>>>> I tried to search one element that is first child of whole >> >>>>>>> document >> >>>>>>> and it took 52 sec. >> >>>>>>> I know the XML file is very big 31GB. How can I optimize the >> >>>>>>> search? >> >>>>>>> >> >>>>>>> declare namespace w="http://www.mediawiki.org/xml/export-0.5/"; >> >>>>>>> >> >>>>>>> let $d := fn:doc ("enwiki-latest-pages-articles")//w:siteinfo >> >>>>>>> return $d >> >>>>>>> >> >>>>>>> Database info: >> >>>>>>> >> >>>>>>> > open enwiki-latest-pages-articles >> >>>>>>> Database 'enwiki-latest-pages-articles' opened in 778.49 ms. >> >>>>>>> > info database >> >>>>>>> Database Properties >> >>>>>>> Name: enwiki-latest-pages-articles >> >>>>>>> Size: 23356 MB >> >>>>>>> Nodes: 228090153 >> >>>>>>> Height: 6 >> >>>>>>> >> >>>>>>> Database Creation >> >>>>>>> Path: /mnt/hgfs/C/tmp/enwiki-latest-pages-articles.xml >> >>>>>>> Time Stamp: 03.04.2011 12:29:15 >> >>>>>>> Input Size: 30025 MB >> >>>>>>> Encoding: UTF-8 >> >>>>>>> Documents: 1 >> >>>>>>> Whitespace Chopping: ON >> >>>>>>> Entity Parsing: OFF >> >>>>>>> >> >>>>>>> Indexes >> >>>>>>> Up-to-date: true >> >>>>>>> Path Summary: ON >> >>>>>>> Text Index: ON >> >>>>>>> Attribute Index: ON >> >>>>>>> Full-Text Index: OFF >> >>>>>>> > >> >>>>>>> >> >>>>>>> >> >>>>>>> Timing info: >> >>>>>>> >> >>>>>>> Query: declare namespace >> >>>>>>> w="http://www.mediawiki.org/xml/export-0.5/"; >> >>>>>>> Compiling: >> >>>>>>> - pre-evaluating fn:doc("enwiki-latest-pages-articles") >> >>>>>>> - optimizing descendant-or-self step(s) >> >>>>>>> - binding static variable $d >> >>>>>>> - removing variable $d >> >>>>>>> - simplifying flwor expression >> >>>>>>> Result: element siteinfo { ... } >> >>>>>>> Timing: >> >>>>>>> - Parsing: 1.4 ms >> >>>>>>> - Compiling: 52599.0 ms >> >>>>>>> - Evaluating: 0.28 ms >> >>>>>>> - Printing: 0.62 ms >> >>>>>>> - Total Time: 52601.32 ms >> >>>>>>> Query plan: >> >>>>>>> <DBNode name="enwiki-latest-pages-articles" pre="5"/> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> Result of query: >> >>>>>>> >> >>>>>>> <siteinfo xmlns="http://www.mediawiki.org/xml/export-0.5/" >> >>>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> >> >>>>>>> <sitename>Wikipedia</sitename> >> >>>>>>> <base>http://en.wikipedia.org/wiki/Main_Page</base> >> >>>>>>> <generator>MediaWiki 1.17wmf1</generator> >> >>>>>>> <case>first-letter</case> >> >>>>>>> <namespaces> >> >>>>>>> <namespace key="-2" case="first-letter">Media</namespace> >> >>>>>>> <namespace key="-1" case="first-letter">Special</namespace> >> >>>>>>> <namespace key="0" case="first-letter"/> >> >>>>>>> <namespace key="1" case="first-letter">Talk</namespace> >> >>>>>>> <namespace key="2" case="first-letter">User</namespace> >> >>>>>>> <namespace key="3" case="first-letter">User talk</namespace> >> >>>>>>> <namespace key="4" case="first-letter">Wikipedia</namespace> >> >>>>>>> <namespace key="5" case="first-letter">Wikipedia >> >>>>>>> talk</namespace> >> >>>>>>> <namespace key="6" case="first-letter">File</namespace> >> >>>>>>> <namespace key="7" case="first-letter">File talk</namespace> >> >>>>>>> <namespace key="8" case="first-letter">MediaWiki</namespace> >> >>>>>>> <namespace key="9" case="first-letter">MediaWiki >> >>>>>>> talk</namespace> >> >>>>>>> <namespace key="10" case="first-letter">Template</namespace> >> >>>>>>> <namespace key="11" case="first-letter">Template >> >>>>>>> talk</namespace> >> >>>>>>> <namespace key="12" case="first-letter">Help</namespace> >> >>>>>>> <namespace key="13" case="first-letter">Help talk</namespace> >> >>>>>>> <namespace key="14" case="first-letter">Category</namespace> >> >>>>>>> <namespace key="15" case="first-letter">Category >> >>>>>>> talk</namespace> >> >>>>>>> <namespace key="100" case="first-letter">Portal</namespace> >> >>>>>>> <namespace key="101" case="first-letter">Portal >> >>>>>>> talk</namespace> >> >>>>>>> <namespace key="108" case="first-letter">Book</namespace> >> >>>>>>> <namespace key="109" case="first-letter">Book talk</namespace> >> >>>>>>> </namespaces> >> >>>>>>> </siteinfo> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> Thanks >> >>>>>>> >> >>>>>>> Erol Akarsu >> >>>>>>> >> >>>>>> >> >>>>>> _______________________________________________ >> >>>>>> BaseX-Talk mailing list >> >>>>>> BaseX-Talk at mailman.uni-konstanz.de >> >>>>>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >>>>>> >> >>>>> >> >>>> >> >>> >> >> >> > >> > >> > _______________________________________________ >> > BaseX-Talk mailing list >> > BaseX-Talk at mailman.uni-konstanz.de >> > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> > >> > > >