BaseX6.5 - how to add documents into collection over wire (probably using ADD command)

Am 10.02.2011 um 15:18 schrieb Jan Vlčinský (CAD): > Hi > I need to insert documents into BaseX collection. > As data are not always available directly as files and as sometime my server is remote to me, I need a method to add documents remotely. > > With older BaseX (I think BaseX 6.4) I was able to use add commend in the way > > add as doc1 <root/> > > This works now. > But I am unable to insert anything, what has any attribute in this manner > > add as doc1 <root att="att"/> > > This keeps complaining and prints back help on the console. > > I am using basexclient.bat and Python API. > > What am I doing wrong? > > Thanks for any advice. > > Jan > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

> Hi Jan, > > use quotes around your xml snippet: > > add as doc1 "<root att='att'/>" > > Kind regards, > Andreas > > Am 10.02.2011 um 15:18 schrieb Jan Vlčinský (CAD): > > Hi > I need to insert documents into BaseX collection. > As data are not always available directly as files and as sometime my > server is remote to me, I need a method to add documents remotely. > > With older BaseX (I think BaseX 6.4) I was able to use add commend in the > way > > add as doc1 <root/> > > This works now. > But I am unable to insert anything, what has any attribute in this manner > > add as doc1 <root att="att"/> > > This keeps complaining and prints back help on the console. > > I am using basexclient.bat and Python API. > > What am I doing wrong? > > Thanks for any advice. > > Jan > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >

> Hi Andreas > Thanks for the trick. > However, it does not make me very happy, this time. > > A little background > I started running XQuery document in production, but it consumes really a lot of CPU. > My need is to take current version of some XML from Internet, run it through XQuery and post the resulting XML out. > Currently, I do not store the source document in BaseX but pass it as text parameter into the query, taking advantage of query.bind() in my Python API > > declare variable $xmlString external := '<TICELogicalModel/>'; (: just setting some initial content:) > let $xml := util:eval($xmlString) > (: let $xml := . uncomment this line to test on locally stored XML doc :) > let $colOpenLR := collection('__PreLocSet_OpenLRBinaryv3_30610_10_3') > let $colPreLoc := collection("__PredefinedLocationSet_30610_02") > > let $creationTime := $xml//PayloadPublication/data(@publicationTime) > return > <Traffic xmlns:openlr="http://www.openlr.org/openlr" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="TTTSP.xsd"> > <creationTime>{$creationTime}</creationTime> > <provider>CE-Traffic</provider>{ > > for $rec in $xml//ElaboratedRecord > let $recId := $rec/data(@id) > ...and go on with actual transformation producing one resulting element for each $rec > > My processing can be probably strongly optimized, as there is probably no use of indices for transformed document. The transformation runs few seconds on localy BaseX stored document, but tenth of seconds in real life scenario with XML being passed as value of variable (source XML has about 8 MB in size and there similar processes running concurrently, 2nd and 3rd are a bit smaller, referenced collections have about 10 MB and 800 kB of size). > > So I decided to test programatically adding the source XML into BaseX and run the XQuery then. > > However, current Python API allows adding documents only by use of add command, and it is forcing me to play special tricks with quotes and double quotes as you advised. There are two reasons, I do not like this - having to fiddle with files, in memory, where I prefer to behave as efficiently as possible; and my previous code was adding documents with older BaseX without troubles, so I have not so strong trust in stability of this solution for future versions. > > Searching for possible solutions > > Ideal solution would be, if Python API would offer some ADD method. Or do something like bind() to pass the value in. > Using command add with local file is not always an option for me. > Another alternative is using JaxRX http://docs.basex.org/wiki/JaxRxPOSTAdd, but this adds extra layer and some complexity of basic infrastructure and I have to use API to run optimize command anyway. > > And there could be extra trick with modification of already inserted node > use API to call simple ADD command with the simplest content > ADD as mydoc <root/> > Use API, to run XQuery, taking advantage of BIND method passing name of document to modify and real content of the XML. > The XQuery would modify the pre-created document to get real full content. > Use API to run OPTIMIZE command > Finally, process my XQuery, which produces my real result > Summary of options > Use Python API, calling execute with ADD command, properly quoting inserted XML string > Use JAX RX to add the document over wire. > Rest of activities done by Python API > Use/modify/create/get modified Python API, which offers ADD method for document > Use Python API with trick, modifying simple XML doc by XQuery having bound full document into variable > Any comments are welcome. > > Jan > > 2011/2/10 Andreas Weiler <andreas.weiler at uni-konstanz.de> > Hi Jan, > > use quotes around your xml snippet: > > add as doc1 "<root att='att'/>" > > Kind regards, > Andreas > > Am 10.02.2011 um 15:18 schrieb Jan Vlčinský (CAD): > >> Hi >> I need to insert documents into BaseX collection. >> As data are not always available directly as files and as sometime my server is remote to me, I need a method to add documents remotely. >> >> With older BaseX (I think BaseX 6.4) I was able to use add commend in the way >> >> add as doc1 <root/> >> >> This works now. >> But I am unable to insert anything, what has any attribute in this manner >> >> add as doc1 <root att="att"/> >> >> This keeps complaining and prints back help on the console. >> >> I am using basexclient.bat and Python API. >> >> What am I doing wrong? >> >> Thanks for any advice. >> >> Jan >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > > -- > Ing. Jan Vlčinský > CAD programy > Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic > tel: +420-597 602 024; mob: +420-608 979 040 > skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com > http://cz.linkedin.com/in/vlcinsky > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

> Hi Jan, > > Have you considered using the REST interface? Using curl it's easy to add > documents from anywhere in the world, post queries and delete documents when > you're done. Just start bin/basexrest and you're ready to go. > > Huib. > > Verstuurd vanaf mijn iPad > > Op 11 feb. 2011 om 22:36 heeft Jan Vlčinský (CAD) < > jan.vlcinsky at cad-programs.com> het volgende geschreven: > > Hi Andreas > Thanks for the trick. > However, it does not make me very happy, this time. > > *A little background* > I started running XQuery document in production, but it consumes really a > lot of CPU. > My need is to take current version of some XML from Internet, run it > through XQuery and post the resulting XML out. > Currently, I do not store the source document in BaseX but pass it as text > parameter into the query, taking advantage of query.bind() in my Python API > > declare variable $xmlString external := '<TICELogicalModel/>'; (: just > setting some initial content:) > let $xml := util:eval($xmlString) > (: let $xml := . uncomment this line to test on locally stored XML doc > :) > let $colOpenLR := collection('__PreLocSet_OpenLRBinaryv3_30610_10_3') > let $colPreLoc := collection("__PredefinedLocationSet_30610_02") > > let $creationTime := $xml//PayloadPublication/data(@publicationTime) > return > <Traffic xmlns:openlr=" <http://www.openlr.org/openlr> > http://www.openlr.org/openlr" xmlns:xsi="<http://www.w3.org/2001/XMLSchema-instance> > http://www.w3.org/2001/XMLSchema-instance" > xsi:noNamespaceSchemaLocation="TTTSP.xsd"> > <creationTime>{$creationTime}</creationTime> > <provider>CE-Traffic</provider>{ > > for $rec in $xml//ElaboratedRecord > let $recId := $rec/data(@id) > ...and go on with actual transformation producing one resulting element for > each $rec > > My processing can be probably strongly optimized, as there is probably no > use of indices for transformed document. The transformation runs few seconds > on localy BaseX stored document, but tenth of seconds in real life scenario > with XML being passed as value of variable (source XML has about 8 MB in > size and there similar processes running concurrently, 2nd and 3rd are a bit > smaller, referenced collections have about 10 MB and 800 kB of size). > > So I decided to test programatically adding the source XML into BaseX and > run the XQuery then. > > However, current Python API allows adding documents only by use of add > command, and it is forcing me to play special tricks with quotes and double > quotes as you advised. There are two reasons, I do not like this - having to > fiddle with files, in memory, where I prefer to behave as efficiently as > possible; and my previous code was adding documents with older BaseX without > troubles, so I have not so strong trust in stability of this solution for > future versions. > > *Searching for possible solutions* > > Ideal solution would be, *if Python API would offer some ADD method*. Or > do something like bind() to pass the value in. > Using command add with local file is not always an option for me. > Another alternative is *using JaxRX* <http://docs.basex.org/wiki/JaxRxPOSTAdd> > http://docs.basex.org/wiki/JaxRxPOSTAdd, but this adds extra layer and > some complexity of basic infrastructure and I have to use API to run > optimize command anyway. > > And there could be extra *trick with modification of already inserted node > * > > 1. use API to call simple ADD command with the simplest content > ADD as mydoc <root/> > 2. Use API, to run XQuery, taking advantage of BIND method passing name > of document to modify and real content of the XML. > The XQuery would modify the pre-created document to get real full > content. > 3. Use API to run OPTIMIZE command > 4. Finally, process my XQuery, which produces my real result > > *Summary of options* > > 1. Use Python API, calling execute with ADD command, properly quoting > inserted XML string > 2. Use JAX RX to add the document over wire. > Rest of activities done by Python API > 3. Use/modify/create/get modified Python API, which offers ADD method > for document > 4. Use Python API with trick, modifying simple XML doc by XQuery having > bound full document into variable > > Any comments are welcome. > > Jan > > 2011/2/10 Andreas Weiler < <andreas.weiler at uni-konstanz.de> > andreas.weiler at uni-konstanz.de> > >> Hi Jan, >> >> use quotes around your xml snippet: >> >> add as doc1 "<root att='att'/>" >> >> Kind regards, >> Andreas >> >> Am 10.02.2011 um 15:18 schrieb Jan Vlčinský (CAD): >> >> Hi >> I need to insert documents into BaseX collection. >> As data are not always available directly as files and as sometime my >> server is remote to me, I need a method to add documents remotely. >> >> With older BaseX (I think BaseX 6.4) I was able to use add commend in the >> way >> >> add as doc1 <root/> >> >> This works now. >> But I am unable to insert anything, what has any attribute in this manner >> >> add as doc1 <root att="att"/> >> >> This keeps complaining and prints back help on the console. >> >> I am using basexclient.bat and Python API. >> >> What am I doing wrong? >> >> Thanks for any advice. >> >> Jan >> >> _______________________________________________ >> BaseX-Talk mailing list >> <BaseX-Talk at mailman.uni-konstanz.de>BaseX-Talk@mailman.uni-konstanz.de >> <https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk> >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >> >> >> _______________________________________________ >> BaseX-Talk mailing list >> <BaseX-Talk at mailman.uni-konstanz.de>BaseX-Talk@mailman.uni-konstanz.de >> <https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk> >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >> > > > -- > *Ing. Jan Vlčinský* > CAD programy > Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic > tel: +420-597 602 024; mob: +420-608 979 040 > skype: janvlcinsky; GoogleTalk: <jan.vlcinsky at gmail.com> > jan.vlcinsky at gmail.com > <http://cz.linkedin.com/in/vlcinsky>http://cz.linkedin.com/in/vlcinsky > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >

> Hi Huib > Yes, I considered, my note about using JAX RX was in fact about using > REST. > I wonder - is there any way, how to optimize collection by REST after new > document is submited? > I assume, that for proper performance of XQuery is optimize needed, as > adding new document probably breaks previous indices. > > And with REST, as I understand, there is no state. Taking into > consideration, that to optimize a collection, it has to be first opened, > then in next command it can be optimized, there is a problem, that the > optimize command has no context of opened collection, as there is no state > preserved. Or do I understand something wrong? > > I know, I can use PythonAPI for opening the database and running optimize, > but I would prefer to keep my solution as simple as possible and using 2 API > (REST + Python API) is more complex then one (just REST). > > Jan > > > 2011/2/11 Huib Verweij <hhv at home.nl> > > Hi Jan, >> >> Have you considered using the REST interface? Using curl it's easy to add >> documents from anywhere in the world, post queries and delete documents when >> you're done. Just start bin/basexrest and you're ready to go. >> >> Huib. >> >> Verstuurd vanaf mijn iPad >> >> Op 11 feb. 2011 om 22:36 heeft Jan Vlčinský (CAD) < >> jan.vlcinsky at cad-programs.com> het volgende geschreven: >> >> Hi Andreas >> Thanks for the trick. >> However, it does not make me very happy, this time. >> >> *A little background* >> I started running XQuery document in production, but it consumes really a >> lot of CPU. >> My need is to take current version of some XML from Internet, run it >> through XQuery and post the resulting XML out. >> Currently, I do not store the source document in BaseX but pass it as text >> parameter into the query, taking advantage of query.bind() in my Python API >> >> declare variable $xmlString external := '<TICELogicalModel/>'; (: just >> setting some initial content:) >> let $xml := util:eval($xmlString) >> (: let $xml := . uncomment this line to test on locally stored XML doc >> :) >> let $colOpenLR := collection('__PreLocSet_OpenLRBinaryv3_30610_10_3') >> let $colPreLoc := collection("__PredefinedLocationSet_30610_02") >> >> let $creationTime := $xml//PayloadPublication/data(@publicationTime) >> return >> <Traffic xmlns:openlr=" <http://www.openlr.org/openlr> >> http://www.openlr.org/openlr" xmlns:xsi="<http://www.w3.org/2001/XMLSchema-instance> >> http://www.w3.org/2001/XMLSchema-instance" >> xsi:noNamespaceSchemaLocation="TTTSP.xsd"> >> <creationTime>{$creationTime}</creationTime> >> <provider>CE-Traffic</provider>{ >> >> for $rec in $xml//ElaboratedRecord >> let $recId := $rec/data(@id) >> ...and go on with actual transformation producing one resulting element >> for each $rec >> >> My processing can be probably strongly optimized, as there is probably no >> use of indices for transformed document. The transformation runs few seconds >> on localy BaseX stored document, but tenth of seconds in real life scenario >> with XML being passed as value of variable (source XML has about 8 MB in >> size and there similar processes running concurrently, 2nd and 3rd are a bit >> smaller, referenced collections have about 10 MB and 800 kB of size). >> >> So I decided to test programatically adding the source XML into BaseX and >> run the XQuery then. >> >> However, current Python API allows adding documents only by use of add >> command, and it is forcing me to play special tricks with quotes and double >> quotes as you advised. There are two reasons, I do not like this - having to >> fiddle with files, in memory, where I prefer to behave as efficiently as >> possible; and my previous code was adding documents with older BaseX without >> troubles, so I have not so strong trust in stability of this solution for >> future versions. >> >> *Searching for possible solutions* >> >> Ideal solution would be, *if Python API would offer some ADD method*. Or >> do something like bind() to pass the value in. >> Using command add with local file is not always an option for me. >> Another alternative is *using JaxRX* <http://docs.basex.org/wiki/JaxRxPOSTAdd> >> http://docs.basex.org/wiki/JaxRxPOSTAdd, but this adds extra layer and >> some complexity of basic infrastructure and I have to use API to run >> optimize command anyway. >> >> And there could be extra *trick with modification of already inserted >> node* >> >> 1. use API to call simple ADD command with the simplest content >> ADD as mydoc <root/> >> 2. Use API, to run XQuery, taking advantage of BIND method passing >> name of document to modify and real content of the XML. >> The XQuery would modify the pre-created document to get real full >> content. >> 3. Use API to run OPTIMIZE command >> 4. Finally, process my XQuery, which produces my real result >> >> *Summary of options* >> >> 1. Use Python API, calling execute with ADD command, properly quoting >> inserted XML string >> 2. Use JAX RX to add the document over wire. >> Rest of activities done by Python API >> 3. Use/modify/create/get modified Python API, which offers ADD method >> for document >> 4. Use Python API with trick, modifying simple XML doc by XQuery >> having bound full document into variable >> >> Any comments are welcome. >> >> Jan >> >> 2011/2/10 Andreas Weiler < <andreas.weiler at uni-konstanz.de> >> andreas.weiler at uni-konstanz.de> >> >>> Hi Jan, >>> >>> use quotes around your xml snippet: >>> >>> add as doc1 "<root att='att'/>" >>> >>> Kind regards, >>> Andreas >>> >>> Am 10.02.2011 um 15:18 schrieb Jan Vlčinský (CAD): >>> >>> Hi >>> I need to insert documents into BaseX collection. >>> As data are not always available directly as files and as sometime my >>> server is remote to me, I need a method to add documents remotely. >>> >>> With older BaseX (I think BaseX 6.4) I was able to use add commend in the >>> way >>> >>> add as doc1 <root/> >>> >>> This works now. >>> But I am unable to insert anything, what has any attribute in this manner >>> >>> add as doc1 <root att="att"/> >>> >>> This keeps complaining and prints back help on the console. >>> >>> I am using basexclient.bat and Python API. >>> >>> What am I doing wrong? >>> >>> Thanks for any advice. >>> >>> Jan >>> >>> _______________________________________________ >>> BaseX-Talk mailing list >>> <BaseX-Talk at mailman.uni-konstanz.de>BaseX-Talk@mailman.uni-konstanz.de >>> <https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk> >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>> >>> >>> >>> _______________________________________________ >>> BaseX-Talk mailing list >>> <BaseX-Talk at mailman.uni-konstanz.de>BaseX-Talk@mailman.uni-konstanz.de >>> <https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk> >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>> >>> >> >> >> -- >> *Ing. Jan Vlčinský* >> CAD programy >> Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic >> tel: +420-597 602 024; mob: +420-608 979 040 >> skype: janvlcinsky; GoogleTalk: <jan.vlcinsky at gmail.com> >> jan.vlcinsky at gmail.com >> <http://cz.linkedin.com/in/vlcinsky>http://cz.linkedin.com/in/vlcinsky >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >> > > > -- > *Ing. Jan Vlčinský* > CAD programy > Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic > tel: +420-597 602 024; mob: +420-608 979 040 > skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com > http://cz.linkedin.com/in/vlcinsky > >

> Is the Java API really working with current version? I did not test it.

>> My need is to take current version of some XML from Internet, run it through XQuery and post the resulting XML out.

Am 12.02.2011 um 04:25 schrieb Jan Vlčinský (CAD): > Hi > I found some very promising information here: https://svn.uni-konstanz.de/dbis/basex/trunk/api/etc/readme.txt > Also Java API shows, how is document added. > However > I am getting problems to make it working at Python - still getting response Expecting commend and not yet familiar with each byte of reading responses in the code > Is the Java API really working with current version? I did not test it. > Is the readme really up to date? As I noticed, BaseX 6.5 and 6.5.1 had some small differences with client > Jan > > PS: One Czech proverb says "Morning is wiser then late night". So resolution might be here relatively soon :-). > > 2011/2/12 Jan Vlčinský (CAD) <jan.vlcinsky at cad-programs.com> > Hi Huib > Yes, I considered, my note about using JAX RX was in fact about using REST. > I wonder - is there any way, how to optimize collection by REST after new document is submited? > I assume, that for proper performance of XQuery is optimize needed, as adding new document probably breaks previous indices. > > And with REST, as I understand, there is no state. Taking into consideration, that to optimize a collection, it has to be first opened, then in next command it can be optimized, there is a problem, that the optimize command has no context of opened collection, as there is no state preserved. Or do I understand something wrong? > > I know, I can use PythonAPI for opening the database and running optimize, but I would prefer to keep my solution as simple as possible and using 2 API (REST + Python API) is more complex then one (just REST). > > Jan > > > 2011/2/11 Huib Verweij <hhv at home.nl> > > Hi Jan, > > Have you considered using the REST interface? Using curl it's easy to add documents from anywhere in the world, post queries and delete documents when you're done. Just start bin/basexrest and you're ready to go. > > Huib. > > Verstuurd vanaf mijn iPad > > Op 11 feb. 2011 om 22:36 heeft Jan Vlčinský (CAD) <jan.vlcinsky at cad-programs.com> het volgende geschreven: > >> Hi Andreas >> Thanks for the trick. >> However, it does not make me very happy, this time. >> >> A little background >> I started running XQuery document in production, but it consumes really a lot of CPU. >> My need is to take current version of some XML from Internet, run it through XQuery and post the resulting XML out. >> Currently, I do not store the source document in BaseX but pass it as text parameter into the query, taking advantage of query.bind() in my Python API >> >> declare variable $xmlString external := '<TICELogicalModel/>'; (: just setting some initial content:) >> let $xml := util:eval($xmlString) >> (: let $xml := . uncomment this line to test on locally stored XML doc :) >> let $colOpenLR := collection('__PreLocSet_OpenLRBinaryv3_30610_10_3') >> let $colPreLoc := collection("__PredefinedLocationSet_30610_02") >> >> let $creationTime := $xml//PayloadPublication/data(@publicationTime) >> return >> <Traffic xmlns:openlr="http://www.openlr.org/openlr" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="TTTSP.xsd"> >> <creationTime>{$creationTime}</creationTime> >> <provider>CE-Traffic</provider>{ >> >> for $rec in $xml//ElaboratedRecord >> let $recId := $rec/data(@id) >> ...and go on with actual transformation producing one resulting element for each $rec >> >> My processing can be probably strongly optimized, as there is probably no use of indices for transformed document. The transformation runs few seconds on localy BaseX stored document, but tenth of seconds in real life scenario with XML being passed as value of variable (source XML has about 8 MB in size and there similar processes running concurrently, 2nd and 3rd are a bit smaller, referenced collections have about 10 MB and 800 kB of size). >> >> So I decided to test programatically adding the source XML into BaseX and run the XQuery then. >> >> However, current Python API allows adding documents only by use of add command, and it is forcing me to play special tricks with quotes and double quotes as you advised. There are two reasons, I do not like this - having to fiddle with files, in memory, where I prefer to behave as efficiently as possible; and my previous code was adding documents with older BaseX without troubles, so I have not so strong trust in stability of this solution for future versions. >> >> Searching for possible solutions >> >> Ideal solution would be, if Python API would offer some ADD method. Or do something like bind() to pass the value in. >> Using command add with local file is not always an option for me. >> Another alternative is using JaxRX http://docs.basex.org/wiki/JaxRxPOSTAdd, but this adds extra layer and some complexity of basic infrastructure and I have to use API to run optimize command anyway. >> >> And there could be extra trick with modification of already inserted node >> use API to call simple ADD command with the simplest content >> ADD as mydoc <root/> >> Use API, to run XQuery, taking advantage of BIND method passing name of document to modify and real content of the XML. >> The XQuery would modify the pre-created document to get real full content. >> Use API to run OPTIMIZE command >> Finally, process my XQuery, which produces my real result >> Summary of options >> Use Python API, calling execute with ADD command, properly quoting inserted XML string >> Use JAX RX to add the document over wire. >> Rest of activities done by Python API >> Use/modify/create/get modified Python API, which offers ADD method for document >> Use Python API with trick, modifying simple XML doc by XQuery having bound full document into variable >> Any comments are welcome. >> >> Jan >> >> 2011/2/10 Andreas Weiler <andreas.weiler at uni-konstanz.de> >> Hi Jan, >> >> use quotes around your xml snippet: >> >> add as doc1 "<root att='att'/>" >> >> Kind regards, >> Andreas >> >> Am 10.02.2011 um 15:18 schrieb Jan Vlčinský (CAD): >> >>> Hi >>> I need to insert documents into BaseX collection. >>> As data are not always available directly as files and as sometime my server is remote to me, I need a method to add documents remotely. >>> >>> With older BaseX (I think BaseX 6.4) I was able to use add commend in the way >>> >>> add as doc1 <root/> >>> >>> This works now. >>> But I am unable to insert anything, what has any attribute in this manner >>> >>> add as doc1 <root att="att"/> >>> >>> This keeps complaining and prints back help on the console. >>> >>> I am using basexclient.bat and Python API. >>> >>> What am I doing wrong? >>> >>> Thanks for any advice. >>> >>> Jan >>> >>> _______________________________________________ >>> BaseX-Talk mailing list >>> BaseX-Talk at mailman.uni-konstanz.de >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >> >> >> >> -- >> Ing. Jan Vlčinský >> CAD programy >> Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic >> tel: +420-597 602 024; mob: +420-608 979 040 >> skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com >> http://cz.linkedin.com/in/vlcinsky >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > > -- > Ing. Jan Vlčinský > CAD programy > Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic > tel: +420-597 602 024; mob: +420-608 979 040 > skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com > http://cz.linkedin.com/in/vlcinsky > > > > > -- > Ing. Jan Vlčinský > CAD programy > Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic > tel: +420-597 602 024; mob: +420-608 979 040 > skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com > http://cz.linkedin.com/in/vlcinsky >

Am 12.02.2011 um 10:33 schrieb Andreas Weiler: > Hi Jan, > >> Is the Java API really working with current version? I did not test it. > yes it is working, i just tested the AddExample. > The readme is up to date, i will implement the add function in python and let you know. > > >>> My need is to take current version of some XML from Internet, run it through XQuery and post the resulting XML out. > > > You could use XQuery itself for that, like: doc('http://files.basex.org/xml/xmark.xml') > > Kind regards, > Andreas > > Am 12.02.2011 um 04:25 schrieb Jan Vlčinský (CAD): > >> Hi >> I found some very promising information here: https://svn.uni-konstanz.de/dbis/basex/trunk/api/etc/readme.txt >> Also Java API shows, how is document added. >> However >> I am getting problems to make it working at Python - still getting response Expecting commend and not yet familiar with each byte of reading responses in the code >> Is the Java API really working with current version? I did not test it. >> Is the readme really up to date? As I noticed, BaseX 6.5 and 6.5.1 had some small differences with client >> Jan >> >> PS: One Czech proverb says "Morning is wiser then late night". So resolution might be here relatively soon :-). >> >> 2011/2/12 Jan Vlčinský (CAD) <jan.vlcinsky at cad-programs.com> >> Hi Huib >> Yes, I considered, my note about using JAX RX was in fact about using REST. >> I wonder - is there any way, how to optimize collection by REST after new document is submited? >> I assume, that for proper performance of XQuery is optimize needed, as adding new document probably breaks previous indices. >> >> And with REST, as I understand, there is no state. Taking into consideration, that to optimize a collection, it has to be first opened, then in next command it can be optimized, there is a problem, that the optimize command has no context of opened collection, as there is no state preserved. Or do I understand something wrong? >> >> I know, I can use PythonAPI for opening the database and running optimize, but I would prefer to keep my solution as simple as possible and using 2 API (REST + Python API) is more complex then one (just REST). >> >> Jan >> >> >> 2011/2/11 Huib Verweij <hhv at home.nl> >> >> Hi Jan, >> >> Have you considered using the REST interface? Using curl it's easy to add documents from anywhere in the world, post queries and delete documents when you're done. Just start bin/basexrest and you're ready to go. >> >> Huib. >> >> Verstuurd vanaf mijn iPad >> >> Op 11 feb. 2011 om 22:36 heeft Jan Vlčinský (CAD) <jan.vlcinsky at cad-programs.com> het volgende geschreven: >> >>> Hi Andreas >>> Thanks for the trick. >>> However, it does not make me very happy, this time. >>> >>> A little background >>> I started running XQuery document in production, but it consumes really a lot of CPU. >>> My need is to take current version of some XML from Internet, run it through XQuery and post the resulting XML out. >>> Currently, I do not store the source document in BaseX but pass it as text parameter into the query, taking advantage of query.bind() in my Python API >>> >>> declare variable $xmlString external := '<TICELogicalModel/>'; (: just setting some initial content:) >>> let $xml := util:eval($xmlString) >>> (: let $xml := . uncomment this line to test on locally stored XML doc :) >>> let $colOpenLR := collection('__PreLocSet_OpenLRBinaryv3_30610_10_3') >>> let $colPreLoc := collection("__PredefinedLocationSet_30610_02") >>> >>> let $creationTime := $xml//PayloadPublication/data(@publicationTime) >>> return >>> <Traffic xmlns:openlr="http://www.openlr.org/openlr" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="TTTSP.xsd"> >>> <creationTime>{$creationTime}</creationTime> >>> <provider>CE-Traffic</provider>{ >>> >>> for $rec in $xml//ElaboratedRecord >>> let $recId := $rec/data(@id) >>> ...and go on with actual transformation producing one resulting element for each $rec >>> >>> My processing can be probably strongly optimized, as there is probably no use of indices for transformed document. The transformation runs few seconds on localy BaseX stored document, but tenth of seconds in real life scenario with XML being passed as value of variable (source XML has about 8 MB in size and there similar processes running concurrently, 2nd and 3rd are a bit smaller, referenced collections have about 10 MB and 800 kB of size). >>> >>> So I decided to test programatically adding the source XML into BaseX and run the XQuery then. >>> >>> However, current Python API allows adding documents only by use of add command, and it is forcing me to play special tricks with quotes and double quotes as you advised. There are two reasons, I do not like this - having to fiddle with files, in memory, where I prefer to behave as efficiently as possible; and my previous code was adding documents with older BaseX without troubles, so I have not so strong trust in stability of this solution for future versions. >>> >>> Searching for possible solutions >>> >>> Ideal solution would be, if Python API would offer some ADD method. Or do something like bind() to pass the value in. >>> Using command add with local file is not always an option for me. >>> Another alternative is using JaxRX http://docs.basex.org/wiki/JaxRxPOSTAdd, but this adds extra layer and some complexity of basic infrastructure and I have to use API to run optimize command anyway. >>> >>> And there could be extra trick with modification of already inserted node >>> use API to call simple ADD command with the simplest content >>> ADD as mydoc <root/> >>> Use API, to run XQuery, taking advantage of BIND method passing name of document to modify and real content of the XML. >>> The XQuery would modify the pre-created document to get real full content. >>> Use API to run OPTIMIZE command >>> Finally, process my XQuery, which produces my real result >>> Summary of options >>> Use Python API, calling execute with ADD command, properly quoting inserted XML string >>> Use JAX RX to add the document over wire. >>> Rest of activities done by Python API >>> Use/modify/create/get modified Python API, which offers ADD method for document >>> Use Python API with trick, modifying simple XML doc by XQuery having bound full document into variable >>> Any comments are welcome. >>> >>> Jan >>> >>> 2011/2/10 Andreas Weiler <andreas.weiler at uni-konstanz.de> >>> Hi Jan, >>> >>> use quotes around your xml snippet: >>> >>> add as doc1 "<root att='att'/>" >>> >>> Kind regards, >>> Andreas >>> >>> Am 10.02.2011 um 15:18 schrieb Jan Vlčinský (CAD): >>> >>>> Hi >>>> I need to insert documents into BaseX collection. >>>> As data are not always available directly as files and as sometime my server is remote to me, I need a method to add documents remotely. >>>> >>>> With older BaseX (I think BaseX 6.4) I was able to use add commend in the way >>>> >>>> add as doc1 <root/> >>>> >>>> This works now. >>>> But I am unable to insert anything, what has any attribute in this manner >>>> >>>> add as doc1 <root att="att"/> >>>> >>>> This keeps complaining and prints back help on the console. >>>> >>>> I am using basexclient.bat and Python API. >>>> >>>> What am I doing wrong? >>>> >>>> Thanks for any advice. >>>> >>>> Jan >>>> >>>> _______________________________________________ >>>> BaseX-Talk mailing list >>>> BaseX-Talk at mailman.uni-konstanz.de >>>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>> >>> >>> _______________________________________________ >>> BaseX-Talk mailing list >>> BaseX-Talk at mailman.uni-konstanz.de >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>> >>> >>> >>> >>> -- >>> Ing. Jan Vlčinský >>> CAD programy >>> Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic >>> tel: +420-597 602 024; mob: +420-608 979 040 >>> skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com >>> http://cz.linkedin.com/in/vlcinsky >>> >>> _______________________________________________ >>> BaseX-Talk mailing list >>> BaseX-Talk at mailman.uni-konstanz.de >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >> >> >> >> -- >> Ing. Jan Vlčinský >> CAD programy >> Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic >> tel: +420-597 602 024; mob: +420-608 979 040 >> skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com >> http://cz.linkedin.com/in/vlcinsky >> >> >> >> >> -- >> Ing. Jan Vlčinský >> CAD programy >> Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic >> tel: +420-597 602 024; mob: +420-608 979 040 >> skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com >> http://cz.linkedin.com/in/vlcinsky >> > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

> Hi Jan, > > i just added the add method to the python api. > > Kind regards, > Andreas > > Am 12.02.2011 um 10:33 schrieb Andreas Weiler: > > Hi Jan, > > > 1. Is the Java API really working with current version? I did not test > it. > > yes it is working, i just tested the AddExample. > The readme is up to date, i will implement the add function in python and > let you know. > > My need is to take current version of some XML from Internet, run it >>> through XQuery and post the resulting XML out. >>> >>> > You could use XQuery itself for that, like: doc(' > http://files.basex.org/xml/xmark.xml') > > Kind regards, > Andreas > > Am 12.02.2011 um 04:25 schrieb Jan Vlčinský (CAD): > > Hi > I found some very promising information here: > https://svn.uni-konstanz.de/dbis/basex/trunk/api/etc/readme.txt > <https://svn.uni-konstanz.de/dbis/basex/trunk/api/etc/readme.txt>Also Java > API shows, how is document added. > However > > 1. I am getting problems to make it working at Python - still getting > response Expecting commend and not yet familiar with each byte of reading > responses in the code > 2. Is the Java API really working with current version? I did not test > it. > 3. Is the readme really up to date? As I noticed, BaseX 6.5 and 6.5.1 > had some small differences with client > > Jan > > PS: One Czech proverb says "Morning is wiser then late night". So > resolution might be here relatively soon :-). > > 2011/2/12 Jan Vlčinský (CAD) <jan.vlcinsky at cad-programs.com> > >> Hi Huib >> Yes, I considered, my note about using JAX RX was in fact about using >> REST. >> I wonder - is there any way, how to optimize collection by REST after new >> document is submited? >> I assume, that for proper performance of XQuery is optimize needed, as >> adding new document probably breaks previous indices. >> >> And with REST, as I understand, there is no state. Taking into >> consideration, that to optimize a collection, it has to be first opened, >> then in next command it can be optimized, there is a problem, that the >> optimize command has no context of opened collection, as there is no state >> preserved. Or do I understand something wrong? >> >> I know, I can use PythonAPI for opening the database and running optimize, >> but I would prefer to keep my solution as simple as possible and using 2 API >> (REST + Python API) is more complex then one (just REST). >> >> Jan >> >> >> 2011/2/11 Huib Verweij <hhv at home.nl> >> >> Hi Jan, >>> >>> Have you considered using the REST interface? Using curl it's easy to add >>> documents from anywhere in the world, post queries and delete documents when >>> you're done. Just start bin/basexrest and you're ready to go. >>> >>> Huib. >>> >>> Verstuurd vanaf mijn iPad >>> >>> Op 11 feb. 2011 om 22:36 heeft Jan Vlčinský (CAD) < >>> jan.vlcinsky at cad-programs.com> het volgende geschreven: >>> >>> Hi Andreas >>> Thanks for the trick. >>> However, it does not make me very happy, this time. >>> >>> *A little background* >>> I started running XQuery document in production, but it consumes really a >>> lot of CPU. >>> My need is to take current version of some XML from Internet, run it >>> through XQuery and post the resulting XML out. >>> Currently, I do not store the source document in BaseX but pass it as >>> text parameter into the query, taking advantage of query.bind() in my Python >>> API >>> >>> declare variable $xmlString external := '<TICELogicalModel/>'; (: just >>> setting some initial content:) >>> let $xml := util:eval($xmlString) >>> (: let $xml := . uncomment this line to test on locally stored XML doc >>> :) >>> let $colOpenLR := collection('__PreLocSet_OpenLRBinaryv3_30610_10_3') >>> let $colPreLoc := collection("__PredefinedLocationSet_30610_02") >>> >>> let $creationTime := $xml//PayloadPublication/data(@publicationTime) >>> return >>> <Traffic xmlns:openlr=" <http://www.openlr.org/openlr> >>> http://www.openlr.org/openlr" xmlns:xsi="<http://www.w3.org/2001/XMLSchema-instance> >>> http://www.w3.org/2001/XMLSchema-instance" >>> xsi:noNamespaceSchemaLocation="TTTSP.xsd"> >>> <creationTime>{$creationTime}</creationTime> >>> <provider>CE-Traffic</provider>{ >>> >>> for $rec in $xml//ElaboratedRecord >>> let $recId := $rec/data(@id) >>> ...and go on with actual transformation producing one resulting element >>> for each $rec >>> >>> My processing can be probably strongly optimized, as there is probably no >>> use of indices for transformed document. The transformation runs few seconds >>> on localy BaseX stored document, but tenth of seconds in real life scenario >>> with XML being passed as value of variable (source XML has about 8 MB in >>> size and there similar processes running concurrently, 2nd and 3rd are a bit >>> smaller, referenced collections have about 10 MB and 800 kB of size). >>> >>> So I decided to test programatically adding the source XML into BaseX and >>> run the XQuery then. >>> >>> However, current Python API allows adding documents only by use of add >>> command, and it is forcing me to play special tricks with quotes and double >>> quotes as you advised. There are two reasons, I do not like this - having to >>> fiddle with files, in memory, where I prefer to behave as efficiently as >>> possible; and my previous code was adding documents with older BaseX without >>> troubles, so I have not so strong trust in stability of this solution for >>> future versions. >>> >>> *Searching for possible solutions* >>> >>> Ideal solution would be, *if Python API would offer some ADD method*. Or >>> do something like bind() to pass the value in. >>> Using command add with local file is not always an option for me. >>> Another alternative is *using JaxRX* <http://docs.basex.org/wiki/JaxRxPOSTAdd> >>> http://docs.basex.org/wiki/JaxRxPOSTAdd, but this adds extra layer and >>> some complexity of basic infrastructure and I have to use API to run >>> optimize command anyway. >>> >>> And there could be extra *trick with modification of already inserted >>> node* >>> >>> 1. use API to call simple ADD command with the simplest content >>> ADD as mydoc <root/> >>> 2. Use API, to run XQuery, taking advantage of BIND method passing >>> name of document to modify and real content of the XML. >>> The XQuery would modify the pre-created document to get real full >>> content. >>> 3. Use API to run OPTIMIZE command >>> 4. Finally, process my XQuery, which produces my real result >>> >>> *Summary of options* >>> >>> 1. Use Python API, calling execute with ADD command, properly quoting >>> inserted XML string >>> 2. Use JAX RX to add the document over wire. >>> Rest of activities done by Python API >>> 3. Use/modify/create/get modified Python API, which offers ADD method >>> for document >>> 4. Use Python API with trick, modifying simple XML doc by XQuery >>> having bound full document into variable >>> >>> Any comments are welcome. >>> >>> Jan >>> >>> 2011/2/10 Andreas Weiler < <andreas.weiler at uni-konstanz.de> >>> andreas.weiler at uni-konstanz.de> >>> >>>> Hi Jan, >>>> >>>> use quotes around your xml snippet: >>>> >>>> add as doc1 "<root att='att'/>" >>>> >>>> Kind regards, >>>> Andreas >>>> >>>> Am 10.02.2011 um 15:18 schrieb Jan Vlčinský (CAD): >>>> >>>> Hi >>>> I need to insert documents into BaseX collection. >>>> As data are not always available directly as files and as sometime my >>>> server is remote to me, I need a method to add documents remotely. >>>> >>>> With older BaseX (I think BaseX 6.4) I was able to use add commend in >>>> the way >>>> >>>> add as doc1 <root/> >>>> >>>> This works now. >>>> But I am unable to insert anything, what has any attribute in this >>>> manner >>>> >>>> add as doc1 <root att="att"/> >>>> >>>> This keeps complaining and prints back help on the console. >>>> >>>> I am using basexclient.bat and Python API. >>>> >>>> What am I doing wrong? >>>> >>>> Thanks for any advice. >>>> >>>> Jan >>>> >>>> _______________________________________________ >>>> BaseX-Talk mailing list >>>> <BaseX-Talk at mailman.uni-konstanz.de>BaseX-Talk@mailman.uni-konstanz.de >>>> <https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk> >>>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>>> >>>> >>>> >>>> _______________________________________________ >>>> BaseX-Talk mailing list >>>> <BaseX-Talk at mailman.uni-konstanz.de>BaseX-Talk@mailman.uni-konstanz.de >>>> <https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk> >>>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>>> >>>> >>> >>> >>> -- >>> *Ing. Jan Vlčinský* >>> CAD programy >>> Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic >>> tel: +420-597 602 024; mob: +420-608 979 040 >>> skype: janvlcinsky; GoogleTalk: <jan.vlcinsky at gmail.com> >>> jan.vlcinsky at gmail.com >>> <http://cz.linkedin.com/in/vlcinsky>http://cz.linkedin.com/in/vlcinsky >>> >>> _______________________________________________ >>> BaseX-Talk mailing list >>> BaseX-Talk at mailman.uni-konstanz.de >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>> >>> >> >> >> -- >> *Ing. Jan Vlčinský* >> CAD programy >> Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic >> tel: +420-597 602 024; mob: +420-608 979 040 >> skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com >> http://cz.linkedin.com/in/vlcinsky >> >> > > > -- > *Ing. Jan Vlčinský* > CAD programy > Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic > tel: +420-597 602 024; mob: +420-608 979 040 > skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com > http://cz.linkedin.com/in/vlcinsky > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > >

> I wonder - is there any way, how to optimize collection by REST after new document is submited?

> I assume, that for proper performance of XQuery is optimize needed, as adding new document probably breaks previous indices.

> > And with REST, as I understand, there is no state. Taking into consideration, that to optimize a collection, it has to be first opened, then in next command it can be optimized, there is a problem, that the optimize command has no context of opened collection, as there is no state preserved. Or do I understand something wrong?

> > I know, I can use PythonAPI for opening the database and running optimize, but I would prefer to keep my solution as simple as possible and using 2 API (REST + Python API) is more complex then one (just REST).

> Hi Jan, > > Op 12 feb 2011, om 01:33 heeft Jan Vlčinský (CAD) het volgende geschreven: > > I wonder - is there any way, how to optimize collection by REST after new > document is submited? > > > On the JAX-RX page (http://docs.basex.org/wiki/JAX-RX_API) it says you can > issue commands like this: > > <command xmlns="http://jax-rx.sourceforge.net"> > <text>show users</text> > <parameter name="wrap" value="no"/> > </command> > > So I guess to optimize you would send this: > > <command xmlns="http://jax-rx.sourceforge.net"> > <text>optimize</text> > </command> > > using this: > > curl -i -X POST -H "Content-Type: application/query+xml" -d at optimize.xml"localhost:8984/basex/jax-rx/factbook/" > > where factbook/ specifies the collection to optimize and the optimize.xml > file contains the above optimize command. > > After that you can send a query like this: > > <query xmlns="http://jax-rx.sourceforge.net"> > <text>for $c in //city return $c/name</text> > <parameter name="count" value="5"/> > <parameter name="output" value="indent=no"/> > </query> > > > > > I assume, that for proper performance of XQuery is optimize needed, as > adding new document probably breaks previous indices. > > > I find BaseX performs really well without regular optimize, but I'm not > sure if it uses the indexes or not and when exactly the indexes are no > longer used, I can't find any info in the Query Info window for example. But > never mind, it works very well and I'll just wait for the incremental > indexes. (In fact I just optimized my collection using the above method and > BaseX responded a little slower after that, even after a while using it (I > thought the caches may have emptied by the optimze - pure speculation here > ;-)). After a restart it was up to speed again.) > > > And with REST, as I understand, there is no state. Taking into > consideration, that to optimize a collection, it has to be first opened, > then in next command it can be optimized, there is a problem, that the > optimize command has no context of opened collection, as there is no state > preserved. Or do I understand something wrong? > > > The state is missing from the HTTP communication, it's not about the > context of an opened collection in BaseX. When using REST you're specifying > the context in the URL, e.g. > http://localhost:8984/basex/jax-rx/factbook?query=//city/name&count=5where factbook is the collection your querying. > > > I know, I can use PythonAPI for opening the database and running optimize, > but I would prefer to keep my solution as simple as possible and using 2 API > (REST + Python API) is more complex then one (just REST). > > > The REST interface is quite simple I find, and it also makes it easy to > query your docs from the command line using curl. > > Hartelijke groet, > > Huib > >