Frequently Updated XML files
Friday, 1 July 2011 · 7 messages · last reply 3 Jul 2011
EJ
Eric Jones1 Jul 2011 08:59 #
Hello all,
I am new to BaseX. I want to know how one handles frequently updated xml files. Do you have to monitor the file with a third party tool and upon change drop the file from BaseX and re-add it?
Regards,
Eric
JC
Jan Vlčinský (CAD)1 Jul 2011 11:05 #
Hi Eric.
Welcome to astonishing word of XQuery and BaseX.
If you mean xml files, which are stored inside of BaseX - you can modify the file (by commands, XQuery or other API), but you cannot delete it from the collection because XQuery itself does not provide such a method. However, you are free to set it to empty element, if you need.
Be sure, that the file inside of BaseX database and on your disk, from where you probably imported them, are two different things living separate life. There are of course tools to handle some syncing, but I do not do this so I cannot help in it much.
With best regards
Jan
*Ing. Jan Vlčinský*
CAD programy
Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic
tel: +420-597 602 024; mob: +420-608 979 040
skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com
http://cz.linkedin.com/in/vlcinsky
Quoted text · Eric Jones · 14 lines
On 1 July 2011 06:53, Eric Jones <ecjones2040 at gmail.com> wrote: > Hello all, > > I am new to BaseX. I want to know how one handles frequently updated > xml files. Do you have to monitor the file with a third party tool and > upon change drop the file from BaseX and re-add it? > > Regards, > Eric > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
EJ
Eric Jones2 Jul 2011 03:09 #
Hello Mike,
Thank you for the response. After a few hours of contemplation and reading I still have yet to come
up with a way to merge changes in an XML file to the BaseX database created with a previous version of that file. Now, I have come up with a bunch of work arounds but nothing clean. Do you know
of any tools that I might look into?
Many Thanks,
Eric
Quoted text · Jan Vlčinský (CAD) · 38 lines
On Fri, Jul 1, 2011 at 5:05 AM, Jan Vlčinský (CAD) <jan.vlcinsky at cad-programs.com> wrote: > Hi Eric. > Welcome to astonishing word of XQuery and BaseX. > If you mean xml files, which are stored inside of BaseX - you can modify the > file (by commands, XQuery or other API), but you cannot delete it from the > collection because XQuery itself does not provide such a method. However, > you are free to set it to empty element, if you need. > Be sure, that the file inside of BaseX database and on your disk, from where > you probably imported them, are two different things living separate life. > There are of course tools to handle some syncing, but I do not do this so I > cannot help in it much. > With best regards > Jan > Ing. Jan Vlčinský > CAD programy > Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic > tel: +420-597 602 024; mob: +420-608 979 040 > skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com > http://cz.linkedin.com/in/vlcinsky > > > On 1 July 2011 06:53, Eric Jones <ecjones2040 at gmail.com> wrote: >> >> Hello all, >> >> I am new to BaseX. I want to know how one handles frequently updated >> xml files. Do you have to monitor the file with a third party tool and >> upon change drop the file from BaseX and re-add it? >> >> Regards, >> Eric >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >
LK
Lukas Kircher2 Jul 2011 07:23 #
Hi Eric,
I can't think of any tool or a convenient workaround for your case at the moment.
How do you change the XML file you want to merge into the
database in the first place?
Everything I have on my mind evolves either around logging the changes as, i.e., XQuery update statements or trying to take the whole process to a BaseX Server scenario where everything will be up to date by nature. But I'm sure you've thought of that already ...
Regards,
Lukas
2011/7/2 Eric Jones <ecjones2040 at gmail.com>
Quoted text · Eric Jones · 60 lines
> Hello Mike, > > Thank you for the response. After a few hours of contemplation and > reading I still have yet to come > up with a way to merge changes in an XML file to the BaseX database created > with a previous version of that file. Now, I have come up with a bunch > of work arounds but nothing clean. Do you know > of any tools that I might look into? > > Many Thanks, > Eric > > On Fri, Jul 1, 2011 at 5:05 AM, Jan Vlčinský (CAD) > <jan.vlcinsky at cad-programs.com> wrote: > > Hi Eric. > > Welcome to astonishing word of XQuery and BaseX. > > If you mean xml files, which are stored inside of BaseX - you can modify > the > > file (by commands, XQuery or other API), but you cannot delete it from > the > > collection because XQuery itself does not provide such a method. However, > > you are free to set it to empty element, if you need. > > Be sure, that the file inside of BaseX database and on your disk, from > where > > you probably imported them, are two different things living separate > life. > > There are of course tools to handle some syncing, but I do not do this so > I > > cannot help in it much. > > With best regards > > Jan > > Ing. Jan Vlčinský > > CAD programy > > Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic > > tel: +420-597 602 024; mob: +420-608 979 040 > > skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com > > http://cz.linkedin.com/in/vlcinsky > > > > > > On 1 July 2011 06:53, Eric Jones <ecjones2040 at gmail.com> wrote: > >> > >> Hello all, > >> > >> I am new to BaseX. I want to know how one handles frequently updated > >> xml files. Do you have to monitor the file with a third party tool and > >> upon change drop the file from BaseX and re-add it? > >> > >> Regards, > >> Eric > >> _______________________________________________ > >> BaseX-Talk mailing list > >> BaseX-Talk at mailman.uni-konstanz.de > >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
HV
Hi Eric,
could you give an example of a use case?
For monitoring changes to files and acting upon them Google tells me I could use inotify or incron.
From your setup I suspect you'll use BaseX as a read-only store, maybe just for searching? So you won't use the XQuery Update Facility for modifying XML in the database?
Hartelijke groet,
Huib Verweij.
--
Drs. Huib Verweij
Senior software developer - The Language Archive
Max Planck Institute for Psycholinguistics
P.O. Box 310
6500 AH Nijmegen
The Netherlands
t +31-24-3521911
e huib.verwey at mpi.nl<mailto:huib.verwey at mpi.nl>
w http://www.mpi.nl/
Op 1 jul 2011, om 06:53 heeft Eric Jones het volgende geschreven:
Hello all,
I am new to BaseX. I want to know how one handles frequently updated xml files. Do you have to monitor the file with a third party tool and upon change drop the file from BaseX and re-add it?
Regards,
Eric
_______________________________________________
BaseX-Talk mailing list
BaseX-Talk at mailman.uni-konstanz.de<mailto:BaseX-Talk at mailman.uni-konstanz.de> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
EJ
Eric Jones3 Jul 2011 03:14 #
Hi All,
A load testing tool (jmeter) has the capability to write out its test results as XML. This is great as I
search and perform metrics calculations on this data using XQuery. However, the number of XML files can grow to some volume and it is hard to tell when a test is actually over which would mean its xml result file is finished being written to. The solution to this is to use Inotify
to monitor the directory where these files are stored and add them to the database if they do not exist or delete the file and re-add it to BaseX if it already has been added and is just in need of an update (do to a running test). So that is the solution I will implement -- if anyone has anything cleaner I am all ears.
I thank the BaseX mailing list for helping me with this.
Regards,
Eric
Quoted text · Huib Verwey · 37 lines
On Sat, Jul 2, 2011 at 3:29 AM, Huib Verwey <Huib.Verwey at mpi.nl> wrote: > Hi Eric, > could you give an example of a use case? > For monitoring changes to files and acting upon them Google tells me I could > use inotify or incron. > From your setup I suspect you'll use BaseX as a read-only store, maybe just > for searching? So you won't use the XQuery Update Facility for modifying XML > in the database? > Hartelijke groet, > Huib Verweij. > -- > Drs. Huib Verweij > Senior software developer - The Language Archive > Max Planck Institute for Psycholinguistics > P.O. Box 310 > 6500 AH Nijmegen > The Netherlands > t +31-24-3521911 > e huib.verwey at mpi.nl > w http://www.mpi.nl/ > > Op 1 jul 2011, om 06:53 heeft Eric Jones het volgende geschreven: > > Hello all, > > I am new to BaseX. I want to know how one handles frequently updated > xml files. Do you have to monitor the file with a third party tool and > upon change drop the file from BaseX and re-add it? > > Regards, > Eric > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >
JC
Jan Vlčinský (CAD)3 Jul 2011 13:05 #
Hi Eric
Your use case is very clear.
It seems you need some nice tool, which would be able to filter information from dynamically growing number of information stored in sort of jmeter "logs".
I am not sure, if BaseX is the best tool for this - it has nice GUI, which allows to query large number of xml data - this is really nice - but I never did anything with so changing number of data.
Be prepared to following limitations (some can correct me, if I mention limitation, which is already overcome):
- data added over command line tool are not synchronised with what you see in GUI. Reopening GUI resolvers this, but do not expect GUI to show latest changed done by some other process in background. I am not sure, if closing and then opening the collection would help, it is quite likely.
- Modification of a collection by adding documents does not automatically updates indices - you have to call explicitly command optimize.
On the other hand, if you would analyse the data over BaseX client (not GUI), then this shall work quite well.
Some hints:
If there would be large amount of data and need to optimize performance, I would think of using multiple collections, having older and not changing data separated form latest ones. This way the old data would be kept optimised without reindexing, only the latest database would have to be optimised (it is possible, rebuilding the latest database would be faster) . I am not sure, how would perform xqueries when they query multiple databases, I did not test it (it is likely, it would not have any problem). It is possible, your xqueries are simple and can perform fast enough even without updated indices - this could simplify the situation as you would not have to care about command optimize.
Jan
*Ing. Jan Vlčinský*
CAD programy
Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic
tel: +420-597 602 024; mob: +420-608 979 040
skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com
http://cz.linkedin.com/in/vlcinsky
Quoted text · Eric Jones · 66 lines
On 3 July 2011 01:14, Eric Jones <ecjones2040 at gmail.com> wrote: > Hi All, > > A load testing tool (jmeter) has the capability to write out its test > results as XML. This is great as I > search and perform metrics calculations on this data using XQuery. > However, the number of XML files can grow to some volume and > it is hard to tell when a test is actually over which would mean its > xml result file is finished being written to. The solution to this is > to use Inotify > to monitor the directory where these files are stored and add them to > the database if they do not exist or delete the file and re-add it to > BaseX if it already has been added and is just in need of an update > (do to a running test). So that is the solution I will implement -- > if anyone has anything cleaner I am all ears. > > I thank the BaseX mailing list for helping me with this. > > Regards, > Eric > > On Sat, Jul 2, 2011 at 3:29 AM, Huib Verwey <Huib.Verwey at mpi.nl> wrote: > > Hi Eric, > > could you give an example of a use case? > > For monitoring changes to files and acting upon them Google tells me I > could > > use inotify or incron. > > From your setup I suspect you'll use BaseX as a read-only store, maybe > just > > for searching? So you won't use the XQuery Update Facility for modifying > XML > > in the database? > > Hartelijke groet, > > Huib Verweij. > > -- > > Drs. Huib Verweij > > Senior software developer - The Language Archive > > Max Planck Institute for Psycholinguistics > > P.O. Box 310 > > 6500 AH Nijmegen > > The Netherlands > > t +31-24-3521911 > > e huib.verwey at mpi.nl > > w http://www.mpi.nl/ > > > > Op 1 jul 2011, om 06:53 heeft Eric Jones het volgende geschreven: > > > > Hello all, > > > > I am new to BaseX. I want to know how one handles frequently updated > > xml files. Do you have to monitor the file with a third party tool and > > upon change drop the file from BaseX and re-add it? > > > > Regards, > > Eric > > _______________________________________________ > > BaseX-Talk mailing list > > BaseX-Talk at mailman.uni-konstanz.de > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
Written for BaseX 6.7. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery