insert node takes really long to complete
Sunday, 27 November 2011 · 3 messages
JC
Jan Vlčinský (CAD)27 Nov 2011 10:59 #
Hi
I am trying to add some attribute into my data, doing it this way:
for $msg in //MSG
let $evi := $msg//TMCE/EVI[1]
let $updclass := $evi/@updateclass
let $tmcloc := $msg//MLOC/TMCL[1]
let $dir := $tmcloc/@direction
let $loccode := $tmcloc/@primarycode
let $alertc-id := concat($loccode, $dir, $updclass)
return
insert node (attribute alertc-id {$alertc-id}) into $msgIf I simply return result similar to this, using following query:
for $msg in //MSG
let $evi := $msg//TMCE/EVI[1]
let $updclass := $evi/@updateclass
let $tmcloc := $msg//MLOC/TMCL[1]
let $dir := $tmcloc/@direction
let $loccode := $tmcloc/@primarycode
let $alertc-id := concat($loccode, $dir, $updclass)
return element enhancedmsg {attribute alertc-id {$alertc-id}, $msg}it takes just 27 s
With the insert node, it takes for ever, I had to cancel the query after 90 minutes.
I wonder, if there is any relation between this long execution time and possible inefficiency with automatic index update (but I am not aware of what is current status of this approach).
Currently, I see, it would be much more efficient to output the result as files, and then import them back, as importing the data I am working on took just 2 minutes.
Attached are few sample xml files in the format, I use in my database. Real database consists of 129097 MSG elements and number of DOC elements and documents is exactly the same.
With best regards
Jan
*Ing. Jan Vlčinský*
CAD programy
Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic
tel: +420-597 602 024; mob: +420-608 979 040
skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com
http://cz.linkedin.com/in/vlcinsky
JC
Jan Vlčinský (CAD)27 Nov 2011 11:02 #
Hi.
Just one addition: I am using version 7.0.1.
Jan
Quoted text · Jan Vlčinský (CAD) · 54 lines
On 27 November 2011 10:59, Jan Vlčinský (CAD) <jan.vlcinsky at cad-programs.com > wrote: > Hi > I am trying to add some attribute into my data, doing it this way: > > for $msg in //MSG > let $evi := $msg//TMCE/EVI[1] > let $updclass := $evi/@updateclass > let $tmcloc := $msg//MLOC/TMCL[1] > let $dir := $tmcloc/@direction > let $loccode := $tmcloc/@primarycode > let $alertc-id := concat($loccode, $dir, $updclass) > return > insert node (attribute alertc-id {$alertc-id}) into $msg > > If I simply return result similar to this, using following query: > > for $msg in //MSG > let $evi := $msg//TMCE/EVI[1] > let $updclass := $evi/@updateclass > let $tmcloc := $msg//MLOC/TMCL[1] > let $dir := $tmcloc/@direction > let $loccode := $tmcloc/@primarycode > let $alertc-id := concat($loccode, $dir, $updclass) > return element enhancedmsg {attribute alertc-id {$alertc-id}, $msg} > > it takes just 27 s > > With the insert node, it takes for ever, I had to cancel the query after > 90 minutes. > I wonder, if there is any relation between this long execution time and > possible inefficiency with automatic index update (but I am not aware of > what is current status of this approach). > > Currently, I see, it would be much more efficient to output the result as > files, and then import them back, as importing the data I am working on > took just 2 minutes. > > Attached are few sample xml files in the format, I use in my database. > Real database consists of 129097 MSG elements and number of DOC elements > and documents is exactly the same. > > > With best regards > > Jan > *Ing. Jan Vlčinský* > CAD programy > Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic > tel: +420-597 602 024; mob: +420-608 979 040 > skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com > http://cz.linkedin.com/in/vlcinsky > >
CG
Hi Jan,
129.000 single update operations are quite a lot, but yet I'm wondering why the query takes really that long. It's difficult to guess what are the reasons for that your particular case, so it would be great if you could do some basic Java profiling and pass on the results via this mail.. .e.g. by the following command-line call:
java -cp path/to/basex.jar -Xmx1g -Xrunhprof:cpu=samples,depth=15 org.basex.BaseX -c"create db testdb path/to/files; run
/path/to/query.xq
Best,
Christian
Quoted text · Jan Vlčinský (CAD) · 50 lines
On Sun, Nov 27, 2011 at 10:59 AM, Jan Vlčinský (CAD) <jan.vlcinsky at cad-programs.com> wrote: > Hi > I am trying to add some attribute into my data, doing it this way: > for $msg in //MSG > let $evi := $msg//TMCE/EVI[1] > let $updclass := $evi/@updateclass > let $tmcloc := $msg//MLOC/TMCL[1] > let $dir := $tmcloc/@direction > let $loccode := $tmcloc/@primarycode > let $alertc-id := concat($loccode, $dir, $updclass) > return > insert node (attribute alertc-id {$alertc-id}) into $msg > If I simply return result similar to this, using following query: > for $msg in //MSG > let $evi := $msg//TMCE/EVI[1] > let $updclass := $evi/@updateclass > let $tmcloc := $msg//MLOC/TMCL[1] > let $dir := $tmcloc/@direction > let $loccode := $tmcloc/@primarycode > let $alertc-id := concat($loccode, $dir, $updclass) > return element enhancedmsg {attribute alertc-id {$alertc-id}, $msg} > it takes just 27 s > With the insert node, it takes for ever, I had to cancel the query after 90 > minutes. > I wonder, if there is any relation between this long execution time and > possible inefficiency with automatic index update (but I am not aware of > what is current status of this approach). > Currently, I see, it would be much more efficient to output the result as > files, and then import them back, as importing the data I am working on took > just 2 minutes. > Attached are few sample xml files in the format, I use in my database. Real > database consists of 129097 MSG elements and number of DOC elements and > documents is exactly the same. > > With best regards > Jan > Ing. Jan Vlčinský > CAD programy > Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic > tel: +420-597 602 024; mob: +420-608 979 040 > skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com > http://cz.linkedin.com/in/vlcinsky > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >
Written for BaseX 7.0. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery