db:add speed proportional to DB size?
Wednesday, 6 January 2016 · 3 messages
HE
Hondros, Constantine (ELS-AMS)6 Jan 2016 10:53 #
Hello all,
Processing hundreds of thousands of zips, using
db:add to to append small XML fragments from each into a single DB, I notice that the process becomes successively slower. Without having done any proper profiling, and aware that I might be looking in the wrong direction here, would it be reasonable to expect db:add to become steadily slower proportional to the size of the database being added to?
Cheers,
C.
________________________________
Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33156677, Registered in The Netherlands.CG
Christian Grün6 Jan 2016 13:57 #
Hi Hondros,
Quoted text · Hondros, Constantine (ELS-AMS) · 6 lines
> Processing hundreds of thousands of zips, using db:add to to append small > XML fragments from each into a single DB, I notice that the process becomes > successively slower. Without having done any proper profiling, and aware > that I might be looking in the wrong direction here, would it be reasonable > to expect db:add to become steadily slower proportional to the size of the > database being added to?
Yes, this may happen – but spontaneously I cannot tell you what might the reason for that in your setup. Feel free to send me some profiling output (e.g. using -Xhrunprof:cpu=samples).
Christian
Quoted text · Hondros, Constantine (ELS-AMS) · 12 lines
> > > > Cheers, > > C. > > > ________________________________ > > Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The > Netherlands, Registration No. 33156677, Registered in The Netherlands.
HE
Hondros, Constantine (ELS-AMS)6 Jan 2016 21:50 #
Thanks Christian, but I've satisfied myself that the slowdown is due to the size of the DB being added to by timing a sample set against an empty database. No big deal, and I can easily work around it.
Cheers,
Constantine
Quoted text · Christian Grün · 28 lines
-----Original Message----- From: Christian Grün [mailto:christian.gruen at gmail.com] Sent: 06 January 2016 13:57 To: Hondros, Constantine (ELS-AMS) Cc: BaseX Subject: Re: [basex-talk] db:add speed proportional to DB size? Hi Hondros, > Processing hundreds of thousands of zips, using db:add to to append > small XML fragments from each into a single DB, I notice that the > process becomes successively slower. Without having done any proper > profiling, and aware that I might be looking in the wrong direction > here, would it be reasonable to expect db:add to become steadily > slower proportional to the size of the database being added to? Yes, this may happen – but spontaneously I cannot tell you what might the reason for that in your setup. Feel free to send me some profiling output (e.g. using -Xhrunprof:cpu=samples). Christian > > > > Cheers, > > C. > > > ________________________________ > > Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The > Netherlands, Registration No. 33156677, Registered in The Netherlands. Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33156677, Registered in The Netherlands.
Written for BaseX 8.3. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery