running in parallel
Sunday, 8 December 2019 · 6 messages · last reply 9 Dec 2019
C
celano8 Dec 2019 16:48 #
Hi,
I am trying to run two BaseX scripts in parallel using:
xquery:fork-join(
(
function() {xquery:eval(xs:anyURI('extract_from_ocr1.xq'))}
,
function (){xquery:eval(xs:anyURI('extract_from_ocr2.xq'))}
)
)As far as I can understand (read below), the scripts are kind of run in parallel, but still the time benefit of that does not seem much in comparison with running in sequence (~25s vs ~28s). The files contain the same function, which reads files from a directory, performs some calculation, and saves the result in a file (the two scripts work on different directories). I infer that the previous script is run in parallel because the files for the results are created at the same time.
I tried to do the same with GNU parallel, and in that case the files are actually run in parallel.
Do we know why the execution time is not (more or less) halved in BaseX? Thanks.
Ciao,
Giuseppe
MW
Markus Wittenberg8 Dec 2019 17:04 #
Hi Giuseppe,
as long as the files are not on physically different disks, you will have the two functions block each other with read and write operations. And BaseX runs lots of code in parallel without you explicitly telling it so.
Best regards,
Markus
Quoted text · celano · 30 lines
Am 08.12.2019 um 16:48 schrieb celano at informatik.uni-leipzig.de: > Hi, > > I am trying to run two BaseX scripts in parallel using: > > xquery:fork-join( > ( > function() {xquery:eval(xs:anyURI('extract_from_ocr1.xq'))} > , > function (){xquery:eval(xs:anyURI('extract_from_ocr2.xq'))} > ) > ) > > As far as I can understand (read below), the scripts are kind of run > in parallel, but still the time benefit of that does not seem much in > comparison with running in sequence (~25s vs ~28s). The files contain > the same function, which reads files from a directory, performs some > calculation, and saves the result in a file (the two scripts work on > different directories). I infer that the previous script is run in > parallel because the files for the results are created at the same time. > > I tried to do the same with GNU parallel, and in that case the files > are actually run in parallel. > > Do we know why the execution time is not (more or less) halved in > BaseX? Thanks. > > Ciao, > Giuseppe >
--
Markus Wittenberg
Tel +49 (0)341 248 475 36
Mail wittenberg at axxepta.de
----
axxepta solutions GmbH
Lehmgrubenweg 17, 88131 Lindau
Amtsgericht Berlin HRB 97544B
Geschäftsführer: Karsten Becke, Maximilian Gärber
CG
Hi Giuseppe,
Maybe you can consult the mailing list archive for more information on concurrent/parallel query processing; there has been a lot of discussion in the past (e.g. [1]).
Cheers,
Christian
[1] https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg12080.html
Quoted text · PM · 31 lines
On Sun, Dec 8, 2019 at 4:49 PM <celano at informatik.uni-leipzig.de> wrote: > > Hi, > > I am trying to run two BaseX scripts in parallel using: > > xquery:fork-join( > ( > function() {xquery:eval(xs:anyURI('extract_from_ocr1.xq'))} > , > function (){xquery:eval(xs:anyURI('extract_from_ocr2.xq'))} > ) > ) > > As far as I can understand (read below), the scripts are kind of run > in parallel, but still the time benefit of that does not seem much in > comparison with running in sequence (~25s vs ~28s). The files contain > the same function, which reads files from a directory, performs some > calculation, and saves the result in a file (the two scripts work on > different directories). I infer that the previous script is run in > parallel because the files for the results are created at the same time. > > I tried to do the same with GNU parallel, and in that case the files > are actually run in parallel. > > Do we know why the execution time is not (more or less) halved in > BaseX? Thanks. > > Ciao, > Giuseppe >
OS
Hi,
I see the same in my application. My two cent of wisdom: I would say most disks today will be fast enough to mask this problem. Let alone SSDs that can happily fetch two files at the (almost) same time. But the thing is: The exist code uses some pretty heavy locks to make sure no two Java threads access the same (database) file at the same time. And unless this is really given some thought for data safety I am glad that it does not allow queries to run in parallel. I would love to solve this in a more state of the art way but got burned in the past by multi threading. So I have great respect for any good, safe and fast implementation multi threading file access implementation. I fear no one did one yet for BaseX.
Best regards
Omar Siam
Quoted text · Markus Wittenberg · 42 lines
Am 08.12.2019 um 17:04 schrieb Markus Wittenberg: > Hi Giuseppe, > > as long as the files are not on physically different disks, you will > have the two functions block each other with read and write > operations. And BaseX runs lots of code in parallel without you > explicitly telling it so. > > Best regards, > > Markus > > Am 08.12.2019 um 16:48 schrieb celano at informatik.uni-leipzig.de: >> Hi, >> >> I am trying to run two BaseX scripts in parallel using: >> >> xquery:fork-join( >> ( >> function() {xquery:eval(xs:anyURI('extract_from_ocr1.xq'))} >> , >> function (){xquery:eval(xs:anyURI('extract_from_ocr2.xq'))} >> ) >> ) >> >> As far as I can understand (read below), the scripts are kind of run >> in parallel, but still the time benefit of that does not seem much in >> comparison with running in sequence (~25s vs ~28s). The files contain >> the same function, which reads files from a directory, performs some >> calculation, and saves the result in a file (the two scripts work on >> different directories). I infer that the previous script is run in >> parallel because the files for the results are created at the same time. >> >> I tried to do the same with GNU parallel, and in that case the files >> are actually run in parallel. >> >> Do we know why the execution time is not (more or less) halved in >> BaseX? Thanks. >> >> Ciao, >> Giuseppe >>
GC
Giuseppe G. A. Celano9 Dec 2019 18:54 #
Thanks for your answers!
I have run an experiment, and I confirm that fork-join() actually works, even if the gain is not as expected. Most importantly, I noticed that the amount of RAM made available is crucial: with 2MB the sequential script was very slow, while with 5/7MB it works fine.
(2,8 GHz Quad-Core Intel Core i7)
Sequential: about 47 s.
fork-join: about 40 s.
GNU parallel: about 30 s.
Best,
Giuseppe
Quoted text · Omar Siam · 41 lines
> On 9. Dec 2019, at 16:58, Omar Siam <Omar.Siam at oeaw.ac.at> wrote: > > Hi, > > I see the same in my application. My two cent of wisdom: I would say most disks today will be fast enough to mask this problem. Let alone SSDs that can happily fetch two files at the (almost) same time. But the thing is: The exist code uses some pretty heavy locks to make sure no two Java threads access the same (database) file at the same time. And unless this is really given some thought for data safety I am glad that it does not allow queries to run in parallel. I would love to solve this in a more state of the art way but got burned in the past by multi threading. So I have great respect for any good, safe and fast implementation multi threading file access implementation. I fear no one did one yet for BaseX. > > Best regards > > Omar Siam > > Am 08.12.2019 um 17:04 schrieb Markus Wittenberg: >> Hi Giuseppe, >> >> as long as the files are not on physically different disks, you will have the two functions block each other with read and write operations. And BaseX runs lots of code in parallel without you explicitly telling it so. >> >> Best regards, >> >> Markus >> >> Am 08.12.2019 um 16:48 schrieb celano at informatik.uni-leipzig.de: >>> Hi, >>> >>> I am trying to run two BaseX scripts in parallel using: >>> >>> xquery:fork-join( >>> ( >>> function() {xquery:eval(xs:anyURI('extract_from_ocr1.xq'))} >>> , >>> function (){xquery:eval(xs:anyURI('extract_from_ocr2.xq'))} >>> ) >>> ) >>> >>> As far as I can understand (read below), the scripts are kind of run in parallel, but still the time benefit of that does not seem much in comparison with running in sequence (~25s vs ~28s). The files contain the same function, which reads files from a directory, performs some calculation, and saves the result in a file (the two scripts work on different directories). I infer that the previous script is run in parallel because the files for the results are created at the same time. >>> >>> I tried to do the same with GNU parallel, and in that case the files are actually run in parallel. >>> >>> Do we know why the execution time is not (more or less) halved in BaseX? Thanks. >>> >>> Ciao, >>> Giuseppe >>>
GC
Giuseppe G. A. Celano9 Dec 2019 21:23 #
I forgot to mention that I often use fork-join() with
proc:execute() to run in parallel more than one instance of the OCR engine “tesseract” (I can have more than 1000 images to OCR): it works fabulously (up to 8 processes on my Quad-core Intel Core i7). More in general, when it comes to running system programs, fork-join() + proc:execute() is extremely useful.
GiuseppeQuoted text · Giuseppe G. A. Celano · 59 lines
> On 9. Dec 2019, at 18:54, Giuseppe G. A. Celano <celano at informatik.uni-leipzig.de> wrote: > > Thanks for your answers! > > I have run an experiment, and I confirm that fork-join() actually works, even if the gain is not as expected. Most importantly, I noticed that the amount of RAM made available is crucial: with 2MB the sequential script was very slow, while with 5/7MB it works fine. > > (2,8 GHz Quad-Core Intel Core i7) > > Sequential: about 47 s. > fork-join: about 40 s. > GNU parallel: about 30 s. > > Best, > Giuseppe > > >> On 9. Dec 2019, at 16:58, Omar Siam <Omar.Siam at oeaw.ac.at> wrote: >> >> Hi, >> >> I see the same in my application. My two cent of wisdom: I would say most disks today will be fast enough to mask this problem. Let alone SSDs that can happily fetch two files at the (almost) same time. But the thing is: The exist code uses some pretty heavy locks to make sure no two Java threads access the same (database) file at the same time. And unless this is really given some thought for data safety I am glad that it does not allow queries to run in parallel. I would love to solve this in a more state of the art way but got burned in the past by multi threading. So I have great respect for any good, safe and fast implementation multi threading file access implementation. I fear no one did one yet for BaseX. >> >> Best regards >> >> Omar Siam >> >> Am 08.12.2019 um 17:04 schrieb Markus Wittenberg: >>> Hi Giuseppe, >>> >>> as long as the files are not on physically different disks, you will have the two functions block each other with read and write operations. And BaseX runs lots of code in parallel without you explicitly telling it so. >>> >>> Best regards, >>> >>> Markus >>> >>> Am 08.12.2019 um 16:48 schrieb celano at informatik.uni-leipzig.de: >>>> Hi, >>>> >>>> I am trying to run two BaseX scripts in parallel using: >>>> >>>> xquery:fork-join( >>>> ( >>>> function() {xquery:eval(xs:anyURI('extract_from_ocr1.xq'))} >>>> , >>>> function (){xquery:eval(xs:anyURI('extract_from_ocr2.xq'))} >>>> ) >>>> ) >>>> >>>> As far as I can understand (read below), the scripts are kind of run in parallel, but still the time benefit of that does not seem much in comparison with running in sequence (~25s vs ~28s). The files contain the same function, which reads files from a directory, performs some calculation, and saves the result in a file (the two scripts work on different directories). I infer that the previous script is run in parallel because the files for the results are created at the same time. >>>> >>>> I tried to do the same with GNU parallel, and in that case the files are actually run in parallel. >>>> >>>> Do we know why the execution time is not (more or less) halved in BaseX? Thanks. >>>> >>>> Ciao, >>>> Giuseppe >>>> > >
Written for BaseX 9.3. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery