Write files in parallel?

On Tue, Oct 1, 2024 at 5:32 PM Thompson, Timothy <timothy.thompson at yale.edu> wrote: > Hello, > > > > Is it possible to call file:write-text-lines in parallel inside a > fork-join operation? I have multiple databases that I would like to run a > query over, in parallel, and write the results as JSON Lines to a file per > database. When I try this, it doesn’t seem to parallelize. > > > > Thanks in advance, > > Tim > > > > > > -- > Tim A. Thompson (*he, him*) > Librarian for Applied Metadata Research > > *Interim Manager, Metadata Services Unit* > > Yale University Library > > www.linkedin.com/in/timathompson > > >

From: Bridger Dyson-Smith <bdysonsmith at gmail.com> Date: Wednesday, October 2, 2024 at 1:05 PM To: Thompson, Timothy <timothy.thompson at yale.edu> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Write files in parallel? hi Tim - hope you are well. In the past (i.e. I don't remember exactly if this was perfectly parallel, it was just "parallel enough"), I have used something like the following for web requests: xquery:fork-join( for $xml in ('calq.xqm','factbook.xml','filesystem.xml','locations.xml','wiki1.zip', 'wiki2.zip','xmark.xml') let $url := 'https://files.basex.org/xml/' return fn() { file:write( '/tmp/fork-test/' || $xml, http:send-request( <http:request method='get'/>, $url || $xml ) ) }, map { 'parallel': '3'} ) Hopefully that's helpful (and apologies to the BaseX team's file server)! Best, Bridger ) ls -l --time-style=full-iso total 11640 -rw-r--r-- 1 bridger bridger 1593 2024-10-02 17:02:51.321251082 +0000 calq.xqm -rw-r--r-- 1 bridger bridger 1763070 2024-10-02 17:02:52.301261520 +0000 factbook.xml -rw-r--r-- 1 bridger bridger 2770290 2024-10-02 17:02:53.331272491 +0000 filesystem.xml -rw-r--r-- 1 bridger bridger 1566322 2024-10-02 17:02:52.497263608 +0000 locations.xml -rw-r--r-- 1 bridger bridger 512686 2024-10-02 17:02:52.670265451 +0000 wiki1.zip -rw-r--r-- 1 bridger bridger 5133340 2024-10-02 17:02:54.046280106 +0000 wiki2.zip -rw-r--r-- 1 bridger bridger 155448 2024-10-02 17:02:52.859267464 +0000 xmark.xml On Tue, Oct 1, 2024 at 5:32 PM Thompson, Timothy <timothy.thompson at yale.edu<mailto:timothy.thompson at yale.edu>> wrote: Hello, Is it possible to call file:write-text-lines in parallel inside a fork-join operation? I have multiple databases that I would like to run a query over, in parallel, and write the results as JSON Lines to a file per database. When I try this, it doesn’t seem to parallelize. Thanks in advance, Tim -- Tim A. Thompson (he, him) Librarian for Applied Metadata Research Interim Manager, Metadata Services Unit Yale University Library www.linkedin.com/in/timathompson<http://www.linkedin.com/in/timathompson>

On Fri, Oct 4, 2024 at 5:53 PM Thompson, Timothy <timothy.thompson at yale.edu> wrote: > Thanks, Bridger! `file:write-text-lines` seems to be the issue. For > example, this query doesn’t run in parallel. > > >

> Is this expected behavior? > > declare variable $PATH := ""; > > > > xquery:fork-join( > > for $_ in (1 to 8) > > return fn() { > > file:write-text-lines( > > $PATH||$_||".json", > > for $i in (1 to 1000000) > > return > > serialize( > > <fn:map> > > <fn:string key="n">{$i}</fn:string> > > </fn:map>, {"method": "json", "escape-solidus": "no", "json": { > > "format": "basic", "indent": "no" > > }} > > ) > > ) > > }, > > { "parallel": "8"} > > ) > > >

> > -- > Tim A. Thompson (*he, him*) > Librarian for Applied Metadata Research > > *Interim Manager, Metadata Services Unit* > > www.linkedin.com/in/timathompson > > > > > > *From: *Bridger Dyson-Smith <bdysonsmith at gmail.com> > *Date: *Wednesday, October 2, 2024 at 1:05 PM > *To: *Thompson, Timothy <timothy.thompson at yale.edu> > *Cc: *BaseX <basex-talk at mailman.uni-konstanz.de> > *Subject: *Re: [basex-talk] Write files in parallel? > > hi Tim - hope you are well. > > In the past (i.e. I don't remember exactly if this was perfectly parallel, > it was just "parallel enough"), I have used something like the following > for web requests: > > xquery:fork-join( > for $xml in > ('calq.xqm','factbook.xml','filesystem.xml','locations.xml','wiki1.zip', > 'wiki2.zip','xmark.xml') > let $url := 'https://files.basex.org/xml/' > return fn() { > file:write( > '/tmp/fork-test/' || $xml, > http:send-request( > <http:request method='get'/>, > $url || $xml > ) > ) > }, > map { 'parallel': '3'} > ) > > Hopefully that's helpful (and apologies to the BaseX team's file server)! > > Best, > > Bridger > > > > ) ls -l --time-style=full-iso > total 11640 > -rw-r--r-- 1 bridger bridger 1593 2024-10-02 17:02:51.321251082 +0000 > calq.xqm > -rw-r--r-- 1 bridger bridger 1763070 2024-10-02 17:02:52.301261520 +0000 > factbook.xml > -rw-r--r-- 1 bridger bridger 2770290 2024-10-02 17:02:53.331272491 +0000 > filesystem.xml > -rw-r--r-- 1 bridger bridger 1566322 2024-10-02 17:02:52.497263608 +0000 > locations.xml > -rw-r--r-- 1 bridger bridger 512686 2024-10-02 17:02:52.670265451 +0000 > wiki1.zip > -rw-r--r-- 1 bridger bridger 5133340 2024-10-02 17:02:54.046280106 +0000 > wiki2.zip > -rw-r--r-- 1 bridger bridger 155448 2024-10-02 17:02:52.859267464 +0000 > xmark.xml > > > > > > On Tue, Oct 1, 2024 at 5:32 PM Thompson, Timothy < > timothy.thompson at yale.edu> wrote: > > Hello, > > > > Is it possible to call file:write-text-lines in parallel inside a > fork-join operation? I have multiple databases that I would like to run a > query over, in parallel, and write the results as JSON Lines to a file per > database. When I try this, it doesn’t seem to parallelize. > > > > Thanks in advance, > > Tim > > > > > > -- > Tim A. Thompson (*he, him*) > Librarian for Applied Metadata Research > > *Interim Manager, Metadata Services Unit* > > Yale University Library > > www.linkedin.com/in/timathompson > > > >

On Fri, Oct 4, 2024 at 11:03 PM Bridger Dyson-Smith <bdysonsmith at gmail.com> wrote: > Hey Tim - > > On Fri, Oct 4, 2024 at 5:53 PM Thompson, Timothy < > timothy.thompson at yale.edu> wrote: > >> Thanks, Bridger! `file:write-text-lines` seems to be the issue. For >> example, this query doesn’t run in parallel. >> >> >> > You're right - apologies for missing this key point in your initial > email. > >> Is this expected behavior? >> >> declare variable $PATH := ""; >> >> >> >> xquery:fork-join( >> >> for $_ in (1 to 8) >> >> return fn() { >> >> file:write-text-lines( >> >> $PATH||$_||".json", >> >> for $i in (1 to 1000000) >> >> return >> >> serialize( >> >> <fn:map> >> >> <fn:string key="n">{$i}</fn:string> >> >> </fn:map>, {"method": "json", "escape-solidus": "no", "json": { >> >> "format": "basic", "indent": "no" >> >> }} >> >> ) >> >> ) >> >> }, >> >> { "parallel": "8"} >> >> ) >> >> >> > It does seem to be the case that the writes in `file:write-text-lines` are > *not* parallel vs a sequential use of the same: > I did the following comparison: > > using your example, > ls -l --time-style=full-iso /tmp/fork-test > total 130860 > -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:39:57.926518544 +0000 > 1.json > -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:40:02.849576119 +0000 > 2.json > -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:40:07.799634010 +0000 > 3.json > -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:40:28.652877890 +0000 > 4.json > -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:40:12.892693574 +0000 > 5.json > -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:40:18.140754950 +0000 > 6.json > -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:40:23.569818443 +0000 > 7.json > -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:40:39.098000046 +0000 > 8.json > -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:40:33.779937851 +0000 > 9.json > > vs > > using a sequential write: > declare variable $PATH := "/tmp/fork-test/sequential/"; > > for $i in (1 to 9) > return > file:write-text-lines( > $PATH || $i || ".json", > for $n in (1 to 1000000) > return > serialize( > <fn:map> > <fn:string key="n">{$n}</fn:string> > </fn:map>, > { "method": "json", "escape-solidus": "no", > "json": { "format": "basic", "indent": "no" } > } > ) > ) > > ls -l --time-style=full-iso /tmp/fork-test/sequential > total 130860 > -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:49:19.841259435 +0000 > 1.json > -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:49:24.820319704 +0000 > 2.json > -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:49:29.838380446 +0000 > 3.json > -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:49:35.041443427 +0000 > 4.json > -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:49:40.182505657 +0000 > 5.json > -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:49:45.305567669 +0000 > 6.json > -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:49:50.535630977 +0000 > 7.json > -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:49:55.703693534 +0000 > 8.json > -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:50:00.948757024 +0000 > 9.json > > each file in both attempts takes about 5ms to write, with the exception > that the writes are non-sequential in the fork-join example. I wonder if > it's due to the appending in `file:write-text-lines`? > Maybe Christian can chime in and let us know :) > > Have a nice weekend! > Best, > Bridger > > >> >> -- >> Tim A. Thompson (*he, him*) >> Librarian for Applied Metadata Research >> >> *Interim Manager, Metadata Services Unit* >> >> www.linkedin.com/in/timathompson >> >> >> >> >> >> *From: *Bridger Dyson-Smith <bdysonsmith at gmail.com> >> *Date: *Wednesday, October 2, 2024 at 1:05 PM >> *To: *Thompson, Timothy <timothy.thompson at yale.edu> >> *Cc: *BaseX <basex-talk at mailman.uni-konstanz.de> >> *Subject: *Re: [basex-talk] Write files in parallel? >> >> hi Tim - hope you are well. >> >> In the past (i.e. I don't remember exactly if this was perfectly >> parallel, it was just "parallel enough"), I have used something like the >> following for web requests: >> >> xquery:fork-join( >> for $xml in >> ('calq.xqm','factbook.xml','filesystem.xml','locations.xml','wiki1.zip', >> 'wiki2.zip','xmark.xml') >> let $url := 'https://files.basex.org/xml/' >> return fn() { >> file:write( >> '/tmp/fork-test/' || $xml, >> http:send-request( >> <http:request method='get'/>, >> $url || $xml >> ) >> ) >> }, >> map { 'parallel': '3'} >> ) >> >> Hopefully that's helpful (and apologies to the BaseX team's file server)! >> >> Best, >> >> Bridger >> >> >> >> ) ls -l --time-style=full-iso >> total 11640 >> -rw-r--r-- 1 bridger bridger 1593 2024-10-02 17:02:51.321251082 +0000 >> calq.xqm >> -rw-r--r-- 1 bridger bridger 1763070 2024-10-02 17:02:52.301261520 +0000 >> factbook.xml >> -rw-r--r-- 1 bridger bridger 2770290 2024-10-02 17:02:53.331272491 +0000 >> filesystem.xml >> -rw-r--r-- 1 bridger bridger 1566322 2024-10-02 17:02:52.497263608 +0000 >> locations.xml >> -rw-r--r-- 1 bridger bridger 512686 2024-10-02 17:02:52.670265451 +0000 >> wiki1.zip >> -rw-r--r-- 1 bridger bridger 5133340 2024-10-02 17:02:54.046280106 +0000 >> wiki2.zip >> -rw-r--r-- 1 bridger bridger 155448 2024-10-02 17:02:52.859267464 +0000 >> xmark.xml >> >> >> >> >> >> On Tue, Oct 1, 2024 at 5:32 PM Thompson, Timothy < >> timothy.thompson at yale.edu> wrote: >> >> Hello, >> >> >> >> Is it possible to call file:write-text-lines in parallel inside a >> fork-join operation? I have multiple databases that I would like to run a >> query over, in parallel, and write the results as JSON Lines to a file per >> database. When I try this, it doesn’t seem to parallelize. >> >> >> >> Thanks in advance, >> >> Tim >> >> >> >> >> >> -- >> Tim A. Thompson (*he, him*) >> Librarian for Applied Metadata Research >> >> *Interim Manager, Metadata Services Unit* >> >> Yale University Library >> >> www.linkedin.com/in/timathompson >> >> >> >>

From: Christian Grün <christian.gruen at gmail.com> Date: Sunday, October 6, 2024 at 7:25 AM To: Bridger Dyson-Smith <bdysonsmith at gmail.com> Cc: Thompson, Timothy <timothy.thompson at yale.edu>, BaseX <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Write files in parallel? Hi Tim, hi Bridger, Some time ago, we have decided to put atomicity first, and to synchronize all concurrent updating file operations, whether they run in parallel or are invoked by different clients. This way, we prevent parallel transactions from being interrupted if they write to the same target. Perhaps we are being overly cautious. We could choose a more fine granular concept and synchronize I/O access to individual files. It would be straightforward for operations on single operations (with file:write and its variants), but it gets more complex for recursive operations like file:copy or file:delete that may affect all files in a directory. We’ll have some more thoughts on this. Best, Christian On Fri, Oct 4, 2024 at 11:03 PM Bridger Dyson-Smith <bdysonsmith at gmail.com<mailto:bdysonsmith at gmail.com>> wrote: Hey Tim - On Fri, Oct 4, 2024 at 5:53 PM Thompson, Timothy <timothy.thompson at yale.edu<mailto:timothy.thompson at yale.edu>> wrote: Thanks, Bridger! `file:write-text-lines` seems to be the issue. For example, this query doesn’t run in parallel. You're right - apologies for missing this key point in your initial email. Is this expected behavior? declare variable $PATH := ""; xquery:fork-join( for $_ in (1 to 8) return fn() { file:write-text-lines( $PATH||$_||".json", for $i in (1 to 1000000) return serialize( <fn:map> <fn:string key="n">{$i}</fn:string> </fn:map>, {"method": "json", "escape-solidus": "no", "json": { "format": "basic", "indent": "no" }} ) ) }, { "parallel": "8"} ) It does seem to be the case that the writes in `file:write-text-lines` are *not* parallel vs a sequential use of the same: I did the following comparison: using your example, ls -l --time-style=full-iso /tmp/fork-test total 130860 -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:39:57.926518544 +0000 1.json -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:40:02.849576119 +0000 2.json -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:40:07.799634010 +0000 3.json -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:40:28.652877890 +0000 4.json -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:40:12.892693574 +0000 5.json -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:40:18.140754950 +0000 6.json -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:40:23.569818443 +0000 7.json -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:40:39.098000046 +0000 8.json -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:40:33.779937851 +0000 9.json vs using a sequential write: declare variable $PATH := "/tmp/fork-test/sequential/"; for $i in (1 to 9) return file:write-text-lines( $PATH || $i || ".json", for $n in (1 to 1000000) return serialize( <fn:map> <fn:string key="n">{$n}</fn:string> </fn:map>, { "method": "json", "escape-solidus": "no", "json": { "format": "basic", "indent": "no" } } ) ) ls -l --time-style=full-iso /tmp/fork-test/sequential total 130860 -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:49:19.841259435 +0000 1.json -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:49:24.820319704 +0000 2.json -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:49:29.838380446 +0000 3.json -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:49:35.041443427 +0000 4.json -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:49:40.182505657 +0000 5.json -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:49:45.305567669 +0000 6.json -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:49:50.535630977 +0000 7.json -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:49:55.703693534 +0000 8.json -rw-r--r-- 1 bridger bridger 14888896 2024-10-04 20:50:00.948757024 +0000 9.json each file in both attempts takes about 5ms to write, with the exception that the writes are non-sequential in the fork-join example. I wonder if it's due to the appending in `file:write-text-lines`? Maybe Christian can chime in and let us know :) Have a nice weekend! Best, Bridger -- Tim A. Thompson (he, him) Librarian for Applied Metadata Research Interim Manager, Metadata Services Unit www.linkedin.com/in/timathompson<http://www.linkedin.com/in/timathompson> From: Bridger Dyson-Smith <bdysonsmith at gmail.com<mailto:bdysonsmith at gmail.com>> Date: Wednesday, October 2, 2024 at 1:05 PM To: Thompson, Timothy <timothy.thompson at yale.edu<mailto:timothy.thompson at yale.edu>> Cc: BaseX <basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de>> Subject: Re: [basex-talk] Write files in parallel? hi Tim - hope you are well. In the past (i.e. I don't remember exactly if this was perfectly parallel, it was just "parallel enough"), I have used something like the following for web requests: xquery:fork-join( for $xml in ('calq.xqm','factbook.xml','filesystem.xml','locations.xml','wiki1.zip', 'wiki2.zip','xmark.xml') let $url := 'https://files.basex.org/xml/' return fn() { file:write( '/tmp/fork-test/' || $xml, http:send-request( <http:request method='get'/>, $url || $xml ) ) }, map { 'parallel': '3'} ) Hopefully that's helpful (and apologies to the BaseX team's file server)! Best, Bridger ) ls -l --time-style=full-iso total 11640 -rw-r--r-- 1 bridger bridger 1593 2024-10-02 17:02:51.321251082 +0000 calq.xqm -rw-r--r-- 1 bridger bridger 1763070 2024-10-02 17:02:52.301261520 +0000 factbook.xml -rw-r--r-- 1 bridger bridger 2770290 2024-10-02 17:02:53.331272491 +0000 filesystem.xml -rw-r--r-- 1 bridger bridger 1566322 2024-10-02 17:02:52.497263608 +0000 locations.xml -rw-r--r-- 1 bridger bridger 512686 2024-10-02 17:02:52.670265451 +0000 wiki1.zip -rw-r--r-- 1 bridger bridger 5133340 2024-10-02 17:02:54.046280106 +0000 wiki2.zip -rw-r--r-- 1 bridger bridger 155448 2024-10-02 17:02:52.859267464 +0000 xmark.xml On Tue, Oct 1, 2024 at 5:32 PM Thompson, Timothy <timothy.thompson at yale.edu<mailto:timothy.thompson at yale.edu>> wrote: Hello, Is it possible to call file:write-text-lines in parallel inside a fork-join operation? I have multiple databases that I would like to run a query over, in parallel, and write the results as JSON Lines to a file per database. When I try this, it doesn’t seem to parallelize. Thanks in advance, Tim -- Tim A. Thompson (he, him) Librarian for Applied Metadata Research Interim Manager, Metadata Services Unit Yale University Library www.linkedin.com/in/timathompson<http://www.linkedin.com/in/timathompson>

> Thanks, Christian, for considering. Being able to call the file:write

On Mon, Oct 7, 2024 at 3:36 PM Christian Grün <christian.gruen at gmail.com> wrote: > > Thanks, Christian, for considering. Being able to call the file:write > functions in parallel would make a big difference! > > I have added an issue for this feature request [1]. > Best, Christian > > > [1] https://github.com/BaseXdb/basex/issues/2338 > > >