file:read-text-lines performance

On Tue, Jan 15, 2019 at 11:21 AM George Sofianos <gsf.greece at gmail.com> wrote: > > Hello, > > I'm trying to read a 4GB text file with 5 million lines and parse its contents. I'm using file:read-text-lines function to do that. I managed to use fork-join and use 16 CPU threads to read the whole file by reading 10000 lines in each iteration, but it still takes 500 seconds for parsing / analyzing the data. Using a profiler I can see that most of the time is wasted reading each line - method readline. I plan to make some changes on the code tonight and see if I can find a way to read it faster, but I thought I should also post it here in case you have any tips. I'm also very inexperienced with using profilers so I hope I read the output correctly :) > > Regards, > > George

On 1/15/19 12:43 PM, Christian Grün wrote: > What are your experiences with using a single thread? If memory > consumption is too exhaustive, you could play with the window clause > of the FLWOR expression [2,3]. It takes some time to explore the full > magic of this XQuery 3.0 extension (the syntax is somewhat verbose), > but it’s often a good alternative to complex functional code.

On Tue, Jan 15, 2019 at 12:16 PM George Sofianos <gsf.greece at gmail.com> wrote: > > Hi Christian, > > On 1/15/19 12:43 PM, Christian Grün wrote: > > What are your experiences with using a single thread? If memory > > consumption is too exhaustive, you could play with the window clause > > of the FLWOR expression [2,3]. It takes some time to explore the full > > magic of this XQuery 3.0 extension (the syntax is somewhat verbose), > > but it’s often a good alternative to complex functional code. > > Using a single thread looks to be OK too, about 10k lines per second, > and I'm not sure reading the same file with 16 threads (on SSD) is the > way to go from an I/O point of view. Searching on stackoverflow there > are many suggestions on how to read a file with one or multiple threads > e.g [1] > > I immediately return the data I need for each line (a small string for > example) so the memory consumption is low, I have provided 12GB but I > never see over 2-3GB of memory usage. My initial thoughts were that > maybe garbage collection was causing delays but after profiling BaseX I > don't think this is an issue. It's interesting to know about the window > function though, I will certainly find a use for it. While I know most > of these functions exist, I can always learn much more about a language. > Only yesterday I managed to use fork-join successfully and I think it > will save me a lot of time and effort for my use cases. I will post > again if I have any updates, thanks again, > > George. > > [1]: > https://stackoverflow.com/questions/40412008/how-to-read-a-file-using-multiple-threads-in-java-when-a-high-throughput3gb-s >

On 1/15/19 1:48 PM, Christian Grün wrote: > Hi George, > > I’m glad to announce that files are now processed in an iterative > manner [1,2]. That’s something I wanted to try a while ago, and your > mail was another motivation to get it done. > > It works pretty fine: I reduced the JVM memory to a tiny maximum of > 4mb, and I managed to count the line numbers of a file with several > gigabytes: > > count(file:read-text-lines('huge.txt')) > > I’d be interested to hear if your code runs faster with the latest snapshot. > Christian > > [1] http://files.basex.org/releases/latest/ > [2] https://github.com/BaseXdb/basex/commit/cfb7a7965de85139ec9595a6e79a45d873da7c25 >

On 1/15/19 1:48 PM, Christian Grün wrote: > Hi George, > > I’m glad to announce that files are now processed in an iterative > manner [1,2]. That’s something I wanted to try a while ago, and your > mail was another motivation to get it done. > > It works pretty fine: I reduced the JVM memory to a tiny maximum of > 4mb, and I managed to count the line numbers of a file with several > gigabytes: > > count(file:read-text-lines('huge.txt')) > > I’d be interested to hear if your code runs faster with the latest snapshot. > Christian > > [1] http://files.basex.org/releases/latest/ > [2] https://github.com/BaseXdb/basex/commit/cfb7a7965de85139ec9595a6e79a45d873da7c25 >

On 1/15/19 1:48 PM, Christian Grün wrote: > Hi George, > > I’m glad to announce that files are now processed in an iterative > manner [1,2]. That’s something I wanted to try a while ago, and your > mail was another motivation to get it done. > > It works pretty fine: I reduced the JVM memory to a tiny maximum of > 4mb, and I managed to count the line numbers of a file with several > gigabytes: > > count(file:read-text-lines('huge.txt')) > > I’d be interested to hear if your code runs faster with the latest snapshot. > Christian > > [1] http://files.basex.org/releases/latest/ > [2] https://github.com/BaseXdb/basex/commit/cfb7a7965de85139ec9595a6e79a45d873da7c25 >

> Using the offset mode looks to still be more memory efficient (stays around 1-1,3GB), but is very slow (both single core and multi core).

> let $data := file:read-text-lines($file, "UTF-8", false()) > let $count := count($data) > > let $all := > xquery:fork-join( > for $i in $data return function() { > parse-json($i)?('object1')?*?('object2')?('object3') > } > )

> 1) return count(file:read-text-lines($file, "UTF-8", false()))

> 2) let $data := file:read-text-lines($file, "UTF-8", false()) > return count($data)

On 1/16/19 1:41 PM, Christian Grün wrote: > The reason for that: file:read-text-lines is a non-deterministic > function. Each invocation might yield different results (as the file > contents may change in the background). This is different with > non-deterministic function calls, such as fn:doc('abc.xml'). If you > call such a function repreatedly, it will always access the same > document, which has been opened and parsed by the first call of this > function. > >> 1) return count(file:read-text-lines($file, "UTF-8", false())) > Here, file processing will be iterative. > >> 2) let $data := file:read-text-lines($file, "UTF-8", false()) >> return count($data) > The file contents will be bound to $data, and counted in a second > step. If the expression of your let clause was deterministic, the > variable would be inlined, and the resulting query plan would be > identical to the one of your first query.

> I can see there is a list with the deterministic functions in the specs [1] but not so sure about the BaseX specific functions. Is it possible to know if a function is deterministic or not?

> I tried file:read-text-lines("/path.txt") is file:read-text-lines("/path.txt") but it doesn't work.

On 1/16/19 1:37 PM, Christian Grün wrote: > This code will potentially create thousands or millions of Java > threads. Maybe you are getting better results by splitting your input > into 4 or 8 parts, and process each part in a dedicated function.

> I would indeed assume that the following code… > > distinct-values( > for $line in file:read-text-lines($file, "UTF-8", false()) > return parse-json($line)?('object1')?*?('object2')?('object3') > ) > > …will be most efficient, even if you process files of 100 GB or more > (especially with the new, iterative approach).