Performance of ft:search function
Wednesday, 27 April 2022 · 4 messages · last reply 30 Apr 2022
TT
Tim Thompson27 Apr 2022 22:16 #
Hello,
I have a largish (5.4G) file with a full-text index that I am using to reconcile names in a local dataset. I've been experimenting with splitting the file into many smaller index files to improve performance. I group the entries by initial character and create a new index file for each distinct initial character. Each smaller file then gets its own full-text index.
I've been following the approach outlined in the documentation for custom index structures
<https://docs.basex.org/wiki/Indexes#Custom_Index_Structures>. Using
prof:track, I've noticed the following performance for different uses of ft:search.
(Here, $db refers to the 5.4G file, and $index refers to a smaller 159MB subindex. Times are averaged across 10 runs of 1000 iterations for each expression.)
1. Direct lookup against large index
Time: 23ms
Expression: ft:search($db, $text)/../..
2. Direct lookup against subindex
Time: 3.3ms
Expression: ft:search($index, $text)/../..
3. Lookup against subindex file with reference to large index Time: 2.9ms
Expression:
let $s :=
ft:search($index, $text)/../..
return db:open-id($db, $s/id)/../..
My question is: why would the third expression be slightly faster (or at least not slower) than the second one, if it involves additional computation?
Thanks in advance,
Tim--
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
Yale University Library
CG
Christian Grün27 Apr 2022 23:09 #
Quoted text · Tim Thompson · 16 lines
> > 2. Direct lookup against subindex > Time: 3.3ms > Expression: ft:search($index, $text)/../.. > > 3. Lookup against subindex file with reference to large index > Time: 2.9ms > Expression: > let $s := > ft:search($index, $text)/../.. > return db:open-id($db, $s/id)/../.. > > My question is: why would the third expression be slightly faster (or at > least not slower) than the second one, if it involves additional > computation? >
I assume it's due to slight variations during your measurements. How many items will be returned by
ft:search? Do you get the same runtime if you run the code 100 or 1000 times?
In the GUI, you can type and execute SET RUNS 100 in the top input bar (in command mode). Your query will then be executed multiple times, and you will get shown the average runtime in the Info View.CG
Exactly: The longer you run a BaseX instance, the faster it gets. That’s particularly noticeable when using the client/server or HTTP architecture.
There are various reasons for that: BaseX caches, OS & main-memory caching, JIT optimizations, …
Quoted text · Tim Thompson · 49 lines
Tim Thompson <timathom at gmail.com> schrieb am Fr., 29. Apr. 2022, 22:40: > Oh, I see--thanks for the tip; I wasn't aware of the SET RUNS feature, > which is really helpful! With 1000 runs, the average execution time is more > in line with expectations: 38.96ms for expression #1 and 12.44ms for #2. > But I notice that with successive executions, #1 gets faster: 38.96ms, > 17.73ms, 12.82ms. Is this a result of caching? > > Best, > Tim > > > -- > Tim A. Thompson (he, him) > Librarian for Applied Metadata Research > Yale University Library > > > > On Wed, Apr 27, 2022 at 5:09 PM Christian Grün <christian.gruen at gmail.com> > wrote: > >> 2. Direct lookup against subindex >>> Time: 3.3ms >>> Expression: ft:search($index, $text)/../.. >>> >>> 3. Lookup against subindex file with reference to large index >>> Time: 2.9ms >>> Expression: >>> let $s := >>> ft:search($index, $text)/../.. >>> return db:open-id($db, $s/id)/../.. >>> >>> My question is: why would the third expression be slightly faster (or at >>> least not slower) than the second one, if it involves additional >>> computation? >>> >> >> I assume it's due to slight variations during your measurements. How many >> items will be returned by ft:search? Do you get the same runtime if you run >> the code 100 or 1000 times? >> >> In the GUI, you can type and execute SET RUNS 100 in the top input bar >> (in command mode). Your query will then be executed multiple times, and you >> will get shown the average runtime in the Info View. >> >> >> >> >>
TT
Oh, I see--thanks for the tip; I wasn't aware of the SET
RUNS feature, which is really helpful! With 1000 runs, the average execution time is more in line with expectations: 38.96ms for expression #1 and 12.44ms for #2. But I notice that with successive executions, #1 gets faster: 38.96ms, 17.73ms, 12.82ms. Is this a result of caching?
Best,
Tim--
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
Yale University Library
Quoted text · Christian Grün · 30 lines
On Wed, Apr 27, 2022 at 5:09 PM Christian Grün <christian.gruen at gmail.com> wrote: > 2. Direct lookup against subindex >> Time: 3.3ms >> Expression: ft:search($index, $text)/../.. >> >> 3. Lookup against subindex file with reference to large index >> Time: 2.9ms >> Expression: >> let $s := >> ft:search($index, $text)/../.. >> return db:open-id($db, $s/id)/../.. >> >> My question is: why would the third expression be slightly faster (or at >> least not slower) than the second one, if it involves additional >> computation? >> > > I assume it's due to slight variations during your measurements. How many > items will be returned by ft:search? Do you get the same runtime if you run > the code 100 or 1000 times? > > In the GUI, you can type and execute SET RUNS 100 in the top input bar (in > command mode). Your query will then be executed multiple times, and you > will get shown the average runtime in the Info View. > > > > >
Written for BaseX 9.7. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery