Full-Text

De : BaseX-Talk [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de ???????? ???????? Envoyé : jeudi 21 juin 2018 16:02 À : BaseX Objet : [basex-talk] Full-Text Hi, everyone! Is there any way to index only imported xml-files? Now, when I import xml-files the full-text index is deleted. After importing I recreate whole full-text index and it takes too much time :( -- С уважением, Ветошкин Владимир Владимирович

21.06.2018, 17:16, "Fabrice ETANCHAUD" <fetanchaud at pch.cerfrance.fr>: > Hi Vladimir, > > I don’t think there is something like a incremental full text index for the moment [1]. > As index is per collection, the recommanded way shall be to split your data in two collections : > - A big readonly collection of all the past updates, indexed once > - A small/medium sized collection whom full text index can be recreated in an acceptable time after each update. > At the end of a predefined time period, you have to add the live collection to the readonly one, reindex it, and truncate the live one. > > Best regards from France, > Fabrice Etanchaud > > [1] http://docs.basex.org/wiki/Indexes#Updates > > > > > De : BaseX-Talk [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de ???????? ???????? > Envoyé : jeudi 21 juin 2018 16:02 > À : BaseX > Objet : [basex-talk] Full-Text > > Hi, everyone! > > > > Is there any way to index only imported xml-files? > > Now, when I import xml-files the full-text index is deleted. > > After importing I recreate whole full-text index and it takes too much time :( > > > > -- > > С уважением, > > Ветошкин Владимир Владимирович > >

De : Ветошкин Владимир [mailto:en-trance at yandex.ru] Envoyé : lundi 25 juin 2018 09:42 À : Fabrice ETANCHAUD; BaseX Objet : Re: [basex-talk] Full-Text Hi, Fabrice! Thank you. All databases constantly change.That is why there is no way to single out "a big readonly collection" :( Maybe it is possible to use some other incremental indexes? I have to index specific xml-files, not all files in database. 21.06.2018, 17:16, "Fabrice ETANCHAUD" <fetanchaud at pch.cerfrance.fr>: Hi Vladimir, I don’t think there is something like a incremental full text index for the moment [1]. As index is per collection, the recommanded way shall be to split your data in two collections : - A big readonly collection of all the past updates, indexed once - A small/medium sized collection whom full text index can be recreated in an acceptable time after each update. At the end of a predefined time period, you have to add the live collection to the readonly one, reindex it, and truncate the live one. Best regards from France, Fabrice Etanchaud [1] http://docs.basex.org/wiki/Indexes#Updates De : BaseX-Talk [mailto:basex-talk-bounces at mailman.uni-konstanz.de<mailto:basex-talk-bounces at mailman.uni-konstanz.de>] De la part de ???????? ???????? Envoyé : jeudi 21 juin 2018 16:02 À : BaseX Objet : [basex-talk] Full-Text Hi, everyone! Is there any way to index only imported xml-files? Now, when I import xml-files the full-text index is deleted. After importing I recreate whole full-text index and it takes too much time :( -- С уважением, Ветошкин Владимир Владимирович -- С уважением, Ветошкин Владимир Владимирович

On Mon, Jun 25, 2018 at 10:42 AM Ветошкин Владимир <en-trance at yandex.ru> wrote: > Hi, Fabrice! > Thank you. > > All databases constantly change.That is why there is no way to single out > "a big readonly collection" :( > Maybe it is possible to use some other incremental indexes? > I have to index specific xml-files, not all files in database. > > 21.06.2018, 17:16, "Fabrice ETANCHAUD" <fetanchaud at pch.cerfrance.fr>: > > Hi Vladimir, > > > > I don’t think there is something like a incremental full text index for > the moment [1]. > > As index is per collection, the recommanded way shall be to split your > data in two collections : > > - A big readonly collection of all the past updates, indexed once > > - A small/medium sized collection whom full text index can be > recreated in an acceptable time after each update. > > At the end of a predefined time period, you have to add the live > collection to the readonly one, reindex it, and truncate the live one. > > > > Best regards from France, > > Fabrice Etanchaud > > > > [1] http://docs.basex.org/wiki/Indexes#Updates > > > > > > > > > > *De :* BaseX-Talk [mailto:basex-talk-bounces at mailman.uni-konstanz.de] *De > la part de* ???????? ???????? > *Envoyé :* jeudi 21 juin 2018 16:02 > *À :* BaseX > *Objet :* [basex-talk] Full-Text > > > > Hi, everyone! > > > > Is there any way to index only imported xml-files? > > Now, when I import xml-files the full-text index is deleted. > > After importing I recreate whole full-text index and it takes too much > time :( > > > > -- > > С уважением, > > Ветошкин Владимир Владимирович > > > > > > -- > С уважением, > Ветошкин Владимир Владимирович > >

25.06.2018, 11:10, "Fabrice ETANCHAUD" <fetanchaud at pch.cerfrance.fr>: > Hi Vladimir, > > So what about storing those specific files in an ad-hoc collection, and only index that one ? > Is there a partition pattern you could use to split these specific files in several collections of medium size ? > > If you do not need fulltext capabilities, you can use incremental text() or attribute indices. > Do you need to index only a specific set of text() or attributes ? If so, you can also specify a list of element or attribute names to be indexed, and by the way reduce the time needed to reindex. > > Best regards, > Fabrice > > De : Ветошкин Владимир [mailto:en-trance at yandex.ru] > Envoyé : lundi 25 juin 2018 09:42 > À : Fabrice ETANCHAUD; BaseX > Objet : Re: [basex-talk] Full-Text > > Hi, Fabrice! > > Thank you. > > > > All databases constantly change.That is why there is no way to single out "a big readonly collection" :( > > Maybe it is possible to use some other incremental indexes? > > I have to index specific xml-files, not all files in database. > > > > 21.06.2018, 17:16, "Fabrice ETANCHAUD" <fetanchaud at pch.cerfrance.fr>: > > > > Hi Vladimir, > > > > I don’t think there is something like a incremental full text index for the moment [1]. > > As index is per collection, the recommanded way shall be to split your data in two collections : > > - A big readonly collection of all the past updates, indexed once > > - A small/medium sized collection whom full text index can be recreated in an acceptable time after each update. > > At the end of a predefined time period, you have to add the live collection to the readonly one, reindex it, and truncate the live one. > > > > Best regards from France, > > Fabrice Etanchaud > > > > [1] http://docs.basex.org/wiki/Indexes#Updates > > > > > > > > > > De : BaseX-Talk [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de ???????? ???????? > > Envoyé : jeudi 21 juin 2018 16:02 > > À : BaseX > > Objet : [basex-talk] Full-Text > > > > Hi, everyone! > > > > > > > > Is there any way to index only imported xml-files? > > > > Now, when I import xml-files the full-text index is deleted. > > > > After importing I recreate whole full-text index and it takes too much time :( > > > > > > > > -- > > > > С уважением, > > > > Ветошкин Владимир Владимирович > > > > > > > > > -- > > С уважением, > > Ветошкин Владимир Владимирович > >

25.06.2018, 11:25, "Alexander Shpack" <shadowkin at gmail.com>: > Hey, Vladimir! > > Just put this specific files to the separated DB and than index it. > You can process it automatically, BaseX allows to create and index DB right from XQuery. > I hope it helps you. Anyhow, you can provide more details about your task and we can figure out the best solution for you. > > > On Mon, Jun 25, 2018 at 10:42 AM Ветошкин Владимир <en-trance at yandex.ru> wrote: > > > Hi, Fabrice! > > Thank you. > > > > All databases constantly change.That is why there is no way to single out "a big readonly collection" :( > > Maybe it is possible to use some other incremental indexes? > > I have to index specific xml-files, not all files in database. > > > > 21.06.2018, 17:16, "Fabrice ETANCHAUD" <fetanchaud at pch.cerfrance.fr>: > > > > > Hi Vladimir, > > > > > > I don’t think there is something like a incremental full text index for the moment [1]. > > > As index is per collection, the recommanded way shall be to split your data in two collections : > > > - A big readonly collection of all the past updates, indexed once > > > - A small/medium sized collection whom full text index can be recreated in an acceptable time after each update. > > > At the end of a predefined time period, you have to add the live collection to the readonly one, reindex it, and truncate the live one. > > > > > > Best regards from France, > > > Fabrice Etanchaud > > > > > > [1] http://docs.basex.org/wiki/Indexes#Updates > > > > > > > > > > > > > > > De : BaseX-Talk [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de ???????? ???????? > > > Envoyé : jeudi 21 juin 2018 16:02 > > > À : BaseX > > > Objet : [basex-talk] Full-Text > > > > > > Hi, everyone! > > > > > > > > > > > > Is there any way to index only imported xml-files? > > > > > > Now, when I import xml-files the full-text index is deleted. > > > > > > After importing I recreate whole full-text index and it takes too much time :( > > > > > > > > > > > > -- > > > > > > С уважением, > > > > > > Ветошкин Владимир Владимирович > > > > > > > > > > > > -- > > С уважением, > > Ветошкин Владимир Владимирович > > > > > --s0rr0w

On Mon, Jun 25, 2018 at 11:50 AM Ветошкин Владимир <en-trance at yandex.ru> wrote: > Hi, Alexander! > Thank you! > > In my previous letter I have described the proccess in short. > I'll think about separated DB. But I'm afraid that this base will also be > very big in future. > Although I can try to split data to several databases - one per year.. > Hmm.. > > 25.06.2018, 11:25, "Alexander Shpack" <shadowkin at gmail.com>: > > Hey, Vladimir! > > Just put this specific files to the separated DB and than index it. > You can process it automatically, BaseX allows to create and index DB > right from XQuery. > > I hope it helps you. Anyhow, you can provide more details about your task > and we can figure out the best solution for you. > > > > On Mon, Jun 25, 2018 at 10:42 AM Ветошкин Владимир <en-trance at yandex.ru> > wrote: > > Hi, Fabrice! > Thank you. > > All databases constantly change.That is why there is no way to single out > "a big readonly collection" :( > Maybe it is possible to use some other incremental indexes? > I have to index specific xml-files, not all files in database. > > 21.06.2018, 17:16, "Fabrice ETANCHAUD" <fetanchaud at pch.cerfrance.fr>: > > Hi Vladimir, > > > > I don’t think there is something like a incremental full text index for > the moment [1]. > > As index is per collection, the recommanded way shall be to split your > data in two collections : > > - A big readonly collection of all the past updates, indexed once > > - A small/medium sized collection whom full text index can be > recreated in an acceptable time after each update. > > At the end of a predefined time period, you have to add the live > collection to the readonly one, reindex it, and truncate the live one. > > > > Best regards from France, > > Fabrice Etanchaud > > > > [1] http://docs.basex.org/wiki/Indexes#Updates > > > > > > > > > > *De :* BaseX-Talk [mailto:basex-talk-bounces at mailman.uni-konstanz.de] *De > la part de* ???????? ???????? > *Envoyé :* jeudi 21 juin 2018 16:02 > *À :* BaseX > *Objet :* [basex-talk] Full-Text > > > > Hi, everyone! > > > > Is there any way to index only imported xml-files? > > Now, when I import xml-files the full-text index is deleted. > > After importing I recreate whole full-text index and it takes too much > time :( > > > > -- > > С уважением, > > Ветошкин Владимир Владимирович > > > > > > -- > С уважением, > Ветошкин Владимир Владимирович > > > > > -- > s0rr0w > > > > -- > С уважением, > Ветошкин Владимир Владимирович > >

25.06.2018, 11:56, "Alexander Shpack" <shadowkin at gmail.com>: > Hey Vladimir, > > You can use sharding approach for you data import and split all DBs even every month. > > On Mon, Jun 25, 2018 at 11:50 AM Ветошкин Владимир <en-trance at yandex.ru> wrote: > > > Hi, Alexander! > > Thank you! > > > > In my previous letter I have described the proccess in short. > > I'll think about separated DB. But I'm afraid that this base will also be very big in future. > > Although I can try to split data to several databases - one per year.. Hmm.. > > > > 25.06.2018, 11:25, "Alexander Shpack" <shadowkin at gmail.com>: > > > > > Hey, Vladimir! > > > > > > Just put this specific files to the separated DB and than index it. > > > You can process it automatically, BaseX allows to create and index DB right from XQuery. > > > I hope it helps you. Anyhow, you can provide more details about your task and we can figure out the best solution for you. > > > > > > > > > On Mon, Jun 25, 2018 at 10:42 AM Ветошкин Владимир <en-trance at yandex.ru> wrote: > > > > > > > Hi, Fabrice! > > > > Thank you. > > > > > > > > All databases constantly change.That is why there is no way to single out "a big readonly collection" :( > > > > Maybe it is possible to use some other incremental indexes? > > > > I have to index specific xml-files, not all files in database. > > > > > > > > 21.06.2018, 17:16, "Fabrice ETANCHAUD" <fetanchaud at pch.cerfrance.fr>: > > > > > > > > > Hi Vladimir, > > > > > > > > > > I don’t think there is something like a incremental full text index for the moment [1]. > > > > > As index is per collection, the recommanded way shall be to split your data in two collections : > > > > > - A big readonly collection of all the past updates, indexed once > > > > > - A small/medium sized collection whom full text index can be recreated in an acceptable time after each update. > > > > > At the end of a predefined time period, you have to add the live collection to the readonly one, reindex it, and truncate the live one. > > > > > > > > > > Best regards from France, > > > > > Fabrice Etanchaud > > > > > > > > > > [1] http://docs.basex.org/wiki/Indexes#Updates > > > > > > > > > > > > > > > > > > > > > > > > > De : BaseX-Talk [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de ???????? ???????? > > > > > Envoyé : jeudi 21 juin 2018 16:02 > > > > > À : BaseX > > > > > Objet : [basex-talk] Full-Text > > > > > > > > > > Hi, everyone! > > > > > > > > > > > > > > > > > > > > Is there any way to index only imported xml-files? > > > > > > > > > > Now, when I import xml-files the full-text index is deleted. > > > > > > > > > > After importing I recreate whole full-text index and it takes too much time :( > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > С уважением, > > > > > > > > > > Ветошкин Владимир Владимирович > > > > > > > > > > > > > > > > > > > > > > -- > > > > С уважением, > > > > Ветошкин Владимир Владимирович > > > > > > > > > > > > > --s0rr0w > > > > > > -- > > С уважением, > > Ветошкин Владимир Владимирович > > > > > --s0rr0w

On Mon, Jun 25, 2018 at 12:32 PM Ветошкин Владимир <en-trance at yandex.ru> wrote: > Hi, Alexander, > > Some questions: > After that, how can I perform a search in all of these databases? > Can I search for substring without fulltext using only text index? > > 25.06.2018, 11:56, "Alexander Shpack" <shadowkin at gmail.com>: > > Hey Vladimir, > > You can use sharding approach for you data import and split all DBs even > every month. > > > > On Mon, Jun 25, 2018 at 11:50 AM Ветошкин Владимир <en-trance at yandex.ru> > wrote: > > Hi, Alexander! > Thank you! > > In my previous letter I have described the proccess in short. > I'll think about separated DB. But I'm afraid that this base will also be > very big in future. > Although I can try to split data to several databases - one per year.. > Hmm.. > > 25.06.2018, 11:25, "Alexander Shpack" <shadowkin at gmail.com>: > > Hey, Vladimir! > > Just put this specific files to the separated DB and than index it. > You can process it automatically, BaseX allows to create and index DB > right from XQuery. > > I hope it helps you. Anyhow, you can provide more details about your task > and we can figure out the best solution for you. > > > > On Mon, Jun 25, 2018 at 10:42 AM Ветошкин Владимир <en-trance at yandex.ru> > wrote: > > Hi, Fabrice! > Thank you. > > All databases constantly change.That is why there is no way to single out > "a big readonly collection" :( > Maybe it is possible to use some other incremental indexes? > I have to index specific xml-files, not all files in database. > > 21.06.2018, 17:16, "Fabrice ETANCHAUD" <fetanchaud at pch.cerfrance.fr>: > > Hi Vladimir, > > > > I don’t think there is something like a incremental full text index for > the moment [1]. > > As index is per collection, the recommanded way shall be to split your > data in two collections : > > - A big readonly collection of all the past updates, indexed once > > - A small/medium sized collection whom full text index can be > recreated in an acceptable time after each update. > > At the end of a predefined time period, you have to add the live > collection to the readonly one, reindex it, and truncate the live one. > > > > Best regards from France, > > Fabrice Etanchaud > > > > [1] http://docs.basex.org/wiki/Indexes#Updates > > > > > > > > > > *De :* BaseX-Talk [mailto:basex-talk-bounces at mailman.uni-konstanz.de] *De > la part de* ???????? ???????? > *Envoyé :* jeudi 21 juin 2018 16:02 > *À :* BaseX > *Objet :* [basex-talk] Full-Text > > > > Hi, everyone! > > > > Is there any way to index only imported xml-files? > > Now, when I import xml-files the full-text index is deleted. > > After importing I recreate whole full-text index and it takes too much > time :( > > > > -- > > С уважением, > > Ветошкин Владимир Владимирович > > > > > > -- > С уважением, > Ветошкин Владимир Владимирович > > > > > -- > s0rr0w > > > > -- > С уважением, > Ветошкин Владимир Владимирович > > > > > -- > s0rr0w > > > > -- > С уважением, > Ветошкин Владимир Владимирович > >

25.06.2018, 13:07, "Alexander Shpack" <shadowkin at gmail.com>: > Hi, Vladimir, > > If you will do db names with the particular prefix, for example "db_", you may use the next code > > let $docs := for $i in db:list()[starts-with(.,"db_")] return db:open($i)return $docs/* > > > On Mon, Jun 25, 2018 at 12:32 PM Ветошкин Владимир <en-trance at yandex.ru> wrote: > > > Hi, Alexander, > > > > Some questions: > > After that, how can I perform a search in all of these databases? > > Can I search for substring without fulltext using only text index? > > > > 25.06.2018, 11:56, "Alexander Shpack" <shadowkin at gmail.com>: > > > > > Hey Vladimir, > > > > > > You can use sharding approach for you data import and split all DBs even every month. > > > > > > On Mon, Jun 25, 2018 at 11:50 AM Ветошкин Владимир <en-trance at yandex.ru> wrote: > > > > > > > Hi, Alexander! > > > > Thank you! > > > > > > > > In my previous letter I have described the proccess in short. > > > > I'll think about separated DB. But I'm afraid that this base will also be very big in future. > > > > Although I can try to split data to several databases - one per year.. Hmm.. > > > > > > > > 25.06.2018, 11:25, "Alexander Shpack" <shadowkin at gmail.com>: > > > > > > > > > Hey, Vladimir! > > > > > > > > > > Just put this specific files to the separated DB and than index it. > > > > > You can process it automatically, BaseX allows to create and index DB right from XQuery. > > > > > I hope it helps you. Anyhow, you can provide more details about your task and we can figure out the best solution for you. > > > > > > > > > > > > > > > On Mon, Jun 25, 2018 at 10:42 AM Ветошкин Владимир <en-trance at yandex.ru> wrote: > > > > > > > > > > > Hi, Fabrice! > > > > > > Thank you. > > > > > > > > > > > > All databases constantly change.That is why there is no way to single out "a big readonly collection" :( > > > > > > Maybe it is possible to use some other incremental indexes? > > > > > > I have to index specific xml-files, not all files in database. > > > > > > > > > > > > 21.06.2018, 17:16, "Fabrice ETANCHAUD" <fetanchaud at pch.cerfrance.fr>: > > > > > > > > > > > > > Hi Vladimir, > > > > > > > > > > > > > > I don’t think there is something like a incremental full text index for the moment [1]. > > > > > > > As index is per collection, the recommanded way shall be to split your data in two collections : > > > > > > > - A big readonly collection of all the past updates, indexed once > > > > > > > - A small/medium sized collection whom full text index can be recreated in an acceptable time after each update. > > > > > > > At the end of a predefined time period, you have to add the live collection to the readonly one, reindex it, and truncate the live one. > > > > > > > > > > > > > > Best regards from France, > > > > > > > Fabrice Etanchaud > > > > > > > > > > > > > > [1] http://docs.basex.org/wiki/Indexes#Updates > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > De : BaseX-Talk [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de ???????? ???????? > > > > > > > Envoyé : jeudi 21 juin 2018 16:02 > > > > > > > À : BaseX > > > > > > > Objet : [basex-talk] Full-Text > > > > > > > > > > > > > > Hi, everyone! > > > > > > > > > > > > > > > > > > > > > > > > > > > > Is there any way to index only imported xml-files? > > > > > > > > > > > > > > Now, when I import xml-files the full-text index is deleted. > > > > > > > > > > > > > > After importing I recreate whole full-text index and it takes too much time :( > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > С уважением, > > > > > > > > > > > > > > Ветошкин Владимир Владимирович > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > С уважением, > > > > > > Ветошкин Владимир Владимирович > > > > > > > > > > > > > > > > > > > > > --s0rr0w > > > > > > > > > > > > -- > > > > С уважением, > > > > Ветошкин Владимир Владимирович > > > > > > > > > > > > > --s0rr0w > > > > > > -- > > С уважением, > > Ветошкин Владимир Владимирович > > > > > --s0rr0w

25.06.2018, 13:07, "Alexander Shpack" <shadowkin at gmail.com>: > Hi, Vladimir, > > If you will do db names with the particular prefix, for example "db_", you may use the next code > > let $docs := for $i in db:list()[starts-with(.,"db_")] return db:open($i)return $docs/* > > > On Mon, Jun 25, 2018 at 12:32 PM Ветошкин Владимир <en-trance at yandex.ru> wrote: > > > Hi, Alexander, > > > > Some questions: > > After that, how can I perform a search in all of these databases? > > Can I search for substring without fulltext using only text index? > > > > 25.06.2018, 11:56, "Alexander Shpack" <shadowkin at gmail.com>: > > > > > Hey Vladimir, > > > > > > You can use sharding approach for you data import and split all DBs even every month. > > > > > > On Mon, Jun 25, 2018 at 11:50 AM Ветошкин Владимир <en-trance at yandex.ru> wrote: > > > > > > > Hi, Alexander! > > > > Thank you! > > > > > > > > In my previous letter I have described the proccess in short. > > > > I'll think about separated DB. But I'm afraid that this base will also be very big in future. > > > > Although I can try to split data to several databases - one per year.. Hmm.. > > > > > > > > 25.06.2018, 11:25, "Alexander Shpack" <shadowkin at gmail.com>: > > > > > > > > > Hey, Vladimir! > > > > > > > > > > Just put this specific files to the separated DB and than index it. > > > > > You can process it automatically, BaseX allows to create and index DB right from XQuery. > > > > > I hope it helps you. Anyhow, you can provide more details about your task and we can figure out the best solution for you. > > > > > > > > > > > > > > > On Mon, Jun 25, 2018 at 10:42 AM Ветошкин Владимир <en-trance at yandex.ru> wrote: > > > > > > > > > > > Hi, Fabrice! > > > > > > Thank you. > > > > > > > > > > > > All databases constantly change.That is why there is no way to single out "a big readonly collection" :( > > > > > > Maybe it is possible to use some other incremental indexes? > > > > > > I have to index specific xml-files, not all files in database. > > > > > > > > > > > > 21.06.2018, 17:16, "Fabrice ETANCHAUD" <fetanchaud at pch.cerfrance.fr>: > > > > > > > > > > > > > Hi Vladimir, > > > > > > > > > > > > > > I don’t think there is something like a incremental full text index for the moment [1]. > > > > > > > As index is per collection, the recommanded way shall be to split your data in two collections : > > > > > > > - A big readonly collection of all the past updates, indexed once > > > > > > > - A small/medium sized collection whom full text index can be recreated in an acceptable time after each update. > > > > > > > At the end of a predefined time period, you have to add the live collection to the readonly one, reindex it, and truncate the live one. > > > > > > > > > > > > > > Best regards from France, > > > > > > > Fabrice Etanchaud > > > > > > > > > > > > > > [1] http://docs.basex.org/wiki/Indexes#Updates > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > De : BaseX-Talk [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de ???????? ???????? > > > > > > > Envoyé : jeudi 21 juin 2018 16:02 > > > > > > > À : BaseX > > > > > > > Objet : [basex-talk] Full-Text > > > > > > > > > > > > > > Hi, everyone! > > > > > > > > > > > > > > > > > > > > > > > > > > > > Is there any way to index only imported xml-files? > > > > > > > > > > > > > > Now, when I import xml-files the full-text index is deleted. > > > > > > > > > > > > > > After importing I recreate whole full-text index and it takes too much time :( > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > С уважением, > > > > > > > > > > > > > > Ветошкин Владимир Владимирович > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > С уважением, > > > > > > Ветошкин Владимир Владимирович > > > > > > > > > > > > > > > > > > > > > --s0rr0w > > > > > > > > > > > > -- > > > > С уважением, > > > > Ветошкин Владимир Владимирович > > > > > > > > > > > > > --s0rr0w > > > > > > -- > > С уважением, > > Ветошкин Владимир Владимирович > > > > > --s0rr0w

> But if I search in "db:list...db:open..." - it takes about 12-15 seconds.

> Example takes ~12-15s: > let $db := for $i in db:list()[starts-with(.,'000999~')] return try {db:open($i)} catch * {} > for $doc in $db/.//*[text() contains text { 'TEN-9258' } any] > return $doc > > Example takes ~180ms (returns 2 rows): > let $db := for $i in db:list()[starts-with(.,'000999~201807')] return db:open($i) > for $doc in $db/.//*[text() contains text { 'TEN-9258' } any] > return $doc > > Example takes ~10ms (returns 2 rows): > for $doc in db:open('000999~201807')/.//*[text() contains text { 'TEN-9258' } any] > return $doc > > Why do the last 2 examples take different times? > How can I improve this? > > Example takes ~2s (returns 0 rows): > let $db := for $i in db:list()[starts-with(.,'000999~201806')] return db:open($i) > for $doc in $db/.//*[text() contains text { 'TEN-9258' } any] > return $doc > > Example takes ~12ms (returns 0 rows): > for $doc in db:open('000999~201806')/.//*[text() contains text { 'TEN-9258' } any] > return $doc > > > 25.06.2018, 13:07, "Alexander Shpack" <shadowkin at gmail.com>: > > Hi, Vladimir, > > If you will do db names with the particular prefix, for example "db_", you may use the next code > > let $docs := for $i in db:list()[starts-with(.,"db_")] return db:open($i)return $docs/* > > > > On Mon, Jun 25, 2018 at 12:32 PM Ветошкин Владимир <en-trance at yandex.ru> wrote: > > Hi, Alexander, > > Some questions: > After that, how can I perform a search in all of these databases? > Can I search for substring without fulltext using only text index? > > 25.06.2018, 11:56, "Alexander Shpack" <shadowkin at gmail.com>: > > Hey Vladimir, > > You can use sharding approach for you data import and split all DBs even every month. > > > > On Mon, Jun 25, 2018 at 11:50 AM Ветошкин Владимир <en-trance at yandex.ru> wrote: > > Hi, Alexander! > Thank you! > > In my previous letter I have described the proccess in short. > I'll think about separated DB. But I'm afraid that this base will also be very big in future. > Although I can try to split data to several databases - one per year.. Hmm.. > > 25.06.2018, 11:25, "Alexander Shpack" <shadowkin at gmail.com>: > > Hey, Vladimir! > > Just put this specific files to the separated DB and than index it. > You can process it automatically, BaseX allows to create and index DB right from XQuery. > > I hope it helps you. Anyhow, you can provide more details about your task and we can figure out the best solution for you. > > > > On Mon, Jun 25, 2018 at 10:42 AM Ветошкин Владимир <en-trance at yandex.ru> wrote: > > Hi, Fabrice! > Thank you. > > All databases constantly change.That is why there is no way to single out "a big readonly collection" :( > Maybe it is possible to use some other incremental indexes? > I have to index specific xml-files, not all files in database. > > 21.06.2018, 17:16, "Fabrice ETANCHAUD" <fetanchaud at pch.cerfrance.fr>: > > Hi Vladimir, > > > > I don’t think there is something like a incremental full text index for the moment [1]. > > As index is per collection, the recommanded way shall be to split your data in two collections : > > - A big readonly collection of all the past updates, indexed once > > - A small/medium sized collection whom full text index can be recreated in an acceptable time after each update. > > At the end of a predefined time period, you have to add the live collection to the readonly one, reindex it, and truncate the live one. > > > > Best regards from France, > > Fabrice Etanchaud > > > > [1] http://docs.basex.org/wiki/Indexes#Updates > > > > > > > > > > De : BaseX-Talk [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de ???????? ???????? > Envoyé : jeudi 21 juin 2018 16:02 > À : BaseX > Objet : [basex-talk] Full-Text > > > > Hi, everyone! > > > > Is there any way to index only imported xml-files? > > Now, when I import xml-files the full-text index is deleted. > > After importing I recreate whole full-text index and it takes too much time :( > > > > -- > > С уважением, > > Ветошкин Владимир Владимирович > > > > > > -- > С уважением, > Ветошкин Владимир Владимирович > > > > > -- > s0rr0w > > > > -- > С уважением, > Ветошкин Владимир Владимирович > > > > > -- > s0rr0w > > > > -- > С уважением, > Ветошкин Владимир Владимирович > > > > > -- > s0rr0w > > > > -- > С уважением, > Ветошкин Владимир Владимирович >

20.07.2018, 11:31, "Christian Grün" <christian.gruen at gmail.com>: > Hi Vladimir, > > > > But if I search in "db:list...db:open..." - it takes about 12-15 seconds. > > If the name of the database is not statically known, the query cannot > be rewritten for index access (because one the targeted database may > not have the required index). I guess you have the full-text index > enabled? > > However, since BaseX 9, you can take advantage of the ENFORCEINDEX > option: All queries will then optimized for index operations, based on > your knowledge that there an index will be available. See [1] for > further details. > > By the way, you can have a look at the compilation section of the Info > View in the GUI to see if indexes will be applied in your query. > > Best, > Christian > > [1] http://docs.basex.org/wiki/Indexes#Enforce_Rewritings > > > > > Example takes ~12-15s: > > let $db := for $i in db:list()[starts-with(.,'000999~')] return try {db:open($i)} catch * {} > > for $doc in $db/.//*[text() contains text { 'TEN-9258' } any] > > return $doc > > > > Example takes ~180ms (returns 2 rows): > > let $db := for $i in db:list()[starts-with(.,'000999~201807')] return db:open($i) > > for $doc in $db/.//*[text() contains text { 'TEN-9258' } any] > > return $doc > > > > Example takes ~10ms (returns 2 rows): > > for $doc in db:open('000999~201807')/.//*[text() contains text { 'TEN-9258' } any] > > return $doc > > > > Why do the last 2 examples take different times? > > How can I improve this? > > > > Example takes ~2s (returns 0 rows): > > let $db := for $i in db:list()[starts-with(.,'000999~201806')] return db:open($i) > > for $doc in $db/.//*[text() contains text { 'TEN-9258' } any] > > return $doc > > > > Example takes ~12ms (returns 0 rows): > > for $doc in db:open('000999~201806')/.//*[text() contains text { 'TEN-9258' } any] > > return $doc > > > > > > 25.06.2018, 13:07, "Alexander Shpack" <shadowkin at gmail.com>: > > > > Hi, Vladimir, > > > > If you will do db names with the particular prefix, for example "db_", you may use the next code > > > > let $docs := for $i in db:list()[starts-with(.,"db_")] return db:open($i)return $docs/* > > > > > > > > On Mon, Jun 25, 2018 at 12:32 PM Ветошкин Владимир <en-trance at yandex.ru> wrote: > > > > Hi, Alexander, > > > > Some questions: > > After that, how can I perform a search in all of these databases? > > Can I search for substring without fulltext using only text index? > > > > 25.06.2018, 11:56, "Alexander Shpack" <shadowkin at gmail.com>: > > > > Hey Vladimir, > > > > You can use sharding approach for you data import and split all DBs even every month. > > > > > > > > On Mon, Jun 25, 2018 at 11:50 AM Ветошкин Владимир <en-trance at yandex.ru> wrote: > > > > Hi, Alexander! > > Thank you! > > > > In my previous letter I have described the proccess in short. > > I'll think about separated DB. But I'm afraid that this base will also be very big in future. > > Although I can try to split data to several databases - one per year.. Hmm.. > > > > 25.06.2018, 11:25, "Alexander Shpack" <shadowkin at gmail.com>: > > > > Hey, Vladimir! > > > > Just put this specific files to the separated DB and than index it. > > You can process it automatically, BaseX allows to create and index DB right from XQuery. > > > > I hope it helps you. Anyhow, you can provide more details about your task and we can figure out the best solution for you. > > > > > > > > On Mon, Jun 25, 2018 at 10:42 AM Ветошкин Владимир <en-trance at yandex.ru> wrote: > > > > Hi, Fabrice! > > Thank you. > > > > All databases constantly change.That is why there is no way to single out "a big readonly collection" :( > > Maybe it is possible to use some other incremental indexes? > > I have to index specific xml-files, not all files in database. > > > > 21.06.2018, 17:16, "Fabrice ETANCHAUD" <fetanchaud at pch.cerfrance.fr>: > > > > Hi Vladimir, > > > > > > > > I don’t think there is something like a incremental full text index for the moment [1]. > > > > As index is per collection, the recommanded way shall be to split your data in two collections : > > > > - A big readonly collection of all the past updates, indexed once > > > > - A small/medium sized collection whom full text index can be recreated in an acceptable time after each update. > > > > At the end of a predefined time period, you have to add the live collection to the readonly one, reindex it, and truncate the live one. > > > > > > > > Best regards from France, > > > > Fabrice Etanchaud > > > > > > > > [1] http://docs.basex.org/wiki/Indexes#Updates > > > > > > > > > > > > > > > > > > > > De : BaseX-Talk [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de ???????? ???????? > > Envoyé : jeudi 21 juin 2018 16:02 > > À : BaseX > > Objet : [basex-talk] Full-Text > > > > > > > > Hi, everyone! > > > > > > > > Is there any way to index only imported xml-files? > > > > Now, when I import xml-files the full-text index is deleted. > > > > After importing I recreate whole full-text index and it takes too much time :( > > > > > > > > -- > > > > С уважением, > > > > Ветошкин Владимир Владимирович > > > > > > > > > > > > -- > > С уважением, > > Ветошкин Владимир Владимирович > > > > > > > > > > -- > > s0rr0w > > > > > > > > -- > > С уважением, > > Ветошкин Владимир Владимирович > > > > > > > > > > -- > > s0rr0w > > > > > > > > -- > > С уважением, > > Ветошкин Владимир Владимирович > > > > > > > > > > -- > > s0rr0w > > > > > > > > -- > > С уважением, > > Ветошкин Владимир Владимирович > >

> let $dbs := for $i in db:list()[starts-with(.,'000999~')] return $i > for $db in $dbs > for $doc in db:open($db)/.//*[(# db:enforceindex #) { text() contains text { 'TEN-9258' } any }] > return $doc

20.07.2018, 12:29, "Christian Grün" <christian.gruen at gmail.com>: > > let $dbs := for $i in db:list()[starts-with(.,'000999~')] return $i > > for $db in $dbs > > for $doc in db:open($db)/.//*[(# db:enforceindex #) { text() contains text { 'TEN-9258' } any }] > > return $doc > > Maybe you’ll have to use the pragma on top of your expression: > > (# db:enforceindex #) { > for $db in db:list()[starts-with(.,'000999~')] > return db:open($db)//*[text() contains text 'TEN-9258'] > } > > As you may have seen, I have simplified `/.//*` to `//*` (but maybe it > was optimized to this representation anyway). > > Cheers, > Christian

> These examples work differently.

> 1. > (# db:enforceindex #) { > for $db in db:list()[starts-with(.,'000999~')] > return db:open($db)//*[text() contains text { 'болт' } any] > } > 378 rows > > let $dbs := for $i in db:list()[starts-with(.,'000999~')] return $i > for $db in $dbs > let $ft := ft:search($db, "болт")/parent::* > for $node in $ft > return $node > 378 rows > > 2. > (# db:enforceindex #) { > for $db in db:list()[starts-with(.,'000999~')] > return db:open($db)//*[text() contains text { 'автомобиль' } any] > } > 0 rows > > let $dbs := for $i in db:list()[starts-with(.,'000999~')] return $i > for $db in $dbs > let $ft := ft:search($db, "автомобиль")/parent::* > for $node in $ft > return $node > 2138 rows > > > Why do they work differently?

20.07.2018, 12:47, "Christian Grün" <christian.gruen at gmail.com>: > > These examples work differently. > > So If I read this correctly, the number of results for 1. is still > identical, right? However, in the second query in 2., no results are > returned. You could report the query plan will give us some insight > into what happens here. > > > > > 1. > > (# db:enforceindex #) { > > for $db in db:list()[starts-with(.,'000999~')] > > return db:open($db)//*[text() contains text { 'болт' } any] > > } > > 378 rows > > > > let $dbs := for $i in db:list()[starts-with(.,'000999~')] return $i > > for $db in $dbs > > let $ft := ft:search($db, "болт")/parent::* > > for $node in $ft > > return $node > > 378 rows > > > > 2. > > (# db:enforceindex #) { > > for $db in db:list()[starts-with(.,'000999~')] > > return db:open($db)//*[text() contains text { 'автомобиль' } any] > > } > > 0 rows > > > > let $dbs := for $i in db:list()[starts-with(.,'000999~')] return $i > > for $db in $dbs > > let $ft := ft:search($db, "автомобиль")/parent::* > > for $node in $ft > > return $node > > 2138 rows > > > > > > Why do they work differently?

On Fri, Jul 20, 2018 at 11:54 AM Ветошкин Владимир <en-trance at yandex.ru> wrote: > > Query plan (0 rows): > > <QueryPlan compiled="true" updating="false"> > <Extension type="element()*"> > <DBPragma value=""> > <QNm type="xs:QName">db:enforceindex</QNm> > </DBPragma> > <GFLWOR type="element()*"> > <For type="xs:string" size="1"> > <Var name="$db" id="0" type="xs:string"/> > <IterFilter type="xs:string*"> > <DbList name="list([database[,path]])" type="xs:string*"/> > <FnStartsWith name="starts-with(string,sub[,collation])" type="xs:boolean" size="1"> > <ContextValue type="xs:string" size="1"/> > <Str type="xs:string">000999~</Str> > </FnStartsWith> > </IterFilter> > </For> > <CachedPath type="element()*"> > <FTIndexAccess type="text()*"> > <IndexDynDb> > <DbOpen name="open(database[,path])" type="document-node()*"> > <VarRef type="xs:string" size="1"> > <Var name="$db" id="0" type="xs:string"/> > </VarRef> > </DbOpen> > </IndexDynDb> > <FTWords type="xs:boolean" size="1"> > <Str type="xs:string">автомобиль</Str> > </FTWords> > </FTIndexAccess> > <IterStep axis="parent" test="*" type="element()*"/> > </CachedPath> > </GFLWOR> > </Extension> > </QueryPlan> > > > > And query plan (2138 rows): > > <QueryPlan compiled="true" updating="false"> > <GFLWOR type="element()*"> > <Let type="xs:string*"> > <Var name="$dbs" id="0" type="xs:string*"/> > <IterFilter type="xs:string*"> > <DbList name="list([database[,path]])" type="xs:string*"/> > <FnStartsWith name="starts-with(string,sub[,collation])" type="xs:boolean" size="1"> > <ContextValue type="xs:string" size="1"/> > <Str type="xs:string">000999~</Str> > </FnStartsWith> > </IterFilter> > </Let> > <For type="xs:string" size="1"> > <Var name="$db" id="2" type="xs:string"/> > <VarRef type="xs:string*"> > <Var name="$dbs" id="0" type="xs:string*"/> > </VarRef> > </For> > <Let type="element()*"> > <Var name="$ft" id="3" type="element()*"/> > <CachedPath type="element()*"> > <FtSearch name="search(database,terms[,options])" type="text()*"> > <VarRef type="xs:string" size="1"> > <Var name="$db" id="2" type="xs:string"/> > </VarRef> > <Str type="xs:string">автомобиль</Str> > </FtSearch> > <IterStep axis="parent" test="*" type="element()*"/> > </CachedPath> > </Let> > <VarRef type="element()*"> > <Var name="$ft" id="3" type="element()*"/> > </VarRef> > </GFLWOR> > </QueryPlan> > > > 20.07.2018, 12:47, "Christian Grün" <christian.gruen at gmail.com>: > > These examples work differently. > > > So If I read this correctly, the number of results for 1. is still > identical, right? However, in the second query in 2., no results are > returned. You could report the query plan will give us some insight > into what happens here. > > > > 1. > (# db:enforceindex #) { > for $db in db:list()[starts-with(.,'000999~')] > return db:open($db)//*[text() contains text { 'болт' } any] > } > 378 rows > > let $dbs := for $i in db:list()[starts-with(.,'000999~')] return $i > for $db in $dbs > let $ft := ft:search($db, "болт")/parent::* > for $node in $ft > return $node > 378 rows > > 2. > (# db:enforceindex #) { > for $db in db:list()[starts-with(.,'000999~')] > return db:open($db)//*[text() contains text { 'автомобиль' } any] > } > 0 rows > > let $dbs := for $i in db:list()[starts-with(.,'000999~')] return $i > for $db in $dbs > let $ft := ft:search($db, "автомобиль")/parent::* > for $node in $ft > return $node > 2138 rows > > > Why do they work differently? > > > > -- > С уважением, > Ветошкин Владимир Владимирович >

20.07.2018, 14:19, "Christian Grün" <christian.gruen at gmail.com>: > I think I found the missing pieces: > > • In your full-text index, you used non-default options (which is > completely fine) > • In the rewritten query, these options cannot applied to your query > (because, once again, they are not known at compile time). > > Your query should yield the expected results if you add the options to > your full-text expression: > > (# db:enforceindex #) { > for $db in db:list()[starts-with(., 'x')] > return db:open($db)//*[text() contains text 'автомобиль' > using stemming using language 'ru'] > } > > I added a note in our documentation [1]. Another option is (as you > already found out) to directly use ft:search. > > Cheers, > Christian > > [1] http://docs.basex.org/wiki/Indexes#Enforce_Rewritings > > > On Fri, Jul 20, 2018 at 11:54 AM Ветошкин Владимир <en-trance at yandex.ru> wrote: > > > Query plan (0 rows): > > > > <QueryPlan compiled="true" updating="false"> > > <Extension type="element()*"> > > <DBPragma value=""> > > <QNm type="xs:QName">db:enforceindex</QNm> > > </DBPragma> > > <GFLWOR type="element()*"> > > <For type="xs:string" size="1"> > > <Var name="$db" id="0" type="xs:string"/> > > <IterFilter type="xs:string*"> > > <DbList name="list([database[,path]])" type="xs:string*"/> > > <FnStartsWith name="starts-with(string,sub[,collation])" type="xs:boolean" size="1"> > > <ContextValue type="xs:string" size="1"/> > > <Str type="xs:string">000999~</Str> > > </FnStartsWith> > > </IterFilter> > > </For> > > <CachedPath type="element()*"> > > <FTIndexAccess type="text()*"> > > <IndexDynDb> > > <DbOpen name="open(database[,path])" type="document-node()*"> > > <VarRef type="xs:string" size="1"> > > <Var name="$db" id="0" type="xs:string"/> > > </VarRef> > > </DbOpen> > > </IndexDynDb> > > <FTWords type="xs:boolean" size="1"> > > <Str type="xs:string">автомобиль</Str> > > </FTWords> > > </FTIndexAccess> > > <IterStep axis="parent" test="*" type="element()*"/> > > </CachedPath> > > </GFLWOR> > > </Extension> > > </QueryPlan> > > > > > > > > And query plan (2138 rows): > > > > <QueryPlan compiled="true" updating="false"> > > <GFLWOR type="element()*"> > > <Let type="xs:string*"> > > <Var name="$dbs" id="0" type="xs:string*"/> > > <IterFilter type="xs:string*"> > > <DbList name="list([database[,path]])" type="xs:string*"/> > > <FnStartsWith name="starts-with(string,sub[,collation])" type="xs:boolean" size="1"> > > <ContextValue type="xs:string" size="1"/> > > <Str type="xs:string">000999~</Str> > > </FnStartsWith> > > </IterFilter> > > </Let> > > <For type="xs:string" size="1"> > > <Var name="$db" id="2" type="xs:string"/> > > <VarRef type="xs:string*"> > > <Var name="$dbs" id="0" type="xs:string*"/> > > </VarRef> > > </For> > > <Let type="element()*"> > > <Var name="$ft" id="3" type="element()*"/> > > <CachedPath type="element()*"> > > <FtSearch name="search(database,terms[,options])" type="text()*"> > > <VarRef type="xs:string" size="1"> > > <Var name="$db" id="2" type="xs:string"/> > > </VarRef> > > <Str type="xs:string">автомобиль</Str> > > </FtSearch> > > <IterStep axis="parent" test="*" type="element()*"/> > > </CachedPath> > > </Let> > > <VarRef type="element()*"> > > <Var name="$ft" id="3" type="element()*"/> > > </VarRef> > > </GFLWOR> > > </QueryPlan> > > > > > > 20.07.2018, 12:47, "Christian Grün" <christian.gruen at gmail.com>: > > > > These examples work differently. > > > > > > So If I read this correctly, the number of results for 1. is still > > identical, right? However, in the second query in 2., no results are > > returned. You could report the query plan will give us some insight > > into what happens here. > > > > > > > > 1. > > (# db:enforceindex #) { > > for $db in db:list()[starts-with(.,'000999~')] > > return db:open($db)//*[text() contains text { 'болт' } any] > > } > > 378 rows > > > > let $dbs := for $i in db:list()[starts-with(.,'000999~')] return $i > > for $db in $dbs > > let $ft := ft:search($db, "болт")/parent::* > > for $node in $ft > > return $node > > 378 rows > > > > 2. > > (# db:enforceindex #) { > > for $db in db:list()[starts-with(.,'000999~')] > > return db:open($db)//*[text() contains text { 'автомобиль' } any] > > } > > 0 rows > > > > let $dbs := for $i in db:list()[starts-with(.,'000999~')] return $i > > for $db in $dbs > > let $ft := ft:search($db, "автомобиль")/parent::* > > for $node in $ft > > return $node > > 2138 rows > > > > > > Why do they work differently? > > > > > > > > -- > > С уважением, > > Ветошкин Владимир Владимирович > >

20.07.2018, 14:26, "Ветошкин Владимир" <en-trance at yandex.ru>: > Christian, you're genius :) > Thank you very much for your help! > > 20.07.2018, 14:19, "Christian Grün" <christian.gruen at gmail.com>: > > > I think I found the missing pieces: > > > > • In your full-text index, you used non-default options (which is > > completely fine) > > • In the rewritten query, these options cannot applied to your query > > (because, once again, they are not known at compile time). > > > > Your query should yield the expected results if you add the options to > > your full-text expression: > > > > (# db:enforceindex #) { > > for $db in db:list()[starts-with(., 'x')] > > return db:open($db)//*[text() contains text 'автомобиль' > > using stemming using language 'ru'] > > } > > > > I added a note in our documentation [1]. Another option is (as you > > already found out) to directly use ft:search. > > > > Cheers, > > Christian > > > > [1] http://docs.basex.org/wiki/Indexes#Enforce_Rewritings > > > > > > On Fri, Jul 20, 2018 at 11:54 AM Ветошкин Владимир <en-trance at yandex.ru> wrote: > > > > > Query plan (0 rows): > > > > > > <QueryPlan compiled="true" updating="false"> > > > <Extension type="element()*"> > > > <DBPragma value=""> > > > <QNm type="xs:QName">db:enforceindex</QNm> > > > </DBPragma> > > > <GFLWOR type="element()*"> > > > <For type="xs:string" size="1"> > > > <Var name="$db" id="0" type="xs:string"/> > > > <IterFilter type="xs:string*"> > > > <DbList name="list([database[,path]])" type="xs:string*"/> > > > <FnStartsWith name="starts-with(string,sub[,collation])" type="xs:boolean" size="1"> > > > <ContextValue type="xs:string" size="1"/> > > > <Str type="xs:string">000999~</Str> > > > </FnStartsWith> > > > </IterFilter> > > > </For> > > > <CachedPath type="element()*"> > > > <FTIndexAccess type="text()*"> > > > <IndexDynDb> > > > <DbOpen name="open(database[,path])" type="document-node()*"> > > > <VarRef type="xs:string" size="1"> > > > <Var name="$db" id="0" type="xs:string"/> > > > </VarRef> > > > </DbOpen> > > > </IndexDynDb> > > > <FTWords type="xs:boolean" size="1"> > > > <Str type="xs:string">автомобиль</Str> > > > </FTWords> > > > </FTIndexAccess> > > > <IterStep axis="parent" test="*" type="element()*"/> > > > </CachedPath> > > > </GFLWOR> > > > </Extension> > > > </QueryPlan> > > > > > > > > > > > > And query plan (2138 rows): > > > > > > <QueryPlan compiled="true" updating="false"> > > > <GFLWOR type="element()*"> > > > <Let type="xs:string*"> > > > <Var name="$dbs" id="0" type="xs:string*"/> > > > <IterFilter type="xs:string*"> > > > <DbList name="list([database[,path]])" type="xs:string*"/> > > > <FnStartsWith name="starts-with(string,sub[,collation])" type="xs:boolean" size="1"> > > > <ContextValue type="xs:string" size="1"/> > > > <Str type="xs:string">000999~</Str> > > > </FnStartsWith> > > > </IterFilter> > > > </Let> > > > <For type="xs:string" size="1"> > > > <Var name="$db" id="2" type="xs:string"/> > > > <VarRef type="xs:string*"> > > > <Var name="$dbs" id="0" type="xs:string*"/> > > > </VarRef> > > > </For> > > > <Let type="element()*"> > > > <Var name="$ft" id="3" type="element()*"/> > > > <CachedPath type="element()*"> > > > <FtSearch name="search(database,terms[,options])" type="text()*"> > > > <VarRef type="xs:string" size="1"> > > > <Var name="$db" id="2" type="xs:string"/> > > > </VarRef> > > > <Str type="xs:string">автомобиль</Str> > > > </FtSearch> > > > <IterStep axis="parent" test="*" type="element()*"/> > > > </CachedPath> > > > </Let> > > > <VarRef type="element()*"> > > > <Var name="$ft" id="3" type="element()*"/> > > > </VarRef> > > > </GFLWOR> > > > </QueryPlan> > > > > > > > > > 20.07.2018, 12:47, "Christian Grün" <christian.gruen at gmail.com>: > > > > > > These examples work differently. > > > > > > > > > So If I read this correctly, the number of results for 1. is still > > > identical, right? However, in the second query in 2., no results are > > > returned. You could report the query plan will give us some insight > > > into what happens here. > > > > > > > > > > > > 1. > > > (# db:enforceindex #) { > > > for $db in db:list()[starts-with(.,'000999~')] > > > return db:open($db)//*[text() contains text { 'болт' } any] > > > } > > > 378 rows > > > > > > let $dbs := for $i in db:list()[starts-with(.,'000999~')] return $i > > > for $db in $dbs > > > let $ft := ft:search($db, "болт")/parent::* > > > for $node in $ft > > > return $node > > > 378 rows > > > > > > 2. > > > (# db:enforceindex #) { > > > for $db in db:list()[starts-with(.,'000999~')] > > > return db:open($db)//*[text() contains text { 'автомобиль' } any] > > > } > > > 0 rows > > > > > > let $dbs := for $i in db:list()[starts-with(.,'000999~')] return $i > > > for $db in $dbs > > > let $ft := ft:search($db, "автомобиль")/parent::* > > > for $node in $ft > > > return $node > > > 2138 rows > > > > > > > > > Why do they work differently? > > > > > > > > > > > > -- > > > С уважением, > > > Ветошкин Владимир Владимирович > > > > > > -- > С уважением, > Ветошкин Владимир Владимирович >

> Hi! > > After some tests of search (using stemming, using language ru) I have > found several problems. > E.g.: > if search for "кузов" - it doesn't find "кузова" > > > 20.07.2018, 14:26, "Ветошкин Владимир" <en-trance at yandex.ru>: > > Christian, you're genius :) > Thank you very much for your help! > > 20.07.2018, 14:19, "Christian Grün" <christian.gruen at gmail.com>: > > I think I found the missing pieces: > > • In your full-text index, you used non-default options (which is > completely fine) > • In the rewritten query, these options cannot applied to your query > (because, once again, they are not known at compile time). > > Your query should yield the expected results if you add the options to > your full-text expression: > > (# db:enforceindex #) { > for $db in db:list()[starts-with(., 'x')] > return db:open($db)//*[text() contains text 'автомобиль' > using stemming using language 'ru'] > } > > I added a note in our documentation [1]. Another option is (as you > already found out) to directly use ft:search. > > Cheers, > Christian > > [1] http://docs.basex.org/wiki/Indexes#Enforce_Rewritings > > > On Fri, Jul 20, 2018 at 11:54 AM Ветошкин Владимир <en-trance at yandex.ru> > wrote: > > > Query plan (0 rows): > > <QueryPlan compiled="true" updating="false"> > <Extension type="element()*"> > <DBPragma value=""> > <QNm type="xs:QName">db:enforceindex</QNm> > </DBPragma> > <GFLWOR type="element()*"> > <For type="xs:string" size="1"> > <Var name="$db" id="0" type="xs:string"/> > <IterFilter type="xs:string*"> > <DbList name="list([database[,path]])" type="xs:string*"/> > <FnStartsWith name="starts-with(string,sub[,collation])" > type="xs:boolean" size="1"> > <ContextValue type="xs:string" size="1"/> > <Str type="xs:string">000999~</Str> > </FnStartsWith> > </IterFilter> > </For> > <CachedPath type="element()*"> > <FTIndexAccess type="text()*"> > <IndexDynDb> > <DbOpen name="open(database[,path])" type="document-node()*"> > <VarRef type="xs:string" size="1"> > <Var name="$db" id="0" type="xs:string"/> > </VarRef> > </DbOpen> > </IndexDynDb> > <FTWords type="xs:boolean" size="1"> > <Str type="xs:string">автомобиль</Str> > </FTWords> > </FTIndexAccess> > <IterStep axis="parent" test="*" type="element()*"/> > </CachedPath> > </GFLWOR> > </Extension> > </QueryPlan> > > > > And query plan (2138 rows): > > <QueryPlan compiled="true" updating="false"> > <GFLWOR type="element()*"> > <Let type="xs:string*"> > <Var name="$dbs" id="0" type="xs:string*"/> > <IterFilter type="xs:string*"> > <DbList name="list([database[,path]])" type="xs:string*"/> > <FnStartsWith name="starts-with(string,sub[,collation])" > type="xs:boolean" size="1"> > <ContextValue type="xs:string" size="1"/> > <Str type="xs:string">000999~</Str> > </FnStartsWith> > </IterFilter> > </Let> > <For type="xs:string" size="1"> > <Var name="$db" id="2" type="xs:string"/> > <VarRef type="xs:string*"> > <Var name="$dbs" id="0" type="xs:string*"/> > </VarRef> > </For> > <Let type="element()*"> > <Var name="$ft" id="3" type="element()*"/> > <CachedPath type="element()*"> > <FtSearch name="search(database,terms[,options])" type="text()*"> > <VarRef type="xs:string" size="1"> > <Var name="$db" id="2" type="xs:string"/> > </VarRef> > <Str type="xs:string">автомобиль</Str> > </FtSearch> > <IterStep axis="parent" test="*" type="element()*"/> > </CachedPath> > </Let> > <VarRef type="element()*"> > <Var name="$ft" id="3" type="element()*"/> > </VarRef> > </GFLWOR> > </QueryPlan> > > > 20.07.2018, 12:47, "Christian Grün" <christian.gruen at gmail.com>: > > These examples work differently. > > > So If I read this correctly, the number of results for 1. is still > identical, right? However, in the second query in 2., no results are > returned. You could report the query plan will give us some insight > into what happens here. > > > > 1. > (# db:enforceindex #) { > for $db in db:list()[starts-with(.,'000999~')] > return db:open($db)//*[text() contains text { 'болт' } any] > } > 378 rows > > let $dbs := for $i in db:list()[starts-with(.,'000999~')] return $i > for $db in $dbs > let $ft := ft:search($db, "болт")/parent::* > for $node in $ft > return $node > 378 rows > > 2. > (# db:enforceindex #) { > for $db in db:list()[starts-with(.,'000999~')] > return db:open($db)//*[text() contains text { 'автомобиль' } any] > } > 0 rows > > let $dbs := for $i in db:list()[starts-with(.,'000999~')] return $i > for $db in $dbs > let $ft := ft:search($db, "автомобиль")/parent::* > for $node in $ft > return $node > 2138 rows > > > Why do they work differently? > > > > -- > С уважением, > Ветошкин Владимир Владимирович > > > > > -- > С уважением, > Ветошкин Владимир Владимирович > > > > > -- > С уважением, > Ветошкин Владимир Владимирович > >

22.07.2018, 18:55, "Christian Grün" <christian.gruen at gmail.com>: > This seems to be a limitation of the Russian stemmer implementation, which we took from the Apache Lucene project. Maybe we could replace it with a more sophisticated implementation. Do you have some experience with other stemmers that are available in the wild? > Ветошкин Владимир <en-trance at yandex.ru> schrieb am So., 22. Juli 2018, 14:30: > > > Hi! > > > > After some tests of search (using stemming, using language ru) I have found several problems. > > E.g.: > > if search for "кузов" - it doesn't find "кузова" > > > > > 20.07.2018, 14:26, "Ветошкин Владимир" <en-trance at yandex.ru>: > > > Christian, you're genius :) > > Thank you very much for your help! > > > > 20.07.2018, 14:19, "Christian Grün" <christian.gruen at gmail.com>: > > > > > I think I found the missing pieces: > > > > > > • In your full-text index, you used non-default options (which is > > > completely fine) > > > • In the rewritten query, these options cannot applied to your query > > > (because, once again, they are not known at compile time). > > > > > > Your query should yield the expected results if you add the options to > > > your full-text expression: > > > > > > (# db:enforceindex #) { > > > for $db in db:list()[starts-with(., 'x')] > > > return db:open($db)//*[text() contains text 'автомобиль' > > > using stemming using language 'ru'] > > > } > > > > > > I added a note in our documentation [1]. Another option is (as you > > > already found out) to directly use ft:search. > > > > > > Cheers, > > > Christian > > > > > > [1] http://docs.basex.org/wiki/Indexes#Enforce_Rewritings > > > > > > > > > On Fri, Jul 20, 2018 at 11:54 AM Ветошкин Владимир <en-trance at yandex.ru> wrote: > > > > > > > Query plan (0 rows): > > > > > > > > <QueryPlan compiled="true" updating="false"> > > > > <Extension type="element()*"> > > > > <DBPragma value=""> > > > > <QNm type="xs:QName">db:enforceindex</QNm> > > > > </DBPragma> > > > > <GFLWOR type="element()*"> > > > > <For type="xs:string" size="1"> > > > > <Var name="$db" id="0" type="xs:string"/> > > > > <IterFilter type="xs:string*"> > > > > <DbList name="list([database[,path]])" type="xs:string*"/> > > > > <FnStartsWith name="starts-with(string,sub[,collation])" type="xs:boolean" size="1"> > > > > <ContextValue type="xs:string" size="1"/> > > > > <Str type="xs:string">000999~</Str> > > > > </FnStartsWith> > > > > </IterFilter> > > > > </For> > > > > <CachedPath type="element()*"> > > > > <FTIndexAccess type="text()*"> > > > > <IndexDynDb> > > > > <DbOpen name="open(database[,path])" type="document-node()*"> > > > > <VarRef type="xs:string" size="1"> > > > > <Var name="$db" id="0" type="xs:string"/> > > > > </VarRef> > > > > </DbOpen> > > > > </IndexDynDb> > > > > <FTWords type="xs:boolean" size="1"> > > > > <Str type="xs:string">автомобиль</Str> > > > > </FTWords> > > > > </FTIndexAccess> > > > > <IterStep axis="parent" test="*" type="element()*"/> > > > > </CachedPath> > > > > </GFLWOR> > > > > </Extension> > > > > </QueryPlan> > > > > > > > > > > > > > > > > And query plan (2138 rows): > > > > > > > > <QueryPlan compiled="true" updating="false"> > > > > <GFLWOR type="element()*"> > > > > <Let type="xs:string*"> > > > > <Var name="$dbs" id="0" type="xs:string*"/> > > > > <IterFilter type="xs:string*"> > > > > <DbList name="list([database[,path]])" type="xs:string*"/> > > > > <FnStartsWith name="starts-with(string,sub[,collation])" type="xs:boolean" size="1"> > > > > <ContextValue type="xs:string" size="1"/> > > > > <Str type="xs:string">000999~</Str> > > > > </FnStartsWith> > > > > </IterFilter> > > > > </Let> > > > > <For type="xs:string" size="1"> > > > > <Var name="$db" id="2" type="xs:string"/> > > > > <VarRef type="xs:string*"> > > > > <Var name="$dbs" id="0" type="xs:string*"/> > > > > </VarRef> > > > > </For> > > > > <Let type="element()*"> > > > > <Var name="$ft" id="3" type="element()*"/> > > > > <CachedPath type="element()*"> > > > > <FtSearch name="search(database,terms[,options])" type="text()*"> > > > > <VarRef type="xs:string" size="1"> > > > > <Var name="$db" id="2" type="xs:string"/> > > > > </VarRef> > > > > <Str type="xs:string">автомобиль</Str> > > > > </FtSearch> > > > > <IterStep axis="parent" test="*" type="element()*"/> > > > > </CachedPath> > > > > </Let> > > > > <VarRef type="element()*"> > > > > <Var name="$ft" id="3" type="element()*"/> > > > > </VarRef> > > > > </GFLWOR> > > > > </QueryPlan> > > > > > > > > > > > > 20.07.2018, 12:47, "Christian Grün" <christian.gruen at gmail.com>: > > > > > > > > These examples work differently. > > > > > > > > > > > > So If I read this correctly, the number of results for 1. is still > > > > identical, right? However, in the second query in 2., no results are > > > > returned. You could report the query plan will give us some insight > > > > into what happens here. > > > > > > > > > > > > > > > > 1. > > > > (# db:enforceindex #) { > > > > for $db in db:list()[starts-with(.,'000999~')] > > > > return db:open($db)//*[text() contains text { 'болт' } any] > > > > } > > > > 378 rows > > > > > > > > let $dbs := for $i in db:list()[starts-with(.,'000999~')] return $i > > > > for $db in $dbs > > > > let $ft := ft:search($db, "болт")/parent::* > > > > for $node in $ft > > > > return $node > > > > 378 rows > > > > > > > > 2. > > > > (# db:enforceindex #) { > > > > for $db in db:list()[starts-with(.,'000999~')] > > > > return db:open($db)//*[text() contains text { 'автомобиль' } any] > > > > } > > > > 0 rows > > > > > > > > let $dbs := for $i in db:list()[starts-with(.,'000999~')] return $i > > > > for $db in $dbs > > > > let $ft := ft:search($db, "автомобиль")/parent::* > > > > for $node in $ft > > > > return $node > > > > 2138 rows > > > > > > > > > > > > Why do they work differently? > > > > > > > > > > > > > > > > -- > > > > С уважением, > > > > Ветошкин Владимир Владимирович > > > > > > > > > > -- > > С уважением, > > Ветошкин Владимир Владимирович > > > > > -- > С уважением, > Ветошкин Владимир Владимирович >