Basex Query Optimization Support
Friday, 4 September 2015 · 10 messages · last reply 18 Sep 2015
AB
Adi Babu4 Sep 2015 08:31 #
Dear Team,
I am facing performance issue taking long time to execute the Xquery in Basex(Client Server architecture), can you please guide me to optimize the query performance.
Below are the statistics & Basex server hardware specifications.
Statistics :
-------------
Total Records count : 6000000
Fetch count : 10000
Hits Parallel hits : 100
Time Taking : 750 seconds
Architecture : Client Server Architecture
Hardware Specifications of Basex Sever :
----------------------------------------------------------
Operating System : Red Hat Enterprise Linux 64bit
OS Version : Linux 2.6.32
Hard Disk : 500 GB
Memory Capacity : 16 GB Ram
Processor Family : Intel (R) Xeon(R) CPU E5 -2680 V2 Processor Speed : 2.80 GHz
Physical Processors : 8
Virtual Processors/cores : 4
Server Version : basex 7.9
Note : I have attached the sample xml data and Xquery .
Thanks & Regards
Adi
MG
Maximilian Gärber8 Sep 2015 17:45 #
Hi Adi,
some general pointers:
* Server version 7.9 is missing all the latest perf improvements, current version is 8.2.3
* You are accessing the files in the file system - try adding them to the database first, then you can make use of the text index etc. See http://docs.basex.org/wiki/Indexes#Text_Index
Regards,
Max
2015-09-04 8:31 GMT+02:00 Adi Babu <adibabu.b at gmail.com>:
Quoted text · Adi Babu · 33 lines
> Dear Team, > I am facing performance issue taking long time to execute the > Xquery in Basex(Client Server architecture), can you please guide me to > optimize the query performance. > > Below are the statistics & Basex server hardware specifications. > > Statistics : > ------------- > Total Records count : 6000000 > Fetch count : 10000 > Hits Parallel hits : 100 > Time Taking : 750 seconds > Architecture : Client Server Architecture > > Hardware Specifications of Basex Sever : > ---------------------------------------------------------- > Operating System : Red Hat Enterprise Linux 64bit > OS Version : Linux 2.6.32 > Hard Disk : 500 GB > Memory Capacity : 16 GB Ram > Processor Family : Intel (R) Xeon(R) CPU E5 -2680 V2 > Processor Speed : 2.80 GHz > Physical Processors : 8 > Virtual Processors/cores : 4 > Server Version : basex 7.9 > > Note : I have attached the sample xml data and Xquery . > > > Thanks & Regards > Adi >
AB
Adi Babu11 Sep 2015 15:07 #
Dear Max,
Currently I am using collection only(size 7.96 GB), I am sharing the query for your reference only. I tried the latest version 8.2.3 also but it takes more time than 7.9.
Can you please assist me to proceed further.
Thanks & Regards,
Adi
Quoted text · Maximilian Gärber · 50 lines
On Tue, Sep 8, 2015 at 9:15 PM, Maximilian Gärber <mgaerber at arcor.de> wrote: > Hi Adi, > > some general pointers: > * Server version 7.9 is missing all the latest perf improvements, > current version is 8.2.3 > * You are accessing the files in the file system - try adding them to > the database first, then you can make use of the text index etc. See > http://docs.basex.org/wiki/Indexes#Text_Index > > Regards, > > Max > > > 2015-09-04 8:31 GMT+02:00 Adi Babu <adibabu.b at gmail.com>: > > Dear Team, > > I am facing performance issue taking long time to execute the > > Xquery in Basex(Client Server architecture), can you please guide me to > > optimize the query performance. > > > > Below are the statistics & Basex server hardware specifications. > > > > Statistics : > > ------------- > > Total Records count : 6000000 > > Fetch count : 10000 > > Hits Parallel hits : 100 > > Time Taking : 750 seconds > > Architecture : Client Server Architecture > > > > Hardware Specifications of Basex Sever : > > ---------------------------------------------------------- > > Operating System : Red Hat Enterprise Linux 64bit > > OS Version : Linux 2.6.32 > > Hard Disk : 500 GB > > Memory Capacity : 16 GB Ram > > Processor Family : Intel (R) Xeon(R) CPU E5 -2680 V2 > > Processor Speed : 2.80 GHz > > Physical Processors : 8 > > Virtual Processors/cores : 4 > > Server Version : basex 7.9 > > > > Note : I have attached the sample xml data and Xquery . > > > > > > Thanks & Regards > > Adi > > >
MF
Martín Ferrari11 Sep 2015 18:34 #
HI Adi, Have you tried Max's suggestion of adding the xml data to a BaseX db first? Based on this: for $details in
(collection("E:\Web-Projects\VodafoneUK\Docs\Cloud_Test_Stats\Performance_Analysis\Freq_Called_Sample_XML_Data.xml") it looks like you're using the file Freq_Called_Sample_XML_Data.xml directly from the filesystem, which means it has no indexes and must be fully scanned to get results. You should create a DB and upload that xml file with
TEXTINDEX and ATTRINDEX enabled (which is the default). You might need to run the OPTIMIZE ALL command after uploading the xml file, you can run the INFO DB command and it will show if your database has up-to-date indexes or not. Once you have a db, you can also use the GUI to check the query plan that's being used for a particular query. Martín.
Date: Fri, 11 Sep 2015 18:37:05 +0530Quoted text · adibabu.b at gmail.com · 50 lines
From: adibabu.b at gmail.com To: mgaerber at arcor.de CC: basex-talk at mailman.uni-konstanz.de Subject: Re: [basex-talk] Fwd: Re : Basex Query Optimization Support Dear Max,Currently I am using collection only(size 7.96 GB), I am sharing the query for your reference only. I tried the latest version 8.2.3 also but it takes more time than 7.9.Can you please assist me to proceed further. Thanks & Regards,Adi On Tue, Sep 8, 2015 at 9:15 PM, Maximilian Gärber <mgaerber at arcor.de> wrote: Hi Adi, some general pointers: * Server version 7.9 is missing all the latest perf improvements, current version is 8.2.3 * You are accessing the files in the file system - try adding them to the database first, then you can make use of the text index etc. See http://docs.basex.org/wiki/Indexes#Text_Index Regards, Max 2015-09-04 8:31 GMT+02:00 Adi Babu <adibabu.b at gmail.com>: > Dear Team, > I am facing performance issue taking long time to execute the > Xquery in Basex(Client Server architecture), can you please guide me to > optimize the query performance. > > Below are the statistics & Basex server hardware specifications. > > Statistics : > ------------- > Total Records count : 6000000 > Fetch count : 10000 > Hits Parallel hits : 100 > Time Taking : 750 seconds > Architecture : Client Server Architecture > > Hardware Specifications of Basex Sever : > ---------------------------------------------------------- > Operating System : Red Hat Enterprise Linux 64bit > OS Version : Linux 2.6.32 > Hard Disk : 500 GB > Memory Capacity : 16 GB Ram > Processor Family : Intel (R) Xeon(R) CPU E5 -2680 V2 > Processor Speed : 2.80 GHz > Physical Processors : 8 > Virtual Processors/cores : 4 > Server Version : basex 7.9 > > Note : I have attached the sample xml data and Xquery . > > > Thanks & Regards > Adi >
AB
Adi Babu15 Sep 2015 12:48 #
Hi Martin,
I am working with collection data base only from starting on wards. Through mail
it is not possible to share the large collection db(more than 7GB) thats why I am sharing sample query(with xml doc but actually I am using collection only).
I have tried all your suggestions in the trail mail ,but it takes more time in 8.2.3 than 7.9 version. Can you please guide me to proceed further.
Thanks & Regards,
Adi
Quoted text · Martín Ferrari · 86 lines
On Fri, Sep 11, 2015 at 10:04 PM, Martín Ferrari <ferrari_martin at hotmail.com > wrote: > HI Adi, > Have you tried Max's suggestion of adding the xml data to a BaseX db > first? Based on this: > > for $details in > (collection("E:\Web-Projects\VodafoneUK\Docs\Cloud_Test_Stats\Performance_Analysis\Freq_Called_Sample_XML_Data.xml") > > > it looks like you're using the file Freq_Called_Sample_XML_Data.xml directly from the filesystem, which means it has no indexes and must be fully scanned to get results. You should create a DB and upload that xml file with TEXTINDEX and ATTRINDEX enabled (which is the default). You might need to run the OPTIMIZE ALL command after uploading the xml file, you can run the INFO DB command and it will show if your database has up-to-date indexes or not. Once you have a db, you can also use the GUI to check the query plan that's being used for a particular query. > > > Martín. > > > ------------------------------ > Date: Fri, 11 Sep 2015 18:37:05 +0530 > From: adibabu.b at gmail.com > To: mgaerber at arcor.de > CC: basex-talk at mailman.uni-konstanz.de > Subject: Re: [basex-talk] Fwd: Re : Basex Query Optimization Support > > Dear Max, > Currently I am using collection only(size 7.96 GB), I am sharing the query > for your reference only. I tried the latest version 8.2.3 also but it takes > more time than 7.9. > Can you please assist me to proceed further. > > Thanks & Regards, > Adi > > On Tue, Sep 8, 2015 at 9:15 PM, Maximilian Gärber <mgaerber at arcor.de> > wrote: > > Hi Adi, > > some general pointers: > * Server version 7.9 is missing all the latest perf improvements, > current version is 8.2.3 > * You are accessing the files in the file system - try adding them to > the database first, then you can make use of the text index etc. See > http://docs.basex.org/wiki/Indexes#Text_Index > > Regards, > > Max > > > 2015-09-04 8:31 GMT+02:00 Adi Babu <adibabu.b at gmail.com>: > > Dear Team, > > I am facing performance issue taking long time to execute the > > Xquery in Basex(Client Server architecture), can you please guide me to > > optimize the query performance. > > > > Below are the statistics & Basex server hardware specifications. > > > > Statistics : > > ------------- > > Total Records count : 6000000 > > Fetch count : 10000 > > Hits Parallel hits : 100 > > Time Taking : 750 seconds > > Architecture : Client Server Architecture > > > > Hardware Specifications of Basex Sever : > > ---------------------------------------------------------- > > Operating System : Red Hat Enterprise Linux 64bit > > OS Version : Linux 2.6.32 > > Hard Disk : 500 GB > > Memory Capacity : 16 GB Ram > > Processor Family : Intel (R) Xeon(R) CPU E5 -2680 V2 > > Processor Speed : 2.80 GHz > > Physical Processors : 8 > > Virtual Processors/cores : 4 > > Server Version : basex 7.9 > > > > Note : I have attached the sample xml data and Xquery . > > > > > > Thanks & Regards > > Adi > > > > >
CG
Christian Grün15 Sep 2015 13:04 #
Hi Adi,
How does your new code look like? How did you rewrite the collection function?
Christian
Quoted text · Adi Babu · 107 lines
On Tue, Sep 15, 2015 at 12:48 PM, Adi Babu <adibabu.b at gmail.com> wrote: > Hi Martin, > I am working with collection data base only from starting on wards. Through > mail it is not possible to share the large collection db(more than 7GB) > thats why I am sharing sample query(with xml doc but actually I am using > collection only). > I have tried all your suggestions in the trail mail ,but it takes more time > in 8.2.3 than 7.9 version. Can you please guide me to proceed further. > > Thanks & Regards, > Adi > > On Fri, Sep 11, 2015 at 10:04 PM, Martín Ferrari > <ferrari_martin at hotmail.com> wrote: >> >> HI Adi, >> Have you tried Max's suggestion of adding the xml data to a BaseX db >> first? Based on this: >> >> for $details in >> >> (collection("E:\Web-Projects\VodafoneUK\Docs\Cloud_Test_Stats\Performance_Analysis\Freq_Called_Sample_XML_Data.xml") >> >> >> it looks like you're using the file Freq_Called_Sample_XML_Data.xml >> directly from the filesystem, which means it has no indexes and must be >> fully scanned to get results. You should create a DB and upload that xml >> file with TEXTINDEX and ATTRINDEX enabled (which is the default). You might >> need to run the OPTIMIZE ALL command after uploading the xml file, you can >> run the INFO DB command and it will show if your database has up-to-date >> indexes or not. Once you have a db, you can also use the GUI to check the >> query plan that's being used for a particular query. >> >> >> Martín. >> >> >> ________________________________ >> Date: Fri, 11 Sep 2015 18:37:05 +0530 >> From: adibabu.b at gmail.com >> To: mgaerber at arcor.de >> CC: basex-talk at mailman.uni-konstanz.de >> Subject: Re: [basex-talk] Fwd: Re : Basex Query Optimization Support >> >> Dear Max, >> Currently I am using collection only(size 7.96 GB), I am sharing the query >> for your reference only. I tried the latest version 8.2.3 also but it takes >> more time than 7.9. >> Can you please assist me to proceed further. >> >> Thanks & Regards, >> Adi >> >> On Tue, Sep 8, 2015 at 9:15 PM, Maximilian Gärber <mgaerber at arcor.de> >> wrote: >> >> Hi Adi, >> >> some general pointers: >> * Server version 7.9 is missing all the latest perf improvements, >> current version is 8.2.3 >> * You are accessing the files in the file system - try adding them to >> the database first, then you can make use of the text index etc. See >> http://docs.basex.org/wiki/Indexes#Text_Index >> >> Regards, >> >> Max >> >> >> 2015-09-04 8:31 GMT+02:00 Adi Babu <adibabu.b at gmail.com>: >> > Dear Team, >> > I am facing performance issue taking long time to execute the >> > Xquery in Basex(Client Server architecture), can you please guide me to >> > optimize the query performance. >> > >> > Below are the statistics & Basex server hardware specifications. >> > >> > Statistics : >> > ------------- >> > Total Records count : 6000000 >> > Fetch count : 10000 >> > Hits Parallel hits : 100 >> > Time Taking : 750 seconds >> > Architecture : Client Server Architecture >> > >> > Hardware Specifications of Basex Sever : >> > ---------------------------------------------------------- >> > Operating System : Red Hat Enterprise Linux 64bit >> > OS Version : Linux 2.6.32 >> > Hard Disk : 500 GB >> > Memory Capacity : 16 GB Ram >> > Processor Family : Intel (R) Xeon(R) CPU E5 -2680 V2 >> > Processor Speed : 2.80 GHz >> > Physical Processors : 8 >> > Virtual Processors/cores : 4 >> > Server Version : basex 7.9 >> > >> > Note : I have attached the sample xml data and Xquery . >> > >> > >> > Thanks & Regards >> > Adi >> > >> >> >
AB
Adi Babu15 Sep 2015 13:19 #
Hai Chistian,
Please find the attachment sample piece of code.
Thanks & Regards
Adi
Quoted text · Christian Grün · 129 lines
On Tue, Sep 15, 2015 at 4:34 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Adi, > > How does your new code look like? How did you rewrite the collection > function? > > Christian > > > > On Tue, Sep 15, 2015 at 12:48 PM, Adi Babu <adibabu.b at gmail.com> wrote: > > Hi Martin, > > I am working with collection data base only from starting on wards. > Through > > mail it is not possible to share the large collection db(more than 7GB) > > thats why I am sharing sample query(with xml doc but actually I am using > > collection only). > > I have tried all your suggestions in the trail mail ,but it takes more > time > > in 8.2.3 than 7.9 version. Can you please guide me to proceed further. > > > > Thanks & Regards, > > Adi > > > > On Fri, Sep 11, 2015 at 10:04 PM, Martín Ferrari > > <ferrari_martin at hotmail.com> wrote: > >> > >> HI Adi, > >> Have you tried Max's suggestion of adding the xml data to a BaseX > db > >> first? Based on this: > >> > >> for $details in > >> > >> > (collection("E:\Web-Projects\VodafoneUK\Docs\Cloud_Test_Stats\Performance_Analysis\Freq_Called_Sample_XML_Data.xml") > >> > >> > >> it looks like you're using the file Freq_Called_Sample_XML_Data.xml > >> directly from the filesystem, which means it has no indexes and must be > >> fully scanned to get results. You should create a DB and upload that xml > >> file with TEXTINDEX and ATTRINDEX enabled (which is the default). You > might > >> need to run the OPTIMIZE ALL command after uploading the xml file, you > can > >> run the INFO DB command and it will show if your database has up-to-date > >> indexes or not. Once you have a db, you can also use the GUI to check > the > >> query plan that's being used for a particular query. > >> > >> > >> Martín. > >> > >> > >> ________________________________ > >> Date: Fri, 11 Sep 2015 18:37:05 +0530 > >> From: adibabu.b at gmail.com > >> To: mgaerber at arcor.de > >> CC: basex-talk at mailman.uni-konstanz.de > >> Subject: Re: [basex-talk] Fwd: Re : Basex Query Optimization Support > >> > >> Dear Max, > >> Currently I am using collection only(size 7.96 GB), I am sharing the > query > >> for your reference only. I tried the latest version 8.2.3 also but it > takes > >> more time than 7.9. > >> Can you please assist me to proceed further. > >> > >> Thanks & Regards, > >> Adi > >> > >> On Tue, Sep 8, 2015 at 9:15 PM, Maximilian Gärber <mgaerber at arcor.de> > >> wrote: > >> > >> Hi Adi, > >> > >> some general pointers: > >> * Server version 7.9 is missing all the latest perf improvements, > >> current version is 8.2.3 > >> * You are accessing the files in the file system - try adding them to > >> the database first, then you can make use of the text index etc. See > >> http://docs.basex.org/wiki/Indexes#Text_Index > >> > >> Regards, > >> > >> Max > >> > >> > >> 2015-09-04 8:31 GMT+02:00 Adi Babu <adibabu.b at gmail.com>: > >> > Dear Team, > >> > I am facing performance issue taking long time to execute the > >> > Xquery in Basex(Client Server architecture), can you please guide me > to > >> > optimize the query performance. > >> > > >> > Below are the statistics & Basex server hardware specifications. > >> > > >> > Statistics : > >> > ------------- > >> > Total Records count : 6000000 > >> > Fetch count : 10000 > >> > Hits Parallel hits : 100 > >> > Time Taking : 750 seconds > >> > Architecture : Client Server Architecture > >> > > >> > Hardware Specifications of Basex Sever : > >> > ---------------------------------------------------------- > >> > Operating System : Red Hat Enterprise Linux 64bit > >> > OS Version : Linux 2.6.32 > >> > Hard Disk : 500 GB > >> > Memory Capacity : 16 GB Ram > >> > Processor Family : Intel (R) Xeon(R) CPU E5 -2680 V2 > >> > Processor Speed : 2.80 GHz > >> > Physical Processors : 8 > >> > Virtual Processors/cores : 4 > >> > Server Version : basex 7.9 > >> > > >> > Note : I have attached the sample xml data and Xquery . > >> > > >> > > >> > Thanks & Regards > >> > Adi > >> > > >> > >> > > >
CG
Christian Grün15 Sep 2015 13:22 #
Hi Adi,
I may not be up-to-date, as I expected you to send me an updated version of your XQuery code. Did you try what Martín suggested?
Christian
Quoted text · Adi Babu · 141 lines
On Tue, Sep 15, 2015 at 1:19 PM, Adi Babu <adibabu.b at gmail.com> wrote: > Hai Chistian, > Please find the attachment sample piece of code. > > Thanks & Regards > Adi > > > On Tue, Sep 15, 2015 at 4:34 PM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> Hi Adi, >> >> How does your new code look like? How did you rewrite the collection >> function? >> >> Christian >> >> >> >> On Tue, Sep 15, 2015 at 12:48 PM, Adi Babu <adibabu.b at gmail.com> wrote: >> > Hi Martin, >> > I am working with collection data base only from starting on wards. >> > Through >> > mail it is not possible to share the large collection db(more than 7GB) >> > thats why I am sharing sample query(with xml doc but actually I am using >> > collection only). >> > I have tried all your suggestions in the trail mail ,but it takes more >> > time >> > in 8.2.3 than 7.9 version. Can you please guide me to proceed further. >> > >> > Thanks & Regards, >> > Adi >> > >> > On Fri, Sep 11, 2015 at 10:04 PM, Martín Ferrari >> > <ferrari_martin at hotmail.com> wrote: >> >> >> >> HI Adi, >> >> Have you tried Max's suggestion of adding the xml data to a BaseX >> >> db >> >> first? Based on this: >> >> >> >> for $details in >> >> >> >> >> >> (collection("E:\Web-Projects\VodafoneUK\Docs\Cloud_Test_Stats\Performance_Analysis\Freq_Called_Sample_XML_Data.xml") >> >> >> >> >> >> it looks like you're using the file Freq_Called_Sample_XML_Data.xml >> >> directly from the filesystem, which means it has no indexes and must be >> >> fully scanned to get results. You should create a DB and upload that >> >> xml >> >> file with TEXTINDEX and ATTRINDEX enabled (which is the default). You >> >> might >> >> need to run the OPTIMIZE ALL command after uploading the xml file, you >> >> can >> >> run the INFO DB command and it will show if your database has >> >> up-to-date >> >> indexes or not. Once you have a db, you can also use the GUI to check >> >> the >> >> query plan that's being used for a particular query. >> >> >> >> >> >> Martín. >> >> >> >> >> >> ________________________________ >> >> Date: Fri, 11 Sep 2015 18:37:05 +0530 >> >> From: adibabu.b at gmail.com >> >> To: mgaerber at arcor.de >> >> CC: basex-talk at mailman.uni-konstanz.de >> >> Subject: Re: [basex-talk] Fwd: Re : Basex Query Optimization Support >> >> >> >> Dear Max, >> >> Currently I am using collection only(size 7.96 GB), I am sharing the >> >> query >> >> for your reference only. I tried the latest version 8.2.3 also but it >> >> takes >> >> more time than 7.9. >> >> Can you please assist me to proceed further. >> >> >> >> Thanks & Regards, >> >> Adi >> >> >> >> On Tue, Sep 8, 2015 at 9:15 PM, Maximilian Gärber <mgaerber at arcor.de> >> >> wrote: >> >> >> >> Hi Adi, >> >> >> >> some general pointers: >> >> * Server version 7.9 is missing all the latest perf improvements, >> >> current version is 8.2.3 >> >> * You are accessing the files in the file system - try adding them to >> >> the database first, then you can make use of the text index etc. See >> >> http://docs.basex.org/wiki/Indexes#Text_Index >> >> >> >> Regards, >> >> >> >> Max >> >> >> >> >> >> 2015-09-04 8:31 GMT+02:00 Adi Babu <adibabu.b at gmail.com>: >> >> > Dear Team, >> >> > I am facing performance issue taking long time to execute the >> >> > Xquery in Basex(Client Server architecture), can you please guide me >> >> > to >> >> > optimize the query performance. >> >> > >> >> > Below are the statistics & Basex server hardware specifications. >> >> > >> >> > Statistics : >> >> > ------------- >> >> > Total Records count : 6000000 >> >> > Fetch count : 10000 >> >> > Hits Parallel hits : 100 >> >> > Time Taking : 750 seconds >> >> > Architecture : Client Server Architecture >> >> > >> >> > Hardware Specifications of Basex Sever : >> >> > ---------------------------------------------------------- >> >> > Operating System : Red Hat Enterprise Linux 64bit >> >> > OS Version : Linux 2.6.32 >> >> > Hard Disk : 500 GB >> >> > Memory Capacity : 16 GB Ram >> >> > Processor Family : Intel (R) Xeon(R) CPU E5 -2680 V2 >> >> > Processor Speed : 2.80 GHz >> >> > Physical Processors : 8 >> >> > Virtual Processors/cores : 4 >> >> > Server Version : basex 7.9 >> >> > >> >> > Note : I have attached the sample xml data and Xquery . >> >> > >> >> > >> >> > Thanks & Regards >> >> > Adi >> >> > >> >> >> >> >> > > >
AB
Adi Babu15 Sep 2015 14:13 #
Hi Chirstian,
I tried what Martin suggested. Please find the below is the sample code what I am using for creating collection.
/*
This method takes the xml files list as input and process the list Create the xml collection DB, adds xml to the collection if already exists replace it
*/
public static void createCollection(ArrayList<String> listFiles){
ClientSession clientSession = null;
String strIndexedFile="";
try {
clientSession = new ClientSession("<<IP_ADDRESS>>", 1984,
"<<USER>>", "<<PWD>>");
String strDbName = "Sample_Collection";
String strKeysFileLocation =
"D:\\TEMP\\7GB_Collection_TestingTeam\\Files";
String res = clientSession.execute("XQUERY db:exists('" +
strDbName + "')");
//DB existence checking
if (!res.equalsIgnoreCase("true")) {
clientSession.execute("CREATE DB " + strDbName);
} else {
clientSession.execute("OPEN " + strDbName);
}
for (int i = 0; i < listFiles.size(); i++) {
strIndexedFile = listFiles.get(i);
String tempFlag=clientSession.execute("XQUERY db:exists('"
+ strDbName + "','" + strIndexedFile + "')");
clientSession.execute("ADD " + strKeysFileLocation +
File.separator + strIndexedFile);
}
System.out.println("InFormation::
"+clientSession.execute("INFO"));
//optimizing the current opend DB using OPTIMIZE ALL command
clientSession.execute("OPTIMIZE ALL");
System.out.println("InFormation::
"+clientSession.execute("INFO"));
} catch (Exception e) {
System.out.println("Exception files DB Updating: " +
e.getMessage());
e.printStackTrace();
} finally {
if (clientSession != null) {
try {
clientSession.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
}Thanks & Regards,
Adi
Quoted text · Christian Grün · 164 lines
On Tue, Sep 15, 2015 at 4:52 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Adi, > > I may not be up-to-date, as I expected you to send me an updated > version of your XQuery code. Did you try what Martín suggested? > > Christian > > > > On Tue, Sep 15, 2015 at 1:19 PM, Adi Babu <adibabu.b at gmail.com> wrote: > > Hai Chistian, > > Please find the attachment sample piece of code. > > > > Thanks & Regards > > Adi > > > > > > On Tue, Sep 15, 2015 at 4:34 PM, Christian Grün < > christian.gruen at gmail.com> > > wrote: > >> > >> Hi Adi, > >> > >> How does your new code look like? How did you rewrite the collection > >> function? > >> > >> Christian > >> > >> > >> > >> On Tue, Sep 15, 2015 at 12:48 PM, Adi Babu <adibabu.b at gmail.com> wrote: > >> > Hi Martin, > >> > I am working with collection data base only from starting on wards. > >> > Through > >> > mail it is not possible to share the large collection db(more than > 7GB) > >> > thats why I am sharing sample query(with xml doc but actually I am > using > >> > collection only). > >> > I have tried all your suggestions in the trail mail ,but it takes more > >> > time > >> > in 8.2.3 than 7.9 version. Can you please guide me to proceed further. > >> > > >> > Thanks & Regards, > >> > Adi > >> > > >> > On Fri, Sep 11, 2015 at 10:04 PM, Martín Ferrari > >> > <ferrari_martin at hotmail.com> wrote: > >> >> > >> >> HI Adi, > >> >> Have you tried Max's suggestion of adding the xml data to a > BaseX > >> >> db > >> >> first? Based on this: > >> >> > >> >> for $details in > >> >> > >> >> > >> >> > (collection("E:\Web-Projects\VodafoneUK\Docs\Cloud_Test_Stats\Performance_Analysis\Freq_Called_Sample_XML_Data.xml") > >> >> > >> >> > >> >> it looks like you're using the file Freq_Called_Sample_XML_Data.xml > >> >> directly from the filesystem, which means it has no indexes and must > be > >> >> fully scanned to get results. You should create a DB and upload that > >> >> xml > >> >> file with TEXTINDEX and ATTRINDEX enabled (which is the default). > You > >> >> might > >> >> need to run the OPTIMIZE ALL command after uploading the xml file, > you > >> >> can > >> >> run the INFO DB command and it will show if your database has > >> >> up-to-date > >> >> indexes or not. Once you have a db, you can also use the GUI to check > >> >> the > >> >> query plan that's being used for a particular query. > >> >> > >> >> > >> >> Martín. > >> >> > >> >> > >> >> ________________________________ > >> >> Date: Fri, 11 Sep 2015 18:37:05 +0530 > >> >> From: adibabu.b at gmail.com > >> >> To: mgaerber at arcor.de > >> >> CC: basex-talk at mailman.uni-konstanz.de > >> >> Subject: Re: [basex-talk] Fwd: Re : Basex Query Optimization Support > >> >> > >> >> Dear Max, > >> >> Currently I am using collection only(size 7.96 GB), I am sharing the > >> >> query > >> >> for your reference only. I tried the latest version 8.2.3 also but it > >> >> takes > >> >> more time than 7.9. > >> >> Can you please assist me to proceed further. > >> >> > >> >> Thanks & Regards, > >> >> Adi > >> >> > >> >> On Tue, Sep 8, 2015 at 9:15 PM, Maximilian Gärber <mgaerber at arcor.de > > > >> >> wrote: > >> >> > >> >> Hi Adi, > >> >> > >> >> some general pointers: > >> >> * Server version 7.9 is missing all the latest perf improvements, > >> >> current version is 8.2.3 > >> >> * You are accessing the files in the file system - try adding them > to > >> >> the database first, then you can make use of the text index etc. See > >> >> http://docs.basex.org/wiki/Indexes#Text_Index > >> >> > >> >> Regards, > >> >> > >> >> Max > >> >> > >> >> > >> >> 2015-09-04 8:31 GMT+02:00 Adi Babu <adibabu.b at gmail.com>: > >> >> > Dear Team, > >> >> > I am facing performance issue taking long time to execute the > >> >> > Xquery in Basex(Client Server architecture), can you please guide > me > >> >> > to > >> >> > optimize the query performance. > >> >> > > >> >> > Below are the statistics & Basex server hardware specifications. > >> >> > > >> >> > Statistics : > >> >> > ------------- > >> >> > Total Records count : 6000000 > >> >> > Fetch count : 10000 > >> >> > Hits Parallel hits : 100 > >> >> > Time Taking : 750 seconds > >> >> > Architecture : Client Server Architecture > >> >> > > >> >> > Hardware Specifications of Basex Sever : > >> >> > ---------------------------------------------------------- > >> >> > Operating System : Red Hat Enterprise Linux 64bit > >> >> > OS Version : Linux 2.6.32 > >> >> > Hard Disk : 500 GB > >> >> > Memory Capacity : 16 GB Ram > >> >> > Processor Family : Intel (R) Xeon(R) CPU E5 -2680 V2 > >> >> > Processor Speed : 2.80 GHz > >> >> > Physical Processors : 8 > >> >> > Virtual Processors/cores : 4 > >> >> > Server Version : basex 7.9 > >> >> > > >> >> > Note : I have attached the sample xml data and Xquery . > >> >> > > >> >> > > >> >> > Thanks & Regards > >> >> > Adi > >> >> > > >> >> > >> >> > >> > > > > > >
MF
Hi Adi, The attached code you sent is for uploading the files (by the way, I believe you don't need the tempFlag on each iteration). I'd think that would take a long time to upload, but it should only run once, right?. So you're query now should be something like this: for $details incollection('Sample_Collection')/
SUBCUSTBRK[AN[@NO = (987654321)]][DOB >='2014-04-01' and DOB <= '2015-03-31']/CNOS/ROW)
Have you isolated that this query is the one that's taking a long time to execute? Have you used the GUI to find out the query plan that's being used? You could start by using just one condition, then add the others. The query plan should show you whether it's using indexes or not. You could also double check that the data has been indexed by using the index:text and
index:attributes functions, though they might return a lot of data, but they will tell you which data has been indexed. Hope it helps, Martín.
Date: Tue, 15 Sep 2015 16:49:22 +0530
Subject: Re: [basex-talk] Fwd: Re : Basex Query Optimization SupportQuoted text · adibabu.b at gmail.com · 116 lines
From: adibabu.b at gmail.com To: christian.gruen at gmail.com CC: ferrari_martin at hotmail.com; basex-talk at mailman.uni-konstanz.de Hai Chistian,Please find the attachment sample piece of code. Thanks & RegardsAdi On Tue, Sep 15, 2015 at 4:34 PM, Christian Grün <christian.gruen at gmail.com> wrote: Hi Adi, How does your new code look like? How did you rewrite the collection function? Christian On Tue, Sep 15, 2015 at 12:48 PM, Adi Babu <adibabu.b at gmail.com> wrote: > Hi Martin, > I am working with collection data base only from starting on wards. Through > mail it is not possible to share the large collection db(more than 7GB) > thats why I am sharing sample query(with xml doc but actually I am using > collection only). > I have tried all your suggestions in the trail mail ,but it takes more time > in 8.2.3 than 7.9 version. Can you please guide me to proceed further. > > Thanks & Regards, > Adi > > On Fri, Sep 11, 2015 at 10:04 PM, Martín Ferrari > <ferrari_martin at hotmail.com> wrote: >> >> HI Adi, >> Have you tried Max's suggestion of adding the xml data to a BaseX db >> first? Based on this: >> >> for $details in >> >> (collection("E:\Web-Projects\VodafoneUK\Docs\Cloud_Test_Stats\Performance_Analysis\Freq_Called_Sample_XML_Data.xml") >> >> >> it looks like you're using the file Freq_Called_Sample_XML_Data.xml >> directly from the filesystem, which means it has no indexes and must be >> fully scanned to get results. You should create a DB and upload that xml >> file with TEXTINDEX and ATTRINDEX enabled (which is the default). You might >> need to run the OPTIMIZE ALL command after uploading the xml file, you can >> run the INFO DB command and it will show if your database has up-to-date >> indexes or not. Once you have a db, you can also use the GUI to check the >> query plan that's being used for a particular query. >> >> >> Martín. >> >> >> ________________________________ >> Date: Fri, 11 Sep 2015 18:37:05 +0530 >> From: adibabu.b at gmail.com >> To: mgaerber at arcor.de >> CC: basex-talk at mailman.uni-konstanz.de >> Subject: Re: [basex-talk] Fwd: Re : Basex Query Optimization Support >> >> Dear Max, >> Currently I am using collection only(size 7.96 GB), I am sharing the query >> for your reference only. I tried the latest version 8.2.3 also but it takes >> more time than 7.9. >> Can you please assist me to proceed further. >> >> Thanks & Regards, >> Adi >> >> On Tue, Sep 8, 2015 at 9:15 PM, Maximilian Gärber <mgaerber at arcor.de> >> wrote: >> >> Hi Adi, >> >> some general pointers: >> * Server version 7.9 is missing all the latest perf improvements, >> current version is 8.2.3 >> * You are accessing the files in the file system - try adding them to >> the database first, then you can make use of the text index etc. See >> http://docs.basex.org/wiki/Indexes#Text_Index >> >> Regards, >> >> Max >> >> >> 2015-09-04 8:31 GMT+02:00 Adi Babu <adibabu.b at gmail.com>: >> > Dear Team, >> > I am facing performance issue taking long time to execute the >> > Xquery in Basex(Client Server architecture), can you please guide me to >> > optimize the query performance. >> > >> > Below are the statistics & Basex server hardware specifications. >> > >> > Statistics : >> > ------------- >> > Total Records count : 6000000 >> > Fetch count : 10000 >> > Hits Parallel hits : 100 >> > Time Taking : 750 seconds >> > Architecture : Client Server Architecture >> > >> > Hardware Specifications of Basex Sever : >> > ---------------------------------------------------------- >> > Operating System : Red Hat Enterprise Linux 64bit >> > OS Version : Linux 2.6.32 >> > Hard Disk : 500 GB >> > Memory Capacity : 16 GB Ram >> > Processor Family : Intel (R) Xeon(R) CPU E5 -2680 V2 >> > Processor Speed : 2.80 GHz >> > Physical Processors : 8 >> > Virtual Processors/cores : 4 >> > Server Version : basex 7.9 >> > >> > Note : I have attached the sample xml data and Xquery . >> > >> > >> > Thanks & Regards >> > Adi >> > >> >> >
Written for BaseX 8.2. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery