BaseX performance issue
Hi BaseX team,
We currently encountered performance issue using BaseX.
We noticed that when # of documents increased in BaseX database, it will double time to retrieve documents in database. Our test cases execution time double when # of documents increased from 200 to 400. And If number of document reach to 5000, The test case execution time increase much more.
Is that any performance testing BaseX team can share with us to compare, or there is any documentation or suggestion for BaseX database turning that we should follow?
Thanks
Yvonne
Hi Yvonne,
could you give us some hints how you retrieve documents? Do you use one of the existing APIs, db:open, or any other functions?
TIA Christian
Quoted text · Wang, Yvonne · 17 lines
> We noticed that when # of documents increased in BaseX database, it will > double time to retrieve documents in database. > Our test cases execution time double when # of documents increased from 200 > to 400. And If number of document reach to 5000, > The test case execution time increase much more. > > Is that any performance testing BaseX team can share with us to compare, or > there is any documentation or suggestion for BaseX database turning that we > should follow? > > > Thanks > > Yvonne > > >
Currently all document we installed is in binary format. So we use bin:decode-String(db:retrieve(‘schema’, path)
Thanks
Yvonne
Quoted text · Christian Grün · 27 lines
On 6/21/16, 1:44 PM, "Christian Grün" <christian.gruen at gmail.com> wrote: >Hi Yvonne, > >could you give us some hints how you retrieve documents? Do you use >one of the existing APIs, db:open, or any other functions? > >TIA >Christian > > >> We noticed that when # of documents increased in BaseX database, it will >> double time to retrieve documents in database. >> Our test cases execution time double when # of documents increased from 200 >> to 400. And If number of document reach to 5000, >> The test case execution time increase much more. >> >> Is that any performance testing BaseX team can share with us to compare, or >> there is any documentation or suggestion for BaseX database turning that we >> should follow? >> >> >> Thanks >> >> Yvonne >> >> >>
Quoted text · Wang, Yvonne · 2 lines
> Currently all document we installed is in binary format. > So we use bin:decode-String(db:retrieve(‘schema’, path)
I see. In that case, the degradation is due to your filesystem (find some details in [1]). You could either try to run BaseX on a partition with a file system that is capable of handling a larger amount of files, or you can distribute your files in different sub-paths.
0/1/file01.txt 1/2/file12.txt
Out of interest: BaseX is particularly good at handling XML structures. Why do you store everything are binary data?
Quoted text · Wang, Yvonne · 38 lines
> > > > Thanks > > Yvonne > > > > > On 6/21/16, 1:44 PM, "Christian Grün" <christian.gruen at gmail.com> wrote: > >>Hi Yvonne, >> >>could you give us some hints how you retrieve documents? Do you use >>one of the existing APIs, db:open, or any other functions? >> >>TIA >>Christian >> >> >>> We noticed that when # of documents increased in BaseX database, it will >>> double time to retrieve documents in database. >>> Our test cases execution time double when # of documents increased from 200 >>> to 400. And If number of document reach to 5000, >>> The test case execution time increase much more. >>> >>> Is that any performance testing BaseX team can share with us to compare, or >>> there is any documentation or suggestion for BaseX database turning that we >>> should follow? >>> >>> >>> Thanks >>> >>> Yvonne >>> >>> >>>
Actually, they are json format files, during testing, we only use very small size of .json format files, But our test cases including insert, update, replace, get and delete using bases db functions.
Thanks
Yvonne
Quoted text · Christian Grün · 56 lines
On 6/21/16, 11:41 PM, "Christian Grün" <christian.gruen at gmail.com> wrote: >> Currently all document we installed is in binary format. >> So we use bin:decode-String(db:retrieve(‘schema’, path) > >I see. In that case, the degradation is due to your filesystem (find >some details in [1]). You could either try to run BaseX on a partition >with a file system that is capable of handling a larger amount of >files, or you can distribute your files in different sub-paths. > > 0/1/file01.txt > 1/2/file12.txt > >Out of interest: BaseX is particularly good at handling XML >structures. Why do you store everything are binary data? > >[1] http://docs.basex.org/wiki/Binary_Data > > >> >> >> >> Thanks >> >> Yvonne >> >> >> >> >> On 6/21/16, 1:44 PM, "Christian Grün" <christian.gruen at gmail.com> wrote: >> >>>Hi Yvonne, >>> >>>could you give us some hints how you retrieve documents? Do you use >>>one of the existing APIs, db:open, or any other functions? >>> >>>TIA >>>Christian >>> >>> >>>> We noticed that when # of documents increased in BaseX database, it will >>>> double time to retrieve documents in database. >>>> Our test cases execution time double when # of documents increased from 200 >>>> to 400. And If number of document reach to 5000, >>>> The test case execution time increase much more. >>>> >>>> Is that any performance testing BaseX team can share with us to compare, or >>>> there is any documentation or suggestion for BaseX database turning that we >>>> should follow? >>>> >>>> >>>> Thanks >>>> >>>> Yvonne >>>> >>>> >>>>
Quoted text · Wang, Yvonne · 2 lines
> Actually, they are json format files, during testing, we only use very small size of .json format files, > But our test cases including insert, update, replace, get and delete using bases db functions.
Did you consider using the JSON Module [1]? It gives you great flexibility when it comes to navigating and updating JSON in the XML format, and it can easily be output as JSON again.
Quoted text · Yvonne · 68 lines
On Wed, Jun 22, 2016 at 7:17 PM, Wang, Yvonne <yvonne.wang at sap.com> wrote: > Actually, they are json format files, during testing, we only use very small size of .json format files, > But our test cases including insert, update, replace, get and delete using bases db functions. > > > Thanks > > Yvonne > > > > On 6/21/16, 11:41 PM, "Christian Grün" <christian.gruen at gmail.com> wrote: > >>> Currently all document we installed is in binary format. >>> So we use bin:decode-String(db:retrieve(‘schema’, path) >> >>I see. In that case, the degradation is due to your filesystem (find >>some details in [1]). You could either try to run BaseX on a partition >>with a file system that is capable of handling a larger amount of >>files, or you can distribute your files in different sub-paths. >> >> 0/1/file01.txt >> 1/2/file12.txt >> >>Out of interest: BaseX is particularly good at handling XML >>structures. Why do you store everything are binary data? >> >>[1] http://docs.basex.org/wiki/Binary_Data >> >> >>> >>> >>> >>> Thanks >>> >>> Yvonne >>> >>> >>> >>> >>> On 6/21/16, 1:44 PM, "Christian Grün" <christian.gruen at gmail.com> wrote: >>> >>>>Hi Yvonne, >>>> >>>>could you give us some hints how you retrieve documents? Do you use >>>>one of the existing APIs, db:open, or any other functions? >>>> >>>>TIA >>>>Christian >>>> >>>> >>>>> We noticed that when # of documents increased in BaseX database, it will >>>>> double time to retrieve documents in database. >>>>> Our test cases execution time double when # of documents increased from 200 >>>>> to 400. And If number of document reach to 5000, >>>>> The test case execution time increase much more. >>>>> >>>>> Is that any performance testing BaseX team can share with us to compare, or >>>>> there is any documentation or suggestion for BaseX database turning that we >>>>> should follow? >>>>> >>>>> >>>>> Thanks >>>>> >>>>> Yvonne >>>>> >>>>> >>>>>
Written for BaseX 8.4. BaseX may have changed since: please check the current documentation.