Windows fast, linux very slow

On Thu, Oct 27, 2011 at 10:36 PM, Sven Rega <sven.rega at gmx.de> wrote: > Hy, > > i'm using the basexserver 6.7.1 on a windows (i7; 8GB; 2,8GHz) and on > different linux machines (i7; 16GB, 3,4GHz). > the same xml file (~2,5GByte) and one simple query has i big difference in > speed between both machines and i don't know, where the bottleneck is coming > from. > on the windows machine i get througput of ~600 to 1000 xml entities per > second. On the linux machine i only get max. 110 xml entities per second. > > I use the same jvm setting on both machines: -server -Xmx3g > I also use the index (text, path, attribtue) on both machines. > > The query: > > "declare function local:ProductValues() { > for $n in //Product[(@ID='A' or @ID='D' or @ID='G')]/Values > return > <pv><utid>{data($n/../@ID)}</utid><id>{data($n/../@SecondID)}</id>{$n}</pv> > }; > local:ProductValues()" > > I read from the basex server over lan with 1GBit. > > Does anybody have a idea, what could be the reason for such speed difference > between windows and linux? > > king regards > Sven > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >

On 27.10.2011 22:40, Christian Grün wrote: > Hi Sven, > > thanks for your mail. Usually, there shouldn't be any notable > differences between Windows and Linux (only the Mac JVM is known to be > slower). Before giving any further advices, it might be interesting to > hear if the problems persist with the latest version of BaseX (Version > 7.01) ? > > Best, > Christian > ___________________________ > > On Thu, Oct 27, 2011 at 10:36 PM, Sven Rega<sven.rega at gmx.de> wrote: >> Hy, >> >> i'm using the basexserver 6.7.1 on a windows (i7; 8GB; 2,8GHz) and on >> different linux machines (i7; 16GB, 3,4GHz). >> the same xml file (~2,5GByte) and one simple query has i big difference in >> speed between both machines and i don't know, where the bottleneck is coming >> from. >> on the windows machine i get througput of ~600 to 1000 xml entities per >> second. On the linux machine i only get max. 110 xml entities per second. >> >> I use the same jvm setting on both machines: -server -Xmx3g >> I also use the index (text, path, attribtue) on both machines. >> >> The query: >> >> "declare function local:ProductValues() { >> for $n in //Product[(@ID='A' or @ID='D' or @ID='G')]/Values >> return >> <pv><utid>{data($n/../@ID)}</utid><id>{data($n/../@SecondID)}</id>{$n}</pv> >> }; >> local:ProductValues()" >> >> I read from the basex server over lan with 1GBit. >> >> Does anybody have a idea, what could be the reason for such speed difference >> between windows and linux? >> >> king regards >> Sven >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>

Am 28.10.2011 12:06, schrieb Sven Rega: > Hy Christian, > > I currently don't plan to change the database to a new version, > because "never change a running system" :-) > But on weekend i will update my process to test the new database > version 7.1. > > Some current Profiling results: > I test the process with the profiling software VisualVM from > sun/oracle and i think, i find the bottleneck: > org.basex.io.in.BufferInput.next() > On the windows machine i see, that this method is called 11150 times > and executes this method in sum in 1890ms. (11150/1890= 5,89 calls per > ms) > On the linux machine i see, that the method is called 255635 times and > needs in sum 834981ms (255635/834981 = 0,306 calls per ms). > So the factor between both executions is around 20. > > Do you have any idea, which reason could be in this method? > > king regards > Sven > > On 27.10.2011 22:40, Christian Grün wrote: >> Hi Sven, >> >> thanks for your mail. Usually, there shouldn't be any notable >> differences between Windows and Linux (only the Mac JVM is known to be >> slower). Before giving any further advices, it might be interesting to >> hear if the problems persist with the latest version of BaseX (Version >> 7.01) ? >> >> Best, >> Christian >> ___________________________ >> >> On Thu, Oct 27, 2011 at 10:36 PM, Sven Rega<sven.rega at gmx.de> wrote: >>> Hy, >>> >>> i'm using the basexserver 6.7.1 on a windows (i7; 8GB; 2,8GHz) and on >>> different linux machines (i7; 16GB, 3,4GHz). >>> the same xml file (~2,5GByte) and one simple query has i big >>> difference in >>> speed between both machines and i don't know, where the bottleneck >>> is coming >>> from. >>> on the windows machine i get througput of ~600 to 1000 xml entities per >>> second. On the linux machine i only get max. 110 xml entities per >>> second. >>> >>> I use the same jvm setting on both machines: -server -Xmx3g >>> I also use the index (text, path, attribtue) on both machines. >>> >>> The query: >>> >>> "declare function local:ProductValues() { >>> for $n in //Product[(@ID='A' or @ID='D' or @ID='G')]/Values >>> return >>> <pv><utid>{data($n/../@ID)}</utid><id>{data($n/../@SecondID)}</id>{$n}</pv> >>> >>> }; >>> local:ProductValues()" >>> >>> I read from the basex server over lan with 1GBit. >>> >>> Does anybody have a idea, what could be the reason for such speed >>> difference >>> between windows and linux? >>> >>> king regards >>> Sven >>> _______________________________________________ >>> BaseX-Talk mailing list >>> BaseX-Talk at mailman.uni-konstanz.de >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>> > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

On 28.10.2011 12:21, Stefan Koch wrote: > Hi Sven, > > which filesystem do you use? > > Some databases perform slow in not optimized XFS systens. > > http://ondrejcertik.blogspot.com/2008/02/xfs-is-20x-slower-than-ext3-with.html > > > May also be hardware related, try to test your disk performance with > hdparm and compare it to your windows machine. > > Greetings, > > > > > > > > Am 28.10.2011 12:06, schrieb Sven Rega: >> Hy Christian, >> >> I currently don't plan to change the database to a new version, >> because "never change a running system" :-) >> But on weekend i will update my process to test the new database >> version 7.1. >> >> Some current Profiling results: >> I test the process with the profiling software VisualVM from >> sun/oracle and i think, i find the bottleneck: >> org.basex.io.in.BufferInput.next() >> On the windows machine i see, that this method is called 11150 times >> and executes this method in sum in 1890ms. (11150/1890= 5,89 calls >> per ms) >> On the linux machine i see, that the method is called 255635 times >> and needs in sum 834981ms (255635/834981 = 0,306 calls per ms). >> So the factor between both executions is around 20. >> >> Do you have any idea, which reason could be in this method? >> >> king regards >> Sven >> >> On 27.10.2011 22:40, Christian Grün wrote: >>> Hi Sven, >>> >>> thanks for your mail. Usually, there shouldn't be any notable >>> differences between Windows and Linux (only the Mac JVM is known to be >>> slower). Before giving any further advices, it might be interesting to >>> hear if the problems persist with the latest version of BaseX (Version >>> 7.01) ? >>> >>> Best, >>> Christian >>> ___________________________ >>> >>> On Thu, Oct 27, 2011 at 10:36 PM, Sven Rega<sven.rega at gmx.de> wrote: >>>> Hy, >>>> >>>> i'm using the basexserver 6.7.1 on a windows (i7; 8GB; 2,8GHz) and on >>>> different linux machines (i7; 16GB, 3,4GHz). >>>> the same xml file (~2,5GByte) and one simple query has i big >>>> difference in >>>> speed between both machines and i don't know, where the bottleneck >>>> is coming >>>> from. >>>> on the windows machine i get througput of ~600 to 1000 xml entities >>>> per >>>> second. On the linux machine i only get max. 110 xml entities per >>>> second. >>>> >>>> I use the same jvm setting on both machines: -server -Xmx3g >>>> I also use the index (text, path, attribtue) on both machines. >>>> >>>> The query: >>>> >>>> "declare function local:ProductValues() { >>>> for $n in //Product[(@ID='A' or @ID='D' or @ID='G')]/Values >>>> return >>>> <pv><utid>{data($n/../@ID)}</utid><id>{data($n/../@SecondID)}</id>{$n}</pv> >>>> >>>> }; >>>> local:ProductValues()" >>>> >>>> I read from the basex server over lan with 1GBit. >>>> >>>> Does anybody have a idea, what could be the reason for such speed >>>> difference >>>> between windows and linux? >>>> >>>> king regards >>>> Sven >>>> _______________________________________________ >>>> BaseX-Talk mailing list >>>> BaseX-Talk at mailman.uni-konstanz.de >>>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>>> >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >

On 27.10.2011 22:40, Christian Grün wrote: > Hi Sven, > > thanks for your mail. Usually, there shouldn't be any notable > differences between Windows and Linux (only the Mac JVM is known to be > slower). Before giving any further advices, it might be interesting to > hear if the problems persist with the latest version of BaseX (Version > 7.01) ? > > Best, > Christian > ___________________________ > > On Thu, Oct 27, 2011 at 10:36 PM, Sven Rega<sven.rega at gmx.de> wrote: >> Hy, >> >> i'm using the basexserver 6.7.1 on a windows (i7; 8GB; 2,8GHz) and on >> different linux machines (i7; 16GB, 3,4GHz). >> the same xml file (~2,5GByte) and one simple query has i big difference in >> speed between both machines and i don't know, where the bottleneck is coming >> from. >> on the windows machine i get througput of ~600 to 1000 xml entities per >> second. On the linux machine i only get max. 110 xml entities per second. >> >> I use the same jvm setting on both machines: -server -Xmx3g >> I also use the index (text, path, attribtue) on both machines. >> >> The query: >> >> "declare function local:ProductValues() { >> for $n in //Product[(@ID='A' or @ID='D' or @ID='G')]/Values >> return >> <pv><utid>{data($n/../@ID)}</utid><id>{data($n/../@SecondID)}</id>{$n}</pv> >> }; >> local:ProductValues()" >> >> I read from the basex server over lan with 1GBit. >> >> Does anybody have a idea, what could be the reason for such speed difference >> between windows and linux? >> >> king regards >> Sven >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>

> Currently i try the basex version 7.0.1. Now, the speed/performance isn't > the problem (is very fast now :-) )

> but i get an problem with my xml data and i think, it is an heisenbug.

> What i'm doing wrong? Or is this a bug? Why this problem was never seen in > the basex 6.7.1 version?

> Since Version 7.0, we use our internal XML parser as default parser. > It would be interesting to hear what happens if you switch back to the > Java XML parser (see [1] for details). > > [1] http://docs.basex.org/wiki/Parser#XML_Parsers

>> Since Version 7.0, we use our internal XML parser as default parser. >> It would be interesting to hear what happens if you switch back to the >> Java XML parser (see [1] for details). >> >> [1] http://docs.basex.org/wiki/Parser#XML_Parsers > That site still says that the SAX parser is the default. Which information is > correct?

On 02.11.2011 18:20, Christian Grün wrote: > Hi Sven, > >> Currently i try the basex version 7.0.1. Now, the speed/performance isn't >> the problem (is very fast now :-) ) > ..always nice to hear.. > >> but i get an problem with my xml data and i think, it is an heisenbug. > ..not that nice to hear ;).. > >> What i'm doing wrong? Or is this a bug? Why this problem was never seen in >> the basex 6.7.1 version? > Since Version 7.0, we use our internal XML parser as default parser. > It would be interesting to hear what happens if you switch back to the > Java XML parser (see [1] for details). > > Best, > Christian > > [1] http://docs.basex.org/wiki/Parser#XML_Parsers

On Thu, Nov 3, 2011 at 1:38 PM, Sven Rega <sven.rega at gmx.de> wrote: > Hy Christian, > > i tested the import with the sax and with the internal parser and results > are also corrupt. > Process: import data from one large xml file (2,5GB) into one empty db > > my data: > <root> > <article ID="xys"> > <values> > .... > <value ..>.....</value> > <value ..>.....</value> > <value ID="abcdefg" refcontent="other article">&t;b&gt;System requirements: > &lt;/b&gt;</value> > </values> > </article> > </root> > > QUERY_1: > for $n in //article[(@ID='Article' or @ID='Other')]/values > return <pv><utid>{data($n/../@ID)}</utid>{$n}</pv> > > QUERY_2: > for $n in //article[(@ID='Article' or @ID='Other') and ID="abc"]/values > return <pv><utid>{data($n/../@ID)}</utid>{$n}</pv> > > > If i query with QUERY_1, i get one corrupt results which looks like: > <pv> > <utid>abc</utid> > .. > <value ...> ...... <value <----- here the closing ">" is missing > .. > </pv> > > but if i use query QUERY_2, which only grab the one specific article with > the corrupt data, i get a correct result, without the missing closing tag. > > So the DB contains all correct information, but the query (QUERY_1) over all > article returns corrupt data. > > Do you have some other ideas for the reason of this problem/bug? Do you > know, which code parts are changed from version 6.7.1 to 7.0.1, which are > involved in this problem? > > regards > Sven > > > On 02.11.2011 18:20, Christian Grün wrote: >> >> Hi Sven, >> >>> Currently i try the basex version 7.0.1. Now, the speed/performance isn't >>> the problem (is very fast now :-) ) >> >> ..always nice to hear.. >> >>> but i get an problem with my xml data and i think, it is an heisenbug. >> >> ..not that nice to hear ;).. >> >>> What i'm doing wrong? Or is this a bug? Why this problem was never seen >>> in >>> the basex 6.7.1 version? >> >> Since Version 7.0, we use our internal XML parser as default parser. >> It would be interesting to hear what happens if you switch back to the >> Java XML parser (see [1] for details). >> >> Best, >> Christian >> >> [1] http://docs.basex.org/wiki/Parser#XML_Parsers > >

On 03.11.2011 13:54, Christian Grün wrote: > Hi Sven, > > I'm not sure what to think about this issue. This could also be a > serialization problem, as angle brackets (<,>) are not stored in the > database at all. Which API do you use to serialize the results/have > you tried another API as well? > > Christian > ___________________________ > > On Thu, Nov 3, 2011 at 1:38 PM, Sven Rega<sven.rega at gmx.de> wrote: >> Hy Christian, >> >> i tested the import with the sax and with the internal parser and results >> are also corrupt. >> Process: import data from one large xml file (2,5GB) into one empty db >> >> my data: >> <root> >> <article ID="xys"> >> <values> >> .... >> <value ..>.....</value> >> <value ..>.....</value> >> <value ID="abcdefg" refcontent="other article">&t;b&gt;System requirements: >> &lt;/b&gt;</value> >> </values> >> </article> >> </root> >> >> QUERY_1: >> for $n in //article[(@ID='Article' or @ID='Other')]/values >> return<pv><utid>{data($n/../@ID)}</utid>{$n}</pv> >> >> QUERY_2: >> for $n in //article[(@ID='Article' or @ID='Other') and ID="abc"]/values >> return<pv><utid>{data($n/../@ID)}</utid>{$n}</pv> >> >> >> If i query with QUERY_1, i get one corrupt results which looks like: >> <pv> >> <utid>abc</utid> >> .. >> <value ...> ......<value<----- here the closing ">" is missing >> .. >> </pv> >> >> but if i use query QUERY_2, which only grab the one specific article with >> the corrupt data, i get a correct result, without the missing closing tag. >> >> So the DB contains all correct information, but the query (QUERY_1) over all >> article returns corrupt data. >> >> Do you have some other ideas for the reason of this problem/bug? Do you >> know, which code parts are changed from version 6.7.1 to 7.0.1, which are >> involved in this problem? >> >> regards >> Sven >> >> >> On 02.11.2011 18:20, Christian Grün wrote: >>> Hi Sven, >>> >>>> Currently i try the basex version 7.0.1. Now, the speed/performance isn't >>>> the problem (is very fast now :-) ) >>> ..always nice to hear.. >>> >>>> but i get an problem with my xml data and i think, it is an heisenbug. >>> ..not that nice to hear ;).. >>> >>>> What i'm doing wrong? Or is this a bug? Why this problem was never seen >>>> in >>>> the basex 6.7.1 version? >>> Since Version 7.0, we use our internal XML parser as default parser. >>> It would be interesting to hear what happens if you switch back to the >>> Java XML parser (see [1] for details). >>> >>> Best, >>> Christian >>> >>> [1] http://docs.basex.org/wiki/Parser#XML_Parsers >>

Am Donnerstag, 3. November 2011, 14:15:31 schrieb Sven Rega: > Hy Christian, > > Some idea from me: could it be a buffer problem (overlapping issue) ? > The Query_1 produce data with around 2,1 GB. With the same query, the > angle bracket is always missing on the same entity on the same (byte) > offset in the result. > May be some output buffer from the server side or some input buffer from > the client side has a bug?

On 03.11.2011 14:34, Dimitar Popov wrote: > Am Donnerstag, 3. November 2011, 14:15:31 schrieb Sven Rega: >> Hy Christian, >> >> Some idea from me: could it be a buffer problem (overlapping issue) ? >> The Query_1 produce data with around 2,1 GB. With the same query, the >> angle bracket is always missing on the same entity on the same (byte) >> offset in the result. >> May be some output buffer from the server side or some input buffer from >> the client side has a bug? > Hi Sven, > > can you try running basex as stand-alone application (i.e. not client-server). > Thus, we'll be able to see if the problem is in the client-server > communication... > > Greetz, > Dimitar

On Thu, Nov 3, 2011 at 5:08 PM, Sven Rega <sven.rega at gmx.de> wrote: > Hy Dimitar, > > With the standalone mode, the described problem is not present. So it could > be really a problem in the client/server communication. > > regards, > Sven > > > On 03.11.2011 14:34, Dimitar Popov wrote: >> >> Am Donnerstag, 3. November 2011, 14:15:31 schrieb Sven Rega: >>> >>> Hy Christian, >>> >>> Some idea from me: could it be a buffer problem (overlapping issue) ? >>> The Query_1 produce data with around 2,1 GB. With the same query, the >>> angle bracket is always missing on the same entity on the same (byte) >>> offset in the result. >>> May be some output buffer from the server side or some input buffer from >>> the client side has a bug? >> >> Hi Sven, >> >> can you try running basex as stand-alone application (i.e. not >> client-server). >> Thus, we'll be able to see if the problem is in the client-server >> communication... >> >> Greetz, >> Dimitar > > >

On 03.11.2011 18:56, Christian Grün wrote: > Hi Sven, > > sounds indeed like a client issue. I have no idea, however, what the > problem could be, as our client code is pretty light-weight. Did you > manage to encounter the same problem by calling ClientQuery.execute > (..or is the result too large to be generated by this command)? It > would be great if you could provide us with a reproducible example - > but I know it's sometimes difficult to generate one. > > Christian > ___________________________ > > On Thu, Nov 3, 2011 at 5:08 PM, Sven Rega<sven.rega at gmx.de> wrote: >> Hy Dimitar, >> >> With the standalone mode, the described problem is not present. So it could >> be really a problem in the client/server communication. >> >> regards, >> Sven >> >> >> On 03.11.2011 14:34, Dimitar Popov wrote: >>> Am Donnerstag, 3. November 2011, 14:15:31 schrieb Sven Rega: >>>> Hy Christian, >>>> >>>> Some idea from me: could it be a buffer problem (overlapping issue) ? >>>> The Query_1 produce data with around 2,1 GB. With the same query, the >>>> angle bracket is always missing on the same entity on the same (byte) >>>> offset in the result. >>>> May be some output buffer from the server side or some input buffer from >>>> the client side has a bug? >>> Hi Sven, >>> >>> can you try running basex as stand-alone application (i.e. not >>> client-server). >>> Thus, we'll be able to see if the problem is in the client-server >>> communication... >>> >>> Greetz, >>> Dimitar >> >>

On 03.11.2011 19:25, Sven Rega wrote: > Hy Christian, > > I think also, that it's a client problem. > With the direct execution of ClientQuery.execute i get an exception: > > Exception in thread "main" java.lang.NegativeArraySizeException > at java.util.Arrays.copyOf(Arrays.java:2786) > at org.basex.io.out.ArrayOutput.write(ArrayOutput.java:37) > at org.basex.server.ClientSession.receive(ClientSession.java:305) > at org.basex.server.ClientSession.receive(ClientSession.java:268) > at org.basex.server.ClientSession.execute(ClientSession.java:174) > at org.basex.server.Session.execute(Session.java:51) > at test1.main(test1.java:66) > > The data, which i receive from the basexDB, has a size around of 2,16 > GByte. The max. int value could be 2,147,483,647 and so its clear, > that the receive-Method get an negative value. > > void receive(final BufferInput bi, final OutputStream os) throws > IOException { > final DecodingInput di = new DecodingInput(bi); > for(int b; (b = di.read()) != -1;) os.write(b); > } > > May be the size of my result with session.query() and the one missing > character depends also on the max integer value?!? > > regars > Sven > > On 03.11.2011 18:56, Christian Grün wrote: >> Hi Sven, >> >> sounds indeed like a client issue. I have no idea, however, what the >> problem could be, as our client code is pretty light-weight. Did you >> manage to encounter the same problem by calling ClientQuery.execute >> (..or is the result too large to be generated by this command)? It >> would be great if you could provide us with a reproducible example - >> but I know it's sometimes difficult to generate one. >> >> Christian >> ___________________________ >> >> On Thu, Nov 3, 2011 at 5:08 PM, Sven Rega<sven.rega at gmx.de> wrote: >>> Hy Dimitar, >>> >>> With the standalone mode, the described problem is not present. So it could >>> be really a problem in the client/server communication. >>> >>> regards, >>> Sven >>> >>> >>> On 03.11.2011 14:34, Dimitar Popov wrote: >>>> Am Donnerstag, 3. November 2011, 14:15:31 schrieb Sven Rega: >>>>> Hy Christian, >>>>> >>>>> Some idea from me: could it be a buffer problem (overlapping issue) ? >>>>> The Query_1 produce data with around 2,1 GB. With the same query, the >>>>> angle bracket is always missing on the same entity on the same (byte) >>>>> offset in the result. >>>>> May be some output buffer from the server side or some input buffer from >>>>> the client side has a bug? >>>> Hi Sven, >>>> >>>> can you try running basex as stand-alone application (i.e. not >>>> client-server). >>>> Thus, we'll be able to see if the problem is in the client-server >>>> communication... >>>> >>>> Greetz, >>>> Dimitar >>> > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

> if i looked into the code of org.basex.server.Query is see the object value > "pos" which is from the java type int. In the basex version 6.7.1 this "pos" > variable did not exists. Also in the new 7.0.1 version some caching > mechanism seems to be implemented, which was not in the old version. > I think, in these methods the problem could be raised.