Flush command and data corruption

On 03/08/2017 06:20 PM, Colin Delacroix wrote: > Hi everyone, > > (warning this is a long read) > > I'm using server version 8.6.1 and writing a Java client to move lots > of documents from a SQL database to a BaseX database. > The basic version works just fine. Tested up to 1M documents. > > Now I'm trying to do exactly the same but with AUTOFLUSH disabled + > manually FLUSH every N documents and it fails after just a few inserts. > > Pseudo code: > (set up client session, create and open new database) > session.execute(new Set(MainOptions.AUTOFLUSH, false)); > int i = 0; > while(doc = nextDocs()) { > session.add(doc.getKey(), doc.getInputStream()); > if(i++ % 10 == 0) { /* value reduced to 10 for testing purposes */ > session.execute(new Flush()); > } > } > (tear down session) > > Remember tha > > After the first FLUSH has been issued, the ADD directive starts > randomly throwing a org.basex.core.BaseXException (when using the > BaseXClient implementation it was its inner ok() method that started > returning false). > > I tracked the root cause in the server: the SAXParser complaining > about the documents not being well-formed (the exact same documents > are all imported fine with AUTOFLUSH on). In the server standard logs > I can see why: parts of those documents are suddenly missing (usually > the beginning + command part) at parse time! > > That is the symptom, now for the cause I'm also observing that the ADD > and FLUSH directive in the logs are out of order, exactly like what > would happen in the FLUSH commands were not properly queued. > > I was expecting something like : > ADD > ADD > ADD > (N times) > FLUSH > ADD > ADD > ADD > (N times) > FLUSH > etc. > > Instead I'm observing > ADD > FLUSH > ADD > FLUSH > ADD > FLUSH > (etc) > then > ADD > ADD > ADD > (lots of them) > ERROR > > > So for now I'm keeping autoflush on :) but is there anything obvious I > could try there? > (server runs on linux openjdk 1.8.0_91, client runs in Tomcat on > Windows Oracle JDK 1.8.0_121) > > Thanks, > Colin >

On Thu, Mar 9, 2017 at 2:03 PM, Dirk Kirsten <dk at basex.org> wrote: > Hello Colin, > > this really sounds like a bug on our side to me. I opened a bug report > on https://github.com/BaseXdb/basex/issues/1434 and I am sure Christian > will quickly fix it when he gets back from a short vacation. > > Cheers > > Dirk > > > On 03/08/2017 06:20 PM, Colin Delacroix wrote: > > Hi everyone, > > > > (warning this is a long read) > > > > I'm using server version 8.6.1 and writing a Java client to move lots > > of documents from a SQL database to a BaseX database. > > The basic version works just fine. Tested up to 1M documents. > > > > Now I'm trying to do exactly the same but with AUTOFLUSH disabled + > > manually FLUSH every N documents and it fails after just a few inserts. > > > > Pseudo code: > > (set up client session, create and open new database) > > session.execute(new Set(MainOptions.AUTOFLUSH, false)); > > int i = 0; > > while(doc = nextDocs()) { > > session.add(doc.getKey(), doc.getInputStream()); > > if(i++ % 10 == 0) { /* value reduced to 10 for testing purposes */ > > session.execute(new Flush()); > > } > > } > > (tear down session) > > > > Remember tha > > > > After the first FLUSH has been issued, the ADD directive starts > > randomly throwing a org.basex.core.BaseXException (when using the > > BaseXClient implementation it was its inner ok() method that started > > returning false). > > > > I tracked the root cause in the server: the SAXParser complaining > > about the documents not being well-formed (the exact same documents > > are all imported fine with AUTOFLUSH on). In the server standard logs > > I can see why: parts of those documents are suddenly missing (usually > > the beginning + command part) at parse time! > > > > That is the symptom, now for the cause I'm also observing that the ADD > > and FLUSH directive in the logs are out of order, exactly like what > > would happen in the FLUSH commands were not properly queued. > > > > I was expecting something like : > > ADD > > ADD > > ADD > > (N times) > > FLUSH > > ADD > > ADD > > ADD > > (N times) > > FLUSH > > etc. > > > > Instead I'm observing > > ADD > > FLUSH > > ADD > > FLUSH > > ADD > > FLUSH > > (etc) > > then > > ADD > > ADD > > ADD > > (lots of them) > > ERROR > > > > > > So for now I'm keeping autoflush on :) but is there anything obvious I > > could try there? > > (server runs on linux openjdk 1.8.0_91, client runs in Tomcat on > > Windows Oracle JDK 1.8.0_121) > > > > Thanks, > > Colin > > > > -- > Dirk Kirsten, BaseX GmbH, http://basexgmbh.de > |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz > |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: > | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle > `-- Phone: 0049 7531 91 68 276, Fax: 0049 7531 20 05 22 > >