Exception when using replace - UPDINDEX problem?
Monday, 9 September 2013 · 4 messages · last reply 15 Sep 2013
JB
James Ball9 Sep 2013 22:29 #
Hello,
I've got a collection with about 1.4 million small XML files in it. I can load all these through the GUI with no problems.
However really I need to be loading them, a few at a time, through PHP and have code to do this. Because the process involves add, query, add and so on I have set
UPDINDEX to true on the database. In most instances everything works fine but sometimes I get the error below (it's not on particular documents, a second run will give the error in different places).
Once this error has been received it seems the OPTIMIZE doesn't work properly on the database - giving an array out of range error.
I don't understand the stack trace enough to know what the most likely cause is. Can anyone provide me with some guidance?
Many thanks, James
exception 'Exception' with message 'Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk at mailman.uni-konstanz.de
Version: BaseX 7.7
Java: Apple Inc., 1.6.0_51
OS: Mac OS X, x86_64
Stack Trace:
java.lang.IllegalStateException: Key should not exist
at org.basex.index.value.UpdatableDiskValues.index(UpdatableDiskValues.java:79) at org.basex.data.DiskData.indexEnd(DiskData.java:356)
at org.basex.data.Data.insert(Data.java:817)
at org.basex.core.cmd.Add.run(Add.java:107)
at org.basex.core.Command.run(Command.java:345)
at org.basex.core.Command.run(Command.java:117)
at org.basex.core.cmd.Replace.run(Replace.java:68)
at org.basex.core.Command.run(Command.java:345)
at org.basex.core.Command.exec(Command.java:321)
at org.basex.core.Command.execute(Command.java:78)
at org.basex.core.Command.execute(Command.java:90)
at org.basex.server.ClientListener.execute(ClientListener.java:386) at org.basex.server.ClientListener.replace(ClientListener.java:365) at org.basex.server.ClientListener.run(ClientListener.java:106) ' in /Library/WebServer/Documents/ISIS/BaseXClient.php:101
Stack trace:
#0 /Library/WebServer/Documents/ISIS/BaseXClient.php(60): Session->sendCmd(12, 'ISIS_IND_BIO/IW...', '<?xml version="...') #1 /Library/WebServer/Documents/ISIS/uploadBioCli.php(79): Session->replace('ISIS_IND_BIO/IW...', '<?xml version="...') #2 {main}Processing file 358 /Users/jb8748/Documents/Data Auditing/ISIS_IND_BIO/IWF/-9458-.xmlCG
Christian Grün10 Sep 2013 00:09 #
Hi James,
this looks like a potential bug, but it might be tricky to reproduce it with sample data. Have you experienced similar feedback when using Java?
Best,
Christian
___________________________
2013/9/9 James Ball <basex-talk at jamesball.co.uk>:
Quoted text · James Ball · 45 lines
> Hello, > > I've got a collection with about 1.4 million small XML files in it. I can load all these through the GUI with no problems. > > However really I need to be loading them, a few at a time, through PHP and have code to do this. Because the process involves add, query, add and so on I have set UPDINDEX to true on the database. In most instances everything works fine but sometimes I get the error below (it's not on particular documents, a second run will give the error in different places). > > Once this error has been received it seems the OPTIMIZE doesn't work properly on the database - giving an array out of range error. > > I don't understand the stack trace enough to know what the most likely cause is. Can anyone provide me with some guidance? > > Many thanks, James > > exception 'Exception' with message 'Improper use? Potential bug? Your feedback is welcome: > Contact: basex-talk at mailman.uni-konstanz.de > Version: BaseX 7.7 > Java: Apple Inc., 1.6.0_51 > OS: Mac OS X, x86_64 > Stack Trace: > java.lang.IllegalStateException: Key should not exist > at org.basex.index.value.UpdatableDiskValues.index(UpdatableDiskValues.java:79) > at org.basex.data.DiskData.indexEnd(DiskData.java:356) > at org.basex.data.Data.insert(Data.java:817) > at org.basex.core.cmd.Add.run(Add.java:107) > at org.basex.core.Command.run(Command.java:345) > at org.basex.core.Command.run(Command.java:117) > at org.basex.core.cmd.Replace.run(Replace.java:68) > at org.basex.core.Command.run(Command.java:345) > at org.basex.core.Command.exec(Command.java:321) > at org.basex.core.Command.execute(Command.java:78) > at org.basex.core.Command.execute(Command.java:90) > at org.basex.server.ClientListener.execute(ClientListener.java:386) > at org.basex.server.ClientListener.replace(ClientListener.java:365) > at org.basex.server.ClientListener.run(ClientListener.java:106) > ' in /Library/WebServer/Documents/ISIS/BaseXClient.php:101 > Stack trace: > #0 /Library/WebServer/Documents/ISIS/BaseXClient.php(60): Session->sendCmd(12, 'ISIS_IND_BIO/IW...', '<?xml version="...') > #1 /Library/WebServer/Documents/ISIS/uploadBioCli.php(79): Session->replace('ISIS_IND_BIO/IW...', '<?xml version="...') > #2 {main}Processing file 358 /Users/jb8748/Documents/Data Auditing/ISIS_IND_BIO/IWF/-9458-.xml > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
JB
James Ball10 Sep 2013 11:24 #
Hi Christian,
Thank you for that information. I haven't tried it with Java (and my Java is a bit rusty at the moment) but I will try it with Java and see what happens. It won't be for at least a few days now.
Whilst I can't share the exact data being used it's of a very regular form. If I find this behaviour is repeatable I will try to generate a test set that replicates it.
I will be in touch.
Thanks, James
Quoted text · Christian Grün · 57 lines
On 9 Sep 2013, at 23:09, Christian Grün <christian.gruen at gmail.com> wrote: > Hi James, > > this looks like a potential bug, but it might be tricky to reproduce > it with sample data. Have you experienced similar feedback when using > Java? > > Best, > Christian > ___________________________ > > 2013/9/9 James Ball <basex-talk at jamesball.co.uk>: >> Hello, >> >> I've got a collection with about 1.4 million small XML files in it. I can load all these through the GUI with no problems. >> >> However really I need to be loading them, a few at a time, through PHP and have code to do this. Because the process involves add, query, add and so on I have set UPDINDEX to true on the database. In most instances everything works fine but sometimes I get the error below (it's not on particular documents, a second run will give the error in different places). >> >> Once this error has been received it seems the OPTIMIZE doesn't work properly on the database - giving an array out of range error. >> >> I don't understand the stack trace enough to know what the most likely cause is. Can anyone provide me with some guidance? >> >> Many thanks, James >> >> exception 'Exception' with message 'Improper use? Potential bug? Your feedback is welcome: >> Contact: basex-talk at mailman.uni-konstanz.de >> Version: BaseX 7.7 >> Java: Apple Inc., 1.6.0_51 >> OS: Mac OS X, x86_64 >> Stack Trace: >> java.lang.IllegalStateException: Key should not exist >> at org.basex.index.value.UpdatableDiskValues.index(UpdatableDiskValues.java:79) >> at org.basex.data.DiskData.indexEnd(DiskData.java:356) >> at org.basex.data.Data.insert(Data.java:817) >> at org.basex.core.cmd.Add.run(Add.java:107) >> at org.basex.core.Command.run(Command.java:345) >> at org.basex.core.Command.run(Command.java:117) >> at org.basex.core.cmd.Replace.run(Replace.java:68) >> at org.basex.core.Command.run(Command.java:345) >> at org.basex.core.Command.exec(Command.java:321) >> at org.basex.core.Command.execute(Command.java:78) >> at org.basex.core.Command.execute(Command.java:90) >> at org.basex.server.ClientListener.execute(ClientListener.java:386) >> at org.basex.server.ClientListener.replace(ClientListener.java:365) >> at org.basex.server.ClientListener.run(ClientListener.java:106) >> ' in /Library/WebServer/Documents/ISIS/BaseXClient.php:101 >> Stack trace: >> #0 /Library/WebServer/Documents/ISIS/BaseXClient.php(60): Session->sendCmd(12, 'ISIS_IND_BIO/IW...', '<?xml version="...') >> #1 /Library/WebServer/Documents/ISIS/uploadBioCli.php(79): Session->replace('ISIS_IND_BIO/IW...', '<?xml version="...') >> #2 {main}Processing file 358 /Users/jb8748/Documents/Data Auditing/ISIS_IND_BIO/IWF/-9458-.xml >> >> >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
DP
Dimitar Popov15 Sep 2013 00:49 #
Quoted text · 10:24:50 · 10 lines
Am Dienstag, 10. September 2013, 10:24:50 schrieb James Ball: > >> I've got a collection with about 1.4 million small XML files in it. I can > >> load all these through the GUI with no problems. > >> > >> However really I need to be loading them, a few at a time, through PHP > >> and have code to do this. Because the process involves add, query, add > >> and so on I have set UPDINDEX to true on the database. In most instances > >> everything works fine but sometimes I get the error below (it's not on > >> particular documents, a second run will give the error in different > >> places).
Can you try to reproduce the problem without using PHP, but a BaseX command script [1] instead (e.g. many XQUERY commands each executing a replace with a few documents). If the problem persists, we will be able to eliminate the PHP binding and you will not have to use java ;)
Quoted text · James Ball · 11 lines
> >> exception 'Exception' with message 'Improper use? Potential bug? Your > >> feedback is welcome: Contact: basex-talk at mailman.uni-konstanz.de > >> Version: BaseX 7.7 > >> Java: Apple Inc., 1.6.0_51 > >> OS: Mac OS X, x86_64 > >> Stack Trace: > >> java.lang.IllegalStateException: Key should not exist > >> > >> at > >> org.basex.index.value.UpdatableDiskValues.index(UpdatableDiskValu > >> es.java:79) at org.basex.data.DiskData.indexEnd(DiskData.java:356)
Can you check if the error occurs always at line 356 in DiskData.java? If yes, will it be possible to send us a list of all attribute values from all documents?
The problem is that a key, which has been determined to be not existing in a previous step, actually exists in the index when trying to insert it. I checked the relevant code and I fail the see any obvious error. In my opinion the following can cause the incorrect behavior:
1. the PHP binding somehow manages to update the database in parallel (the locking mechanism should already prevent that)
2. there is a bug in the index update code: possible, although I fail to see it; either reproducible example should be provided or another pair of eyes
3. something else (e.g. our implementation of hash table contains duplicate keys, which is very unlikely).
Best regards,
Dimitar
[1] http://docs.basex.org/wiki/Commands#Command_Scripts
Written for BaseX 7.7. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery