Main Page » Mailing List » September 2014

Corruption?

Wednesday, 17 September 2014 · 6 messages · last reply 18 Sep 2014
GJ
Gerald de Jong17 Sep 2014 19:58 #
Hello Folks, Is there anything I can do when opening my database results in "Datbase blabla is being updated, or update was not completed."? A program was pumping serious amounts of XML into BaseX with a few processes concurrently and then things seem to have gone haywire - big CPU usage for a long time but that's all. Nothing seems to tell me what's wrong, and now I don't know how to find out, or maybe hopefully repair. I've killed the server, and now I just run bin/basex to look around. But it seems I can't even find out what went wrong. I'd like to know what went on I can prevent this next time. Do you think the problem was too much concurrency, or not enough memory, or something else? Any clues? - Gerald PS. My project is in Cultural Heritage. https://github.com/delving/narthex
-- Delving BV, Vasteland 8, Rotterdam http://www.delving.eu http://twitter.com/fluxe skype: beautifulcode +31629339805
GJ
Gerald de Jong18 Sep 2014 09:41 #
Ah I found out how to remove the lock from http://docs.basex.org/wiki/Transaction_Management. Just deleted the upd file. After that and a restart it seems to be working again, but the reality is that it died on me. I still need clues to figure out why. Are there some tricks, like increasing memory or something, which will prevent it being overwhelmed? (if that's what happened).
Quoted text · Gerald de Jong · 32 lines
On Wed, Sep 17, 2014 at 7:58 PM, Gerald de Jong <gerald at delving.eu> wrote: > Hello Folks, > > Is there anything I can do when opening my database results in "Datbase > blabla is being updated, or update was not completed."? > > A program was pumping serious amounts of XML into BaseX with a few > processes concurrently and then things seem to have gone haywire - big CPU > usage for a long time but that's all. Nothing seems to tell me what's > wrong, and now I don't know how to find out, or maybe hopefully repair. > > I've killed the server, and now I just run bin/basex to look around. But > it seems I can't even find out what went wrong. > > I'd like to know what went on I can prevent this next time. Do you think > the problem was too much concurrency, or not enough memory, or something > else? > > Any clues? > > - Gerald > > PS. My project is in Cultural Heritage. > https://github.com/delving/narthex > > -- > Delving BV, Vasteland 8, Rotterdam > http://www.delving.eu > http://twitter.com/fluxe > skype: beautifulcode > +31629339805 >
-- Delving BV, Vasteland 8, Rotterdam http://www.delving.eu http://twitter.com/fluxe skype: beautifulcode +31629339805
GJ
Gerald de Jong18 Sep 2014 09:56 #
Another clue:
Exception in thread "Thread-1637" java.lang.IllegalMonitorStateException
at
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:374)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260)
at
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1131)
at org.basex.core.DBLocking.release(DBLocking.java:219)
at org.basex.core.Context.unregister(Context.java:249)
at org.basex.core.Command.execute(Command.java:104)
at org.basex.core.Command.execute(Command.java:117)
at org.basex.server.ClientListener.execute(ClientListener.java:384)
at org.basex.server.ClientListener.add(ClientListener.java:355)
at org.basex.server.ClientListener.run(ClientListener.java:100)
Exception in thread "Thread-2277" java.lang.IllegalMonitorStateException
at
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:374)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260)
at
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1131)
at org.basex.core.DBLocking.release(DBLocking.java:219)
at org.basex.core.Context.unregister(Context.java:249)
at org.basex.core.Command.execute(Command.java:104)
at org.basex.server.ClientListener.run(ClientListener.java:145)
This seems to be at the moment of freezing.
Quoted text · Gerald de Jong · 53 lines
On Thu, Sep 18, 2014 at 9:41 AM, Gerald de Jong <gerald at delving.eu> wrote: > Ah I found out how to remove the lock from > http://docs.basex.org/wiki/Transaction_Management. Just deleted the upd > file. > > After that and a restart it seems to be working again, but the reality is > that it died on me. I still need clues to figure out why. Are there some > tricks, like increasing memory or something, which will prevent it being > overwhelmed? (if that's what happened). > > On Wed, Sep 17, 2014 at 7:58 PM, Gerald de Jong <gerald at delving.eu> wrote: > >> Hello Folks, >> >> Is there anything I can do when opening my database results in "Datbase >> blabla is being updated, or update was not completed."? >> >> A program was pumping serious amounts of XML into BaseX with a few >> processes concurrently and then things seem to have gone haywire - big CPU >> usage for a long time but that's all. Nothing seems to tell me what's >> wrong, and now I don't know how to find out, or maybe hopefully repair. >> >> I've killed the server, and now I just run bin/basex to look around. But >> it seems I can't even find out what went wrong. >> >> I'd like to know what went on I can prevent this next time. Do you think >> the problem was too much concurrency, or not enough memory, or something >> else? >> >> Any clues? >> >> - Gerald >> >> PS. My project is in Cultural Heritage. >> https://github.com/delving/narthex >> >> -- >> Delving BV, Vasteland 8, Rotterdam >> http://www.delving.eu >> http://twitter.com/fluxe >> skype: beautifulcode >> +31629339805 >> > > > > -- > Delving BV, Vasteland 8, Rotterdam > http://www.delving.eu > http://twitter.com/fluxe > skype: beautifulcode > +31629339805 >
-- Delving BV, Vasteland 8, Rotterdam http://www.delving.eu http://twitter.com/fluxe skype: beautifulcode +31629339805
CG
Christian Grünin reply to Gerald de Jong, 18 Sep 2014, 09:4118 Sep 2014 13:29 #
Hi Gerald,
Quoted text · Gerald de Jong · 8 lines
> Ah I found out how to remove the lock from > http://docs.basex.org/wiki/Transaction_Management. Just deleted the upd > file. > > After that and a restart it seems to be working again, but the reality is > that it died on me. I still need clues to figure out why. Are there some > tricks, like increasing memory or something, which will prevent it being > overwhelmed? (if that's what happened).
It's difficult to say in general what may have causes the faulty behavior. You could check out the information stored in the database logs [1]. Maybe there was one particular query that caused the slowdown?
Quoted text · Gerald de Jong · 11 lines
> Exception in thread "Thread-1637" java.lang.IllegalMonitorStateException > at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:374) > at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260) > at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1131) > at org.basex.core.DBLocking.release(DBLocking.java:219) > at org.basex.core.Context.unregister(Context.java:249) > at org.basex.core.Command.execute(Command.java:104) > at org.basex.core.Command.execute(Command.java:117) > at org.basex.server.ClientListener.execute(ClientListener.java:384) > at org.basex.server.ClientListener.add(ClientListener.java:355) > at org.basex.server.ClientListener.run(ClientListener.java:100)
This usually indicates that the client/server code is used in some unorthodox way. Do you think you can provide us with an SSCCE? Thanks, Christian [1] http://docs.basex.org/wiki/Logging
GJ
Gerald de Jong18 Sep 2014 15:45 #
Hi Christian, The client code is tiny and is all based on the Java API (although it's in Scala). Even for people not familar with Scala it should be readable. I'm curious if somebody sees a fatal flaw or something that could be improved. This is it:
  def withSession[T](block: ClientSession => T): T = {
    val session = new ClientSession(host, port, user, pass)
    try {
      block(session)
    }
    finally {
      session.close()
    }
  }
def withDbSession[T](database: String)(block: ClientSession => T): T = { withSession { session => session.execute(new Open(database)) session.execute(new Set("autoflush", "false")) val result = block(session) session.execute(new Flush()) result } } def add(database: String, path: String, document: String) = withDbSession(database)(_.add(path, new ByteArrayInputStream(document.getBytes("utf-8")))) This add() function gets called about a million times, ha ha, literally. Creating an SSCCE might be tricky for such a concurrency issue. I'll see if the crashing persists and maybe find a way to reproduce it if so. I've really come to enjoy doing everything on the back end with XQuery/BaseX. Quite fun composing queries using Scala multi-line string interpolation. def getIds(since: String): String = db { session => val q = s""" | | let $$records := collection('$recordDb')/narthex | return | <ids>{ | for $$narthex in $$records | where $$narthex/@mod >= ${quote(since)} | order by $$narthex/@mod descending | return | <id>{$$narthex/@mod}{string($$narthex/@id)}</id> | }</ids> | |""".stripMargin.trim session.query(q).execute() } Gerald
Quoted text · Christian Grün · 38 lines
> Ah I found out how to remove the lock from > > http://docs.basex.org/wiki/Transaction_Management. Just deleted the upd > > file. > > > > After that and a restart it seems to be working again, but the reality is > > that it died on me. I still need clues to figure out why. Are there > some > > tricks, like increasing memory or something, which will prevent it being > > overwhelmed? (if that's what happened). > > It's difficult to say in general what may have causes the faulty > behavior. You could check out the information stored in the database > logs [1]. Maybe there was one particular query that caused the > slowdown? > > > Exception in thread "Thread-1637" java.lang.IllegalMonitorStateException > > at > java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:374) > > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260) > > at > java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1131) > > at org.basex.core.DBLocking.release(DBLocking.java:219) > > at org.basex.core.Context.unregister(Context.java:249) > > at org.basex.core.Command.execute(Command.java:104) > > at org.basex.core.Command.execute(Command.java:117) > > at org.basex.server.ClientListener.execute(ClientListener.java:384) > > at org.basex.server.ClientListener.add(ClientListener.java:355) > > at org.basex.server.ClientListener.run(ClientListener.java:100) > > This usually indicates that the client/server code is used in some > unorthodox way. Do you think you can provide us with an SSCCE? > > Thanks, > Christian > > [1] http://docs.basex.org/wiki/Logging >
-- Delving BV, Vasteland 8, Rotterdam http://www.delving.eu http://twitter.com/fluxe skype: beautifulcode +31629339805
EA
Erol Akarsuin reply to Gerald de Jong, 17 Sep 2014, 19:5818 Sep 2014 18:33 #
Hi, I am not able use map in xquery scripts in current basex version. How can I create a map and add values to it? Basex documentation says it merge and put functions that can help us to create a map but those are in Basex 8.0. I remember I was creating map with previous versions of Basex. Can you remind my how? I had to download the latest snaphost of Basex 8.0 and run basexguui.bat but getting error org.basex.BaseXGUI class could not be found. Thanks

Written for BaseX 7.9. BaseX may have changed since: please check the current documentation.

« strange webdav behaviour...Multi threaded application »
⚡Generated with XQuery