DB inconsistency
Monday, 26 January 2015 · 4 messages
ML
Marco Lettere26 Jan 2015 10:51 #
Hi all,
we have an installation that has been running in production for nearly two months now.
It's very insertion intensive and from time to time we need to run a
db:optimize to keep it on track because the time it takes to serve a request turns from about 2 seconds to 100k seconds. But after the optimize it again runs with the expected performance.
Today we noticed again a strange difficulty in serving requests so we tried the usual approach but we got this exception during the optimize:
[SENR0001] Attributes cannot be serialized:attribute determinerCode {"INSTA....
Upon running an inspect command, we got a long list of EOF exceptions followed by the resume stating about 85K wrong parent/descendants relationships.
Is there any idea on what could have gone wrongly and if possible any way to recover from this?
Thank you very much.
Regards,
Marco.
java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:446) at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424) at
org.basex.io.random.TableDiskAccess.readBlock(TableDiskAccess.java:525) at
org.basex.io.random.TableDiskAccess.readPage(TableDiskAccess.java:489) at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:468) at org.basex.io.random.TableDiskAccess.read1(TableDiskAccess.java:148) at org.basex.data.Data.kind(Data.java:290)
at org.basex.core.cmd.Inspect.inspect(Inspect.java:50)
at org.basex.core.cmd.Inspect.run(Inspect.java:27)
at org.basex.core.Command.run(Command.java:360)
at org.basex.core.Command.execute(Command.java:94)
at org.basex.api.client.LocalSession.execute(LocalSession.java:122) at org.basex.api.client.Session.execute(Session.java:36) at org.basex.core.CLI.execute(CLI.java:106)
at org.basex.core.CLI.execute(CLI.java:90)
at org.basex.BaseX.console(BaseX.java:197)
at org.basex.BaseX.<init>(BaseX.java:171)
at org.basex.BaseX.main(BaseX.java:42)
Checking main table (43626334 nodes):
- 0 invalid node kinds
- 0 invalid parent references
- 84663 wrong parent/descendant relationships (pre: 6299342,..) Warning: Database is inconsistent.
'db' inspected in 21789.24 ms.CG
Christian Grün26 Jan 2015 12:13 #
Hi Marco,
I'm sorry to hear your database got corrupted. Once a database has inconsistent structures, it's difficult to restore its original state. Some questions that might help:
1. Can you do a full optimization via
db:optimize('db', true())? Before calling the command, I would recommend you to create a backup of your database.
2. Can you successfully serialize or export your database?
3. Did you do any backups during the last months that may help us to find the first corrupt database version?
4. Which version(s) of BaseX did you use?
Hope this helps,
ChristianQuoted text · Marco Lettere · 53 lines
On Mon, Jan 26, 2015 at 10:51 AM, Marco Lettere <marco.lettere at dedalus.eu> wrote: > Hi all, > we have an installation that has been running in production for nearly two > months now. > It's very insertion intensive and from time to time we need to run a > db:optimize to keep it on track because the time it takes to serve a request > turns from about 2 seconds to 100k seconds. But after the optimize it again > runs with the expected performance. > Today we noticed again a strange difficulty in serving requests so we tried > the usual approach but we got this exception during the optimize: > > [SENR0001] Attributes cannot be serialized:attribute determinerCode > {"INSTA.... > > Upon running an inspect command, we got a long list of EOF exceptions > followed by the resume stating about 85K wrong parent/descendants > relationships. > > Is there any idea on what could have gone wrongly and if possible any way to > recover from this? > Thank you very much. > Regards, > Marco. > > java.io.EOFException > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:446) > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424) > at > org.basex.io.random.TableDiskAccess.readBlock(TableDiskAccess.java:525) > at > org.basex.io.random.TableDiskAccess.readPage(TableDiskAccess.java:489) > at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:468) > at org.basex.io.random.TableDiskAccess.read1(TableDiskAccess.java:148) > at org.basex.data.Data.kind(Data.java:290) > at org.basex.core.cmd.Inspect.inspect(Inspect.java:50) > at org.basex.core.cmd.Inspect.run(Inspect.java:27) > at org.basex.core.Command.run(Command.java:360) > at org.basex.core.Command.execute(Command.java:94) > at org.basex.api.client.LocalSession.execute(LocalSession.java:122) > at org.basex.api.client.Session.execute(Session.java:36) > at org.basex.core.CLI.execute(CLI.java:106) > at org.basex.core.CLI.execute(CLI.java:90) > at org.basex.BaseX.console(BaseX.java:197) > at org.basex.BaseX.<init>(BaseX.java:171) > at org.basex.BaseX.main(BaseX.java:42) > Checking main table (43626334 nodes): > - 0 invalid node kinds > - 0 invalid parent references > - 84663 wrong parent/descendant relationships (pre: 6299342,..) > Warning: Database is inconsistent. > 'db' inspected in 21789.24 ms. >
ML
Marco Lettere26 Jan 2015 12:32 #
Responses inline.
Quoted text · Christian Grün · 6 lines
On 26/01/2015 12:13, Christian Grün wrote: > Hi Marco, > > I'm sorry to hear your database got corrupted. Once a database has > inconsistent structures, it's difficult to restore its original state. > Some questions that might help:
Yeah. I was aware of this. I hoped that the problem could be at least identifiable because of the clear inspect outcome.
Quoted text · Christian Grün · 4 lines
> > 1. Can you do a full optimization via db:optimize('db', true())? > Before calling the command, I would recommend you to create a backup > of your database.
No. We try to run it through command "optimize all" which should be the same but this is where we get the attribute serializetion error.
Quoted text · Christian Grün · 1 line
> 2. Can you successfully serialize or export your database?
No. We copied the DB to another machine and tried to import into another basex. The import actually succeeds but a simple query on the root tag of all the documents shows up only 10 percent of the actual data.
Quoted text · Christian Grün · 2 lines
> 3. Did you do any backups during the last months that may help us to > find the first corrupt database version?
Yes. The problem showed up in Friday's logs files and we have a nightly backup (Thursday to Friday) which looks healty.
Quoted text · Christian Grün · 1 line
> 4. Which version(s) of BaseX did you use?
It's "BaseX 8.0 beta 2cc72a0".
M.
Quoted text · Christian Grün · 51 lines
>> Hi all, >> we have an installation that has been running in production for nearly two >> months now. >> It's very insertion intensive and from time to time we need to run a >> db:optimize to keep it on track because the time it takes to serve a request >> turns from about 2 seconds to 100k seconds. But after the optimize it again >> runs with the expected performance. >> Today we noticed again a strange difficulty in serving requests so we tried >> the usual approach but we got this exception during the optimize: >> >> [SENR0001] Attributes cannot be serialized:attribute determinerCode >> {"INSTA.... >> >> Upon running an inspect command, we got a long list of EOF exceptions >> followed by the resume stating about 85K wrong parent/descendants >> relationships. >> >> Is there any idea on what could have gone wrongly and if possible any way to >> recover from this? >> Thank you very much. >> Regards, >> Marco. >> >> java.io.EOFException >> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:446) >> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424) >> at >> org.basex.io.random.TableDiskAccess.readBlock(TableDiskAccess.java:525) >> at >> org.basex.io.random.TableDiskAccess.readPage(TableDiskAccess.java:489) >> at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:468) >> at org.basex.io.random.TableDiskAccess.read1(TableDiskAccess.java:148) >> at org.basex.data.Data.kind(Data.java:290) >> at org.basex.core.cmd.Inspect.inspect(Inspect.java:50) >> at org.basex.core.cmd.Inspect.run(Inspect.java:27) >> at org.basex.core.Command.run(Command.java:360) >> at org.basex.core.Command.execute(Command.java:94) >> at org.basex.api.client.LocalSession.execute(LocalSession.java:122) >> at org.basex.api.client.Session.execute(Session.java:36) >> at org.basex.core.CLI.execute(CLI.java:106) >> at org.basex.core.CLI.execute(CLI.java:90) >> at org.basex.BaseX.console(BaseX.java:197) >> at org.basex.BaseX.<init>(BaseX.java:171) >> at org.basex.BaseX.main(BaseX.java:42) >> Checking main table (43626334 nodes): >> - 0 invalid node kinds >> - 0 invalid parent references >> - 84663 wrong parent/descendant relationships (pre: 6299342,..) >> Warning: Database is inconsistent. >> 'db' inspected in 21789.24 ms. >>
CG
Christian Grün26 Jan 2015 12:43 #
Quoted text · Marco Lettere · 2 lines
> Yes. The problem showed up in Friday's logs files and we have a nightly > backup (Thursday to Friday) which looks healty.
Glad to hear that..
Quoted text · Marco Lettere · 3 lines
>> 4. Which version(s) of BaseX did you use? > > It's "BaseX 8.0 beta 2cc72a0".
I looked up the version number; it dates back to August 26. Since that date, I found two fixes on our storage unit:
* One of them fixed problems when the first document of a database was replaced [1]. If you several documents in your database, and if you use the REPLACE command or db:replace, this could have been the reason
* Another fix handled a rare case in which namespace references were wrongly assigned [2].
I assume it's hardly possible to reproduce the updates that had been performed in your application. If I should be wrong, I'll be glad to look into this further.
Christian
[1] https://github.com/BaseXdb/basex/commit/d91205eced3ef2d45ef4e6a1c00f38a249c133b8 [2] https://github.com/BaseXdb/basex/commit/9ffe53e3b6034bc443e25854bf05c7a795f8e152
Written for BaseX 7.9. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery