db:replace() in DBA resulting in duplicates
Friday, 26 April 2019 · 9 messages · last reply 20 May 2019
ZD
Zimmel, Daniel26 Apr 2019 16:45 #
Hello,
something has gone awry in my database after a db:replace():
query:
db:open('DB','/path/file.xml')/base-uri()
result:
/DB/path/file.xml
This is as expected. The result is the replaced file. But when I
query:
db:open('DB')/root[@id='id']/base-uri()
result is a sequence:
/DB/path/file.xml
/DB/path/file.xml
The first result is the file before the db:replace(), the second after.
Can somebody explain why db:open() with path parameter is returning one result, but without parameter two results? Database statistics say there are indeed two documents. But when I export the database, only one gets exported.
This happens with db:replace() on 9.2 from the DBA browser interface. When I use the same db:replace() XQuery on 9.2 with my local database, all is well (document gets replaced, no duplicates). Settings should be the default ones.
Thanks, Daniel
CG
Christian Grün29 Apr 2019 11:08 #
Hi Daniel,
seems hard to reproduce. Could you tell me what you did step by step? This is what I tried:
1. I created a DB database.
2. I added the document <root id='id'/> to the database path /path/file.xml.
3. I replaced this document with the same document via the DBA.
4. Your query db:open('DB')/root[@id='id']/base-uri() returned a single string.
Thanks in advance,
Christian
PS: The form parameter handling in 9.2 contains a little bug (as a result, I couldn’t invoke the replace functionality with the current version). I just uploaded a new snapshot. Both the current and the snapshot version should behave identically in terms of adding and replacing documents.
Quoted text · Daniel · 32 lines
On Fri, Apr 26, 2019 at 4:45 PM Zimmel, Daniel <D.Zimmel at esvmedien.de> wrote: > > Hello, > > something has gone awry in my database after a db:replace(): > > query: > db:open('DB','/path/file.xml')/base-uri() > > result: > /DB/path/file.xml > > This is as expected. The result is the replaced file. But when I > > query: > db:open('DB')/root[@id='id']/base-uri() > > result is a sequence: > /DB/path/file.xml > /DB/path/file.xml > > The first result is the file before the db:replace(), the second after. > > Can somebody explain why db:open() with path parameter is returning one result, but without parameter two results? > Database statistics say there are indeed two documents. But when I export the database, only one gets exported. > > This happens with db:replace() on 9.2 from the DBA browser interface. > When I use the same db:replace() XQuery on 9.2 with my local database, all is well (document gets replaced, no duplicates). Settings should be the default ones. > > Thanks, Daniel > >
ZD
Zimmel, Daniel29 Apr 2019 13:16 #
Oddly enough, I cannot reproduce by myself!
Still, when I browse the database, the /DB/path only exists once (as far as I can see from the sorting), but querying is giving me two documents, according to
db:path() living in the same path. This should never happen, should it?
I will simply drop the database, but I wish I could understand what went wrong. I just did the same updating query and it is politely replacing the second file without adding another one (still, the initial file remains unreplaced).
Thanks,
DanielQuoted text · Christian Grün · 65 lines
> -----Ursprüngliche Nachricht----- > Von: Christian Grün [mailto:christian.gruen at gmail.com] > Gesendet: Montag, 29. April 2019 11:08 > An: Zimmel, Daniel > Cc: BaseX > Betreff: Re: [basex-talk] db:replace() in DBA resulting in duplicates > > Hi Daniel, > > seems hard to reproduce. Could you tell me what you did step by step? > This is what I tried: > > 1. I created a DB database. > 2. I added the document <root id='id'/> to the database path /path/file.xml. > 3. I replaced this document with the same document via the DBA. > 4. Your query db:open('DB')/root[@id='id']/base-uri() returned a single string. > > Thanks in advance, > Christian > > PS: The form parameter handling in 9.2 contains a little bug (as a > result, I couldn’t invoke the replace functionality with the current > version). I just uploaded a new snapshot. Both the current and the > snapshot version should behave identically in terms of adding and > replacing documents. > > > > On Fri, Apr 26, 2019 at 4:45 PM Zimmel, Daniel <D.Zimmel at esvmedien.de> > wrote: > > > > Hello, > > > > something has gone awry in my database after a db:replace(): > > > > query: > > db:open('DB','/path/file.xml')/base-uri() > > > > result: > > /DB/path/file.xml > > > > This is as expected. The result is the replaced file. But when I > > > > query: > > db:open('DB')/root[@id='id']/base-uri() > > > > result is a sequence: > > /DB/path/file.xml > > /DB/path/file.xml > > > > The first result is the file before the db:replace(), the second after. > > > > Can somebody explain why db:open() with path parameter is returning one > result, but without parameter two results? > > Database statistics say there are indeed two documents. But when I export the > database, only one gets exported. > > > > This happens with db:replace() on 9.2 from the DBA browser interface. > > When I use the same db:replace() XQuery on 9.2 with my local database, all is > well (document gets replaced, no duplicates). Settings should be the default > ones. > > > > Thanks, Daniel > > > >
CG
Christian Grün29 Apr 2019 16:00 #
Quoted text · Zimmel, Daniel · 2 lines
> Still, when I browse the database, the /DB/path only exists once (as far as I can see from the sorting), but querying is giving me two documents, according to db:path() living in the same path. > This should never happen, should it?
This is surprising indeed.
Right now,
db:add is still faster than db:replace, because the check for existing database paths can be completely skipped. This allows high-performance database insertions.In a future version of BaseX, I would love to get rid of db:add and db:replace and replace these functions with a db:update function, which will enforce that each document path exists only once in the database. This would surely reduce patterns like the one you encountered.
If you still manage to get this reproduced, just come back to us.MS
I’m glad I caught this thread. Replace was working for me in basexgui but not from command line. ( And I needed to generate replacement list from the shell using ‘find … -newer basex/data/collection’ )
I pulled the basex/lib/basex-api-9.2.1-SNAPSHOT.jar file out of the snapshot zip file and copied it to home-brew /usr/local/Cellar/basex/9.2/libexec/lib/basex-api-9.2.jar and replacement from the basex shell command appears to work now.
Previously, I was getting this error, trying several variations of absolute/relative filenames or file: URIs.
published$ basex -i published -c "replace jmu/vihart00338.xml /projects/vh-migrate/published/jmu/vihart00338.xml"
Improper use? Potential bug? Your feedback is welcome:
Contact: basex-talk at mailman.uni-konstanz.de
Version: BaseX 9.2
Java: Oracle Corporation, 1.8.0_201
OS: Mac OS X, x86_64
Stack Trace:
java.lang.RuntimeException: published: lock file does not exist.
at org.basex.util.Util.notExpected(Util.java:61)
at org.basex.data.DiskData.finishUpdate(DiskData.java:246)
at org.basex.core.cmd.ACreate.update(ACreate.java:97)
at org.basex.core.cmd.Replace.run(Replace.java:55)
at org.basex.core.Command.run(Command.java:257)
at org.basex.core.Command.execute(Command.java:93)
at org.basex.api.client.LocalSession.execute(LocalSession.java:132)
at org.basex.api.client.Session.execute(Session.java:36)
at org.basex.core.CLI.execute(CLI.java:92)
at org.basex.core.CLI.execute(CLI.java:76)
at org.basex.core.CLI.execute(CLI.java:63)
at org.basex.BaseX.<init>(BaseX.java:80)
at org.basex.BaseX.main(BaseX.java:42)— Steve Majewski
Quoted text · Christian Grün · 55 lines
> On Apr 29, 2019, at 5:08 AM, Christian Grün <christian.gruen at gmail.com> wrote: > > Hi Daniel, > > seems hard to reproduce. Could you tell me what you did step by step? > This is what I tried: > > 1. I created a DB database. > 2. I added the document <root id='id'/> to the database path /path/file.xml. > 3. I replaced this document with the same document via the DBA. > 4. Your query db:open('DB')/root[@id='id']/base-uri() returned a single string. > > Thanks in advance, > Christian > > PS: The form parameter handling in 9.2 contains a little bug (as a > result, I couldn’t invoke the replace functionality with the current > version). I just uploaded a new snapshot. Both the current and the > snapshot version should behave identically in terms of adding and > replacing documents. > > > > On Fri, Apr 26, 2019 at 4:45 PM Zimmel, Daniel <D.Zimmel at esvmedien.de> wrote: >> >> Hello, >> >> something has gone awry in my database after a db:replace(): >> >> query: >> db:open('DB','/path/file.xml')/base-uri() >> >> result: >> /DB/path/file.xml >> >> This is as expected. The result is the replaced file. But when I >> >> query: >> db:open('DB')/root[@id='id']/base-uri() >> >> result is a sequence: >> /DB/path/file.xml >> /DB/path/file.xml >> >> The first result is the file before the db:replace(), the second after. >> >> Can somebody explain why db:open() with path parameter is returning one result, but without parameter two results? >> Database statistics say there are indeed two documents. But when I export the database, only one gets exported. >> >> This happens with db:replace() on 9.2 from the DBA browser interface. >> When I use the same db:replace() XQuery on 9.2 with my local database, all is well (document gets replaced, no duplicates). Settings should be the default ones. >> >> Thanks, Daniel >> >>
ZD
Follow-up, some details:
* Create large (5GB) DB in BaseX GUI
* Copy DB folder manually to another server
* Optimize DB
* The first db:replace() via RESTXQ creates a second file with the same base-uri()
* The second db:replace() via RESTXQ replaces the earlier created second file (duplicate)
I cannot reproduce with a small newly created db.
The initial database was created with BaseX 9.0.2, update operation runs with BaseX 9.2.1
Should I try to optimize the database with 9.1.2 before copying and try again? Perhaps this is a database-structure related issue?
Quoted text · Christian Grün · 23 lines
> -----Ursprüngliche Nachricht----- > Von: Christian Grün [mailto:christian.gruen at gmail.com] > Gesendet: Montag, 29. April 2019 16:00 > An: Zimmel, Daniel > Cc: BaseX > Betreff: Re: [basex-talk] db:replace() in DBA resulting in duplicates > > > Still, when I browse the database, the /DB/path only exists once (as far as I can > see from the sorting), but querying is giving me two documents, according to > db:path() living in the same path. > > This should never happen, should it? > > This is surprising indeed. > > Right now, db:add is still faster than db:replace, because the check > for existing database paths can be completely skipped. This allows > high-performance database insertions.In a future version of BaseX, I > would love to get rid of db:add and db:replace and replace these > functions with a db:update function, which will enforce that each > document path exists only once in the database. This would surely > reduce patterns like the one you encountered. > > If you still manage to get this reproduced, just come back to us.
CG
Christian Grün17 May 2019 19:22 #
Quoted text · Zimmel, Daniel · 7 lines
> > * Optimize DB > * The first db:replace() via RESTXQ creates a second file with the same > base-uri() > * The second db:replace() via RESTXQ replaces the earlier created second > file (duplicate) >
Do you perform the next steps in the database instance that has been copied to another server, or does the problem also happen on the original instance? In the first case, do both servers use the same operating system (either Linux or Windows)?
Quoted text · Zimmel, Daniel · 2 lines
Should I try to optimize the database with 9.1.2 before copying and try > again?
Definitely worth trying. If you export your data and create a new database with 9.2.x, and if the problem persists, we could have a look at it.
this is a database-structure related issue?
Quoted text · Zimmel, Daniel · 27 lines
> > > -----Ursprüngliche Nachricht----- > > Von: Christian Grün [mailto:christian.gruen at gmail.com] > > Gesendet: Montag, 29. April 2019 16:00 > > An: Zimmel, Daniel > > Cc: BaseX > > Betreff: Re: [basex-talk] db:replace() in DBA resulting in duplicates > > > > > Still, when I browse the database, the /DB/path only exists once (as > far as I can > > see from the sorting), but querying is giving me two documents, > according to > > db:path() living in the same path. > > > This should never happen, should it? > > > > This is surprising indeed. > > > > Right now, db:add is still faster than db:replace, because the check > > for existing database paths can be completely skipped. This allows > > high-performance database insertions.In a future version of BaseX, I > > would love to get rid of db:add and db:replace and replace these > > functions with a db:update function, which will enforce that each > > document path exists only once in the database. This would surely > > reduce patterns like the one you encountered. > > > > If you still manage to get this reproduced, just come back to us. >
ZD
Zimmel, Daniel20 May 2019 15:05 #
Quoted text · Christian Grün · 4 lines
> > * Optimize DB > > * The first db:replace() via RESTXQ creates a second file with the same base-uri() > > * The second db:replace() via RESTXQ replaces the earlier created second file (duplicate) > Do you perform the next steps in the database instance that has been copied to another server, or does the problem also happen on the original instance? In the first case, do both servers use the same operating system (either Linux or Windows)?
* both; does not happen in the original instance, does only happen in the copied instance
* cannot reproduce with small fresh database created with 9.2.1
* not the same OS; original was created in Win 7, copied to linux server
OK, with a removal of the target database folder, a database optimization on the source and target and a fresh copy I cannot reproduce. I will definitely write a note to myself to always optimize the database before/after moving them and be sure to remove any artifacts.
Daniel
.
CG
Christian Grün20 May 2019 15:15 #
Quoted text · Zimmel, Daniel · 1 line
> * not the same OS; original was created in Win 7, copied to linux server
So this might be related to OS-specific case sensitivity issues; I remember we had other users discussing this on the list. Maybe it would be consistent in the long term to treat all operating systems identically.
Written for BaseX 9.2. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery