Documents added by db:add(...) not visible for other existing database connections?
Hi all,
I noticed that TCP/IP clients are not seeing new documents that are created using db:add(...) in another client:
Once a query has been executed, subsequent queries using the very same database connection do not see/return new documents that were meanwhile created using another connection. When creating some (different) document using one's own existing connection, then everything seems to be in-sync for a moment. But right after that, any new documents are again not seen.
Updates made in other clients are visible just fine. So, the documents that are returned are up-to-date. It's just that not all documents are retrieved.
I can see that in the query plans, the size in <DBNodeSeq size="..."> does not change, and is lower than the size that clients with a newer connection would show.
Should it be okay to keep a connection open? Or is the expected behaviour to need a new connection to see new documents?
Details below.
Thanks, Arjan.
1. Start the server and a client in debugging mode:
./basexserver -d -S ./basexclient -U admin -P admin -d -x
2. Create and open a database:
create database db1 open db1
3. Keep the client open, and make another client insert some documents. Like every second for about 2 minutes, each time using a new connection:
for i in {1..120}; \
do doc="<test><a>uuidgen</a></test>"; \
echo $i, $doc; \
./basexclient -U admin -P admin "db:add('db1', '$doc', 'dummy')"; \ sleep 1; \
done
4. While Step 3 is running, back in the first client, execute:
xquery count(/test)
The log in data/.logs shows:
REQUEST XQUERY count(/test) OK Query executed in 25.74 ms. 223.77 ms
So far, so good.
5. Repeat Step 4 to see the result NOT change, while the loop from Step 3 is still adding new documents. Logs show a faster result, but otherwise are the same. Also the query plans are always the same. (Typically: the size in <DBNodeSeq size="..."> does not change.)
REQUEST XQUERY count(/test) OK Query executed in 5.48 ms. 7.29 ms
6. Using "close" followed by "open db1" gets one a more up-to-date value for "count". Also, using the connection to add some random other type of document, updates the value for "count(/test)" as well:
xquery db:add('db1', '<another-test><xyz>123</xyz></another-test>', 'dummy')
7. It's not just "count" that does not change its value. Also, one can only query documents up to that count as returned by Step 4. Like if Step 4 returned "15", then the following would return 5 results, not 10:
xquery (/test)[position()=10 to 20]
Also, the following would only return the count as returned earlier:
xquery /test
Like noted, UPDATES made in other clients are visible just fine:
xquery replace value of node /another-test/xyz with '456'
xquery for $n in /test/a return replace value of node $n with
fn:generate-id($n)Tested in both 7.3 and 7.5-20120929.201251 from http://files.basex.org/releases/latest/, in Oracle Java 7 on Mac OS X 10.8.2. I first noticed that my running application did not see new documents created by a test script. But the command line tools show the same issue.
Any clue?
Hi Arjan,
thanks again for your helpful report. It pointed me to a generally buggy behavior, which may occur when document updates occur in databases that are opened by multiple clients. I have opened a new issue [1]. As I'll be on the road for some (more) days, it may take some time until this is fixed, but be sure that this is something we won't sweep under the carpet.
Christian
Quoted text · Arjan van Bentem · 106 lines
On Sun, Sep 30, 2012 at 12:24 AM, Arjan van Bentem <arjan.vanbentem at bidnetwork.org> wrote: > Hi all, > > I noticed that TCP/IP clients are not seeing new documents that are created > using db:add(...) in another client: > > Once a query has been executed, subsequent queries using the very same > database connection do not see/return new documents that were meanwhile > created using another connection. When creating some (different) document > using one's own existing connection, then everything seems to be in-sync for > a moment. But right after that, any new documents are again not seen. > > Updates made in other clients are visible just fine. So, the documents that > are returned are up-to-date. It's just that not all documents are retrieved. > > I can see that in the query plans, the size in <DBNodeSeq size="..."> does > not change, and is lower than the size that clients with a newer connection > would show. > > Should it be okay to keep a connection open? Or is the expected behaviour to > need a new connection to see new documents? > > Details below. > > Thanks, > Arjan. > > > 1. Start the server and a client in debugging mode: > > ./basexserver -d -S > ./basexclient -U admin -P admin -d -x > > 2. Create and open a database: > > create database db1 > open db1 > > 3. Keep the client open, and make another client insert some documents. Like > every second for about 2 minutes, each time using a new connection: > > for i in {1..120}; \ > do doc="<test><a>`uuidgen`</a></test>"; \ > echo $i, $doc; \ > ./basexclient -U admin -P admin "db:add('db1', '$doc', 'dummy')"; \ > sleep 1; \ > done > > 4. While Step 3 is running, back in the first client, execute: > > xquery count(/test) > > The log in data/.logs shows: > > REQUEST XQUERY count(/test) > OK Query executed in 25.74 ms. 223.77 ms > > So far, so good. > > 5. Repeat Step 4 to see the result NOT change, while the loop from Step 3 is > still adding new documents. Logs show a faster result, but otherwise are the > same. Also the query plans are always the same. (Typically: the size in > <DBNodeSeq size="..."> does not change.) > > REQUEST XQUERY count(/test) > OK Query executed in 5.48 ms. 7.29 ms > > 6. Using "close" followed by "open db1" gets one a more up-to-date value for > "count". Also, using the connection to add some random other type of > document, updates the value for "count(/test)" as well: > > xquery db:add('db1', '<another-test><xyz>123</xyz></another-test>', > 'dummy') > > 7. It's not just "count" that does not change its value. Also, one can only > query documents up to that count as returned by Step 4. Like if Step 4 > returned "15", then the following would return 5 results, not 10: > > xquery (/test)[position()=10 to 20] > > Also, the following would only return the count as returned earlier: > > xquery /test > > Like noted, UPDATES made in other clients are visible just fine: > > xquery replace value of node /another-test/xyz with '456' > xquery for $n in /test/a return replace value of node $n with > fn:generate-id($n) > > > Tested in both 7.3 and 7.5-20120929.201251 from > http://files.basex.org/releases/latest/, in Oracle Java 7 on Mac OS X > 10.8.2. I first noticed that my running application did not see new > documents created by a test script. But the command line tools show the same > issue. > > Any clue? > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
Hi Arjan,
it has taken a while, but I'm glad to report that the issue you reported has now been fixed [1]; as usual, I'm recommending the latest snapshot that I've just uploaded [2].
Best, Christian
[1] https://github.com/BaseXdb/basex/issues/575 [2] http://files.basex.org/releases/latest/
Quoted text · Arjan van Bentem · 106 lines
On Sun, Sep 30, 2012 at 12:24 AM, Arjan van Bentem <arjan.vanbentem at bidnetwork.org> wrote: > Hi all, > > I noticed that TCP/IP clients are not seeing new documents that are created > using db:add(...) in another client: > > Once a query has been executed, subsequent queries using the very same > database connection do not see/return new documents that were meanwhile > created using another connection. When creating some (different) document > using one's own existing connection, then everything seems to be in-sync for > a moment. But right after that, any new documents are again not seen. > > Updates made in other clients are visible just fine. So, the documents that > are returned are up-to-date. It's just that not all documents are retrieved. > > I can see that in the query plans, the size in <DBNodeSeq size="..."> does > not change, and is lower than the size that clients with a newer connection > would show. > > Should it be okay to keep a connection open? Or is the expected behaviour to > need a new connection to see new documents? > > Details below. > > Thanks, > Arjan. > > > 1. Start the server and a client in debugging mode: > > ./basexserver -d -S > ./basexclient -U admin -P admin -d -x > > 2. Create and open a database: > > create database db1 > open db1 > > 3. Keep the client open, and make another client insert some documents. Like > every second for about 2 minutes, each time using a new connection: > > for i in {1..120}; \ > do doc="<test><a>`uuidgen`</a></test>"; \ > echo $i, $doc; \ > ./basexclient -U admin -P admin "db:add('db1', '$doc', 'dummy')"; \ > sleep 1; \ > done > > 4. While Step 3 is running, back in the first client, execute: > > xquery count(/test) > > The log in data/.logs shows: > > REQUEST XQUERY count(/test) > OK Query executed in 25.74 ms. 223.77 ms > > So far, so good. > > 5. Repeat Step 4 to see the result NOT change, while the loop from Step 3 is > still adding new documents. Logs show a faster result, but otherwise are the > same. Also the query plans are always the same. (Typically: the size in > <DBNodeSeq size="..."> does not change.) > > REQUEST XQUERY count(/test) > OK Query executed in 5.48 ms. 7.29 ms > > 6. Using "close" followed by "open db1" gets one a more up-to-date value for > "count". Also, using the connection to add some random other type of > document, updates the value for "count(/test)" as well: > > xquery db:add('db1', '<another-test><xyz>123</xyz></another-test>', > 'dummy') > > 7. It's not just "count" that does not change its value. Also, one can only > query documents up to that count as returned by Step 4. Like if Step 4 > returned "15", then the following would return 5 results, not 10: > > xquery (/test)[position()=10 to 20] > > Also, the following would only return the count as returned earlier: > > xquery /test > > Like noted, UPDATES made in other clients are visible just fine: > > xquery replace value of node /another-test/xyz with '456' > xquery for $n in /test/a return replace value of node $n with > fn:generate-id($n) > > > Tested in both 7.3 and 7.5-20120929.201251 from > http://files.basex.org/releases/latest/, in Oracle Java 7 on Mac OS X > 10.8.2. I first noticed that my running application did not see new > documents created by a test script. But the command line tools show the same > issue. > > Any clue? > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
Written for BaseX 7.3. BaseX may have changed since: please check the current documentation.