Streaming results
Thursday, 19 April 2018 · 5 messages · last reply 20 Apr 2018
BV
Bram Vanroy19 Apr 2018 11:02 #
Dear list
I am curious to find out whether it is possible to stream results of a query, i.e. where a submitted query behaves as a generator. Especially for web interfaces this is useful: a user is presented with a result as soon as it is found, and doesn't have to wait for all results to be gathered. Especially useful with large databases of course.
Preferably a way that works with the nodejs, PHP client but I am open to other suggestions.
Thanks in advance
Bram
CG
Christian Grün19 Apr 2018 23:01 #
Hi Bram,
There is no comparable feature in BaseX right now. But maybe it’s already helpful for you to know that all server-side results will always be streamed to the output stream of the HTTP response. As a result, it can easily happen that the client has received the first 10 results while the server is still evaluating your query.
If you want to have more control, you can define a result window via variables (e.g. $start and $length). Usually, if your query succeeds in quickly delivering the result 1-10, it shouldn’t take much longer to deliver result 11-20, 21-30, etc., so you can send multiple iterative requests to the server to receive the subsequent results.
Best,
Christian
Quoted text · Bram Vanroy · 25 lines
On Thu, Apr 19, 2018 at 11:02 AM, Bram Vanroy <Bram.Vanroy at ugent.be> wrote: > Dear list > > > > I am curious to find out whether it is possible to stream results of a > query, i.e. where a submitted query behaves as a generator. Especially for > web interfaces this is useful: a user is presented with a result as soon as > it is found, and doesn’t have to wait for all results to be gathered. > Especially useful with large databases of course. > > > > Preferably a way that works with the nodejs, PHP client but I am open to > other suggestions. > > > > > > Thanks in advance > > > > Bram
KD
Kirsten, Dirk20 Apr 2018 09:12 #
Hi Bram, hi Christian,
I am surprised, I would have expected you to point to the upcoming websocket implementation (see https://github.com/BaseXdb/basex/issues/1449)? To me the scenario described by Bram seems like a perfect fit for websockets, where you simply subscribe to such a generator and the server then delivers the results by notifying every subscriber.
Cheers
Dirk
Senacor Technologies Aktiengesellschaft - Sitz: Eschborn - Amtsgericht Frankfurt am Main - Reg.-Nr.: HRB 105546 Vorstand: Matthias Tomann, Marcus Purzer - Aufsichtsratsvorsitzender: Daniel Grözinger
Quoted text · Christian Grün · 1 line
On 19. Apr 2018, at 23:01, Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> wrote:
Hi Bram,
There is no comparable feature in BaseX right now. But maybe it’s already helpful for you to know that all server-side results will always be streamed to the output stream of the HTTP response. As a result, it can easily happen that the client has received the first 10 results while the server is still evaluating your query.
If you want to have more control, you can define a result window via variables (e.g. $start and $length). Usually, if your query succeeds in quickly delivering the result 1-10, it shouldn’t take much longer to deliver result 11-20, 21-30, etc., so you can send multiple iterative requests to the server to receive the subsequent results.
Best,
Christian
Quoted text · Bram Vanroy · 1 line
On Thu, Apr 19, 2018 at 11:02 AM, Bram Vanroy <Bram.Vanroy at ugent.be<mailto:Bram.Vanroy at ugent.be>> wrote:
Dear list
I am curious to find out whether it is possible to stream results of a query, i.e. where a submitted query behaves as a generator. Especially for web interfaces this is useful: a user is presented with a result as soon as it is found, and doesn’t have to wait for all results to be gathered. Especially useful with large databases of course.
Preferably a way that works with the nodejs, PHP client but I am open to other suggestions.
Thanks in advance
Bram
BV
Bram Vanroy20 Apr 2018 09:42 #
Dear Christian, dear Christian
Thank you for the input. In the past we’d indeed use manual index points and query iteratively. (I think I remember us using something like (query)[position() = 1 to 10]?) But this isn’t the most ‘pretty’ solution and doesn’t leverage any heuristics of found results, i.e. it’ll still just wait until all the items in a given position are found rather than notifying the browser that results are found directly.
Therefore I’m happy to see that a websocket implementation is in the making! We’re planning to drop Internet Explorer support any way, which gives us a bit more freedom in implementing ground-breaking features, which I think WS are.
I can see the feature implementation is tagged as 9.1. Any idea when this version will roll-out? Or is an implementation already available in the latest snapshott?
Thanks in advance!
Bram
Quoted text · Kirsten, Dirk · 37 lines
Van: Kirsten, Dirk <Dirk.Kirsten at senacor.com> Verzonden: vrijdag 20 april 2018 9:13 Aan: Christian Grün <christian.gruen at gmail.com> CC: Bram Vanroy <Bram.Vanroy at UGent.be>; BaseX <basex-talk at mailman.uni-konstanz.de> Onderwerp: Re: [basex-talk] Streaming results Hi Bram, hi Christian, I am surprised, I would have expected you to point to the upcoming websocket implementation (see https://github.com/BaseXdb/basex/issues/1449)? To me the scenario described by Bram seems like a perfect fit for websockets, where you simply subscribe to such a generator and the server then delivers the results by notifying every subscriber. Cheers Dirk Senacor Technologies Aktiengesellschaft - Sitz: Eschborn - Amtsgericht Frankfurt am Main - Reg.-Nr.: HRB 105546 Vorstand: Matthias Tomann, Marcus Purzer - Aufsichtsratsvorsitzender: Daniel Grözinger On 19. Apr 2018, at 23:01, Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> wrote: Hi Bram, There is no comparable feature in BaseX right now. But maybe it’s already helpful for you to know that all server-side results will always be streamed to the output stream of the HTTP response. As a result, it can easily happen that the client has received the first 10 results while the server is still evaluating your query. If you want to have more control, you can define a result window via variables (e.g. $start and $length). Usually, if your query succeeds in quickly delivering the result 1-10, it shouldn’t take much longer to deliver result 11-20, 21-30, etc., so you can send multiple iterative requests to the server to receive the subsequent results. Best, Christian On Thu, Apr 19, 2018 at 11:02 AM, Bram Vanroy <Bram.Vanroy at ugent.be<mailto:Bram.Vanroy at ugent.be>> wrote: Dear list I am curious to find out whether it is possible to stream results of a query, i.e. where a submitted query behaves as a generator. Especially for web interfaces this is useful: a user is presented with a result as soon as it is found, and doesn’t have to wait for all results to be gathered. Especially useful with large databases of course. Preferably a way that works with the nodejs, PHP client but I am open to other suggestions. Thanks in advance Bram
CG
Christian Grün20 Apr 2018 10:20 #
Hi Bram,
Quoted text · Bram Vanroy · 3 lines
> I can see the feature implementation is tagged as 9.1. Any idea when this > version will roll-out? Or is an implementation already available in the > latest snapshott?
We are still investigating what are the best options to integrate WebSockets in BaseX, as we mostly want to avoid building a solution that cannot be properly enhanced once new feature requests are coming in from our users.
To be more specific: We’ll probably build our solution on top of the Jetty WebSocket Server API. While this will give us not as much freedom as when implementing WebSockets completely on our own, and while creating a new Jetty dependency (it won’t work with Tomcat, etc.), we believe it frees us from the need to take care of all minor details, and it allows us to benefit from various optimizations for concurrent requests and heavy traffic that will be available out of the box.
One drawback is that the Jetty servlet approach might hinder us from including support for the 9.0 permissions layer [1], but this is still work in progress. Just follow our Github issue to stay up-to-date.
By the way, this is a good opportunity to thank
Nubisware and Marco Lettere
for sponsoring parts of this new feature! If anyone out there is interested in speeding up the implementation process, we invite you to get connected with us.
Best,
Christian
[1] http://docs.basex.org/wiki/Permissions
[2] https://github.com/BaseXdb/basex/issues/1449#issuecomment-372127950
Quoted text · Bram Vanroy · 84 lines
> Van: Kirsten, Dirk <Dirk.Kirsten at senacor.com> > Verzonden: vrijdag 20 april 2018 9:13 > Aan: Christian Grün <christian.gruen at gmail.com> > CC: Bram Vanroy <Bram.Vanroy at UGent.be>; BaseX > <basex-talk at mailman.uni-konstanz.de> > Onderwerp: Re: [basex-talk] Streaming results > > > > Hi Bram, hi Christian, > > > > I am surprised, I would have expected you to point to the upcoming websocket > implementation (see https://github.com/BaseXdb/basex/issues/1449)? > > To me the scenario described by Bram seems like a perfect fit for > websockets, where you simply subscribe to such a generator and the server > then delivers the results by notifying every subscriber. > > > > Cheers > > Dirk > > > > > Senacor Technologies Aktiengesellschaft - Sitz: Eschborn - Amtsgericht > Frankfurt am Main - Reg.-Nr.: HRB 105546 > Vorstand: Matthias Tomann, Marcus Purzer - Aufsichtsratsvorsitzender: Daniel > Grözinger > > On 19. Apr 2018, at 23:01, Christian Grün <christian.gruen at gmail.com> wrote: > > > > Hi Bram, > > There is no comparable feature in BaseX right now. But maybe it’s > already helpful for you to know that all server-side results will > always be streamed to the output stream of the HTTP response. As a > result, it can easily happen that the client has received the first 10 > results while the server is still evaluating your query. > > If you want to have more control, you can define a result window via > variables (e.g. $start and $length). Usually, if your query succeeds > in quickly delivering the result 1-10, it shouldn’t take much longer > to deliver result 11-20, 21-30, etc., so you can send multiple > iterative requests to the server to receive the subsequent results. > > Best, > Christian > > > On Thu, Apr 19, 2018 at 11:02 AM, Bram Vanroy <Bram.Vanroy at ugent.be> wrote: > > Dear list > > > > I am curious to find out whether it is possible to stream results of a > query, i.e. where a submitted query behaves as a generator. Especially for > web interfaces this is useful: a user is presented with a result as soon as > it is found, and doesn’t have to wait for all results to be gathered. > Especially useful with large databases of course. > > > > Preferably a way that works with the nodejs, PHP client but I am open to > other suggestions. > > > > > > Thanks in advance > > > > Bram > >
Written for BaseX 9.0. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery