HTTP API DB access
Saturday, 15 October 2011 · 10 messages · last reply 19 Oct 2011
RJ
Ralf Jung15 Oct 2011 14:21 #
Hi list,
while debugging my application, I noticed that when I send a request to the HTTP API of basex, the request is then sent by basex to itself via a localhost connection. That is, the HTTP API part of basex sends a message to the BaseX native API through TCP.
Now, I wonder, is this really necessary? After all, both reside in the same process, so going through the OS feels a bit wasteful here. I noticed that I can change this behaviour if I pass the "-l" flag to basexhttp ("local mode"), so the code to directly use the in-process DB obviously exists. I wonder why it can only be used in the local mode which entirely disables the "native" BaseX API?
Kind regards,
Ralf
CG
Christian Grün16 Oct 2011 17:30 #
Hi Ralf,
yes, that's a good point. We'd probably have to check somewhere (e.g. in the HTTPSession.local() method) if a server exists in the same JVM. If you have a spontaneous idea how to perform this check.. Feel free to tell us ;)
Thanks,
Christian
Quoted text · Ralf Jung · 21 lines
On Sat, Oct 15, 2011 at 2:21 PM, Ralf Jung <ralfjung-e at gmx.de> wrote: > Hi list, > > while debugging my application, I noticed that when I send a request to the > HTTP API of basex, the request is then sent by basex to itself via a localhost > connection. That is, the HTTP API part of basex sends a message to the BaseX > native API through TCP. > Now, I wonder, is this really necessary? After all, both reside in the same > process, so going through the OS feels a bit wasteful here. I noticed that I > can change this behaviour if I pass the "-l" flag to basexhttp ("local mode"), > so the code to directly use the in-process DB obviously exists. I wonder why > it can only be used in the local mode which entirely disables the "native" > BaseX API? > > Kind regards, > Ralf > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
RJ
Ralf Jung17 Oct 2011 15:26 #
Hi,
Quoted text · Christian Grün · 4 lines
> yes, that's a good point. We'd probably have to check somewhere (e.g. > in the HTTPSession.local() method) if a server exists in the same JVM. > If you have a spontaneous idea how to perform this check.. Feel free > to tell us ;)
Is using LocalSession safe if there can be concurrent queries through the native interface, i.e. concurrent ClientSession (and of course other LocalSession) accesses?
The decision (Local vs. Client) is complicated by the fact that the logic when a native BaseX server is launched, is quite complicated: It is started if the -l flag is NOT present AND the
HOST property is set to something different from "localhost". I would have expected to control that directly via some flags, for example:
* "basexhttp" starts the API plus a local native BaseX server
* "basexhttp -l" starts just the API (local mode)
* "basexhttp -r" starts the API and connects to a remote BaseX server to execute the queries etc. on (remote mode)
In the first two modes, LocalSession could be used (if it can be used concurrently, see above). It must not be possible to pass both the -l and -r flag.
This would keep the default behaviour, and would remove the "magic" of the server launch happening or not. IMHO that's a quite significant difference, because it decides which data is going to be used.
But maybe I just have a strange intuition ;-)
Kind regards,
RalfRJ
Ralf Jung17 Oct 2011 15:40 #
Hi again,
to clarify what I mean, I roughly implemented it in attached patch... or at least I think I did, it is *COMPLETELY UNTESTED* and just serves the purpose of stating more precisely what I described.
Kind regards,
Ralf
Quoted text · Ralf Jung · 35 lines
On Monday 17 October 2011 15:26:15 Ralf Jung wrote: > Hi, > > > yes, that's a good point. We'd probably have to check somewhere (e.g. > > in the HTTPSession.local() method) if a server exists in the same JVM. > > If you have a spontaneous idea how to perform this check.. Feel free > > to tell us ;) > > Is using LocalSession safe if there can be concurrent queries through the > native interface, i.e. concurrent ClientSession (and of course other > LocalSession) accesses? > > The decision (Local vs. Client) is complicated by the fact that the logic > when a native BaseX server is launched, is quite complicated: It is > started if the -l flag is NOT present AND the HOST property is set to > something different from "localhost". I would have expected to control > that directly via some flags, for example: > * "basexhttp" starts the API plus a local native BaseX server > * "basexhttp -l" starts just the API (local mode) > * "basexhttp -r" starts the API and connects to a remote BaseX server to > execute the queries etc. on (remote mode) > In the first two modes, LocalSession could be used (if it can be used > concurrently, see above). It must not be possible to pass both the -l and > -r flag. > This would keep the default behaviour, and would remove the "magic" of the > server launch happening or not. IMHO that's a quite significant difference, > because it decides which data is going to be used. > But maybe I just have a strange intuition ;-) > > Kind regards, > Ralf > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
CG
Quoted text · Ralf Jung · 3 lines
> Is using LocalSession safe if there can be concurrent queries through the > native interface, i.e. concurrent ClientSession (and of course other > LocalSession) accesses?
Yes; both LocalSession and ClientSession instances take advantage of our locking architecture and can be used in parallel (as long as the server instance and the LocalSession use the same instance of the Context class).
Quoted text · Ralf Jung · 3 lines
> * "basexhttp -l" starts just the API (local mode) > * "basexhttp -r" starts the API and connects to a remote BaseX server to > execute the queries etc. on (remote mode)
Your proposal is worth discussing. I'm still not sure, though, how to name such an additional option, and how to make it as self-explanatory as possible: if no alternative host name will be specified by the user, "remote" is rather confusing imho, as the database server will be located on "localhost", i.e., the same machine. – After all, the reason for the current approach was to reduce the number of options that would otherwise need more explanation, and could make things even more complicated (we even thought about completely discarding the local flag).
Quoted text · Ralf Jung · 1 line
> But maybe I just have a strange intuition ;-)
Not at all.. And I still like the idea of directly communicating to the server and avoiding the ClientSession if it resides in the same JVM (it's mainly a question of how to implement it most properly).
Thanks for caring,
Christian
RJ
Ralf Jung18 Oct 2011 09:38 #
Hi,
Quoted text · Christian Grün · 4 lines
> Yes; both LocalSession and ClientSession instances take advantage of > our locking architecture and can be used in parallel (as long as the > server instance and the LocalSession use the same instance of the > Context class).
Great!
Quoted text · Christian Grün · 13 lines
> > * "basexhttp -l" starts just the API (local mode) > > * "basexhttp -r" starts the API and connects to a remote BaseX server to > > execute the queries etc. on (remote mode) > > Your proposal is worth discussing. I'm still not sure, though, how to > name such an additional option, and how to make it as self-explanatory > as possible: if no alternative host name will be specified by the > user, "remote" is rather confusing imho, as the database server will > be located on "localhost", i.e., the same machine. – After all, the > reason for the current approach was to reduce the number of options > that would otherwise need more explanation, and could make things even > more complicated (we even thought about completely discarding the > local flag).
From the point of view of the server, it is still "remote" as it is accessed via TCP, even if that connection is a localhost connection. From my experience, it is quite unusual to treat a TCP connection to localhost differently than a TCP connection to any other host - and technically, after all, there is no difference. But then, I know how this stuff works, other who don't might find it less intuitive. Maybe "external" would be a better name? (as in, external to the JVM/OS process the API is running in). The current setup however makes it impossible to start a BaseX native server on a host, and then start an API frontend on the same machine but in a different process that connects to it - the remote IP would be "localhost" and the API would start a 2nd database server instance. So this would require work-arounds like adding a name to the /etc/hosts that resolves to 127.0.0.1 but is not recognized by BaseX as localhost.
You might find that a strange setup, but together with SSH tunnelling it even becomes useful: The server runs on a machine far away that you can't directly connect to (or don't want to, unencrypted, as the network in between is untrusted). So you SSH in that machine and create a tunnel from the local BaseX port to the one on the other machine, and let the API connect to that local port (which in reality is a remote BaseX server). Okay, this might still be strange, but I think you get the point ;-)
Quoted text · Christian Grün · 5 lines
> > But maybe I just have a strange intuition ;-) > > Not at all.. And I still like the idea of directly communicating to > the server and avoiding the ClientSession if it resides in the same > JVM (it's mainly a question of how to implement it most properly).
Well, I could easily come up with a patch that essentially duplicates the "start server in JVM or not" logic and passes the info on to HTTPSession (I actually already had that patch done before I came up with the other one). But that would still be confusing to me, and the problem mentioned above would persist.
At the very least, this magic behaviour should be documented (in the wiki and maybe also in the -h help text). I only learned how it works by looking at the source.
Kind regards,
Ralf
RJ
Ralf Jung18 Oct 2011 10:09 #
Hi again,
Quoted text · Ralf Jung · 3 lines
> Well, I could easily come up with a patch that essentially duplicates the > "start server in JVM or not" logic and passes the info on to HTTPSession (I > actually already had that patch done before I came up with the other one).
You can find that patch attached, again *COMPLETELY UNTESTED*. It should get rid of the pointless TCP overhead, but it does not fix the "magic" behaviour.
Kind regards,
Ralf
CG
Christian Grün18 Oct 2011 10:40 #
Looks reasonable (and could surely be worse for being "completely untested" ;). I'll give you feedback some time later today.
Thanks again,
Christian
Quoted text · Ralf Jung · 8 lines
>> Well, I could easily come up with a patch that essentially duplicates the >> "start server in JVM or not" logic and passes the info on to HTTPSession (I >> actually already had that patch done before I came up with the other one). > You can find that patch attached, again *COMPLETELY UNTESTED*. It should get > rid of the pointless TCP overhead, but it does not fix the "magic" behaviour. > > Kind regards, > Ralf
CG
Hi Ralf,
I've just committed a new solution that should reflect your change request. We decided to call the flag "client" (-c), in order to indicate that the HTTP Server talks to the database server as just another client.
Feedback welcome,
Christian
Quoted text · Ralf Jung · 12 lines
On Tue, Oct 18, 2011 at 10:09 AM, Ralf Jung <ralfjung-e at gmx.de> wrote: > Hi again, > >> Well, I could easily come up with a patch that essentially duplicates the >> "start server in JVM or not" logic and passes the info on to HTTPSession (I >> actually already had that patch done before I came up with the other one). > You can find that patch attached, again *COMPLETELY UNTESTED*. It should get > rid of the pointless TCP overhead, but it does not fix the "magic" behaviour. > > Kind regards, > Ralf >
RJ
Ralf Jung19 Oct 2011 10:52 #
Hi,
Quoted text · Christian Grün · 4 lines
> I've just committed a new solution that should reflect your change > request. We decided to call the flag "client" (-c), in order to > indicate that the HTTP Server talks to the database server as just > another client.
That's a good name... thanks!
Kind regards,
Ralf
Written for BaseX 6.7. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery