Main Page » Mailing List » January 2020

How to escape/encode a search term using BaseX REST XQ

Monday, 20 January 2020 · 6 messages · last reply 24 Jan 2020

Ivan Kanakarakis20 Jan 2020 10:54 #

Hello everyone, I am using BaseX 8.44 and the REST XQ interface (ie, http://docs.basex.org/wiki/RESTXQ). I have an endpoint that, when invoked with GET, it does a full text search (using "$db-nodes[text() contains text { $term } all]"), gets the results, constructs a JSON response and sends it back. That's all fine and works great. However, I am not sure how I should be doing the queries I describe bellow. _Note: the query is initiated by a SPA javascript client, thus when I say encode/uri-escape, what I mean is that I invoke the encodeURIComponent function (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent). _Note 2: for the sake of conversation let's consider the example endpoint declared as: %rest:GET %rest:path("/search/{$term}") 1. I want to search for "tea". That is the basic query. A single term, no problem.

    curl -s "https://example.com/search/tea"

2. I want to search for "tea time". Now, this query has a space in between the two words. What I expect to get back, is any node that contains both words (thus I have used "contains text" with "all"), even if they may be a few words apart. - Should I be sending an encoded/uri-escape version of this, ie, "tea%20time"? - Or, should I be replacing the space with "+", ie "tea+time"? - Or, some other advice?

    curl -s "https://example.com/search/tea%20time"
    curl -s "https://example.com/search/tea+time"

3. I want to search for "tea/time". This is even trickier. What I expect to get back, is any node that contains "tea/time", ie a search result for a single term. How do I do this? - If I do not do anything, the slash is treated as part of the URL, thus not matching a route. - If I encoded/uri-escape this term, I get "tea%2Ftime". But, when I invoke the endpoint I get the same as if there was a slash. - I am not sure how I should deal with the slash. How should I escape/encode this?

    curl -s "https://example.com/search/tea/time"
    curl -s "https://example.com/search/tea%2Ftime"

Thank you,

Christian Grün20 Jan 2020 18:36 #

Hi Ivan, A more common approach is to supply search terms as query parameters (URL?query=...); in that case, your path won’t have new segments. If you prefer paths, you can use a regular expression in your RESTXQ path pattern [1]: "search/{$query=.+}" In both cases, encodeURIComponent should be the appropriate function to encode special characters. Hope this helps, Christian [1] http://docs.basex.org/wiki/RESTXQ#Paths

Quoted text · Ivan Kanakarakis · 58 lines

On Mon, Jan 20, 2020 at 10:54 AM Ivan Kanakarakis <ivan.kanak+basex-talk at gmail.com> wrote: > > Hello everyone, > > I am using BaseX 8.44 and the REST XQ interface (ie, > http://docs.basex.org/wiki/RESTXQ). I have an endpoint that, when > invoked with GET, it does a full text search (using "$db-nodes[text() > contains text { $term } all]"), gets the results, constructs a JSON > response and sends it back. > > That's all fine and works great. However, I am not sure how I should > be doing the queries I describe bellow. > > _Note: the query is initiated by a SPA javascript client, thus when I > say encode/uri-escape, what I mean is that I invoke the > encodeURIComponent function > (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent). > _Note 2: for the sake of conversation let's consider the example > endpoint declared as: > > %rest:GET > %rest:path("/search/{$term}") > > > 1. I want to search for "tea". That is the basic query. A single term, > no problem. > > curl -s "https://example.com/search/tea" > > > 2. I want to search for "tea time". Now, this query has a space in > between the two words. What I expect to get back, is any node that > contains both words (thus I have used "contains text" with "all"), > even if they may be a few words apart. > - Should I be sending an encoded/uri-escape version of this, ie, "tea%20time"? > - Or, should I be replacing the space with "+", ie "tea+time"? > - Or, some other advice? > > curl -s "https://example.com/search/tea%20time" > curl -s "https://example.com/search/tea+time" > > > 3. I want to search for "tea/time". This is even trickier. What I > expect to get back, is any node that contains "tea/time", ie a search > result for a single term. How do I do this? > - If I do not do anything, the slash is treated as part of the URL, > thus not matching a route. > - If I encoded/uri-escape this term, I get "tea%2Ftime". But, when I > invoke the endpoint I get the same as if there was a slash. > - I am not sure how I should deal with the slash. How should I > escape/encode this? > > curl -s "https://example.com/search/tea/time" > curl -s "https://example.com/search/tea%2Ftime" > > > Thank you,

Ivan Kanakarakis24 Jan 2020 13:54 #

Hi Christian, thanks for the quick reply. It definitely helps, but it still keeps this behaviour in the "weird" domain. I do not see a reason to be decoding the URI before it gets to match a route. What is the reason for this? What you propose works, but if I have a route like "/search/{$query=.+}/page/{$page}", then the query will match everything including "/page/...". If the path was not decoded, I do not think I would need the regex, neither any other special operation on the route. It should work with "/search/{$query}/page/{$page}" and it should return "tea%2Ftime". Why do I have to make workarounds to try to guess how a part of the URL was encoded, when the URL I hit has that part encoded? I don't think it makes sense, and I don't see a use case for this. When the framework receives the payload, it is responsible to match a route. By matching the route, it will provide me with the binded parts of the route that I requested. Then, *I* am responsible to decode those parts as I see fit and handle the request as I need. If the framework decodes the URL before matching a route, that is a problem to me - I do not have the control I need. If the framework decodes the URL parts before binding the route variables, this is fine - it saves me an operation. While, I now refactored the endpoint handlers to work with query params, and this is no longer a problem for me, it is a problem in general. Cheers,

Quoted text · Christian Grün · 81 lines

On Mon, 20 Jan 2020 at 19:36, Christian Grün <christian.gruen at gmail.com> wrote: > > Hi Ivan, > > A more common approach is to supply search terms as query parameters > (URL?query=...); in that case, your path won’t have new segments. If > you prefer paths, you can use a regular expression in your RESTXQ path > pattern [1]: > > "search/{$query=.+}" > > In both cases, encodeURIComponent should be the appropriate function > to encode special characters. > > Hope this helps, > Christian > > [1] http://docs.basex.org/wiki/RESTXQ#Paths > > > > > > On Mon, Jan 20, 2020 at 10:54 AM Ivan Kanakarakis > <ivan.kanak+basex-talk at gmail.com> wrote: > > > > Hello everyone, > > > > I am using BaseX 8.44 and the REST XQ interface (ie, > > http://docs.basex.org/wiki/RESTXQ). I have an endpoint that, when > > invoked with GET, it does a full text search (using "$db-nodes[text() > > contains text { $term } all]"), gets the results, constructs a JSON > > response and sends it back. > > > > That's all fine and works great. However, I am not sure how I should > > be doing the queries I describe bellow. > > > > _Note: the query is initiated by a SPA javascript client, thus when I > > say encode/uri-escape, what I mean is that I invoke the > > encodeURIComponent function > > (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent). > > _Note 2: for the sake of conversation let's consider the example > > endpoint declared as: > > > > %rest:GET > > %rest:path("/search/{$term}") > > > > > > 1. I want to search for "tea". That is the basic query. A single term, > > no problem. > > > > curl -s "https://example.com/search/tea" > > > > > > 2. I want to search for "tea time". Now, this query has a space in > > between the two words. What I expect to get back, is any node that > > contains both words (thus I have used "contains text" with "all"), > > even if they may be a few words apart. > > - Should I be sending an encoded/uri-escape version of this, ie, "tea%20time"? > > - Or, should I be replacing the space with "+", ie "tea+time"? > > - Or, some other advice? > > > > curl -s "https://example.com/search/tea%20time" > > curl -s "https://example.com/search/tea+time" > > > > > > 3. I want to search for "tea/time". This is even trickier. What I > > expect to get back, is any node that contains "tea/time", ie a search > > result for a single term. How do I do this? > > - If I do not do anything, the slash is treated as part of the URL, > > thus not matching a route. > > - If I encoded/uri-escape this term, I get "tea%2Ftime". But, when I > > invoke the endpoint I get the same as if there was a slash. > > - I am not sure how I should deal with the slash. How should I > > escape/encode this? > > > > curl -s "https://example.com/search/tea/time" > > curl -s "https://example.com/search/tea%2Ftime" > > > > > > Thank you,

Imsieke, Gerrit, le-tex24 Jan 2020 14:36 #

While moving the URI parameter to the query string seems like an acceptable workaround, I, too, suggest that if *reserved* URI characters such as '/' appear percent-encoded, they should not be converted to their decoded character prior to analyzing the URI, in line with Sect. 2.2 of RFC 3986 [1]. If I enter an escaped colon (%3A) in a path segment, it will be kept as %3A by BaseX, rather than converted to the reserved character ':'. The RESTXQ specification [2] doesn’t seem to contain detailed instructions on how to decode the submitted URI before extracting path parameters, therefore I think RFC 3986 should prevail. So I agree, BaseX should not interpret escaped slashes as if they were regular slashes, thereby disallowing them as part of RESTXQ path pa Gerrit [1] https://tools.ietf.org/html/rfc3986#section-2.2 [2] http://exquery.github.io/exquery/exquery-restxq-specification/restxq-1.0-specification.html

Quoted text · Ivan Kanakarakis · 119 lines

On 24.01.2020 13:54, Ivan Kanakarakis wrote: > Hi Christian, > > thanks for the quick reply. It definitely helps, but it still keeps > this behaviour in the "weird" domain. > I do not see a reason to be decoding the URI before it gets to match a > route. What is the reason for this? > > What you propose works, but if I have a route like > "/search/{$query=.+}/page/{$page}", then the query will match > everything including "/page/...". If the path was not decoded, I do > not think I would need the regex, neither any other special operation > on the route. It should work with "/search/{$query}/page/{$page}" and > it should return "tea%2Ftime". Why do I have to make workarounds to > try to guess how a part of the URL was encoded, when the URL I hit has > that part encoded? > I don't think it makes sense, and I don't see a use case for this. > > When the framework receives the payload, it is responsible to match a route. > By matching the route, it will provide me with the binded parts of the > route that I requested. > Then, *I* am responsible to decode those parts as I see fit and handle > the request as I need. > > If the framework decodes the URL before matching a route, that is a > problem to me - I do not have the control I need. > If the framework decodes the URL parts before binding the route > variables, this is fine - it saves me an operation. > > While, I now refactored the endpoint handlers to work with query > params, and this is no longer a problem for me, it is a problem in > general. > > > Cheers, > > > > On Mon, 20 Jan 2020 at 19:36, Christian Grün <christian.gruen at gmail.com> wrote: >> >> Hi Ivan, >> >> A more common approach is to supply search terms as query parameters >> (URL?query=...); in that case, your path won’t have new segments. If >> you prefer paths, you can use a regular expression in your RESTXQ path >> pattern [1]: >> >> "search/{$query=.+}" >> >> In both cases, encodeURIComponent should be the appropriate function >> to encode special characters. >> >> Hope this helps, >> Christian >> >> [1] http://docs.basex.org/wiki/RESTXQ#Paths >> >> >> >> >> >> On Mon, Jan 20, 2020 at 10:54 AM Ivan Kanakarakis >> <ivan.kanak+basex-talk at gmail.com> wrote: >>> >>> Hello everyone, >>> >>> I am using BaseX 8.44 and the REST XQ interface (ie, >>> http://docs.basex.org/wiki/RESTXQ). I have an endpoint that, when >>> invoked with GET, it does a full text search (using "$db-nodes[text() >>> contains text { $term } all]"), gets the results, constructs a JSON >>> response and sends it back. >>> >>> That's all fine and works great. However, I am not sure how I should >>> be doing the queries I describe bellow. >>> >>> _Note: the query is initiated by a SPA javascript client, thus when I >>> say encode/uri-escape, what I mean is that I invoke the >>> encodeURIComponent function >>> (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent). >>> _Note 2: for the sake of conversation let's consider the example >>> endpoint declared as: >>> >>> %rest:GET >>> %rest:path("/search/{$term}") >>> >>> >>> 1. I want to search for "tea". That is the basic query. A single term, >>> no problem. >>> >>> curl -s "https://example.com/search/tea" >>> >>> >>> 2. I want to search for "tea time". Now, this query has a space in >>> between the two words. What I expect to get back, is any node that >>> contains both words (thus I have used "contains text" with "all"), >>> even if they may be a few words apart. >>> - Should I be sending an encoded/uri-escape version of this, ie, "tea%20time"? >>> - Or, should I be replacing the space with "+", ie "tea+time"? >>> - Or, some other advice? >>> >>> curl -s "https://example.com/search/tea%20time" >>> curl -s "https://example.com/search/tea+time" >>> >>> >>> 3. I want to search for "tea/time". This is even trickier. What I >>> expect to get back, is any node that contains "tea/time", ie a search >>> result for a single term. How do I do this? >>> - If I do not do anything, the slash is treated as part of the URL, >>> thus not matching a route. >>> - If I encoded/uri-escape this term, I get "tea%2Ftime". But, when I >>> invoke the endpoint I get the same as if there was a slash. >>> - I am not sure how I should deal with the slash. How should I >>> escape/encode this? >>> >>> curl -s "https://example.com/search/tea/time" >>> curl -s "https://example.com/search/tea%2Ftime" >>> >>> >>> Thank you,

Imsieke, Gerrit, le-tex24 Jan 2020 14:38 #

Quoted text · le-tex · 3 lines

On 24.01.2020 14:36, Imsieke, Gerrit, le-tex wrote: > So I agree, BaseX should not interpret escaped slashes as if they were > regular slashes, thereby disallowing them as part of RESTXQ path pa

…rameters.

Christian Grün24 Jan 2020 18:35 #

Hi Ivan, hi Gerrit, Thanks for your assessments. Most design decisions in RESTXQ have been taken from Java’s JAX-RS API [1]. The semantics for accessing paths is a bit more complex, though: JAX-RS provides two annotations @Path and @PathParam to access the full path and segments of the path, and the segments are automatically decoded. Automatic decoding can be disabled via an optional @Encoded annotation. In RESTXQ, we only have a single %rest:path annotations, which contains both the full path as well as variables for path segments. Requests with wrongly encoded URLs, such as http://localhost:8984/a%2, are already rejected by Jetty (and, I guess, any other web servers). They are rejected before any RESTXQ code can intervene. If a URLs is correctly encoded, the Java servlet function getPathInfo() is used to obtain the path. I noticed there is an alternative function getRequestURI() that could be used to access the original URL. Maybe the introduction of a %rest:encoded annotation could be discussed in the EXQuery/RESTXQ repository [2]? Best, Christian [1] https://download.oracle.com/otndocs/jcp/jaxrs-2_0-fr-eval-spec/index.html [2] https://github.com/exquery/exquery/issues

Quoted text · le-tex · 10 lines

On Fri, Jan 24, 2020 at 2:38 PM Imsieke, Gerrit, le-tex <gerrit.imsieke at le-tex.de> wrote: > > > > On 24.01.2020 14:36, Imsieke, Gerrit, le-tex wrote: > > So I agree, BaseX should not interpret escaped slashes as if they were > > regular slashes, thereby disallowing them as part of RESTXQ path pa > > …rameters.

Written for BaseX 9.3. BaseX may have changed since: please check the current documentation.

⚡Generated with XQuery

`J`	Select the next message
`K`	Select the previous message
`N`	Next thread
`P`	Previous thread
`U`	Go up one level
`/`	Jump to the search field