Content-Encoding: gzip on incoming RESTXQ POST
Tuesday, 2 November 2021 · 8 messages · last reply 8 Nov 2021
JB
James Ball2 Nov 2021 18:29 #
Hello,
I have a RESTXQ function that receives an XML file via HTTP POST. This works perfectly.
The client would now like to compress the data with gzip and set the Content-Encoding: gzip header.
I set
GZIP = true in my .basex file based on the documentation [1]
But these gzipped post requests fail - my function is called but content seems to arrive at my function still compressed.
(Responses from the server become compressed dynamically as expected - and that works great).
Is there something else I need to do to make it work for requests?
Many thanks, James
[1] https://docs.basex.org/wiki/Options#GZIPCG
Christian Grün4 Nov 2021 09:22 #
Hi James,
This is due to the default gzip settings of Jetty, which exclude POST requests. There had been a similar question in the past, and I have now decided to add the POST and PUT method to the Jetty defaults [1,2].
A new snapshot is available!
Cheers,
Christian
[1] https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg12281.html [2] https://github.com/BaseXdb/basex/issues/1748
Quoted text · James Ball · 20 lines
On Tue, Nov 2, 2021 at 6:29 PM James Ball <basex-talk at jamesball.co.uk> wrote: > > Hello, > > I have a RESTXQ function that receives an XML file via HTTP POST. This works perfectly. > > The client would now like to compress the data with gzip and set the Content-Encoding: gzip header. > > I set GZIP = true in my .basex file based on the documentation [1] > > But these gzipped post requests fail - my function is called but content seems to arrive at my function still compressed. > > (Responses from the server become compressed dynamically as expected - and that works great). > > Is there something else I need to do to make it work for requests? > > Many thanks, James > > > [1] https://docs.basex.org/wiki/Options#GZIP
JB
James Ball5 Nov 2021 13:20 #
Hi Christian,
My Google-fu was definitely letting me down. I completely missed that POST and PUT had to be explicitly included.
Thank you for the snapshot - which I’ve downloaded and tried. Sadly - same error.
I *think* I’ve found yet another default configuration for GZIPHandler getting in the way [1]:
"Both Request uncompress and Response compress are gated by a configurable DispatcherType check on the GzipHandler … (Default: DispatcherType.REQUEST).”
I understand this to mean that requests will NOT be decompressed by default. Would you read that the same way?
Do you think adding a line like the below would be the answer? My Java is getting out of date but if you think I’m on the right track I will try it and prepare a pull request for you.
Many thanks, James
BaseXHTTP.java [2]
import javax.servlet.*
gzip.setDispatcherTypes(DispatcherType.REQUEST, DispatcherType.RESPONSE)
[1] https://www.eclipse.org/jetty/javadoc/jetty-9/org/eclipse/jetty/server/handler/gzip/GzipHandler.html#setDispatcherTypes(javax.servlet.DispatcherType...) [2] https://github.com/BaseXdb/basex/blob/5f4f4316c77f781275a3e8afd58c151792d20f40/basex-api/src/main/java/org/basex/BaseXHTTP.java
Quoted text · Christian Grün · 39 lines
> On 4 Nov 2021, at 08:22, Christian Grün <christian.gruen at gmail.com> wrote: > > Hi James, > > This is due to the default gzip settings of Jetty, which exclude POST > requests. There had been a similar question in the past, and I have > now decided to add the POST and PUT method to the Jetty defaults > [1,2]. > > A new snapshot is available! > > Cheers, > Christian > > [1] https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg12281.html > [2] https://github.com/BaseXdb/basex/issues/1748 > > > > On Tue, Nov 2, 2021 at 6:29 PM James Ball <basex-talk at jamesball.co.uk> wrote: >> >> Hello, >> >> I have a RESTXQ function that receives an XML file via HTTP POST. This works perfectly. >> >> The client would now like to compress the data with gzip and set the Content-Encoding: gzip header. >> >> I set GZIP = true in my .basex file based on the documentation [1] >> >> But these gzipped post requests fail - my function is called but content seems to arrive at my function still compressed. >> >> (Responses from the server become compressed dynamically as expected - and that works great). >> >> Is there something else I need to do to make it work for requests? >> >> Many thanks, James >> >> >> [1] https://docs.basex.org/wiki/Options#GZIP
CG
Christian Grün5 Nov 2021 13:32 #
Hi James,
Sounds like a good hint! I’ll be happy to have a look at your pull request.
Thanks in advance,
Christian
Quoted text · James Ball · 71 lines
On Fri, Nov 5, 2021 at 1:20 PM James Ball <basex-talk at jamesball.co.uk> wrote: > > Hi Christian, > > My Google-fu was definitely letting me down. I completely missed that POST and PUT had to be explicitly included. > > Thank you for the snapshot - which I’ve downloaded and tried. Sadly - same error. > > I *think* I’ve found yet another default configuration for GZIPHandler getting in the way [1]: > > "Both Request uncompress and Response compress are gated by a configurable DispatcherType check on the GzipHandler … (Default: DispatcherType.REQUEST).” > > I understand this to mean that requests will NOT be decompressed by default. Would you read that the same way? > > Do you think adding a line like the below would be the answer? My Java is getting out of date but if you think I’m on the right track I will try it and prepare a pull request for you. > > > Many thanks, James > > > BaseXHTTP.java [2] > > import javax.servlet.* > > gzip.setDispatcherTypes(DispatcherType.REQUEST, DispatcherType.RESPONSE) > > > > [1] https://www.eclipse.org/jetty/javadoc/jetty-9/org/eclipse/jetty/server/handler/gzip/GzipHandler.html#setDispatcherTypes(javax.servlet.DispatcherType...) > [2] https://github.com/BaseXdb/basex/blob/5f4f4316c77f781275a3e8afd58c151792d20f40/basex-api/src/main/java/org/basex/BaseXHTTP.java > > > On 4 Nov 2021, at 08:22, Christian Grün <christian.gruen at gmail.com> wrote: > > > > Hi James, > > > > This is due to the default gzip settings of Jetty, which exclude POST > > requests. There had been a similar question in the past, and I have > > now decided to add the POST and PUT method to the Jetty defaults > > [1,2]. > > > > A new snapshot is available! > > > > Cheers, > > Christian > > > > [1] https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg12281.html > > [2] https://github.com/BaseXdb/basex/issues/1748 > > > > > > > > On Tue, Nov 2, 2021 at 6:29 PM James Ball <basex-talk at jamesball.co.uk> wrote: > >> > >> Hello, > >> > >> I have a RESTXQ function that receives an XML file via HTTP POST. This works perfectly. > >> > >> The client would now like to compress the data with gzip and set the Content-Encoding: gzip header. > >> > >> I set GZIP = true in my .basex file based on the documentation [1] > >> > >> But these gzipped post requests fail - my function is called but content seems to arrive at my function still compressed. > >> > >> (Responses from the server become compressed dynamically as expected - and that works great). > >> > >> Is there something else I need to do to make it work for requests? > >> > >> Many thanks, James > >> > >> > >> [1] https://docs.basex.org/wiki/Options#GZIP >
JR
Jonathan Robie8 Nov 2021 16:40 #
Is it possible to encrypt the zip file for HTTP requests? I have some documents that are not freely licensed, and I need some way to protect the IP of the copyright holder if I want to enable their texts.
Jonathan
Quoted text · Christian Grün · 102 lines
On Fri, Nov 5, 2021 at 8:32 AM Christian Grün <christian.gruen at gmail.com> wrote: > Hi James, > > Sounds like a good hint! I’ll be happy to have a look at your pull request. > > Thanks in advance, > Christian > > > > On Fri, Nov 5, 2021 at 1:20 PM James Ball <basex-talk at jamesball.co.uk> > wrote: > > > > Hi Christian, > > > > My Google-fu was definitely letting me down. I completely missed that > POST and PUT had to be explicitly included. > > > > Thank you for the snapshot - which I’ve downloaded and tried. Sadly - > same error. > > > > I *think* I’ve found yet another default configuration for GZIPHandler > getting in the way [1]: > > > > "Both Request uncompress and Response compress are gated by a > configurable DispatcherType check on the GzipHandler … (Default: > DispatcherType.REQUEST).” > > > > I understand this to mean that requests will NOT be decompressed by > default. Would you read that the same way? > > > > Do you think adding a line like the below would be the answer? My Java > is getting out of date but if you think I’m on the right track I will try > it and prepare a pull request for you. > > > > > > Many thanks, James > > > > > > BaseXHTTP.java [2] > > > > import javax.servlet.* > > > > gzip.setDispatcherTypes(DispatcherType.REQUEST, DispatcherType.RESPONSE) > > > > > > > > [1] > https://www.eclipse.org/jetty/javadoc/jetty-9/org/eclipse/jetty/server/handler/gzip/GzipHandler.html#setDispatcherTypes(javax.servlet.DispatcherType.. > .) > > [2] > https://github.com/BaseXdb/basex/blob/5f4f4316c77f781275a3e8afd58c151792d20f40/basex-api/src/main/java/org/basex/BaseXHTTP.java > > > > > On 4 Nov 2021, at 08:22, Christian Grün <christian.gruen at gmail.com> > wrote: > > > > > > Hi James, > > > > > > This is due to the default gzip settings of Jetty, which exclude POST > > > requests. There had been a similar question in the past, and I have > > > now decided to add the POST and PUT method to the Jetty defaults > > > [1,2]. > > > > > > A new snapshot is available! > > > > > > Cheers, > > > Christian > > > > > > [1] > https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg12281.html > > > [2] https://github.com/BaseXdb/basex/issues/1748 > > > > > > > > > > > > On Tue, Nov 2, 2021 at 6:29 PM James Ball <basex-talk at jamesball.co.uk> > wrote: > > >> > > >> Hello, > > >> > > >> I have a RESTXQ function that receives an XML file via HTTP POST. > This works perfectly. > > >> > > >> The client would now like to compress the data with gzip and set the > Content-Encoding: gzip header. > > >> > > >> I set GZIP = true in my .basex file based on the documentation [1] > > >> > > >> But these gzipped post requests fail - my function is called but > content seems to arrive at my function still compressed. > > >> > > >> (Responses from the server become compressed dynamically as expected > - and that works great). > > >> > > >> Is there something else I need to do to make it work for requests? > > >> > > >> Many thanks, James > > >> > > >> > > >> [1] https://docs.basex.org/wiki/Options#GZIP > > >
CG
Christian Grün8 Nov 2021 17:52 #
Hi Jonathan, do you have a chance to use HTTPS?
Quoted text · Jonathan Robie · 88 lines
On Mon, Nov 8, 2021 at 3:40 PM Jonathan Robie <jonathan.robie at gmail.com> wrote: > > Is it possible to encrypt the zip file for HTTP requests? I have some documents that are not freely licensed, and I need some way to protect the IP of the copyright holder if I want to enable their texts. > > Jonathan > > On Fri, Nov 5, 2021 at 8:32 AM Christian Grün <christian.gruen at gmail.com> wrote: >> >> Hi James, >> >> Sounds like a good hint! I’ll be happy to have a look at your pull request. >> >> Thanks in advance, >> Christian >> >> >> >> On Fri, Nov 5, 2021 at 1:20 PM James Ball <basex-talk at jamesball.co.uk> wrote: >> > >> > Hi Christian, >> > >> > My Google-fu was definitely letting me down. I completely missed that POST and PUT had to be explicitly included. >> > >> > Thank you for the snapshot - which I’ve downloaded and tried. Sadly - same error. >> > >> > I *think* I’ve found yet another default configuration for GZIPHandler getting in the way [1]: >> > >> > "Both Request uncompress and Response compress are gated by a configurable DispatcherType check on the GzipHandler … (Default: DispatcherType.REQUEST).” >> > >> > I understand this to mean that requests will NOT be decompressed by default. Would you read that the same way? >> > >> > Do you think adding a line like the below would be the answer? My Java is getting out of date but if you think I’m on the right track I will try it and prepare a pull request for you. >> > >> > >> > Many thanks, James >> > >> > >> > BaseXHTTP.java [2] >> > >> > import javax.servlet.* >> > >> > gzip.setDispatcherTypes(DispatcherType.REQUEST, DispatcherType.RESPONSE) >> > >> > >> > >> > [1] https://www.eclipse.org/jetty/javadoc/jetty-9/org/eclipse/jetty/server/handler/gzip/GzipHandler.html#setDispatcherTypes(javax.servlet.DispatcherType...) >> > [2] https://github.com/BaseXdb/basex/blob/5f4f4316c77f781275a3e8afd58c151792d20f40/basex-api/src/main/java/org/basex/BaseXHTTP.java >> > >> > > On 4 Nov 2021, at 08:22, Christian Grün <christian.gruen at gmail.com> wrote: >> > > >> > > Hi James, >> > > >> > > This is due to the default gzip settings of Jetty, which exclude POST >> > > requests. There had been a similar question in the past, and I have >> > > now decided to add the POST and PUT method to the Jetty defaults >> > > [1,2]. >> > > >> > > A new snapshot is available! >> > > >> > > Cheers, >> > > Christian >> > > >> > > [1] https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg12281.html >> > > [2] https://github.com/BaseXdb/basex/issues/1748 >> > > >> > > >> > > >> > > On Tue, Nov 2, 2021 at 6:29 PM James Ball <basex-talk at jamesball.co.uk> wrote: >> > >> >> > >> Hello, >> > >> >> > >> I have a RESTXQ function that receives an XML file via HTTP POST. This works perfectly. >> > >> >> > >> The client would now like to compress the data with gzip and set the Content-Encoding: gzip header. >> > >> >> > >> I set GZIP = true in my .basex file based on the documentation [1] >> > >> >> > >> But these gzipped post requests fail - my function is called but content seems to arrive at my function still compressed. >> > >> >> > >> (Responses from the server become compressed dynamically as expected - and that works great). >> > >> >> > >> Is there something else I need to do to make it work for requests? >> > >> >> > >> Many thanks, James >> > >> >> > >> >> > >> [1] https://docs.basex.org/wiki/Options#GZIP >> >
JR
Jonathan Robie8 Nov 2021 18:57 #
Yes, I do. And yeah, that's really the right solution ...
Thanks!
Jonathan
Quoted text · Christian Grün · 119 lines
On Mon, Nov 8, 2021 at 11:52 AM Christian Grün <christian.gruen at gmail.com> wrote: > Hi Jonathan, do you have a chance to use HTTPS? > > On Mon, Nov 8, 2021 at 3:40 PM Jonathan Robie <jonathan.robie at gmail.com> > wrote: > > > > Is it possible to encrypt the zip file for HTTP requests? I have some > documents that are not freely licensed, and I need some way to protect the > IP of the copyright holder if I want to enable their texts. > > > > Jonathan > > > > On Fri, Nov 5, 2021 at 8:32 AM Christian Grün <christian.gruen at gmail.com> > wrote: > >> > >> Hi James, > >> > >> Sounds like a good hint! I’ll be happy to have a look at your pull > request. > >> > >> Thanks in advance, > >> Christian > >> > >> > >> > >> On Fri, Nov 5, 2021 at 1:20 PM James Ball <basex-talk at jamesball.co.uk> > wrote: > >> > > >> > Hi Christian, > >> > > >> > My Google-fu was definitely letting me down. I completely missed that > POST and PUT had to be explicitly included. > >> > > >> > Thank you for the snapshot - which I’ve downloaded and tried. Sadly - > same error. > >> > > >> > I *think* I’ve found yet another default configuration for > GZIPHandler getting in the way [1]: > >> > > >> > "Both Request uncompress and Response compress are gated by a > configurable DispatcherType check on the GzipHandler … (Default: > DispatcherType.REQUEST).” > >> > > >> > I understand this to mean that requests will NOT be decompressed by > default. Would you read that the same way? > >> > > >> > Do you think adding a line like the below would be the answer? My > Java is getting out of date but if you think I’m on the right track I will > try it and prepare a pull request for you. > >> > > >> > > >> > Many thanks, James > >> > > >> > > >> > BaseXHTTP.java [2] > >> > > >> > import javax.servlet.* > >> > > >> > gzip.setDispatcherTypes(DispatcherType.REQUEST, > DispatcherType.RESPONSE) > >> > > >> > > >> > > >> > [1] > https://www.eclipse.org/jetty/javadoc/jetty-9/org/eclipse/jetty/server/handler/gzip/GzipHandler.html#setDispatcherTypes(javax.servlet.DispatcherType.. > .) > >> > [2] > https://github.com/BaseXdb/basex/blob/5f4f4316c77f781275a3e8afd58c151792d20f40/basex-api/src/main/java/org/basex/BaseXHTTP.java > >> > > >> > > On 4 Nov 2021, at 08:22, Christian Grün <christian.gruen at gmail.com> > wrote: > >> > > > >> > > Hi James, > >> > > > >> > > This is due to the default gzip settings of Jetty, which exclude > POST > >> > > requests. There had been a similar question in the past, and I have > >> > > now decided to add the POST and PUT method to the Jetty defaults > >> > > [1,2]. > >> > > > >> > > A new snapshot is available! > >> > > > >> > > Cheers, > >> > > Christian > >> > > > >> > > [1] > https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg12281.html > >> > > [2] https://github.com/BaseXdb/basex/issues/1748 > >> > > > >> > > > >> > > > >> > > On Tue, Nov 2, 2021 at 6:29 PM James Ball < > basex-talk at jamesball.co.uk> wrote: > >> > >> > >> > >> Hello, > >> > >> > >> > >> I have a RESTXQ function that receives an XML file via HTTP POST. > This works perfectly. > >> > >> > >> > >> The client would now like to compress the data with gzip and set > the Content-Encoding: gzip header. > >> > >> > >> > >> I set GZIP = true in my .basex file based on the documentation [1] > >> > >> > >> > >> But these gzipped post requests fail - my function is called but > content seems to arrive at my function still compressed. > >> > >> > >> > >> (Responses from the server become compressed dynamically as > expected - and that works great). > >> > >> > >> > >> Is there something else I need to do to make it work for requests? > >> > >> > >> > >> Many thanks, James > >> > >> > >> > >> > >> > >> [1] https://docs.basex.org/wiki/Options#GZIP > >> > >
JB
To close this for the archive and for anyone who needs this in the future:
In Jetty the GzipHandler has a number of defaults that will limit what it does unless explicitly overridden.
For example it will compress responses only with GET requests and will not decompress requests.
The configuration of the GzipHandler in BaseX has been changed (thank you Christian!):
• It now explicitly includes POST and PUT methods
• The inflate buffer size is set greater than 0 - which enables transparent decompression of gzipped requests
So something like:
echo '<Test/>' | gzip | curl -v -H "Content-Type: text/xml" -H "Content-Encoding: gzip" --data-binary @- http://localhost:8984/yourFunction
will now deliver XML to yourFunction on a system that has
GZIP set to true.
Regards, JamesQuoted text · Christian Grün · 39 lines
> On 4 Nov 2021, at 08:22, Christian Grün <christian.gruen at gmail.com> wrote: > > Hi James, > > This is due to the default gzip settings of Jetty, which exclude POST > requests. There had been a similar question in the past, and I have > now decided to add the POST and PUT method to the Jetty defaults > [1,2]. > > A new snapshot is available! > > Cheers, > Christian > > [1] https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg12281.html > [2] https://github.com/BaseXdb/basex/issues/1748 > > > > On Tue, Nov 2, 2021 at 6:29 PM James Ball <basex-talk at jamesball.co.uk> wrote: >> >> Hello, >> >> I have a RESTXQ function that receives an XML file via HTTP POST. This works perfectly. >> >> The client would now like to compress the data with gzip and set the Content-Encoding: gzip header. >> >> I set GZIP = true in my .basex file based on the documentation [1] >> >> But these gzipped post requests fail - my function is called but content seems to arrive at my function still compressed. >> >> (Responses from the server become compressed dynamically as expected - and that works great). >> >> Is there something else I need to do to make it work for requests? >> >> Many thanks, James >> >> >> [1] https://docs.basex.org/wiki/Options#GZIP
Written for BaseX 9.6. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery