HTTPServer + gzip compression
Tuesday, 24 September 2019 · 17 messages · last reply 22 Jan 2020
SK
Stefan Koch24 Sep 2019 20:09 #
Hi,
I'm struggling to get gzip compression working.
I'm using the REST module.
http://docs.basex.org/wiki/REST
Version: BaseX 9.2.2
Tried to add the handler in jetty.xml as descriped here:
https://www.eclipse.org/jetty/documentation/current/gzip-filter.html But it didn't work. Tried gzip filter via web.xml - but it is deprecated since jetty 9.3.
Gzip handler is the correct way to do it.
Searched the mailling list, similar problem was reported here: https://mailman.uni-konstanz.de/pipermail/basex-talk/2019-February/014160.html Unfortunately no solution :(
Not much experience with embedded jetty - but adding gzip compression is straight forward in tomcat or apache - no clue why it doesn't work.Shouldn't be that hard - and it's kinda a default feature. Did anybody get it to work? Any tipps?
Thanks
Stefan Koch
B&IT Solutions
Dipl.-Kfm. Rudolf Markus Petri
Lietzenburger Str. 77
10719 Berlin
Tel: 0049 30 8867 6099
Fax: 0049 30 8867 6159
Mail: koch at buit-solutions.com
Web: www.buit-solutions.com
CG
Christian Grün1 Oct 2019 14:47 #
Hi Stefan,
Do you work with an embedded Jetty instance (i.e., did you run basexhttp), or do you use BaseX as servlet? In the latter case, this may need to be tackled by the Jetty developers. Did you address this on their mailing list?
Another alternative could be to include
GZIP support in RESTXQ, and send gzipped responses whenever the client send a corresponding Accept-Encoding header. I’ll have some more thoughts on that.
Best,
ChristianQuoted text · Stefan Koch · 30 lines
On Tue, Sep 24, 2019 at 8:09 PM Stefan Koch <koch at buit-solutions.com> wrote: > > Hi, > I'm struggling to get gzip compression working. > I'm using the REST module. > http://docs.basex.org/wiki/REST > Version: BaseX 9.2.2 > Tried to add the handler in jetty.xml as descriped here: > https://www.eclipse.org/jetty/documentation/current/gzip-filter.html > But it didn't work. Tried gzip filter via web.xml - but it is deprecated > since jetty 9.3. > Gzip handler is the correct way to do it. > Searched the mailling list, similar problem was reported here: > https://mailman.uni-konstanz.de/pipermail/basex-talk/2019-February/014160.html > Unfortunately no solution :( > Not much experience with embedded jetty - but adding gzip compression is > straight forward in tomcat or apache - no clue why it doesn't > work.Shouldn't be that hard - and it's kinda a default feature. > Did anybody get it to work? Any tipps? > Thanks > Stefan Koch > > B&IT Solutions > Dipl.-Kfm. Rudolf Markus Petri > Lietzenburger Str. 77 > 10719 Berlin > Tel: 0049 30 8867 6099 > Fax: 0049 30 8867 6159 > Mail: koch at buit-solutions.com > Web: www.buit-solutions.com
SK
Stefan Koch1 Oct 2019 16:17 #
Hi Christian,
Thx for your reply.
I'm using the embedded Jett via basexhttp service.
Not using RESTXQ - basic REST is what we need for this project.
Good idea to ask the Jetty devs - I'll try that.
kind regards,
Stefan
Quoted text · Christian Grün · 46 lines
-----Ursprüngliche Nachricht----- Von: Christian Grün [mailto:christian.gruen at gmail.com] Gesendet: Dienstag, 1. Oktober 2019 14:47 An: Stefan Koch <koch at buit-solutions.com> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Betreff: Re: [basex-talk] HTTPServer + gzip compression Hi Stefan, Do you work with an embedded Jetty instance (i.e., did you run basexhttp), or do you use BaseX as servlet? In the latter case, this may need to be tackled by the Jetty developers. Did you address this on their mailing list? Another alternative could be to include GZIP support in RESTXQ, and send gzipped responses whenever the client send a corresponding Accept-Encoding header. I’ll have some more thoughts on that. Best, Christian On Tue, Sep 24, 2019 at 8:09 PM Stefan Koch <koch at buit-solutions.com> wrote: > > Hi, > I'm struggling to get gzip compression working. > I'm using the REST module. > http://docs.basex.org/wiki/REST > Version: BaseX 9.2.2 > Tried to add the handler in jetty.xml as descriped here: > https://www.eclipse.org/jetty/documentation/current/gzip-filter.html > But it didn't work. Tried gzip filter via web.xml - but it is deprecated > since jetty 9.3. > Gzip handler is the correct way to do it. > Searched the mailling list, similar problem was reported here: > https://mailman.uni-konstanz.de/pipermail/basex-talk/2019-February/014160.html > Unfortunately no solution :( > Not much experience with embedded jetty - but adding gzip compression is > straight forward in tomcat or apache - no clue why it doesn't > work.Shouldn't be that hard - and it's kinda a default feature. > Did anybody get it to work? Any tipps? > Thanks > Stefan Koch > > B&IT Solutions > Dipl.-Kfm. Rudolf Markus Petri > Lietzenburger Str. 77 > 10719 Berlin > Tel: 0049 30 8867 6099 > Fax: 0049 30 8867 6159 > Mail: koch at buit-solutions.com > Web: www.buit-solutions.com
CG
Christian Grün3 Oct 2019 11:51 #
Hi Stefan,
There’s a StackOverflow entry that has previously been referenced on this list (by Michael Seiferle, see [2]). Based on this thread, in which Joakim Erdfelt (the magnificent Jetty core developer) explains why the existing approaches for enabling
GZIP compression don’t work anymore, I see three choices:
1. to wrap all HTTP responses in a GZIP output stream;
2. to initialize GZipHandler in our basexhttp code; or
3. enable GZIP compression outside BaseX,
Alternative 1 would give us most control, but it would raise new questions that would need to be solved. Alternative 2 may be the better approach: It only works if basexhttp is used, but we could benefit from existing optimizations and tweaks from the Jetty implementation [2]. Alternative 3 is already available: You can use another light-weight web server as proxy (caddy, nginx), or you can start Jetty as described by Joakim.Quoted text · Stefan Koch · 1 line
> Good idea to ask the Jetty devs - I'll try that.
Thanks; feel free to keep us updated,
Christian
[1] https://stackoverflow.com/questions/38635262/jetty-9-and-gziphandler [2] https://www.eclipse.org/jetty/documentation/current/gzip-filter.html
SK
Stefan Koch7 Oct 2019 13:40 #
Hi Christian,
thx for your reply. Got it :)
Solution 2 would be cool - maybe feature request?
But I can live with a workaround.
kind regards,
Stefan
Quoted text · Christian Grün · 27 lines
-----Ursprüngliche Nachricht----- Von: Christian Grün [mailto:christian.gruen at gmail.com] Gesendet: Donnerstag, 3. Oktober 2019 11:51 An: Stefan Koch <koch at buit-solutions.com> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Betreff: Re: [basex-talk] HTTPServer + gzip compression Hi Stefan, There’s a StackOverflow entry that has previously been referenced on this list (by Michael Seiferle, see [2]). Based on this thread, in which Joakim Erdfelt (the magnificent Jetty core developer) explains why the existing approaches for enabling GZIP compression don’t work anymore, I see three choices: 1. to wrap all HTTP responses in a GZIP output stream; 2. to initialize GZipHandler in our basexhttp code; or 3. enable GZIP compression outside BaseX, Alternative 1 would give us most control, but it would raise new questions that would need to be solved. Alternative 2 may be the better approach: It only works if basexhttp is used, but we could benefit from existing optimizations and tweaks from the Jetty implementation [2]. Alternative 3 is already available: You can use another light-weight web server as proxy (caddy, nginx), or you can start Jetty as described by Joakim. > Good idea to ask the Jetty devs - I'll try that. Thanks; feel free to keep us updated, Christian [1] https://stackoverflow.com/questions/38635262/jetty-9-and-gziphandler [2] https://www.eclipse.org/jetty/documentation/current/gzip-filter.html
CG
Christian Grün12 Nov 2019 14:52 #
Hi Stefan,
Jetty’s
GZIP feature can now be enabled in BaseX [1,2].
Looking forward to your testing feedback,
Christian
[1] http://docs.basex.org/wiki/Options#GZIP
[2] http://files.basex.org/releases/latest/Quoted text · Stefan Koch · 49 lines
On Mon, Oct 7, 2019 at 1:40 PM Stefan Koch <koch at buit-solutions.com> wrote: > Hi Christian, > > thx for your reply. Got it :) > Solution 2 would be cool - maybe feature request? > > But I can live with a workaround. > > kind regards, > > Stefan > > > -----Ursprüngliche Nachricht----- > *Von:* Christian Grün [mailto:christian.gruen at gmail.com] > *Gesendet:* Donnerstag, 3. Oktober 2019 11:51 > *An:* Stefan Koch <koch at buit-solutions.com> > *Cc:* BaseX <basex-talk at mailman.uni-konstanz.de> > *Betreff:* Re: [basex-talk] HTTPServer + gzip compression > > > Hi Stefan, > > There’s a StackOverflow entry that has previously been referenced on > this list (by Michael Seiferle, see [2]). Based on this thread, in > which Joakim Erdfelt (the magnificent Jetty core developer) explains > why the existing approaches for enabling GZIP compression don’t work > anymore, I see three choices: > > 1. to wrap all HTTP responses in a GZIP output stream; > 2. to initialize GZipHandler in our basexhttp code; or > 3. enable GZIP compression outside BaseX, > > Alternative 1 would give us most control, but it would raise new > questions that would need to be solved. Alternative 2 may be the > better approach: It only works if basexhttp is used, but we could > benefit from existing optimizations and tweaks from the Jetty > implementation [2]. Alternative 3 is already available: You can use > another light-weight web server as proxy (caddy, nginx), or you can > start Jetty as described by Joakim. > > > Good idea to ask the Jetty devs - I'll try that. > > Thanks; feel free to keep us updated, > Christian > > [1] https://stackoverflow.com/questions/38635262/jetty-9-and-gziphandler > [2] https://www.eclipse.org/jetty/documentation/current/gzip-filter.html >
OS
Omar Siam18 Nov 2019 18:41 #
I just wanted to try gzip http compression and downloaded the current build BaseX93-20191116. The option is not there anymore. What happened?
Best regards
Omar Siam
CG
Christian Grün19 Nov 2019 01:28 #
The snapshot has been updated.
Is it working now?
Quoted text · Omar Siam · 7 lines
On Mon, Nov 18, 2019 at 6:41 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: > > I just wanted to try gzip http compression and downloaded the current build BaseX93-20191116. The option is not there anymore. What happened? > > Best regards > > Omar Siam
OS
Omar Siam20 Nov 2019 12:41 #
Yes, it works! 1000 entries of XML in a JSON wrapper from my data are now transfered as 193 KB instead of 1.6 MB. Not bad. Unfortunately on localhost that doesn't make much of a difference in overall timing as measured in Chrome's development tools. For real internet connections I think this is quite an improvement.
Best regards
Omar Siam
Quoted text · Christian Grün · 12 lines
Am 19.11.2019 um 01:28 schrieb Christian Grün: > The snapshot has been updated. > Is it working now? > > > > On Mon, Nov 18, 2019 at 6:41 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: >> I just wanted to try gzip http compression and downloaded the current build BaseX93-20191116. The option is not there anymore. What happened? >> >> Best regards >> >> Omar Siam
CG
Christian Grün20 Nov 2019 13:26 #
Thanks for the confirmation!
Quoted text · Omar Siam · 25 lines
Omar Siam <Omar.Siam at oeaw.ac.at> schrieb am Mi., 20. Nov. 2019, 12:41: > Yes, it works! 1000 entries of XML in a JSON wrapper from my data are > now transfered as 193 KB instead of 1.6 MB. Not bad. Unfortunately on > localhost that doesn't make much of a difference in overall timing as > measured in Chrome's development tools. For real internet connections I > think this is quite an improvement. > > Best regards > > Omar Siam > > Am 19.11.2019 um 01:28 schrieb Christian Grün: > > The snapshot has been updated. > > Is it working now? > > > > > > > > On Mon, Nov 18, 2019 at 6:41 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: > >> I just wanted to try gzip http compression and downloaded the current > build BaseX93-20191116. The option is not there anymore. What happened? > >> > >> Best regards > >> > >> Omar Siam >
SK
Stefan Koch20 Nov 2019 14:45 #
Hi,
great to hear! So it's coming in 9.3
We have been running base.war via tomcat with gzip instead - but it wasn't optimal for our solution.
Thx for fixing
Stefan
Quoted text · Christian Grün · 30 lines
-----Ursprüngliche Nachricht----- Von: Christian Grün [mailto:christian.gruen at gmail.com] Gesendet: Mittwoch, 20. November 2019 13:26 An: Omar Siam <Omar.Siam at oeaw.ac.at> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Betreff: Re: [basex-talk] HTTPServer + gzip compression Thanks for the confirmation! Omar Siam <Omar.Siam at oeaw.ac.at> schrieb am Mi., 20. Nov. 2019, 12:41: > Yes, it works! 1000 entries of XML in a JSON wrapper from my data are > now transfered as 193 KB instead of 1.6 MB. Not bad. Unfortunately on > localhost that doesn't make much of a difference in overall timing as > measured in Chrome's development tools. For real internet connections I > think this is quite an improvement. > > Best regards > > Omar Siam > > Am 19.11.2019 um 01:28 schrieb Christian Grün: > > The snapshot has been updated. > > Is it working now? > > > > > > > > On Mon, Nov 18, 2019 at 6:41 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: > >> I just wanted to try gzip http compression and downloaded the current build BaseX93-20191116. The option is not there anymore. What happened? > >> > >> Best regards > >> > >> Omar Siam
SK
Stefan Koch13 Dec 2019 12:24 #
Hi,
GZIP once again. It works fine with 9.3.
However:
http://docs.basex.org/wiki/Options#GZIP
"Sane defaults" - so if you click on the link:
_methods.include(HttpMethod.GET.asString());
So only for GET - not POST.
Our current Webservices rely heavily on POST (+Response) though :(
So my question is - do you plan to change this and support POST responses with GZIP (Feature Request!) in the near future or is this not in your current pipeline? We can work with a external server und run the BaseX.war instead - but it would be cool if POST responses would get supported.
Kind regards and keep up the awesome work,
StefanQuoted text · Stefan Koch · 40 lines
-----Ursprüngliche Nachricht----- Von: Stefan Koch [mailto:koch at buit-solutions.com] Gesendet: Mittwoch, 20. November 2019 14:45 An: basex-talk at mailman.uni-konstanz.de Betreff: Re: [basex-talk] HTTPServer + gzip compression Hi, great to hear! So it's coming in 9.3 We have been running base.war via tomcat with gzip instead - but it wasn't optimal for our solution. Thx for fixing Stefan -----Ursprüngliche Nachricht----- Von: Christian Grün [mailto:christian.gruen at gmail.com] Gesendet: Mittwoch, 20. November 2019 13:26 An: Omar Siam <Omar.Siam at oeaw.ac.at> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Betreff: Re: [basex-talk] HTTPServer + gzip compression Thanks for the confirmation! Omar Siam <Omar.Siam at oeaw.ac.at> schrieb am Mi., 20. Nov. 2019, 12:41: > Yes, it works! 1000 entries of XML in a JSON wrapper from my data are > now transfered as 193 KB instead of 1.6 MB. Not bad. Unfortunately on > localhost that doesn't make much of a difference in overall timing as > measured in Chrome's development tools. For real internet connections I > think this is quite an improvement. > > Best regards > > Omar Siam > > Am 19.11.2019 um 01:28 schrieb Christian Grün: > > The snapshot has been updated. > > Is it working now? > > > > > > > > On Mon, Nov 18, 2019 at 6:41 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: > >> I just wanted to try gzip http compression and downloaded the current build BaseX93-20191116. The option is not there anymore. What happened? > >> > >> Best regards > >> > >> Omar Siam
CG
Christian Grün17 Dec 2019 07:59 #
Hi Stefan,
We could consider an extension of the current solution.
Quoted text · Stefan Koch · 2 lines
> So only for GET - not POST. > Our current Webservices rely heavily on POST (+Response) though :(
Have you checked the remaining default options? What other custom settings may you need to rely on?
Cheers,
Christian
Quoted text · Stefan Koch · 56 lines
> -----Ursprüngliche Nachricht----- > Von: Stefan Koch [mailto:koch at buit-solutions.com] > Gesendet: Mittwoch, 20. November 2019 14:45 > An: basex-talk at mailman.uni-konstanz.de > Betreff: Re: [basex-talk] HTTPServer + gzip compression > > > Hi, > > great to hear! So it's coming in 9.3 > > We have been running base.war via tomcat with gzip instead - but it wasn't optimal for our solution. > > > Thx for fixing > > Stefan > > > > -----Ursprüngliche Nachricht----- > Von: Christian Grün [mailto:christian.gruen at gmail.com] > Gesendet: Mittwoch, 20. November 2019 13:26 > An: Omar Siam <Omar.Siam at oeaw.ac.at> > Cc: BaseX <basex-talk at mailman.uni-konstanz.de> > Betreff: Re: [basex-talk] HTTPServer + gzip compression > > > Thanks for the confirmation! > > > > Omar Siam <Omar.Siam at oeaw.ac.at> schrieb am Mi., 20. Nov. 2019, 12:41: >> >> Yes, it works! 1000 entries of XML in a JSON wrapper from my data are >> now transfered as 193 KB instead of 1.6 MB. Not bad. Unfortunately on >> localhost that doesn't make much of a difference in overall timing as >> measured in Chrome's development tools. For real internet connections I >> think this is quite an improvement. >> >> Best regards >> >> Omar Siam >> >> Am 19.11.2019 um 01:28 schrieb Christian Grün: >> > The snapshot has been updated. >> > Is it working now? >> > >> > >> > >> > On Mon, Nov 18, 2019 at 6:41 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: >> >> I just wanted to try gzip http compression and downloaded the current build BaseX93-20191116. The option is not there anymore. What happened? >> >> >> >> Best regards >> >> >> >> Omar Siam
SK
Stefan Koch23 Dec 2019 19:16 #
Hi Christian,
Checked the
GZIP Handler options.
https://www.eclipse.org/jetty/documentation/current/gzip-filter.html
It should support:
includedMethods
(GET, POST, PUT) - PATCH is not used by BaseX REST API - not needed.
However these are response settings.
Jetty does support request compression - if BaseX works for XML-Webservices, you wanna POST/PUT large XML docs that are compressed, I guess.
As far as I could figure out: "inflateBufferSize" should be set to 1 (0 is default) (it's in the official jetty-gzip.xml not in the link from above). No default option in jetty, shouldn't be default in BaseX REST client, I guess. This should enable request compression.
HTTP is dead, long live HTTPS!
For productive use this is needed (otherwise you have to run NGINX as proxy or do similar workarounds). Currently only HTTP is supported.
Downloaded jetty - in /etc there are the config xmls - relevant config options are:
jetty-https,jetty-ssl, jetty-ssl-context (you wanna have a keystore/certificate for SSL).
productive use:
jetty-threadpool, jetty-threadlimit and jetty-connectionlimit might come in handy for the admins - but have no clue about that. Not really a priority, though.
There is a bunch of other stuff, but IMO not really important.
Best wishes and nice holydays,
StefanQuoted text · Christian Grün · 70 lines
-----Ursprüngliche Nachricht----- Von: Christian Grün [mailto:christian.gruen at gmail.com] Gesendet: Dienstag, 17. Dezember 2019 07:59 An: Stefan Koch <koch at buit-solutions.com> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Betreff: Re: [basex-talk] HTTPServer + gzip compression Hi Stefan, We could consider an extension of the current solution. > So only for GET - not POST. > Our current Webservices rely heavily on POST (+Response) though :( Have you checked the remaining default options? What other custom settings may you need to rely on? Cheers, Christian > -----Ursprüngliche Nachricht----- > Von: Stefan Koch [mailto:koch at buit-solutions.com] > Gesendet: Mittwoch, 20. November 2019 14:45 > An: basex-talk at mailman.uni-konstanz.de > Betreff: Re: [basex-talk] HTTPServer + gzip compression > > > Hi, > > great to hear! So it's coming in 9.3 > > We have been running base.war via tomcat with gzip instead - but it wasn't optimal for our solution. > > > Thx for fixing > > Stefan > > > > -----Ursprüngliche Nachricht----- > Von: Christian Grün [mailto:christian.gruen at gmail.com] > Gesendet: Mittwoch, 20. November 2019 13:26 > An: Omar Siam <Omar.Siam at oeaw.ac.at> > Cc: BaseX <basex-talk at mailman.uni-konstanz.de> > Betreff: Re: [basex-talk] HTTPServer + gzip compression > > > Thanks for the confirmation! > > > > Omar Siam <Omar.Siam at oeaw.ac.at> schrieb am Mi., 20. Nov. 2019, 12:41: >> >> Yes, it works! 1000 entries of XML in a JSON wrapper from my data are >> now transfered as 193 KB instead of 1.6 MB. Not bad. Unfortunately on >> localhost that doesn't make much of a difference in overall timing as >> measured in Chrome's development tools. For real internet connections I >> think this is quite an improvement. >> >> Best regards >> >> Omar Siam >> >> Am 19.11.2019 um 01:28 schrieb Christian Grün: >> > The snapshot has been updated. >> > Is it working now? >> > >> > >> > >> > On Mon, Nov 18, 2019 at 6:41 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: >> >> I just wanted to try gzip http compression and downloaded the current build BaseX93-20191116. The option is not there anymore. What happened? >> >> >> >> Best regards >> >> >> >> Omar Siam
SY
Hi Christian,
we are facing on reports generation using xQuery/xPath. in this case we are getting data from 54000 xml files. it is taking more than 23 hours for generating report.
My hardware :
Linex and ram 25 Gb
And Please check given below my query:
Multilevel activity Summary report (service number level drill down report):1- day
declare function local:getLabelName($serNo as xs:string) { let $doc :=
collection('LabelCollection_1573625140685') for $label in $doc//row[@ser_no
= $serNo] order by $label/@upDate descending return $label };declare
function local:getHHMMSSFromNumber($num as xs:integer) { let $ss :=
xs:integer($num mod 60) let $mm1 := xs:integer($num div 60) let $mm :=
xs:integer($mm1 mod 60) let $hh := xs:integer($mm1 div 60) let $hhFinal
:= if(string-length(xs:string($hh)) >= 2) then $hh else (concat('0', $hh))
let $mmFinal := if(string-length(xs:string($mm)) >= 2) then $mm else
(concat('0', $mm)) let $ssFinal := if(string-length(xs:string($ss)) >= 2)
then $ss else (concat('0', $ss)) return
concat($hhFinal,':',$mmFinal,':',$ssFinal) }; declare function
local:getMinutesFromDutation($duration as xs:string) { let $duration :=
tokenize($duration, ":") let $hh := $duration[1] let $mm :=
$duration[2] let $ss := $duration[3] return number(data($hh)) * 60 +
number(data($mm)) + floor(number(data($ss)) div 60) }; declare function
local:getViewPontChargesData() { let $Rows :=
(collection("867509_Voice_OCTOBER-19_Unbilled_1")/SUBCUSTBRK[AccNo[@NO =
(6945045)]][Site[@NAME='']][DOB >='2019-10-01' and DOB <='2019-10-31'])
let $serviceNumbers :=
string-join(distinct-values(data($Rows/CONN/@NO)),',') let
$eachServiceNumber := tokenize($serviceNumbers, ",") for $service
at $index in $eachServiceNumber let $serviceNumber := $service
let $serviceData := $Rows[CONN[@NO =
$serviceNumber]]/DETAIL/TRANSACTION/SUB_SECTION let $nCalls :=
xs:decimal(sum(data($serviceData/@Calls))) let $totalAmt :=
round-half-to-even(sum(data($serviceData/@Amt)), 3) let $BNCount
:= count(data(distinct-values($serviceData/../../../CONN/@NO))) let
$durSum := sum(data($serviceData/@Duration)) let $totDur :=
local:getHHMMSSFromNumber(xs:integer($durSum)) let $tMs :=
local:getMinutesFromDutation($totDur) let $totalDurSec := $durSum
return <ROW><SNB>{ $serviceNumber }</SNB><SNL>{
distinct-values(local:getLabelName($serviceNumber)[position()=1])
}</SNL><CLS>{ $nCalls }</CLS><DUR>{ $totDur
}</DUR><DUS>{$totalDurSec}</DUS><MIN>{ $tMs }</MIN><AM>{
if($tMs > 0 and $nCalls > 0) then (let
$avgMin := round-half-to-even($tMs div $nCalls, 2)
return $avgMin) else (0.00)
}</AM><AMT>{ $totalAmt }</AMT><PC>{
if($totalAmt > 0 and $nCalls > 0) then (let
$avgPeRCall := round-half-to-even($totalAmt div $nCalls, 2)
return $avgPeRCall) else
(0.00) }</PC><PM>{
if($totalAmt > 0 and $tMs > 0) then
(let $avgPerMin := round-half-to-even($totalAmt div $tMs, 3)
return $avgPerMin) else
(0.00) }</PM><PN>{
if($totalAmt > 0 and $BNCount > 0) then
(let $avgPerNum := round-half-to-even($totalAmt div $BNCount, 3)
return $avgPerNum)
else (0.000) }</PN></ROW> }; let
$viewPointCData := local:getViewPontChargesData() let $TotCalls :=
xs:decimal(sum(data($viewPointCData/CLS))) let $TotAmt :=
sum(data($viewPointCData/AMT)) let $totalDur :=
local:getHHMMSSFromNumber(xs:integer(sum($viewPointCData/DUS))) let
$totalMins := local:getMinutesFromDutation($totalDur) let $TAPAmt :=
sum(data($viewPointCData/PN)) let $TotNoofAccs :=
count(data($viewPointCData/SNB)) return ( if(count($viewPointCData) >
0) then (for $eachRow in $viewPointCData let $eachAmount :=
$eachRow/AMT return <R>{ $eachRow/node(), <PER>{
if($eachAmount > 0 and $TotAmt > 0) then (let $perAll :=
round-half-to-even(($eachAmount div $TotAmt) * 100, 2) return
$perAll) else (0.00) }</PER> }</R>,
<R><SNB>Total/Average</SNB><CLS>{ $TotCalls }</CLS><DUR>{ $totalDur
}</DUR><MIN>{ format-number($totalMins,'0.000') }</MIN><AM>{
if($totalMins > 0 and $TotCalls > 0) then (let
$avgMinutes := round-half-to-even($totalMins div $TotCalls, 2)
return format-number($avgMinutes,'0.000')) else
(0.000) }</AM><AMT>{ format-number($TotAmt,'0.000')
}</AMT><PC>{ if($TotAmt > 0 and $TotCalls > 0) then
(let $avgCallAmt := round-half-to-even($TotAmt div $TotCalls,
2) return format-number($avgCallAmt,'0.000'))
else (0.000) }</PC><PM>{
if($TotAmt > 0 and $totalMins > 0) then
(let $avgPerMin := round-half-to-even($TotAmt div $totalMins, 3)
return format-number($avgPerMin,'0.000'))
else (0.000) }</PM><PN>{
if($TAPAmt > 0 and $TotNoofAccs > 0) then
(let $avgNumAmt := round-half-to-even($TAPAmt div $TotNoofAccs, 3)
return format-number($avgNumAmt,'0.000'))
else (0.000) }</PN><PER>100</PER></R>)
else () )Please suggest way forward.
Regards,
YSL
Quoted text · Christian Grün · 78 lines
On Tue, Dec 17, 2019 at 12:29 PM Christian Grün <christian.gruen at gmail.com> wrote: > Hi Stefan, > > We could consider an extension of the current solution. > > > So only for GET - not POST. > > Our current Webservices rely heavily on POST (+Response) though :( > > Have you checked the remaining default options? What other custom > settings may you need to rely on? > > Cheers, > Christian > > > > > -----Ursprüngliche Nachricht----- > > Von: Stefan Koch [mailto:koch at buit-solutions.com] > > Gesendet: Mittwoch, 20. November 2019 14:45 > > An: basex-talk at mailman.uni-konstanz.de > > Betreff: Re: [basex-talk] HTTPServer + gzip compression > > > > > > Hi, > > > > great to hear! So it's coming in 9.3 > > > > We have been running base.war via tomcat with gzip instead - but it > wasn't optimal for our solution. > > > > > > Thx for fixing > > > > Stefan > > > > > > > > -----Ursprüngliche Nachricht----- > > Von: Christian Grün [mailto:christian.gruen at gmail.com] > > Gesendet: Mittwoch, 20. November 2019 13:26 > > An: Omar Siam <Omar.Siam at oeaw.ac.at> > > Cc: BaseX <basex-talk at mailman.uni-konstanz.de> > > Betreff: Re: [basex-talk] HTTPServer + gzip compression > > > > > > Thanks for the confirmation! > > > > > > > > Omar Siam <Omar.Siam at oeaw.ac.at> schrieb am Mi., 20. Nov. 2019, 12:41: > >> > >> Yes, it works! 1000 entries of XML in a JSON wrapper from my data are > >> now transfered as 193 KB instead of 1.6 MB. Not bad. Unfortunately on > >> localhost that doesn't make much of a difference in overall timing as > >> measured in Chrome's development tools. For real internet connections I > >> think this is quite an improvement. > >> > >> Best regards > >> > >> Omar Siam > >> > >> Am 19.11.2019 um 01:28 schrieb Christian Grün: > >> > The snapshot has been updated. > >> > Is it working now? > >> > > >> > > >> > > >> > On Mon, Nov 18, 2019 at 6:41 PM Omar Siam <Omar.Siam at oeaw.ac.at> > wrote: > >> >> I just wanted to try gzip http compression and downloaded the > current build BaseX93-20191116. The option is not there anymore. What > happened? > >> >> > >> >> Best regards > >> >> > >> >> Omar Siam >
SK
Stefan Koch22 Jan 2020 18:33 #
Hi,
Sorry for self-bumping - but topic might have been overlooked because of holydays.
While gzip may be my personal wishlist - It would be could if others could chime in and say what they think about it.
HTTPS however is pretty important - independent of my personal wishlist. If there is an embedded jetty already - it should support HTTPS without using proxy workarounds, IMO. See recent issue with a lot of repostitories stopping HTTP support, breaking a couple of build chains. (https://www.alphabot.com/security/blog/2020/java/Your-Java-builds-might-break-starting-January-13th.html)
HTTP is gonna cause more issues/bugs in the future ...
For security reasons HTTPS should be mandatory and default, IMO. Trusted SSL certificates are free nowadays (self signed still works for services) ...
Keep up the great work,
Stefan
Quoted text · Sreenivasulu Yadavalli · 96 lines
-----Ursprüngliche Nachricht----- Von: Stefan Koch [mailto:koch at buit-solutions.com] Gesendet: Montag, 23. Dezember 2019 19:16 An: basex-talk at mailman.uni-konstanz.de Betreff: Re: [basex-talk] HTTPServer + gzip compression Hi Christian, Checked the GZIP Handler options. https://www.eclipse.org/jetty/documentation/current/gzip-filter.html It should support: includedMethods (GET, POST, PUT) - PATCH is not used by BaseX REST API - not needed. However these are response settings. Jetty does support request compression - if BaseX works for XML-Webservices, you wanna POST/PUT large XML docs that are compressed, I guess. As far as I could figure out: "inflateBufferSize" should be set to 1 (0 is default) (it's in the official jetty-gzip.xml not in the link from above). No default option in jetty, shouldn't be default in BaseX REST client, I guess. This should enable request compression. HTTP is dead, long live HTTPS! For productive use this is needed (otherwise you have to run NGINX as proxy or do similar workarounds). Currently only HTTP is supported. Downloaded jetty - in /etc there are the config xmls - relevant config options are: jetty-https,jetty-ssl, jetty-ssl-context (you wanna have a keystore/certificate for SSL). productive use: jetty-threadpool, jetty-threadlimit and jetty-connectionlimit might come in handy for the admins - but have no clue about that. Not really a priority, though. There is a bunch of other stuff, but IMO not really important. Best wishes and nice holydays, Stefan -----Ursprüngliche Nachricht----- Von: Christian Grün [mailto:christian.gruen at gmail.com] Gesendet: Dienstag, 17. Dezember 2019 07:59 An: Stefan Koch <koch at buit-solutions.com> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Betreff: Re: [basex-talk] HTTPServer + gzip compression Hi Stefan, We could consider an extension of the current solution. > So only for GET - not POST. > Our current Webservices rely heavily on POST (+Response) though :( Have you checked the remaining default options? What other custom settings may you need to rely on? Cheers, Christian > -----Ursprüngliche Nachricht----- > Von: Stefan Koch [mailto:koch at buit-solutions.com] > Gesendet: Mittwoch, 20. November 2019 14:45 > An: basex-talk at mailman.uni-konstanz.de > Betreff: Re: [basex-talk] HTTPServer + gzip compression > > > Hi, > > great to hear! So it's coming in 9.3 > > We have been running base.war via tomcat with gzip instead - but it wasn't optimal for our solution. > > > Thx for fixing > > Stefan > > > > -----Ursprüngliche Nachricht----- > Von: Christian Grün [mailto:christian.gruen at gmail.com] > Gesendet: Mittwoch, 20. November 2019 13:26 > An: Omar Siam <Omar.Siam at oeaw.ac.at> > Cc: BaseX <basex-talk at mailman.uni-konstanz.de> > Betreff: Re: [basex-talk] HTTPServer + gzip compression > > > Thanks for the confirmation! > > > > Omar Siam <Omar.Siam at oeaw.ac.at> schrieb am Mi., 20. Nov. 2019, 12:41: >> >> Yes, it works! 1000 entries of XML in a JSON wrapper from my data are >> now transfered as 193 KB instead of 1.6 MB. Not bad. Unfortunately on >> localhost that doesn't make much of a difference in overall timing as >> measured in Chrome's development tools. For real internet connections I >> think this is quite an improvement. >> >> Best regards >> >> Omar Siam >> >> Am 19.11.2019 um 01:28 schrieb Christian Grün: >> > The snapshot has been updated. >> > Is it working now? >> > >> > >> > >> > On Mon, Nov 18, 2019 at 6:41 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: >> >> I just wanted to try gzip http compression and downloaded the current build BaseX93-20191116. The option is not there anymore. What happened? >> >> >> >> Best regards >> >> >> >> Omar Siam
CG
Christian Grün22 Jan 2020 18:52 #
Hi Stefan,
I had reopened the corresponding Github issue to remember your request [1]. I’ve just added a link to the mailing list thread.
Best,
Christian
PS: Would anyone else be interested in assigning custom
GZIP options?
[1] https://github.com/BaseXdb/basex/issues/1748Quoted text · Stefan Koch · 145 lines
On Wed, Jan 22, 2020 at 6:33 PM Stefan Koch <koch at buit-solutions.com> wrote: > > Hi, > > Sorry for self-bumping - but topic might have been overlooked because of holydays. > > While gzip may be my personal wishlist - It would be could if others could chime in and say what they think about it. > > HTTPS however is pretty important - independent of my personal wishlist. If there is an embedded jetty already - it should support HTTPS without using proxy workarounds, IMO. > See recent issue with a lot of repostitories stopping HTTP support, breaking a couple of build chains. (https://www.alphabot.com/security/blog/2020/java/Your-Java-builds-might-break-starting-January-13th.html) > > HTTP is gonna cause more issues/bugs in the future ... > For security reasons HTTPS should be mandatory and default, IMO. Trusted SSL certificates are free nowadays (self signed still works for services) ... > > Keep up the great work, > > Stefan > > > > -----Ursprüngliche Nachricht----- > Von: Stefan Koch [mailto:koch at buit-solutions.com] > Gesendet: Montag, 23. Dezember 2019 19:16 > An: basex-talk at mailman.uni-konstanz.de > Betreff: Re: [basex-talk] HTTPServer + gzip compression > > > Hi Christian, > > Checked the GZIP Handler options. > > https://www.eclipse.org/jetty/documentation/current/gzip-filter.html > > It should support: > > includedMethods > (GET, POST, PUT) - PATCH is not used by BaseX REST API - not needed. > > However these are response settings. > > Jetty does support request compression - if BaseX works for XML-Webservices, you wanna POST/PUT large XML docs that are compressed, I guess. > > As far as I could figure out: "inflateBufferSize" should be set to 1 (0 is default) (it's in the official jetty-gzip.xml not in the link from above). No default option in jetty, shouldn't be default in BaseX REST client, I guess. > This should enable request compression. > > HTTP is dead, long live HTTPS! > > For productive use this is needed (otherwise you have to run NGINX as proxy or do similar workarounds). > Currently only HTTP is supported. > > Downloaded jetty - in /etc there are the config xmls - relevant config options are: > > jetty-https,jetty-ssl, jetty-ssl-context (you wanna have a keystore/certificate for SSL). > > productive use: > > jetty-threadpool, jetty-threadlimit and jetty-connectionlimit might come in handy for the admins - but have no clue about that. > Not really a priority, though. > > There is a bunch of other stuff, but IMO not really important. > > Best wishes and nice holydays, > > Stefan > > > -----Ursprüngliche Nachricht----- > Von: Christian Grün [mailto:christian.gruen at gmail.com] > Gesendet: Dienstag, 17. Dezember 2019 07:59 > An: Stefan Koch <koch at buit-solutions.com> > Cc: BaseX <basex-talk at mailman.uni-konstanz.de> > Betreff: Re: [basex-talk] HTTPServer + gzip compression > > > Hi Stefan, > > We could consider an extension of the current solution. > > > So only for GET - not POST. > > Our current Webservices rely heavily on POST (+Response) though :( > > Have you checked the remaining default options? What other custom > settings may you need to rely on? > > Cheers, > Christian > > > > > -----Ursprüngliche Nachricht----- > > Von: Stefan Koch [mailto:koch at buit-solutions.com] > > Gesendet: Mittwoch, 20. November 2019 14:45 > > An: basex-talk at mailman.uni-konstanz.de > > Betreff: Re: [basex-talk] HTTPServer + gzip compression > > > > > > Hi, > > > > great to hear! So it's coming in 9.3 > > > > We have been running base.war via tomcat with gzip instead - but it wasn't optimal for our solution. > > > > > > Thx for fixing > > > > Stefan > > > > > > > > -----Ursprüngliche Nachricht----- > > Von: Christian Grün [mailto:christian.gruen at gmail.com] > > Gesendet: Mittwoch, 20. November 2019 13:26 > > An: Omar Siam <Omar.Siam at oeaw.ac.at> > > Cc: BaseX <basex-talk at mailman.uni-konstanz.de> > > Betreff: Re: [basex-talk] HTTPServer + gzip compression > > > > > > Thanks for the confirmation! > > > > > > > > Omar Siam <Omar.Siam at oeaw.ac.at> schrieb am Mi., 20. Nov. 2019, 12:41: > >> > >> Yes, it works! 1000 entries of XML in a JSON wrapper from my data are > >> now transfered as 193 KB instead of 1.6 MB. Not bad. Unfortunately on > >> localhost that doesn't make much of a difference in overall timing as > >> measured in Chrome's development tools. For real internet connections I > >> think this is quite an improvement. > >> > >> Best regards > >> > >> Omar Siam > >> > >> Am 19.11.2019 um 01:28 schrieb Christian Grün: > >> > The snapshot has been updated. > >> > Is it working now? > >> > > >> > > >> > > >> > On Mon, Nov 18, 2019 at 6:41 PM Omar Siam <Omar.Siam at oeaw.ac.at> wrote: > >> >> I just wanted to try gzip http compression and downloaded the current build BaseX93-20191116. The option is not there anymore. What happened? > >> >> > >> >> Best regards > >> >> > >> >> Omar Siam
Written for BaseX 9.2, 9.3. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery