Reducing logs
Thursday, 2 February 2023 · 4 messages · last reply 3 Feb 2023
ML
Marco Lettere2 Feb 2023 13:31 #
Dear all,
when using the Basex Xquery server (9.7.3) we see that logs such as [1] are produced for every query execution.
Since we have a scenario which involves polling and thus we have a lot of query executions the log files increase to hundreds om MBs during a day.
This makes them unmanageable and even cause dba UI to crash with OOM exceptions when opening the logs page.
Is there a way to disable this logging (without affecting other logs such as HTTP)?
Is it really necessary to log server side query executions at this grain by default? Maybe it could be made optional?
Thanks for any support.
Regards,
Marco.
[1]
12:22:27.849 10.0.4.15:50424 admin OK 0.02 CLOSE[0] 12:22:27.848 10.0.4.15:50434 admin OK 1.40 FULL[0] 12:22:27.848 10.0.4.15:50424 admin OK 0.83 FULL[0] 12:22:27.847 10.0.4.15:50420 admin OK 0.05 CLOSE[0] 12:22:27.847 10.0.4.15:50412 admin OK 0.04 CLOSE[0] 12:22:27.847 10.0.4.15:50434 admin OK 0.05 BIND[0] db=infrastructures as xs:string
12:22:27.846 10.0.4.15:50410 admin OK 0.22 CLOSE[0] 12:22:27.846 10.0.4.15:50434 admin OK 0.06 BIND[0] id=ontheroad-lxd as xs:string
12:22:27.846 10.0.4.15:50424 admin OK 0.05 BIND[0] db=infrastructures as xs:string
12:22:27.846 10.0.4.15:50420 admin OK 0.62 FULL[0] 12:22:27.846 10.0.4.15:50412 admin OK 0.90 FULL[0] 12:22:27.846 10.0.4.15:50424 admin OK 0.06 BIND[0] id=ontheroad-lxd as xs:string
12:22:27.845 10.0.4.15:50380 admin OK 0.02 CLOSE[0] 12:22:27.845 10.0.4.15:50366 admin OK 0.01 CLOSE[0] 12:22:27.845 10.0.4.15:50402 admin OK 0.02 CLOSE[0] 12:22:27.845 10.0.4.15:50420 admin OK 0.04 BIND[0] db=infrastructures as xs:string
12:22:27.845 10.0.4.15:50386 admin OK 0.02 CLOSE[0] 12:22:27.845 10.0.4.15:50410 admin OK 0.88 FULL[0] 12:22:27.845 10.0.4.15:50420 admin OK 0.06 BIND[0] id=ontheroad-lxd as xs:string
12:22:27.845 10.0.4.15:50412 admin OK 0.04 BIND[0] db=infrastructures as xs:string
12:22:27.845 10.0.4.15:50434 admin OK 0.04 QUERY[0] declare variable $id external; declare variable $db external;
db:open($db)[json/id = $id]
CG
Christian Grün2 Feb 2023 14:41 #
Hi Marco,
As this requirement has been reported back to us repeatedly, it’s time to create an issue for it [1].
I think we’ll replace the general LOG option with a more flexible variant. We’ll still need to define which levels to offers; any feedback is welcome. The new option will probably be made available with BaseX 11 (to be expected around spring).
Ciao,
Christian
[1] https://github.com/BaseXdb/basex/issues/2168
Quoted text · Marco Lettere · 57 lines
On Thu, Feb 2, 2023 at 1:37 PM Marco Lettere <m.lettere at gmail.com> wrote: > > Dear all, > > when using the Basex Xquery server (9.7.3) we see that logs such as [1] > are produced for every query execution. > > Since we have a scenario which involves polling and thus we have a lot > of query executions the log files increase to hundreds om MBs during a day. > > This makes them unmanageable and even cause dba UI to crash with OOM > exceptions when opening the logs page. > > Is there a way to disable this logging (without affecting other logs > such as HTTP)? > > Is it really necessary to log server side query executions at this grain > by default? Maybe it could be made optional? > > Thanks for any support. > > Regards, > > Marco. > > [1] > 12:22:27.849 10.0.4.15:50424 admin OK 0.02 CLOSE[0] > 12:22:27.848 10.0.4.15:50434 admin OK 1.40 FULL[0] > 12:22:27.848 10.0.4.15:50424 admin OK 0.83 FULL[0] > 12:22:27.847 10.0.4.15:50420 admin OK 0.05 CLOSE[0] > 12:22:27.847 10.0.4.15:50412 admin OK 0.04 CLOSE[0] > 12:22:27.847 10.0.4.15:50434 admin OK 0.05 BIND[0] > db=infrastructures as xs:string > 12:22:27.846 10.0.4.15:50410 admin OK 0.22 CLOSE[0] > 12:22:27.846 10.0.4.15:50434 admin OK 0.06 BIND[0] > id=ontheroad-lxd as xs:string > 12:22:27.846 10.0.4.15:50424 admin OK 0.05 BIND[0] > db=infrastructures as xs:string > 12:22:27.846 10.0.4.15:50420 admin OK 0.62 FULL[0] > 12:22:27.846 10.0.4.15:50412 admin OK 0.90 FULL[0] > 12:22:27.846 10.0.4.15:50424 admin OK 0.06 BIND[0] > id=ontheroad-lxd as xs:string > 12:22:27.845 10.0.4.15:50380 admin OK 0.02 CLOSE[0] > 12:22:27.845 10.0.4.15:50366 admin OK 0.01 CLOSE[0] > 12:22:27.845 10.0.4.15:50402 admin OK 0.02 CLOSE[0] > 12:22:27.845 10.0.4.15:50420 admin OK 0.04 BIND[0] > db=infrastructures as xs:string > 12:22:27.845 10.0.4.15:50386 admin OK 0.02 CLOSE[0] > 12:22:27.845 10.0.4.15:50410 admin OK 0.88 FULL[0] > 12:22:27.845 10.0.4.15:50420 admin OK 0.06 BIND[0] > id=ontheroad-lxd as xs:string > 12:22:27.845 10.0.4.15:50412 admin OK 0.04 BIND[0] > db=infrastructures as xs:string > 12:22:27.845 10.0.4.15:50434 admin OK 0.04 QUERY[0] declare > variable $id external; declare variable $db external; > db:open($db)[json/id = $id] >
EK
Eliot Kimber2 Feb 2023 16:36 #
For me, being able to log HTTP requests separately (different log file) from XQuery-generated would be very helpful. For our Mirabel server we’ll want to be able to mine usage statistics from the HTTP request logs but for general administration and monitoring, it’s the messages generated by the XQuery code serving the pages that are useful.
Cheers,
E.
_____________________________________________
Eliot Kimber
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com<https://www.servicenow.com>
LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
Quoted text · BaseX-Talk · 73 lines
From: BaseX-Talk <basex-talk-bounces at mailman.uni-konstanz.de> on behalf of Christian Grün <christian.gruen at gmail.com> Date: Thursday, February 2, 2023 at 7:42 AM To: Marco Lettere <m.lettere at gmail.com> Cc: basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Reducing logs [External Email] Hi Marco, As this requirement has been reported back to us repeatedly, it’s time to create an issue for it [1]. I think we’ll replace the general LOG option with a more flexible variant. We’ll still need to define which levels to offers; any feedback is welcome. The new option will probably be made available with BaseX 11 (to be expected around spring). Ciao, Christian [1] https://github.com/BaseXdb/basex/issues/2168<https://github.com/BaseXdb/basex/issues/2168> On Thu, Feb 2, 2023 at 1:37 PM Marco Lettere <m.lettere at gmail.com> wrote: > > Dear all, > > when using the Basex Xquery server (9.7.3) we see that logs such as [1] > are produced for every query execution. > > Since we have a scenario which involves polling and thus we have a lot > of query executions the log files increase to hundreds om MBs during a day. > > This makes them unmanageable and even cause dba UI to crash with OOM > exceptions when opening the logs page. > > Is there a way to disable this logging (without affecting other logs > such as HTTP)? > > Is it really necessary to log server side query executions at this grain > by default? Maybe it could be made optional? > > Thanks for any support. > > Regards, > > Marco. > > [1] > 12:22:27.849 10.0.4.15:50424 admin OK 0.02 CLOSE[0] > 12:22:27.848 10.0.4.15:50434 admin OK 1.40 FULL[0] > 12:22:27.848 10.0.4.15:50424 admin OK 0.83 FULL[0] > 12:22:27.847 10.0.4.15:50420 admin OK 0.05 CLOSE[0] > 12:22:27.847 10.0.4.15:50412 admin OK 0.04 CLOSE[0] > 12:22:27.847 10.0.4.15:50434 admin OK 0.05 BIND[0] > db=infrastructures as xs:string > 12:22:27.846 10.0.4.15:50410 admin OK 0.22 CLOSE[0] > 12:22:27.846 10.0.4.15:50434 admin OK 0.06 BIND[0] > id=ontheroad-lxd as xs:string > 12:22:27.846 10.0.4.15:50424 admin OK 0.05 BIND[0] > db=infrastructures as xs:string > 12:22:27.846 10.0.4.15:50420 admin OK 0.62 FULL[0] > 12:22:27.846 10.0.4.15:50412 admin OK 0.90 FULL[0] > 12:22:27.846 10.0.4.15:50424 admin OK 0.06 BIND[0] > id=ontheroad-lxd as xs:string > 12:22:27.845 10.0.4.15:50380 admin OK 0.02 CLOSE[0] > 12:22:27.845 10.0.4.15:50366 admin OK 0.01 CLOSE[0] > 12:22:27.845 10.0.4.15:50402 admin OK 0.02 CLOSE[0] > 12:22:27.845 10.0.4.15:50420 admin OK 0.04 BIND[0] > db=infrastructures as xs:string > 12:22:27.845 10.0.4.15:50386 admin OK 0.02 CLOSE[0] > 12:22:27.845 10.0.4.15:50410 admin OK 0.88 FULL[0] > 12:22:27.845 10.0.4.15:50420 admin OK 0.06 BIND[0] > id=ontheroad-lxd as xs:string > 12:22:27.845 10.0.4.15:50412 admin OK 0.04 BIND[0] > db=infrastructures as xs:string > 12:22:27.845 10.0.4.15:50434 admin OK 0.04 QUERY[0] declare > variable $id external; declare variable $db external; > db:open($db)[json/id = $id] >
ML
Marco Lettere3 Feb 2023 12:24 #
Eliot' s point is ok for us too. We also mine info about HTTP traffic out of the log files.
In contradiction to some comments on [1] there is a good motivation for writing to stdout/err and it's related to the execution of servers inside docker containers.
"docker logs [my_container_with_basex] -f" would be of great value. Particularly when running a service on tools such as Portainer that do not allow users to "exec into" a container. This is currently not exploitable because the only thing you get is the logs of the server which is less useful when developing a RESTXQ service for instance
"docker logs" intercepts stdout/stderr. There are some workarounds to map /dev/stdout to a file. But again, BaseX switches the file every day so that would also not be feasible.
So, trying to resume:
- put code such as [2] to a trace level reducing impact on standard log to a on or two liner as for REST calls
- have a standardized logging system (log4j? slf4j?) that allows to differentiate logs (HTTP, Server) and levels redirecting to file or console and configuring rotations different from changing the file.
- make the dba GUI not trying to open by default the last log file because if it is large this causes an OOM that a user cannot avoid
- Bonus: logging options changeable through the dba UI would be incredibly useful but I can understand possible difficulties here.
O2C :-)
Thanks a lot Christian for paying usual high level attention to your users'!
Regards,
M.
[1] https://github.com/BaseXdb/basex/issues/940
[2]
Quoted text · Eliot Kimber · 122 lines
> 12:22:27.849 10.0.4.15:50424 admin OK 0.02 CLOSE[0] > 12:22:27.848 10.0.4.15:50434 admin OK 1.40 FULL[0] > 12:22:27.848 10.0.4.15:50424 admin OK 0.83 FULL[0] > 12:22:27.847 10.0.4.15:50420 admin OK 0.05 CLOSE[0] > 12:22:27.847 10.0.4.15:50412 admin OK 0.04 CLOSE[0] > 12:22:27.847 10.0.4.15:50434 admin OK 0.05 BIND[0] On 02/02/23 16:36, Eliot Kimber wrote: > > For me, being able to log HTTP requests separately (different log > file) from XQuery-generated would be very helpful. For our Mirabel > server we’ll want to be able to mine usage statistics from the HTTP > request logs but for general administration and monitoring, it’s the > messages generated by the XQuery code serving the pages that are useful. > > Cheers, > > E. > > _____________________________________________ > > *Eliot Kimber* > > Sr Staff Content Engineer > > O: 512 554 9368 > > M: 512 554 9368 > > servicenow.com <https://www.servicenow.com> > > LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter > <https://twitter.com/servicenow> | YouTube > <https://www.youtube.com/user/servicenowinc> | Facebook > <https://www.facebook.com/servicenow> > > *From: *BaseX-Talk <basex-talk-bounces at mailman.uni-konstanz.de> on > behalf of Christian Grün <christian.gruen at gmail.com> > *Date: *Thursday, February 2, 2023 at 7:42 AM > *To: *Marco Lettere <m.lettere at gmail.com> > *Cc: *basex-talk at mailman.uni-konstanz.de > <basex-talk at mailman.uni-konstanz.de> > *Subject: *Re: [basex-talk] Reducing logs > > [External Email] > > Hi Marco, > > As this requirement has been reported back to us repeatedly, it’s time > to create an issue for it [1]. > > I think we’ll replace the general LOG option with a more flexible > variant. We’ll still need to define which levels to offers; any > feedback is welcome. The new option will probably be made available > with BaseX 11 (to be expected around spring). > > Ciao, > Christian > > [1] https://github.com/BaseXdb/basex/issues/2168 > > > > > On Thu, Feb 2, 2023 at 1:37 PM Marco Lettere <m.lettere at gmail.com> wrote: > > > > Dear all, > > > > when using the Basex Xquery server (9.7.3) we see that logs such as [1] > > are produced for every query execution. > > > > Since we have a scenario which involves polling and thus we have a lot > > of query executions the log files increase to hundreds om MBs during > a day. > > > > This makes them unmanageable and even cause dba UI to crash with OOM > > exceptions when opening the logs page. > > > > Is there a way to disable this logging (without affecting other logs > > such as HTTP)? > > > > Is it really necessary to log server side query executions at this grain > > by default? Maybe it could be made optional? > > > > Thanks for any support. > > > > Regards, > > > > Marco. > > > > [1] > > 12:22:27.849 10.0.4.15:50424 admin OK 0.02 CLOSE[0] > > 12:22:27.848 10.0.4.15:50434 admin OK 1.40 FULL[0] > > 12:22:27.848 10.0.4.15:50424 admin OK 0.83 FULL[0] > > 12:22:27.847 10.0.4.15:50420 admin OK 0.05 CLOSE[0] > > 12:22:27.847 10.0.4.15:50412 admin OK 0.04 CLOSE[0] > > 12:22:27.847 10.0.4.15:50434 admin OK 0.05 BIND[0] > > db=infrastructures as xs:string > > 12:22:27.846 10.0.4.15:50410 admin OK 0.22 CLOSE[0] > > 12:22:27.846 10.0.4.15:50434 admin OK 0.06 BIND[0] > > id=ontheroad-lxd as xs:string > > 12:22:27.846 10.0.4.15:50424 admin OK 0.05 BIND[0] > > db=infrastructures as xs:string > > 12:22:27.846 10.0.4.15:50420 admin OK 0.62 FULL[0] > > 12:22:27.846 10.0.4.15:50412 admin OK 0.90 FULL[0] > > 12:22:27.846 10.0.4.15:50424 admin OK 0.06 BIND[0] > > id=ontheroad-lxd as xs:string > > 12:22:27.845 10.0.4.15:50380 admin OK 0.02 CLOSE[0] > > 12:22:27.845 10.0.4.15:50366 admin OK 0.01 CLOSE[0] > > 12:22:27.845 10.0.4.15:50402 admin OK 0.02 CLOSE[0] > > 12:22:27.845 10.0.4.15:50420 admin OK 0.04 BIND[0] > > db=infrastructures as xs:string > > 12:22:27.845 10.0.4.15:50386 admin OK 0.02 CLOSE[0] > > 12:22:27.845 10.0.4.15:50410 admin OK 0.88 FULL[0] > > 12:22:27.845 10.0.4.15:50420 admin OK 0.06 BIND[0] > > id=ontheroad-lxd as xs:string > > 12:22:27.845 10.0.4.15:50412 admin OK 0.04 BIND[0] > > db=infrastructures as xs:string > > 12:22:27.845 10.0.4.15:50434 admin OK 0.04 QUERY[0] declare > > variable $id external; declare variable $db external; > > db:open($db)[json/id = $id] > > >
Written for BaseX 10.4. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery