Log Analysis Tips?

Hans-Juergen Rennau <hrennau at yahoo.de> schrieb am Mi., 10. Aug. 2022, 20:26: > Hi Eliot, a very general remark (not really related to BaseX logs, but > logs in general): I have practiced the following approach already a number > of times, doing everything with BaseX, as it goes without saying, and it > has always been very satisfying: > > (1) Parse the log files into an XML representation, using regex etc. to > transform the text into highly structured information > (2) Mindset: a log event is a thing with properties; a property has a name > and a value; properties model: "event-type", "time", "msg", (a) > generic-properties (like IP-address, user, ...), (b) event-type-specific > properties (e.g. return-code) (b may be extracted from the msg) > (3) And then you can generate nice reports and also implement a powerful > Online-Log-Query API (command-line or GUI) > (4) The event model can be extended iteratively - defining more event > types, adding more event type specific properties > > Kind regards, > Hans > > > Am Mittwoch, 10. August 2022 um 19:52:01 MESZ hat Eliot Kimber < > eliot.kimber at servicenow.com> Folgendes geschrieben: > > > We have released version 1.0 of our Project Mirabel server to our internal > users. > > > > Now we need to know who’s using it, so time for log analysis. > > > > I haven’t given any thought yet to what log analysis might mean. So > wondering if there are any tips and tricks for analyzing the BaseX logs > when using the BaseX HTTP server. > > Note that we don’t yet have users implemented, so we’re just looking at > HTTP requests with different IP addresses. > > > > Thanks, > > > > E. > > > > > > _____________________________________________ > > *Eliot Kimber* > > Sr Staff Content Engineer > > O: 512 554 9368 > > M: 512 554 9368 > > servicenow.com <https://www.servicenow.com> > > LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter > <https://twitter.com/servicenow> | YouTube > <https://www.youtube.com/user/servicenowinc> | Facebook > <https://www.facebook.com/servicenow> >

From: Christian Grün <christian.gruen at gmail.com> Date: Wednesday, August 10, 2022 at 1:31 PM To: Hans-Juergen Rennau <hrennau at yahoo.de> Cc: Eliot Kimber <eliot.kimber at servicenow.com>, BaseX <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Log Analysis Tips? [External Email] Hi Eliot, The DBA relies on admin:logs($date, true()), which returns a postprocessed XML representation of the raw log files [1]. Hope this helps Christian [1] https://docs.basex.org/wiki/Admin_Module<https://urldefense.com/v3/__https:/docs.basex.org/wiki/Admin_Module__;!!N4vogdjhuJM!BBsWtys4F9BPPQp4lQdcRrk1r0OMumsIuBPVYRWHCKUBFb-oorinHEARPF3grBB1eYL9RHP6PgnI_37wOYUiufeRhx0xLKw$> Hans-Juergen Rennau <hrennau at yahoo.de<mailto:hrennau at yahoo.de>> schrieb am Mi., 10. Aug. 2022, 20:26: Hi Eliot, a very general remark (not really related to BaseX logs, but logs in general): I have practiced the following approach already a number of times, doing everything with BaseX, as it goes without saying, and it has always been very satisfying: (1) Parse the log files into an XML representation, using regex etc. to transform the text into highly structured information (2) Mindset: a log event is a thing with properties; a property has a name and a value; properties model: "event-type", "time", "msg", (a) generic-properties (like IP-address, user, ...), (b) event-type-specific properties (e.g. return-code) (b may be extracted from the msg) (3) And then you can generate nice reports and also implement a powerful Online-Log-Query API (command-line or GUI) (4) The event model can be extended iteratively - defining more event types, adding more event type specific properties Kind regards, Hans Am Mittwoch, 10. August 2022 um 19:52:01 MESZ hat Eliot Kimber <eliot.kimber at servicenow.com<mailto:eliot.kimber at servicenow.com>> Folgendes geschrieben: We have released version 1.0 of our Project Mirabel server to our internal users. Now we need to know who’s using it, so time for log analysis. I haven’t given any thought yet to what log analysis might mean. So wondering if there are any tips and tricks for analyzing the BaseX logs when using the BaseX HTTP server. Note that we don’t yet have users implemented, so we’re just looking at HTTP requests with different IP addresses. Thanks, E. Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://urldefense.com/v3/__https:/www.linkedin.com/company/servicenow__;!!N4vogdjhuJM!BBsWtys4F9BPPQp4lQdcRrk1r0OMumsIuBPVYRWHCKUBFb-oorinHEARPF3grBB1eYL9RHP6PgnI_37wOYUiufeR5Ky5A8s$> | Twitter<https://urldefense.com/v3/__https:/twitter.com/servicenow__;!!N4vogdjhuJM!BBsWtys4F9BPPQp4lQdcRrk1r0OMumsIuBPVYRWHCKUBFb-oorinHEARPF3grBB1eYL9RHP6PgnI_37wOYUiufeR5571jKM$> | YouTube<https://urldefense.com/v3/__https:/www.youtube.com/user/servicenowinc__;!!N4vogdjhuJM!BBsWtys4F9BPPQp4lQdcRrk1r0OMumsIuBPVYRWHCKUBFb-oorinHEARPF3grBB1eYL9RHP6PgnI_37wOYUiufeRpKKwUeg$> | Facebook<https://urldefense.com/v3/__https:/www.facebook.com/servicenow__;!!N4vogdjhuJM!BBsWtys4F9BPPQp4lQdcRrk1r0OMumsIuBPVYRWHCKUBFb-oorinHEARPF3grBB1eYL9RHP6PgnI_37wOYUiufeRI0wIPlc$>