Consistent NPE loading content on one server, other server works

From: Eliot Kimber <eliot.kimber at servicenow.com> Date: Tuesday, January 23, 2024 at 11:51 AM To: basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Consistent NPE loading content on one server, other server works I have two servers running the same code, both on 10.7. I have a REST API handler that takes data as input and stores it in a database, applying some preprocessing to the data first. The data successfully loads on my dev server and consistently fails on the production server, with this Java failure: [INFO] Posting "/Users/eliot.kimber/git-basex/product-content-analytics/analytics/adobe/adobe-analytics-Utah_Pages_Viewed_Aug2023-clean.csv" [INFO] as filename "adobe-analytics-Utah_Pages_Viewed_Aug2023-clean.csv" [INFO] to Mirabel server "http://mirabel.corp.service-now.com:9984"... Unexpected error: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk at mailman.uni-konstanz.de Version: BaseX 10.7 Java: Oracle Corporation, 17.0.8 OS: Linux, amd64 Stack Trace: java.lang.NullPointerException: Cannot invoke "org.basex.index.IdPreMap.write(org.basex.io.IOFile)" because "this.idmap" is null at org.basex.data.DiskData.write(DiskData.java:151) at org.basex.data.DiskData.close(DiskData.java:160) at org.basex.core.Datas.unpin(Datas.java:52) at org.basex.core.cmd.Close.close(Close.java:45) at org.basex.core.cmd.OptimizeAll.optimizeAll(OptimizeAll.java:124) at org.basex.query.up.primitives.db.DBOptimize.apply(DBOptimize.java:119) at org.basex.query.up.DataUpdates.applyDbUpdates(DataUpdates.java:213) at org.basex.query.up.DataUpdates.apply(DataUpdates.java:172) at org.basex.query.up.ContextModifier.apply(ContextModifier.java:120) at org.basex.query.up.Updates.apply(Updates.java:179) at org.basex.query.QueryContext.update(QueryContext.java:663) at org.basex.query.QueryContext.lambda$iter$4(QueryContext.java:357) at org.basex.query.QueryContext.run(QueryContext.java:766) at org.basex.query.QueryContext.iter(QueryContext.java:357) at org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:78) at org.basex.http.web.WebResponse.create(WebResponse.java:58) at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:72) at org.basex.http.BaseXServlet.service(BaseXServlet.java:69) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:319) at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:273) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Thread.java:833)[INFO] Data loaded. The two servers are essentially identical Linux servers, although the production server has more resources. Any idea what might cause this failure or what I can do to diagnose it? There must be some non-obvious difference in these two servers but I don’t know what to look for. The code doing the load to a database is: let $reportPath as xs:string := ``[/`{$analyticsmgmt:csvPathRoot}`/`{$source}`/`{$reportType}`/`{$reportName}`.xml]`` let $msg := prof:dump(``[[INFO] Storing analytics report "`{$reportPath}`"]``) let $csv := $csv transform with { insert node attribute {'timestamp'} {$timeStampStr} into ./* } let $csv := $csv transform with { analyticsmgmt:dispatchAnalysticsStorePreProcessing($reportName, ., $debug) } return try { (db:put($analyticsDb, $csv, $reportPath), db:optimize($analyticsDb, true(), map{ 'attrindex' : true(), 'tokenindex' : true(), 'updindex' : true() }) ) } catch * { util:logToConsole( 'analyticsmgmt:storeAnalyticsCsvReport', ``[`{$err:code}`: `{$err:description}`]``, 'error' ) } Thanks, Eliot Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

On Wed, Jan 24, 2024 at 12:22 AM Eliot Kimber <eliot.kimber at servicenow.com> wrote: > I found the proximate cause: Not having UPDINDEX set to true() on the > failing server. > > When I set UPDINDEX to true on the target database, then the load > succeeded. > > Tracking through the Java code it looks like a failure to initialize the > idmap member of the DiskData object. This code looks like it could be the > issue: > > // open data and indexes > init(); > if(meta.updindex) { > idmap = new IdPreMap(meta.dbFile(DATAIDP)); > if(meta.textindex) textIndex = new UpdatableDiskValues(this, > IndexType.TEXT); > if(meta.attrindex) attrIndex = new UpdatableDiskValues(this, > IndexType.ATTRIBUTE); > if(meta.tokenindex) tokenIndex = new UpdatableDiskValues(this, > IndexType.TOKEN); > } else { > if(meta.textindex) textIndex = new DiskValues(this, IndexType.TEXT); > if(meta.attrindex) attrIndex = new DiskValues(this, > IndexType.ATTRIBUTE); > if(meta.tokenindex) tokenIndex = new DiskValues(this, IndexType.TOKEN); > } > if(meta.ftindex) ftIndex = new FTIndex(this); > > > > The idmap is only set of meta.updIndex is true. > > > > Cheers, > > > > E. > > _____________________________________________ > > *Eliot Kimber* > > Sr Staff Content Engineer > > O: 512 554 9368 > > M: 512 554 9368 > > servicenow.com <https://www.servicenow.com> > > LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter > <https://twitter.com/servicenow> | YouTube > <https://www.youtube.com/user/servicenowinc> | Facebook > <https://www.facebook.com/servicenow> > > > > *From: *Eliot Kimber <eliot.kimber at servicenow.com> > *Date: *Tuesday, January 23, 2024 at 11:51 AM > *To: *basex-talk at mailman.uni-konstanz.de < > basex-talk at mailman.uni-konstanz.de> > *Subject: *Consistent NPE loading content on one server, other server > works > > I have two servers running the same code, both on 10.7. I have a REST API > handler that takes data as input and stores it in a database, applying some > preprocessing to the data first. > > The data successfully loads on my dev server and consistently fails on the > production server, with this Java failure: > > [INFO] Posting > "/Users/eliot.kimber/git-basex/product-content-analytics/analytics/adobe/adobe-analytics-Utah_Pages_Viewed_Aug2023-clean.csv" > > [INFO] as filename "adobe-analytics-Utah_Pages_Viewed_Aug2023-clean.csv" > > [INFO] to Mirabel server "http://mirabel.corp.service-now.com:9984"... > > Unexpected error: Improper use? Potential bug? Your feedback is welcome: > > Contact: basex-talk at mailman.uni-konstanz.de > > Version: BaseX 10.7 > > Java: Oracle Corporation, 17.0.8 > > OS: Linux, amd64 > > Stack Trace: > > java.lang.NullPointerException: Cannot invoke > "org.basex.index.IdPreMap.write(org.basex.io.IOFile)" because "this.idmap" > is null > > at org.basex.data.DiskData.write(DiskData.java:151) > > at org.basex.data.DiskData.close(DiskData.java:160) > > at org.basex.core.Datas.unpin(Datas.java:52) > > at org.basex.core.cmd.Close.close(Close.java:45) > > at > org.basex.core.cmd.OptimizeAll.optimizeAll(OptimizeAll.java:124) > > at > org.basex.query.up.primitives.db.DBOptimize.apply(DBOptimize.java:119) > > at > org.basex.query.up.DataUpdates.applyDbUpdates(DataUpdates.java:213) > > at > org.basex.query.up.DataUpdates.apply(DataUpdates.java:172) > > at > org.basex.query.up.ContextModifier.apply(ContextModifier.java:120) > > at org.basex.query.up.Updates.apply(Updates.java:179) > > at > org.basex.query.QueryContext.update(QueryContext.java:663) > > at > org.basex.query.QueryContext.lambda$iter$4(QueryContext.java:357) > > at org.basex.query.QueryContext.run(QueryContext.java:766) > > at org.basex.query.QueryContext.iter(QueryContext.java:357) > > at > org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:78) > > at > org.basex.http.web.WebResponse.create(WebResponse.java:58) > > at > org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:72) > > at > org.basex.http.BaseXServlet.service(BaseXServlet.java:69) > > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > > at > org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459) > > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) > > at > org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) > > at > org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:319) > > at > org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:273) > > at > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201) > > at > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) > > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) > > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) > > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) > > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) > > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) > > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) > > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) > > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) > > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) > > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) > > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) > > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > > at org.eclipse.jetty.server.Server.handle(Server.java:516) > > at > org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) > > at > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) > > at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) > > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) > > at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) > > at > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) > > at > org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) > > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) > > at > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) > > at java.base/java.lang.Thread.run(Thread.java:833)[INFO] > Data loaded. > > > > The two servers are essentially identical Linux servers, although the > production server has more resources. > > Any idea what might cause this failure or what I can do to diagnose it? > > There must be some non-obvious difference in these two servers but I don’t > know what to look for. > > The code doing the load to a database is: > > let $reportPath as xs:string := > > > ``[/`{$analyticsmgmt:csvPathRoot}`/`{$source}`/`{$reportType}`/`{$reportName}`.xml]`` > > let $msg := prof:dump(``[[INFO] Storing analytics report > "`{$reportPath}`"]``) > > let $csv := $csv transform with { insert node attribute {'timestamp'} > {$timeStampStr} into ./* } > > let $csv := $csv transform with { > analyticsmgmt:dispatchAnalysticsStorePreProcessing($reportName, ., $debug) } > > return > > try { > > (db:put($analyticsDb, $csv, $reportPath), > > db:optimize($analyticsDb, > > true(), > > map{ > > 'attrindex' : true(), > > 'tokenindex' : true(), > > 'updindex' : true() > > }) > > ) > > } catch * { > > util:logToConsole( > > 'analyticsmgmt:storeAnalyticsCsvReport', > > ``[`{$err:code}`: `{$err:description}`]``, > > 'error' > > ) > > } > > > > Thanks, > > > > Eliot > > _____________________________________________ > > *Eliot Kimber* > > Sr Staff Content Engineer > > O: 512 554 9368 > > M: 512 554 9368 > > servicenow.com <https://www.servicenow.com> > > LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter > <https://twitter.com/servicenow> | YouTube > <https://www.youtube.com/user/servicenowinc> | Facebook > <https://www.facebook.com/servicenow> >

On Wed, Jan 24, 2024 at 4:14 PM Christian Grün <christian.gruen at gmail.com> wrote: > Hi Eliot, > > Thanks for your observations, and some first debugging. > > The idmap file is only created and required if UPDINDEX is enabled. Well, > let’s say it should: Maybe this file is not created in due time. > > I haven’t managed to reproduce the bug yet. If you are more successful, > please let us know. > > Cheers, > Christian > > > On Wed, Jan 24, 2024 at 12:22 AM Eliot Kimber <eliot.kimber at servicenow.com> > wrote: > >> I found the proximate cause: Not having UPDINDEX set to true() on the >> failing server. >> >> When I set UPDINDEX to true on the target database, then the load >> succeeded. >> >> Tracking through the Java code it looks like a failure to initialize the >> idmap member of the DiskData object. This code looks like it could be the >> issue: >> >> // open data and indexes >> init(); >> if(meta.updindex) { >> idmap = new IdPreMap(meta.dbFile(DATAIDP)); >> if(meta.textindex) textIndex = new UpdatableDiskValues(this, >> IndexType.TEXT); >> if(meta.attrindex) attrIndex = new UpdatableDiskValues(this, >> IndexType.ATTRIBUTE); >> if(meta.tokenindex) tokenIndex = new UpdatableDiskValues(this, >> IndexType.TOKEN); >> } else { >> if(meta.textindex) textIndex = new DiskValues(this, IndexType.TEXT); >> if(meta.attrindex) attrIndex = new DiskValues(this, >> IndexType.ATTRIBUTE); >> if(meta.tokenindex) tokenIndex = new DiskValues(this, IndexType.TOKEN); >> } >> if(meta.ftindex) ftIndex = new FTIndex(this); >> >> >> >> The idmap is only set of meta.updIndex is true. >> >> >> >> Cheers, >> >> >> >> E. >> >> _____________________________________________ >> >> *Eliot Kimber* >> >> Sr Staff Content Engineer >> >> O: 512 554 9368 >> >> M: 512 554 9368 >> >> servicenow.com <https://www.servicenow.com> >> >> LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter >> <https://twitter.com/servicenow> | YouTube >> <https://www.youtube.com/user/servicenowinc> | Facebook >> <https://www.facebook.com/servicenow> >> >> >> >> *From: *Eliot Kimber <eliot.kimber at servicenow.com> >> *Date: *Tuesday, January 23, 2024 at 11:51 AM >> *To: *basex-talk at mailman.uni-konstanz.de < >> basex-talk at mailman.uni-konstanz.de> >> *Subject: *Consistent NPE loading content on one server, other server >> works >> >> I have two servers running the same code, both on 10.7. I have a REST API >> handler that takes data as input and stores it in a database, applying some >> preprocessing to the data first. >> >> The data successfully loads on my dev server and consistently fails on >> the production server, with this Java failure: >> >> [INFO] Posting >> "/Users/eliot.kimber/git-basex/product-content-analytics/analytics/adobe/adobe-analytics-Utah_Pages_Viewed_Aug2023-clean.csv" >> >> [INFO] as filename "adobe-analytics-Utah_Pages_Viewed_Aug2023-clean.csv" >> >> [INFO] to Mirabel server "http://mirabel.corp.service-now.com:9984"... >> >> Unexpected error: Improper use? Potential bug? Your feedback is welcome: >> >> Contact: basex-talk at mailman.uni-konstanz.de >> >> Version: BaseX 10.7 >> >> Java: Oracle Corporation, 17.0.8 >> >> OS: Linux, amd64 >> >> Stack Trace: >> >> java.lang.NullPointerException: Cannot invoke >> "org.basex.index.IdPreMap.write(org.basex.io.IOFile)" because "this.idmap" >> is null >> >> at org.basex.data.DiskData.write(DiskData.java:151) >> >> at org.basex.data.DiskData.close(DiskData.java:160) >> >> at org.basex.core.Datas.unpin(Datas.java:52) >> >> at org.basex.core.cmd.Close.close(Close.java:45) >> >> at >> org.basex.core.cmd.OptimizeAll.optimizeAll(OptimizeAll.java:124) >> >> at >> org.basex.query.up.primitives.db.DBOptimize.apply(DBOptimize.java:119) >> >> at >> org.basex.query.up.DataUpdates.applyDbUpdates(DataUpdates.java:213) >> >> at >> org.basex.query.up.DataUpdates.apply(DataUpdates.java:172) >> >> at >> org.basex.query.up.ContextModifier.apply(ContextModifier.java:120) >> >> at org.basex.query.up.Updates.apply(Updates.java:179) >> >> at >> org.basex.query.QueryContext.update(QueryContext.java:663) >> >> at >> org.basex.query.QueryContext.lambda$iter$4(QueryContext.java:357) >> >> at org.basex.query.QueryContext.run(QueryContext.java:766) >> >> at >> org.basex.query.QueryContext.iter(QueryContext.java:357) >> >> at >> org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:78) >> >> at >> org.basex.http.web.WebResponse.create(WebResponse.java:58) >> >> at >> org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:72) >> >> at >> org.basex.http.BaseXServlet.service(BaseXServlet.java:69) >> >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) >> >> at >> org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459) >> >> at >> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) >> >> at >> org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) >> >> at >> org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:319) >> >> at >> org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:273) >> >> at >> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201) >> >> at >> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) >> >> at >> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) >> >> at >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) >> >> at >> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) >> >> at >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) >> >> at >> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) >> >> at >> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) >> >> at >> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) >> >> at >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) >> >> at >> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) >> >> at >> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) >> >> at >> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) >> >> at >> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) >> >> at >> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) >> >> at >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >> >> at >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) >> >> at org.eclipse.jetty.server.Server.handle(Server.java:516) >> >> at >> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) >> >> at >> org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) >> >> at >> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) >> >> at >> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) >> >> at >> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) >> >> at >> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) >> >> at >> org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) >> >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) >> >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) >> >> at java.base/java.lang.Thread.run(Thread.java:833)[INFO] >> Data loaded. >> >> >> >> The two servers are essentially identical Linux servers, although the >> production server has more resources. >> >> Any idea what might cause this failure or what I can do to diagnose it? >> >> There must be some non-obvious difference in these two servers but I >> don’t know what to look for. >> >> The code doing the load to a database is: >> >> let $reportPath as xs:string := >> >> >> ``[/`{$analyticsmgmt:csvPathRoot}`/`{$source}`/`{$reportType}`/`{$reportName}`.xml]`` >> >> let $msg := prof:dump(``[[INFO] Storing analytics report >> "`{$reportPath}`"]``) >> >> let $csv := $csv transform with { insert node attribute {'timestamp'} >> {$timeStampStr} into ./* } >> >> let $csv := $csv transform with { >> analyticsmgmt:dispatchAnalysticsStorePreProcessing($reportName, ., $debug) } >> >> return >> >> try { >> >> (db:put($analyticsDb, $csv, $reportPath), >> >> db:optimize($analyticsDb, >> >> true(), >> >> map{ >> >> 'attrindex' : true(), >> >> 'tokenindex' : true(), >> >> 'updindex' : true() >> >> }) >> >> ) >> >> } catch * { >> >> util:logToConsole( >> >> 'analyticsmgmt:storeAnalyticsCsvReport', >> >> ``[`{$err:code}`: `{$err:description}`]``, >> >> 'error' >> >> ) >> >> } >> >> >> >> Thanks, >> >> >> >> Eliot >> >> _____________________________________________ >> >> *Eliot Kimber* >> >> Sr Staff Content Engineer >> >> O: 512 554 9368 >> >> M: 512 554 9368 >> >> servicenow.com <https://www.servicenow.com> >> >> LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter >> <https://twitter.com/servicenow> | YouTube >> <https://www.youtube.com/user/servicenowinc> | Facebook >> <https://www.facebook.com/servicenow> >> >

From: Christian Grün <christian.gruen at gmail.com> Date: Wednesday, January 24, 2024 at 9:27 AM To: Eliot Kimber <eliot.kimber at servicenow.com> Cc: basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Consistent NPE loading content on one server, other server works [External Email] …spending much time on a problem rarely helps. It’s often better to write an e-mail, and discover the solution right after that. A new stable snapshot with the bug fix is online [1,2]. Cheers, Christian [1] https://github.com/BaseXdb/basex/issues/2272 [2] https://files.basex.org/releases/latest/<https://files.basex.org/releases/latest/> On Wed, Jan 24, 2024 at 4:14 PM Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> wrote: Hi Eliot, Thanks for your observations, and some first debugging. The idmap file is only created and required if UPDINDEX is enabled. Well, let’s say it should: Maybe this file is not created in due time. I haven’t managed to reproduce the bug yet. If you are more successful, please let us know. Cheers, Christian On Wed, Jan 24, 2024 at 12:22 AM Eliot Kimber <eliot.kimber at servicenow.com<mailto:eliot.kimber at servicenow.com>> wrote: I found the proximate cause: Not having UPDINDEX set to true() on the failing server. When I set UPDINDEX to true on the target database, then the load succeeded. Tracking through the Java code it looks like a failure to initialize the idmap member of the DiskData object. This code looks like it could be the issue: // open data and indexes init(); if(meta.updindex) { idmap = new IdPreMap(meta.dbFile(DATAIDP)); if(meta.textindex) textIndex = new UpdatableDiskValues(this, IndexType.TEXT); if(meta.attrindex) attrIndex = new UpdatableDiskValues(this, IndexType.ATTRIBUTE); if(meta.tokenindex) tokenIndex = new UpdatableDiskValues(this, IndexType.TOKEN); } else { if(meta.textindex) textIndex = new DiskValues(this, IndexType.TEXT); if(meta.attrindex) attrIndex = new DiskValues(this, IndexType.ATTRIBUTE); if(meta.tokenindex) tokenIndex = new DiskValues(this, IndexType.TOKEN); } if(meta.ftindex) ftIndex = new FTIndex(this); The idmap is only set of meta.updIndex is true. Cheers, E. Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Eliot Kimber <eliot.kimber at servicenow.com<mailto:eliot.kimber at servicenow.com>> Date: Tuesday, January 23, 2024 at 11:51 AM To: basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de> <basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de>> Subject: Consistent NPE loading content on one server, other server works I have two servers running the same code, both on 10.7. I have a REST API handler that takes data as input and stores it in a database, applying some preprocessing to the data first. The data successfully loads on my dev server and consistently fails on the production server, with this Java failure: [INFO] Posting "/Users/eliot.kimber/git-basex/product-content-analytics/analytics/adobe/adobe-analytics-Utah_Pages_Viewed_Aug2023-clean.csv" [INFO] as filename "adobe-analytics-Utah_Pages_Viewed_Aug2023-clean.csv" [INFO] to Mirabel server "http://mirabel.corp.service-now.com:9984"... Unexpected error: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de> Version: BaseX 10.7 Java: Oracle Corporation, 17.0.8 OS: Linux, amd64 Stack Trace: java.lang.NullPointerException: Cannot invoke "org.basex.index.IdPreMap.write(org.basex.io.IOFile)" because "this.idmap" is null at org.basex.data.DiskData.write(DiskData.java:151) at org.basex.data.DiskData.close(DiskData.java:160) at org.basex.core.Datas.unpin(Datas.java:52) at org.basex.core.cmd.Close.close(Close.java:45) at org.basex.core.cmd.OptimizeAll.optimizeAll(OptimizeAll.java:124) at org.basex.query.up.primitives.db.DBOptimize.apply(DBOptimize.java:119) at org.basex.query.up.DataUpdates.applyDbUpdates(DataUpdates.java:213) at org.basex.query.up.DataUpdates.apply(DataUpdates.java:172) at org.basex.query.up.ContextModifier.apply(ContextModifier.java:120) at org.basex.query.up.Updates.apply(Updates.java:179) at org.basex.query.QueryContext.update(QueryContext.java:663) at org.basex.query.QueryContext.lambda$iter$4(QueryContext.java:357) at org.basex.query.QueryContext.run(QueryContext.java:766) at org.basex.query.QueryContext.iter(QueryContext.java:357) at org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:78) at org.basex.http.web.WebResponse.create(WebResponse.java:58) at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:72) at org.basex.http.BaseXServlet.service(BaseXServlet.java:69) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:319) at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:273) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Thread.java:833)[INFO] Data loaded. The two servers are essentially identical Linux servers, although the production server has more resources. Any idea what might cause this failure or what I can do to diagnose it? There must be some non-obvious difference in these two servers but I don’t know what to look for. The code doing the load to a database is: let $reportPath as xs:string := ``[/`{$analyticsmgmt:csvPathRoot}`/`{$source}`/`{$reportType}`/`{$reportName}`.xml]`` let $msg := prof:dump(``[[INFO] Storing analytics report "`{$reportPath}`"]``) let $csv := $csv transform with { insert node attribute {'timestamp'} {$timeStampStr} into ./* } let $csv := $csv transform with { analyticsmgmt:dispatchAnalysticsStorePreProcessing($reportName, ., $debug) } return try { (db:put($analyticsDb, $csv, $reportPath), db:optimize($analyticsDb, true(), map{ 'attrindex' : true(), 'tokenindex' : true(), 'updindex' : true() }) ) } catch * { util:logToConsole( 'analyticsmgmt:storeAnalyticsCsvReport', ``[`{$err:code}`: `{$err:description}`]``, 'error' ) } Thanks, Eliot Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>