Main Page » Mailing List » July 2010

error importing one million documents using console

Wednesday, 7 July 2010 · 5 messages · last reply 9 Jul 2010
PS
Patrick Schäfer7 Jul 2010 17:05 #
Hi, I am trying to import one million documents within a directory to basex using the console as described on the faq-page. The command I am using is: java -cp BaseX61.jar org.basex.BaseXClient -vc "create DB /home/tuscreen/cml tuscreen " Immediately after execution the server throws an exception:
Exception in thread "Thread-6" java.lang.NullPointerException
         at org.basex.build.DiskBuilder.close(DiskBuilder.java:102)
         at org.basex.build.DiskBuilder.abort(DiskBuilder.java:88)
         at org.basex.core.Progress.abort(Progress.java:98)
         at org.basex.core.Proc.run(Proc.java:146)
         at org.basex.core.Proc.exec(Proc.java:109)
         at org.basex.server.ServerProcess.run(ServerProcess.java:128)
When importing a small fraction of the data, everything works fine. Is there any way to import all documents? Thank you Patrick
CG
Christian Grün7 Jul 2010 17:46 #
Patrick, thanks for your report. Have you managed to reproduce the bug (i.e., has it occurred a second time?) As a shorthand, I'd suggest to try the latest, stable version of BaseX, which you find at: http://www.inf.uni-konstanz.de/dbis/basex/maven/org/basex/basex/6.1.9/ Looking forward to your feedback, Christian
Quoted text · Patrick Schäfer · 24 lines
On Wed, Jul 7, 2010 at 5:05 PM, Patrick Schäfer <patrick.schaefer at zib.de> wrote: > Hi, > > I am trying to import one million documents within a directory to basex > using the console as described on the faq-page. > > The command I am using is: > java -cp BaseX61.jar org.basex.BaseXClient -vc "create DB /home/tuscreen/cml > tuscreen " > > Immediately after execution the server throws an exception: > Exception in thread "Thread-6" java.lang.NullPointerException > at org.basex.build.DiskBuilder.close(DiskBuilder.java:102) > at org.basex.build.DiskBuilder.abort(DiskBuilder.java:88) > at org.basex.core.Progress.abort(Progress.java:98) > at org.basex.core.Proc.run(Proc.java:146) > at org.basex.core.Proc.exec(Proc.java:109) > at org.basex.server.ServerProcess.run(ServerProcess.java:128) > > When importing a small fraction of the data, everything works fine. Is there > any way to import all documents? > > Thank you > Patrick
PS
Patrick Schäfer9 Jul 2010 13:56 #
Hi Christian,
Quoted text · Christian Grün · 3 lines
> * Console: > open tuscreen > xquery //*:molecule[@*:id="11178"]
Wow, thank you. That gave a huge boost in performance:
   for $i in
//*:molecule[@*:id="298038"]/*:identifier[@*:convention="iupac:smile"]
     return
<smile>{$i/@value}</smile>
Takes 80ms now. But unfortunately it doesn't work for a query like:
   let $ids := (1 to 10)
   for $i in //*:molecule[@*:id =
$ids]/*:identifier[@*:convention="iupac:smile"]
     return
<smile>{$i/@value}</smile>
Query executed in 3688.12 ms. So it still takes >3500 ms The Query execution plan is quite different I noticed: First Query: Result: element { "smiles" } { for $i in IndexAccess("298038", ATV)/self::*:id/parent::*:molecule/*:identifier[@*:convention = "iupac:smile"] return element { "smile" } { $i/@value } }
Second Query: Result: for $i in IndexAccess("iupac:smile",
ATV)/self::*:convention/parent::*:identifier[parent::*:molecule[@*:id =
(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)]] return element { "smile" } { $i/@value }
So the second plan doesn't make use of the attribute index for locating the id-attributes.
Quoted text · Christian Grün · 3 lines
> Sorry, zu spät gesehen… Jederzeit ;) Je mehr Mails allerdings an die > Liste gehen, desto besser; dann bleiben auch die anderen auf dem > Laufenden.
Dann antworte ich besser an die Liste ;) Patrick
CG
Christian Grün9 Jul 2010 14:23 #
Quoted text · Patrick Schäfer · 9 lines
> The Query execution plan is quite different I noticed: > > First Query: Result: element { "smiles" } { for $i in IndexAccess("298038", > ATV)/self::*:id/parent::*:molecule/*:identifier[@*:convention = > "iupac:smile"] return element { "smile" } { $i/@value } } > > Second Query: Result: for $i in IndexAccess("iupac:smile", > ATV)/self::*:convention/parent::*:identifier[parent::*:molecule[@*:id = (1, > 2, 3, 4, 5, 6, 7, 8, 9, 10)]] return element { "smile" } { $i/@value }
Query optimizer tries to find the cheapest index evaluation. As the costs for queries on variables cannot be estimated in advance, the optimizer will choose the index access for "iupac:simple". If you want to enforce index access on a certain expression, you could try to rewrite your query, so that the other expressions cannot be optimized. Next, the optimizer prefers single string values (although range queries are generally supported as well). In your query, it might help to first convert your ids to strings, such as shown here:
  let $ids :=
    for $i in 1 to 10
    return string($i)
  for $i in //*:molecule[@*:id = $ids]/*:identifier[@*:convention="iupac:smile"]
  return $i
Hope this helps, Christian
PS
Patrick Schäfer9 Jul 2010 14:50 #
Quoted text · Christian Grün · 9 lines
> Next, the optimizer prefers single string values (although range > queries are generally supported as well). In your query, it might help > to first convert your ids to strings, such as shown here: > > let $ids := > for $i in 1 to 10 > return string($i) > for $i in //*:molecule[@*:id = $ids]/*:identifier[@*:convention="iupac:smile"] > return $i
Thanks! That was the solution! "Query executed in 45.11 ms." Patrick

Written for BaseX 6. BaseX may have changed since: please check the current documentation.

« Feature RequestsAdding xml to an embedded database collection »
⚡Generated with XQuery