Main Page » Mailing List » November 2016

Large text files

Friday, 25 November 2016 · 4 messages
CG
Christian Grünin reply to Tim Thompson, 25 Nov 2016, 19:0825 Nov 2016 18:21 #
Hi Tim, In BaseX, texts/strings are internally represented as byte arrays. Due to the 32 bit limitation of Java arrays, the file will be too large to be oped as single text in main-memory. To be honest, I didn’t have a similar use case before, so I guess the best solution for now will be to split the file into smaller chunks before processing it with BaseX. Cheers, Christian
Quoted text · Tim Thompson · 53 lines
On Fri, Nov 25, 2016 at 6:07 PM, Tim Thompson <timathom at gmail.com> wrote: > Hello, > > I have a large file[1] (3.5G unzipped) in the n-triples RDF format that I > would like to work with in BaseX. When I try to read in the file using > file:read-text(), I get the following error: > > Error: > Version: BaseX 8.6 beta 8fa97ca > Java: Oracle Corporation, 1.8.0_73 > OS: Linux, amd64 > Stack Trace: > java.lang.NegativeArraySizeException > at java.util.Arrays.copyOf(Arrays.java:3236) > at org.basex.util.TokenBuilder.addByte(TokenBuilder.java:247) > at org.basex.util.TokenBuilder.add(TokenBuilder.java:176) > at org.basex.io.in.TextInput.cache(TextInput.java:143) > at org.basex.io.in.TextInput.content(TextInput.java:132) > at org.basex.query.value.item.StrStream.materialize(StrStream.java:71) > at org.basex.query.value.item.StrStream.string(StrStream.java:44) > at org.basex.query.expr.ParseExpr.toToken(ParseExpr.java:273) > at org.basex.query.expr.ParseExpr.toEmptyToken(ParseExpr.java:261) > at org.basex.query.func.fn.FnSubstring.item(FnSubstring.java:22) > at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:44) > at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:99) > at org.basex.query.scope.MainModule$1.next(MainModule.java:122) > at org.basex.query.QueryContext.cache(QueryContext.java:648) > at org.basex.query.QueryProcessor.cache(QueryProcessor.java:116) > at org.basex.core.cmd.AQuery.query(AQuery.java:87) > at org.basex.core.cmd.XQuery.run(XQuery.java:22) > at org.basex.core.Command.run(Command.java:255) > at org.basex.core.Command.execute(Command.java:93) > at org.basex.gui.GUI.exec(GUI.java:479) > at org.basex.gui.GUI.access$3(GUI.java:433) > at org.basex.gui.GUI$7.run(GUI.java:421) > > When I try to create a text database using the GUI, I get an error stating > that the file could not be parsed. > > Is it possible to work with text files that are this large using BaseX? > > Thank you, > Tim > > [1] Available for download here: > http://www.bne.es/es/Inicio/Perfiles/Bibliotecarios/DatosEnlazados/DescargaFicheros/ > (http://datos.bne.es/datadumps/autoridades.nt.bz2) > > -- > Tim A. Thompson > Metadata Librarian (Spanish/Portuguese Specialty) > Princeton University Library >
NJ
Neven Jovanović25 Nov 2016 18:47 #
Hi Tim, may I suggest that you convert the n-triples file to RDF/XML format (I'm using Pythong rdflib for such tasks, <http://rdflib.readthedocs.io/en/stable/>)? Perhaps it would be easier for BaseX to ingest the XML instead of text format (which it thinks the n-triples are). Best, Neven Neven Jovanovic, Zagreb
Quoted text · Christian Grün · 69 lines
On 25 November 2016 at 18:21, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Tim, > > In BaseX, texts/strings are internally represented as byte arrays. Due > to the 32 bit limitation of Java arrays, the file will be too large to > be oped as single text in main-memory. > > To be honest, I didn’t have a similar use case before, so I guess the > best solution for now will be to split the file into smaller chunks > before processing it with BaseX. > > Cheers, > Christian > > > > On Fri, Nov 25, 2016 at 6:07 PM, Tim Thompson <timathom at gmail.com> wrote: >> Hello, >> >> I have a large file[1] (3.5G unzipped) in the n-triples RDF format that I >> would like to work with in BaseX. When I try to read in the file using >> file:read-text(), I get the following error: >> >> Error: >> Version: BaseX 8.6 beta 8fa97ca >> Java: Oracle Corporation, 1.8.0_73 >> OS: Linux, amd64 >> Stack Trace: >> java.lang.NegativeArraySizeException >> at java.util.Arrays.copyOf(Arrays.java:3236) >> at org.basex.util.TokenBuilder.addByte(TokenBuilder.java:247) >> at org.basex.util.TokenBuilder.add(TokenBuilder.java:176) >> at org.basex.io.in.TextInput.cache(TextInput.java:143) >> at org.basex.io.in.TextInput.content(TextInput.java:132) >> at org.basex.query.value.item.StrStream.materialize(StrStream.java:71) >> at org.basex.query.value.item.StrStream.string(StrStream.java:44) >> at org.basex.query.expr.ParseExpr.toToken(ParseExpr.java:273) >> at org.basex.query.expr.ParseExpr.toEmptyToken(ParseExpr.java:261) >> at org.basex.query.func.fn.FnSubstring.item(FnSubstring.java:22) >> at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:44) >> at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:99) >> at org.basex.query.scope.MainModule$1.next(MainModule.java:122) >> at org.basex.query.QueryContext.cache(QueryContext.java:648) >> at org.basex.query.QueryProcessor.cache(QueryProcessor.java:116) >> at org.basex.core.cmd.AQuery.query(AQuery.java:87) >> at org.basex.core.cmd.XQuery.run(XQuery.java:22) >> at org.basex.core.Command.run(Command.java:255) >> at org.basex.core.Command.execute(Command.java:93) >> at org.basex.gui.GUI.exec(GUI.java:479) >> at org.basex.gui.GUI.access$3(GUI.java:433) >> at org.basex.gui.GUI$7.run(GUI.java:421) >> >> When I try to create a text database using the GUI, I get an error stating >> that the file could not be parsed. >> >> Is it possible to work with text files that are this large using BaseX? >> >> Thank you, >> Tim >> >> [1] Available for download here: >> http://www.bne.es/es/Inicio/Perfiles/Bibliotecarios/DatosEnlazados/DescargaFicheros/ >> (http://datos.bne.es/datadumps/autoridades.nt.bz2) >> >> -- >> Tim A. Thompson >> Metadata Librarian (Spanish/Portuguese Specialty) >> Princeton University Library >>
TT
Tim Thompson25 Nov 2016 19:08 #
Hello, I have a large file[1] (3.5G unzipped) in the n-triples RDF format that I would like to work with in BaseX. When I try to read in the file using file:read-text(), I get the following error: Error: Version: BaseX 8.6 beta 8fa97ca Java: Oracle Corporation, 1.8.0_73 OS: Linux, amd64 Stack Trace:
java.lang.NegativeArraySizeException
    at java.util.Arrays.copyOf(Arrays.java:3236)
    at org.basex.util.TokenBuilder.addByte(TokenBuilder.java:247)
    at org.basex.util.TokenBuilder.add(TokenBuilder.java:176)
    at org.basex.io.in.TextInput.cache(TextInput.java:143)
    at org.basex.io.in.TextInput.content(TextInput.java:132)
    at org.basex.query.value.item.StrStream.materialize(StrStream.java:71)
    at org.basex.query.value.item.StrStream.string(StrStream.java:44)
    at org.basex.query.expr.ParseExpr.toToken(ParseExpr.java:273)
    at org.basex.query.expr.ParseExpr.toEmptyToken(ParseExpr.java:261)
    at org.basex.query.func.fn.FnSubstring.item(FnSubstring.java:22)
    at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:44)
    at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:99)
    at org.basex.query.scope.MainModule$1.next(MainModule.java:122)
    at org.basex.query.QueryContext.cache(QueryContext.java:648)
    at org.basex.query.QueryProcessor.cache(QueryProcessor.java:116)
    at org.basex.core.cmd.AQuery.query(AQuery.java:87)
    at org.basex.core.cmd.XQuery.run(XQuery.java:22)
    at org.basex.core.Command.run(Command.java:255)
    at org.basex.core.Command.execute(Command.java:93)
    at org.basex.gui.GUI.exec(GUI.java:479)
    at org.basex.gui.GUI.access$3(GUI.java:433)
    at org.basex.gui.GUI$7.run(GUI.java:421)
When I try to create a text database using the GUI, I get an error stating that the file could not be parsed. Is it possible to work with text files that are this large using BaseX? Thank you, Tim [1] Available for download here: http://www.bne.es/es/Inicio/Perfiles/Bibliotecarios/DatosEnlazados/DescargaFicheros/ (http://datos.bne.es/datadumps/autoridades.nt.bz2)
-- Tim A. Thompson Metadata Librarian (Spanish/Portuguese Specialty) Princeton University Library
TT
Tim Thompsonin reply to Neven Jovanović, 25 Nov 2016, 18:4725 Nov 2016 20:12 #
Hi, Neven, Yes, that's what I'll probably end up doing. My real motive for trying to process the file as text was to test a BaseX extension function for parsing n-triples, using one of Gunther Rademacher's REx Parser Generator parsers[1]. I added a Java module with the parser to BaseX, and it works very well with small to medium sized files, generating a parse tree based on the EBNF for the format. I was just curious to see how it would perform with very large files. Best, Tim [1] http://www.bottlecaps.de/rex/
-- Tim A. Thompson Metadata Librarian (Spanish/Portuguese Specialty) Princeton University Library
Quoted text · Neven Jovanović · 92 lines
On Fri, Nov 25, 2016 at 12:47 PM, Neven Jovanović <filologanoga at gmail.com> wrote: > Hi Tim, > > may I suggest that you convert the n-triples file to RDF/XML format > (I'm using Pythong rdflib for such tasks, > <http://rdflib.readthedocs.io/en/stable/>)? Perhaps it would be easier > for BaseX to ingest the XML instead of text format (which it thinks > the n-triples are). > > Best, > > Neven > > Neven Jovanovic, Zagreb > > On 25 November 2016 at 18:21, Christian Grün <christian.gruen at gmail.com> > wrote: > > Hi Tim, > > > > In BaseX, texts/strings are internally represented as byte arrays. Due > > to the 32 bit limitation of Java arrays, the file will be too large to > > be oped as single text in main-memory. > > > > To be honest, I didn’t have a similar use case before, so I guess the > > best solution for now will be to split the file into smaller chunks > > before processing it with BaseX. > > > > Cheers, > > Christian > > > > > > > > On Fri, Nov 25, 2016 at 6:07 PM, Tim Thompson <timathom at gmail.com> > wrote: > >> Hello, > >> > >> I have a large file[1] (3.5G unzipped) in the n-triples RDF format that > I > >> would like to work with in BaseX. When I try to read in the file using > >> file:read-text(), I get the following error: > >> > >> Error: > >> Version: BaseX 8.6 beta 8fa97ca > >> Java: Oracle Corporation, 1.8.0_73 > >> OS: Linux, amd64 > >> Stack Trace: > >> java.lang.NegativeArraySizeException > >> at java.util.Arrays.copyOf(Arrays.java:3236) > >> at org.basex.util.TokenBuilder.addByte(TokenBuilder.java:247) > >> at org.basex.util.TokenBuilder.add(TokenBuilder.java:176) > >> at org.basex.io.in.TextInput.cache(TextInput.java:143) > >> at org.basex.io.in.TextInput.content(TextInput.java:132) > >> at org.basex.query.value.item.StrStream.materialize( > StrStream.java:71) > >> at org.basex.query.value.item.StrStream.string(StrStream.java:44) > >> at org.basex.query.expr.ParseExpr.toToken(ParseExpr.java:273) > >> at org.basex.query.expr.ParseExpr.toEmptyToken(ParseExpr.java:261) > >> at org.basex.query.func.fn.FnSubstring.item(FnSubstring.java:22) > >> at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:44) > >> at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:99) > >> at org.basex.query.scope.MainModule$1.next(MainModule.java:122) > >> at org.basex.query.QueryContext.cache(QueryContext.java:648) > >> at org.basex.query.QueryProcessor.cache(QueryProcessor.java:116) > >> at org.basex.core.cmd.AQuery.query(AQuery.java:87) > >> at org.basex.core.cmd.XQuery.run(XQuery.java:22) > >> at org.basex.core.Command.run(Command.java:255) > >> at org.basex.core.Command.execute(Command.java:93) > >> at org.basex.gui.GUI.exec(GUI.java:479) > >> at org.basex.gui.GUI.access$3(GUI.java:433) > >> at org.basex.gui.GUI$7.run(GUI.java:421) > >> > >> When I try to create a text database using the GUI, I get an error > stating > >> that the file could not be parsed. > >> > >> Is it possible to work with text files that are this large using BaseX? > >> > >> Thank you, > >> Tim > >> > >> [1] Available for download here: > >> http://www.bne.es/es/Inicio/Perfiles/Bibliotecarios/DatosEnlazados/ > DescargaFicheros/ > >> (http://datos.bne.es/datadumps/autoridades.nt.bz2) > >> > >> -- > >> Tim A. Thompson > >> Metadata Librarian (Spanish/Portuguese Specialty) > >> Princeton University Library > >> >

Written for BaseX 8.5. BaseX may have changed since: please check the current documentation.

⚡Generated with XQuery