Streaming file loads
Tuesday, 19 January 2016 · 2 messages
CM
Christophe Marchand19 Jan 2016 12:20 #
Hello !
We have to load many files (>20.000), large size (>30Gb) into BaseX. Today, that's done vis a BXS script which accesses files thru a network share.
I would like to optimize network transfert ; tar / compression at source, uncompress / untar at destination. But I would like to avoid to create temporary files (at source, on NAS server, and on destination server, BaseX host).
So, is there an API which I could use to load a "file" thru a stream ? I think I can write an inputstream that redirects data to outpustreams, cutting input data in many output streams, one per file. But may I chain this outputstream to an inputstream usable by BaseX loader ?
Best regards,
Christophe
EF
Etanchaud Fabrice19 Jan 2016 12:54 #
Hello Christophe,
Are you using java ?
Maybe you should have a look at the add() method in :
https://github.com/BaseXdb/basex/blob/master/basex-examples/src/main/java/org/basex/examples/api/BaseXClient.java
I can also remember that command line tools can read standard input stream with the '-' argument.
Cordialement,
Fabrice Etanchaud
Horanet Smart Objects
-----Message d'origine-----
Quoted text · Christophe Marchand · 11 lines
De : basex-talk-bounces at mailman.uni-konstanz.de [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de Christophe Marchand Envoyé : mardi 19 janvier 2016 12:20 À : basex-talk at mailman.uni-konstanz.de Objet : [basex-talk] Streaming file loads Hello ! We have to load many files (>20.000), large size (>30Gb) into BaseX. Today, that's done vis a BXS script which accesses files thru a network share. I would like to optimize network transfert ; tar / compression at source, uncompress / untar at destination. But I would like to avoid to create temporary files (at source, on NAS server, and on destination server, BaseX host). So, is there an API which I could use to load a "file" thru a stream ? I think I can write an inputstream that redirects data to outpustreams, cutting input data in many output streams, one per file. But may I chain this outputstream to an inputstream usable by BaseX loader ? Best regards, Christophe
Written for BaseX 8.3. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery