Large CSV File (~1M rows)
Sunday, 16 October 2016 · 4 messages
SF
Shaun Flynn16 Oct 2016 10:59 #
Hello there,
The parsing runs out memory. I also tried doubling the memory to 1024MB, and still the same issue.
Kind regards
Quoted text · Christian Grün · 1 line
On 16 Oct 2016 09:15, "Christian Grün" <christian.gruen at gmail.com> wrote:
Hi Shaun,
Quoted text · 2 lines
> I do this through the MS DOS prompt, but -Xmx does not appear to be a > parameter any more,
If you work with the ZIP distributions of BaseX, you can adjust the memory setting in the start scripts (in the bin directory). Otherwise, you’ll need to pass it to to java, not BaseX itself.
Quoted text · 3 lines
> Is there a better method for parsing large CSV files? I then want to add the > resulting file tlk.xml to a new database.
Did you check if it’s the CSV parsing or the
fn:put call that causes the error?
Christian
[1] http://docs.basex.org/wiki/Database_Module#db:addCG
Christian Grün16 Oct 2016 11:01 #
Quoted text · Shaun Flynn · 2 lines
> The parsing runs out memory. I also tried doubling the memory to 1024MB, and > still the same issue.
Memory consumption should be much lower if you create a database from your input (e.g. via the GUI).
Does this help?
Christian
Quoted text · Shaun Flynn · 26 lines
> > Kind regards > > > On 16 Oct 2016 09:15, "Christian Grün" <christian.gruen at gmail.com> wrote: > > Hi Shaun, > >> I do this through the MS DOS prompt, but -Xmx does not appear to be a >> parameter any more, > > If you work with the ZIP distributions of BaseX, you can adjust the > memory setting in the start scripts (in the bin directory). Otherwise, > you’ll need to pass it to to java, not BaseX itself. > >> Is there a better method for parsing large CSV files? I then want to add >> the >> resulting file tlk.xml to a new database. > > Did you check if it’s the CSV parsing or the fn:put call that causes the > error? > Christian > > [1] http://docs.basex.org/wiki/Database_Module#db:add > >
SF
Shaun Flynn16 Oct 2016 11:13 #
Thank you -- I will give a go and let you know.
SCF xx
Quoted text · Christian Grün · 41 lines
On 16 Oct 2016 10:02, "Christian Grün" <christian.gruen at gmail.com> wrote: > > The parsing runs out memory. I also tried doubling the memory to 1024MB, > and > > still the same issue. > > Memory consumption should be much lower if you create a database from > your input (e.g. via the GUI). > > Does this help? > Christian > > > > > > > Kind regards > > > > > > On 16 Oct 2016 09:15, "Christian Grün" <christian.gruen at gmail.com> > wrote: > > > > Hi Shaun, > > > >> I do this through the MS DOS prompt, but -Xmx does not appear to be a > >> parameter any more, > > > > If you work with the ZIP distributions of BaseX, you can adjust the > > memory setting in the start scripts (in the bin directory). Otherwise, > > you’ll need to pass it to to java, not BaseX itself. > > > >> Is there a better method for parsing large CSV files? I then want to add > >> the > >> resulting file tlk.xml to a new database. > > > > Did you check if it’s the CSV parsing or the fn:put call that causes the > > error? > > Christian > > > > [1] http://docs.basex.org/wiki/Database_Module#db:add > > > > >
SF
Shaun Flynn16 Oct 2016 11:22 #
That worked! Thank you.
Quoted text · Shaun Flynn · 49 lines
On 16 Oct 2016 10:13, "Shaun Flynn" <scfy12 at googlemail.com> wrote: > Thank you -- I will give a go and let you know. > > SCF xx > > On 16 Oct 2016 10:02, "Christian Grün" <christian.gruen at gmail.com> wrote: > >> > The parsing runs out memory. I also tried doubling the memory to >> 1024MB, and >> > still the same issue. >> >> Memory consumption should be much lower if you create a database from >> your input (e.g. via the GUI). >> >> Does this help? >> Christian >> >> >> >> > >> > Kind regards >> > >> > >> > On 16 Oct 2016 09:15, "Christian Grün" <christian.gruen at gmail.com> >> wrote: >> > >> > Hi Shaun, >> > >> >> I do this through the MS DOS prompt, but -Xmx does not appear to be a >> >> parameter any more, >> > >> > If you work with the ZIP distributions of BaseX, you can adjust the >> > memory setting in the start scripts (in the bin directory). Otherwise, >> > you’ll need to pass it to to java, not BaseX itself. >> > >> >> Is there a better method for parsing large CSV files? I then want to >> add >> >> the >> >> resulting file tlk.xml to a new database. >> > >> > Did you check if it’s the CSV parsing or the fn:put call that causes the >> > error? >> > Christian >> > >> > [1] http://docs.basex.org/wiki/Database_Module#db:add >> > >> > >> >
Written for BaseX 8.5. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery