Create DB from large XML extraction without root element?
Tuesday, 19 May 2015 · 5 messages
HE
Hondros, Constantine (ELS-AMS)19 May 2015 11:32 #
Hello all,
I have a huge extraction of XML documents in a single file, without a root element, something along these lines:
<a>
<b/>
</a>
<a>
<b/>
</a>
...Other than by editing it to add a root element, is there a clever way of creating a Basex DB from this?
Thanks for any pointers,
C.
________________________________
Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33156677, Registered in The Netherlands.
DK
Dirk Kirsten19 May 2015 11:45 #
Hello Constantine,
as this is not a valid XML document there is no way to add this to BaseX. You could write an XQuery, which reads in the document as a text file and chops the file into valid XML documents (of course you could also do this with some external line-processing tool like sed).
Of course, as you mentioned, adding a root node is the easiest option.
Cheers
Dirk
Quoted text · Constantine (ELS-AMS) · 41 lines
On Tue, May 19, 2015 at 11:32 AM, Hondros, Constantine (ELS-AMS) < C.Hondros at elsevier.com> wrote: > Hello all, > > I have a huge extraction of XML documents in a single file, without a root > element, something along these lines: > > > > <a> > > <b/> > > </a> > > <a> > > <b/> > > </a> > > … > > > > > > Other than by editing it to add a root element, is there a clever way of > creating a Basex DB from this? > > > > Thanks for any pointers, > > C. > > ------------------------------ > > Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The > Netherlands, Registration No. 33156677, Registered in The Netherlands. >
--
Dirk Kirsten, BaseX GmbH, http://basex.org
|-- Firmensitz: Blarerstrasse 56, 78462 Konstanz
|-- Registergericht Freiburg, HRB: 708285, Geschäftsführer:
| Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle `-- Phone: 0049 7531 28 28 676, Fax: 0049 7531 20 05 22
HE
Hondros, Constantine (ELS-AMS)19 May 2015 11:58 #
Thanks, I suspected I would have to tokenise on an end element.
Cheers,
Constantine
Quoted text · Dirk Kirsten · 31 lines
From: Dirk Kirsten [mailto:dk at basex.org] Sent: 19 May 2015 11:46 To: Hondros, Constantine (ELS-AMS) Cc: basex-talk at mailman.uni-konstanz.de Subject: Re: [basex-talk] Create DB from large XML extraction without root element? Hello Constantine, as this is not a valid XML document there is no way to add this to BaseX. You could write an XQuery, which reads in the document as a text file and chops the file into valid XML documents (of course you could also do this with some external line-processing tool like sed). Of course, as you mentioned, adding a root node is the easiest option. Cheers Dirk On Tue, May 19, 2015 at 11:32 AM, Hondros, Constantine (ELS-AMS) <C.Hondros at elsevier.com<mailto:C.Hondros at elsevier.com>> wrote: Hello all, I have a huge extraction of XML documents in a single file, without a root element, something along these lines: <a> <b/> </a> <a> <b/> </a> … Other than by editing it to add a root element, is there a clever way of creating a Basex DB from this? Thanks for any pointers, C. Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33156677, Registered in The Netherlands. -- Dirk Kirsten, BaseX GmbH, http://basex.org<http://basex.org/> |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle `-- Phone: 0049 7531 28 28 676, Fax: 0049 7531 20 05 22 Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33156677, Registered in The Netherlands.
CG
Christian Grün19 May 2015 12:14 #
Hi Constantine,
you could use some Java code (see the attached file).
By the way, have you already created the database with CHOP=true?
Cheers,
Christian
Quoted text · Constantine (ELS-AMS) · 97 lines
On Tue, May 19, 2015 at 11:58 AM, Hondros, Constantine (ELS-AMS) <C.Hondros at elsevier.com> wrote: > Thanks, I suspected I would have to tokenise on an end element. > > > > Cheers, > > Constantine > > > > From: Dirk Kirsten [mailto:dk at basex.org] > Sent: 19 May 2015 11:46 > To: Hondros, Constantine (ELS-AMS) > Cc: basex-talk at mailman.uni-konstanz.de > Subject: Re: [basex-talk] Create DB from large XML extraction without root > element? > > > > Hello Constantine, > > as this is not a valid XML document there is no way to add this to BaseX. > You could write an XQuery, which reads in the document as a text file and > chops the file into valid XML documents (of course you could also do this > with some external line-processing tool like sed). > > Of course, as you mentioned, adding a root node is the easiest option. > > Cheers > > Dirk > > > > On Tue, May 19, 2015 at 11:32 AM, Hondros, Constantine (ELS-AMS) > <C.Hondros at elsevier.com> wrote: > > Hello all, > > I have a huge extraction of XML documents in a single file, without a root > element, something along these lines: > > > > <a> > > <b/> > > </a> > > <a> > > <b/> > > </a> > > … > > > > > > Other than by editing it to add a root element, is there a clever way of > creating a Basex DB from this? > > > > Thanks for any pointers, > > C. > > > > ________________________________ > > Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The > Netherlands, Registration No. 33156677, Registered in The Netherlands. > > > > > -- > > Dirk Kirsten, BaseX GmbH, http://basex.org > > |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz > |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: > | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle > `-- Phone: 0049 7531 28 28 676, Fax: 0049 7531 20 05 22 > > > ________________________________ > > Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The > Netherlands, Registration No. 33156677, Registered in The Netherlands.
HE
Hondros, Constantine (ELS-AMS)19 May 2015 12:49 #
Hi Christian,
Thanks, that's very helpful. And no, I haven't yet recreated the database with CHOP=true - I'll let you know the resulting DB size when I have.
Incidentally it might be of interest just what I am doing: I am trying to analyse a single DB I created containing some 30 million scientific references. I seem to be at the upper limit of what my hardware + BaseX can reasonably achieve. For example, attempting to splice a new DB with a couple of million entries using XQuery alone causes BaseX to hang. About the only successful operation I can perform is
file:append using a for loop (throwing results into a variable and extracting a well-formed document also hangs).
So I am in work-around mode, but always a pleasure to be hacking with BaseX - I still have yet to find an XML-related problem that I can't solve somehow with BaseX.
C.Quoted text · Christian Grün · 108 lines
-----Original Message----- From: Christian Grün [mailto:christian.gruen at gmail.com] Sent: 19 May 2015 12:14 To: Hondros, Constantine (ELS-AMS) Cc: Dirk Kirsten; basex-talk at mailman.uni-konstanz.de Subject: Re: [basex-talk] Create DB from large XML extraction without root element? Hi Constantine, you could use some Java code (see the attached file). By the way, have you already created the database with CHOP=true? Cheers, Christian On Tue, May 19, 2015 at 11:58 AM, Hondros, Constantine (ELS-AMS) <C.Hondros at elsevier.com> wrote: > Thanks, I suspected I would have to tokenise on an end element. > > > > Cheers, > > Constantine > > > > From: Dirk Kirsten [mailto:dk at basex.org] > Sent: 19 May 2015 11:46 > To: Hondros, Constantine (ELS-AMS) > Cc: basex-talk at mailman.uni-konstanz.de > Subject: Re: [basex-talk] Create DB from large XML extraction without > root element? > > > > Hello Constantine, > > as this is not a valid XML document there is no way to add this to BaseX. > You could write an XQuery, which reads in the document as a text file > and chops the file into valid XML documents (of course you could also > do this with some external line-processing tool like sed). > > Of course, as you mentioned, adding a root node is the easiest option. > > Cheers > > Dirk > > > > On Tue, May 19, 2015 at 11:32 AM, Hondros, Constantine (ELS-AMS) > <C.Hondros at elsevier.com> wrote: > > Hello all, > > I have a huge extraction of XML documents in a single file, without a > root element, something along these lines: > > > > <a> > > <b/> > > </a> > > <a> > > <b/> > > </a> > > … > > > > > > Other than by editing it to add a root element, is there a clever way > of creating a Basex DB from this? > > > > Thanks for any pointers, > > C. > > > > ________________________________ > > Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The > Netherlands, Registration No. 33156677, Registered in The Netherlands. > > > > > -- > > Dirk Kirsten, BaseX GmbH, http://basex.org > > |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz > |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: > | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle > `-- Phone: 0049 7531 28 28 676, Fax: 0049 7531 20 05 22 > > > ________________________________ > > Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The > Netherlands, Registration No. 33156677, Registered in The Netherlands. Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33156677, Registered in The Netherlands.
Written for BaseX 8.1. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery