Main Page » Mailing List » May 2016

Replacing node sets in a large file

Tuesday, 24 May 2016 · 16 messages · last reply 31 May 2016

Christian Grünin reply to Michael Sanborn, 24 May 2016, 22:5424 May 2016 21:16 #

Hi Michael, Yes, this can easily be done with XQuery. There are many ways to do this; here is just one: 1. First, create a database from your input file (e.g. with the BaseX GUI) 2. Second, run the following query to replace wrap your partinfo elements with part elements: //unit/partinfo/(replace node . with <part>{ . }</part>) 3. Third, write all page elements to disk:

  for $page at $c in //page
  return file:write($c || '.xml', $page)

Hope this helps, Christian

Quoted text · Michael Sanborn · 85 lines

On Tue, May 24, 2016 at 8:54 PM, Michael Sanborn <galethog at gmail.com> wrote: > I need to perform a transformation that would be simple in XSLT, but the > input is a file about 250 MBs in size. I'm wondering whether XQuery and > BaseX in particular would be the most efficient way of doing it. I'm new to > XQuery, and I've come up with a couple of ways to do this, but they turn out > to be very time-consuming, so I'm sure I'm Doing It Wrong. Hoping to find > out the proper way of doing this. > > The input consists of 2 sections. There are about 3600 page elements with > this structure: > > <page> > [misc page content...] > <list> > <unit> > [misc unit content 1...] > <partinfo> > <number>54321</number> > <manuf>A321</manuf> > </partinfo> > <partinfo> > <number>12345</number> > <manuf>B123</manuf> > </partinfo> > [misc unit content 2...] > </unit> > [multiple units...] > </list> > </page> > > Each unit can have 1 or 2 partinfo elements. The other section has about > 82000 part elements like this: > > <part> > <partinfo> > <number>54321</number> > <manuf>A321</manuf> > </partinfo> > [misc part content 1] > </part> > [...] > <part> > <partinfo> > <number>12345</number> > <manuf>B123</manuf> > </partinfo> > [misc part content 2] > </part> > > I want to replace each unit/partinfo with the correpsonding part, like this: > > <page> > [misc page content...] > <list> > <unit> > [misc unit content 1...] > <part> > <partinfo> > <number>54321</number> > <manuf>A321</manuf> > </partinfo> > [misc part content 1] > </part> > <part> > <partinfo> > <number>12345</number> > <manuf>B123</manuf> > </partinfo> > [misc part content 2] > </part> > [misc unit content 2...] > </unit> > [multiple units...] > </list> > </page> > > Is BaseX a good tool for this task? If so, how does one go about it? > > Finally, it would help to be able to output each page element in a separate > file. Would it be better to have BaseX do this, or to output the whole > database and chunk it with another tool? > > Thanks, > > Michael

Michael Sanborn24 May 2016 22:54 #

I need to perform a transformation that would be simple in XSLT, but the input is a file about 250 MBs in size. I'm wondering whether XQuery and BaseX in particular would be the most efficient way of doing it. I'm new to XQuery, and I've come up with a couple of ways to do this, but they turn out to be very time-consuming, so I'm sure I'm Doing It Wrong. Hoping to find out the proper way of doing this. The input consists of 2 sections. There are about 3600 page elements with this structure:

<page>
    [misc page content...]
    <list>
        <unit>
            [misc unit content 1...]
            <partinfo>
                <number>54321</number>
                <manuf>A321</manuf>
            </partinfo>
            <partinfo>
                <number>12345</number>
                <manuf>B123</manuf>
            </partinfo>
            [misc unit content 2...]
        </unit>
        [multiple units...]
    </list>
</page>

Each unit can have 1 or 2 partinfo elements. The other section has about 82000 part elements like this:

<part>
<partinfo>
<number>54321</number>
<manuf>A321</manuf>
</partinfo>
[misc part content 1]
</part>
[...]
<part>
<partinfo>
<number>12345</number>
<manuf>B123</manuf>
</partinfo>
[misc part content 2]
</part>

I want to replace each unit/partinfo with the correpsonding part, like this:

<page>
    [misc page content...]
    <list>
        <unit>
            [misc unit content 1...]
            <part>
                <partinfo>
                    <number>54321</number>
                    <manuf>A321</manuf>
                </partinfo>
                [misc part content 1]
            </part>
            <part>
                <partinfo>
                    <number>12345</number>
                    <manuf>B123</manuf>
                </partinfo>
                [misc part content 2]
            </part>
            [misc unit content 2...]
        </unit>
        [multiple units...]
    </list>
</page>

Is BaseX a good tool for this task? If so, how does one go about it? Finally, it would help to be able to output each page element in a separate file. Would it be better to have BaseX do this, or to output the whole database and chunk it with another tool? Thanks, Michael

Christian Grünin reply to Michael Sanborn, 25 May 2016, 00:5424 May 2016 23:10 #

Maybe you need something like this:

  for $partinfo in //unit/partinfo
  for $part in //part[deep-equal(partinfo, $partinfo)]
  return replace node $partinfo with $part/node()

The deep-equal will be pretty slow. If the value of the number element is unique, you could do something like this:

  for $partinfo in //unit/partinfo
  let $number := $partinfo/number
  let $part := //part[partinfo/number, $number]
  return replace node $partinfo with $part/node()

Using a map will even be faster:

  let $map := map:merge(//part/map:entry(partinfo/number/text(), .))
  for $partinfo in //unit/partinfo
  let $part := $map($partinfo/number)
  return replace node $partinfo with $part/node()

If you need to consider both number and manuf, you could e.g. combine these two in the map:

  let $map := map:merge(
    for $part in //part
    return map:entry(string-join($part/partinfo/*, '/'), $part)
  )
  for $partinfo in //unit/partinfo
  let $part := $map(string-join($partinfo/*, '/'))
  return replace node $partinfo with $part/node()

Does this help? Christian

Quoted text · Michael Sanborn · 130 lines

On Tue, May 24, 2016 at 10:54 PM, Michael Sanborn <galethog at gmail.com> wrote: > Thanks for that. The trouble in step 2 is, just wrapping partinfo with the > part element doesn't get me what I've labelled "misc part content 1" and > "misc part content 2". It's not sufficient to have just the tags - I need > all the content of the corresponding part elements in the later part of the > file. Is that something that can be done without too much difficulty? > > Thanks, > > Michael > > On Tue, May 24, 2016 at 12:16 PM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> Hi Michael, >> >> Yes, this can easily be done with XQuery. There are many ways to do >> this; here is just one: >> >> 1. First, create a database from your input file (e.g. with the BaseX GUI) >> >> 2. Second, run the following query to replace wrap your partinfo >> elements with part elements: >> >> //unit/partinfo/(replace node . with <part>{ . }</part>) >> >> 3. Third, write all page elements to disk: >> >> for $page at $c in //page >> return file:write($c || '.xml', $page) >> >> Hope this helps, >> Christian >> >> >> >> On Tue, May 24, 2016 at 8:54 PM, Michael Sanborn <galethog at gmail.com> >> wrote: >> > I need to perform a transformation that would be simple in XSLT, but the >> > input is a file about 250 MBs in size. I'm wondering whether XQuery and >> > BaseX in particular would be the most efficient way of doing it. I'm new >> > to >> > XQuery, and I've come up with a couple of ways to do this, but they turn >> > out >> > to be very time-consuming, so I'm sure I'm Doing It Wrong. Hoping to >> > find >> > out the proper way of doing this. >> > >> > The input consists of 2 sections. There are about 3600 page elements >> > with >> > this structure: >> > >> > <page> >> > [misc page content...] >> > <list> >> > <unit> >> > [misc unit content 1...] >> > <partinfo> >> > <number>54321</number> >> > <manuf>A321</manuf> >> > </partinfo> >> > <partinfo> >> > <number>12345</number> >> > <manuf>B123</manuf> >> > </partinfo> >> > [misc unit content 2...] >> > </unit> >> > [multiple units...] >> > </list> >> > </page> >> > >> > Each unit can have 1 or 2 partinfo elements. The other section has about >> > 82000 part elements like this: >> > >> > <part> >> > <partinfo> >> > <number>54321</number> >> > <manuf>A321</manuf> >> > </partinfo> >> > [misc part content 1] >> > </part> >> > [...] >> > <part> >> > <partinfo> >> > <number>12345</number> >> > <manuf>B123</manuf> >> > </partinfo> >> > [misc part content 2] >> > </part> >> > >> > I want to replace each unit/partinfo with the correpsonding part, like >> > this: >> > >> > <page> >> > [misc page content...] >> > <list> >> > <unit> >> > [misc unit content 1...] >> > <part> >> > <partinfo> >> > <number>54321</number> >> > <manuf>A321</manuf> >> > </partinfo> >> > [misc part content 1] >> > </part> >> > <part> >> > <partinfo> >> > <number>12345</number> >> > <manuf>B123</manuf> >> > </partinfo> >> > [misc part content 2] >> > </part> >> > [misc unit content 2...] >> > </unit> >> > [multiple units...] >> > </list> >> > </page> >> > >> > Is BaseX a good tool for this task? If so, how does one go about it? >> > >> > Finally, it would help to be able to output each page element in a >> > separate >> > file. Would it be better to have BaseX do this, or to output the whole >> > database and chunk it with another tool? >> > >> > Thanks, >> > >> > Michael > >

Michael Sanbornin reply to Christian Grün, 24 May 2016, 21:1625 May 2016 00:54 #

Thanks for that. The trouble in step 2 is, just wrapping partinfo with the part element doesn't get me what I've labelled "misc part content 1" and "misc part content 2". It's not sufficient to have just the tags - I need all the content of the corresponding part elements in the later part of the file. Is that something that can be done without too much difficulty? Thanks, Michael

Quoted text · Christian Grün · 115 lines

On Tue, May 24, 2016 at 12:16 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Michael, > > Yes, this can easily be done with XQuery. There are many ways to do > this; here is just one: > > 1. First, create a database from your input file (e.g. with the BaseX GUI) > > 2. Second, run the following query to replace wrap your partinfo > elements with part elements: > > //unit/partinfo/(replace node . with <part>{ . }</part>) > > 3. Third, write all page elements to disk: > > for $page at $c in //page > return file:write($c || '.xml', $page) > > Hope this helps, > Christian > > > > On Tue, May 24, 2016 at 8:54 PM, Michael Sanborn <galethog at gmail.com> > wrote: > > I need to perform a transformation that would be simple in XSLT, but the > > input is a file about 250 MBs in size. I'm wondering whether XQuery and > > BaseX in particular would be the most efficient way of doing it. I'm new > to > > XQuery, and I've come up with a couple of ways to do this, but they turn > out > > to be very time-consuming, so I'm sure I'm Doing It Wrong. Hoping to find > > out the proper way of doing this. > > > > The input consists of 2 sections. There are about 3600 page elements with > > this structure: > > > > <page> > > [misc page content...] > > <list> > > <unit> > > [misc unit content 1...] > > <partinfo> > > <number>54321</number> > > <manuf>A321</manuf> > > </partinfo> > > <partinfo> > > <number>12345</number> > > <manuf>B123</manuf> > > </partinfo> > > [misc unit content 2...] > > </unit> > > [multiple units...] > > </list> > > </page> > > > > Each unit can have 1 or 2 partinfo elements. The other section has about > > 82000 part elements like this: > > > > <part> > > <partinfo> > > <number>54321</number> > > <manuf>A321</manuf> > > </partinfo> > > [misc part content 1] > > </part> > > [...] > > <part> > > <partinfo> > > <number>12345</number> > > <manuf>B123</manuf> > > </partinfo> > > [misc part content 2] > > </part> > > > > I want to replace each unit/partinfo with the correpsonding part, like > this: > > > > <page> > > [misc page content...] > > <list> > > <unit> > > [misc unit content 1...] > > <part> > > <partinfo> > > <number>54321</number> > > <manuf>A321</manuf> > > </partinfo> > > [misc part content 1] > > </part> > > <part> > > <partinfo> > > <number>12345</number> > > <manuf>B123</manuf> > > </partinfo> > > [misc part content 2] > > </part> > > [misc unit content 2...] > > </unit> > > [multiple units...] > > </list> > > </page> > > > > Is BaseX a good tool for this task? If so, how does one go about it? > > > > Finally, it would help to be able to output each page element in a > separate > > file. Would it be better to have BaseX do this, or to output the whole > > database and chunk it with another tool? > > > > Thanks, > > > > Michael >

Michael Sanbornin reply to Christian Grün, 24 May 2016, 23:1025 May 2016 01:53 #

Seems like this would be perfect. I do need both number and manuf. Using your combination map, I'm now getting an "Out of Main Memory" error. Tried on a second computer - same issue. Would it be more likely to work if I tried it from the command line rather than the GUI? If so, I'll need to look up how to create a database that way, but I'm sure it's close to hand. Or is there a better workaround (besides buying a computer with more than 8GB of RAM)? Thanks again, Michael

Quoted text · Christian Grün · 181 lines

On Tue, May 24, 2016 at 2:10 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Maybe you need something like this: > > for $partinfo in //unit/partinfo > for $part in //part[deep-equal(partinfo, $partinfo)] > return replace node $partinfo with $part/node() > > The deep-equal will be pretty slow. If the value of the number element > is unique, you could do something like this: > > for $partinfo in //unit/partinfo > let $number := $partinfo/number > let $part := //part[partinfo/number, $number] > return replace node $partinfo with $part/node() > > Using a map will even be faster: > > let $map := map:merge(//part/map:entry(partinfo/number/text(), .)) > for $partinfo in //unit/partinfo > let $part := $map($partinfo/number) > return replace node $partinfo with $part/node() > > If you need to consider both number and manuf, you could e.g. combine > these two in the map: > > let $map := map:merge( > for $part in //part > return map:entry(string-join($part/partinfo/*, '/'), $part) > ) > for $partinfo in //unit/partinfo > let $part := $map(string-join($partinfo/*, '/')) > return replace node $partinfo with $part/node() > > Does this help? > Christian > > > > > On Tue, May 24, 2016 at 10:54 PM, Michael Sanborn <galethog at gmail.com> > wrote: > > Thanks for that. The trouble in step 2 is, just wrapping partinfo with > the > > part element doesn't get me what I've labelled "misc part content 1" and > > "misc part content 2". It's not sufficient to have just the tags - I need > > all the content of the corresponding part elements in the later part of > the > > file. Is that something that can be done without too much difficulty? > > > > Thanks, > > > > Michael > > > > On Tue, May 24, 2016 at 12:16 PM, Christian Grün < > christian.gruen at gmail.com> > > wrote: > >> > >> Hi Michael, > >> > >> Yes, this can easily be done with XQuery. There are many ways to do > >> this; here is just one: > >> > >> 1. First, create a database from your input file (e.g. with the BaseX > GUI) > >> > >> 2. Second, run the following query to replace wrap your partinfo > >> elements with part elements: > >> > >> //unit/partinfo/(replace node . with <part>{ . }</part>) > >> > >> 3. Third, write all page elements to disk: > >> > >> for $page at $c in //page > >> return file:write($c || '.xml', $page) > >> > >> Hope this helps, > >> Christian > >> > >> > >> > >> On Tue, May 24, 2016 at 8:54 PM, Michael Sanborn <galethog at gmail.com> > >> wrote: > >> > I need to perform a transformation that would be simple in XSLT, but > the > >> > input is a file about 250 MBs in size. I'm wondering whether XQuery > and > >> > BaseX in particular would be the most efficient way of doing it. I'm > new > >> > to > >> > XQuery, and I've come up with a couple of ways to do this, but they > turn > >> > out > >> > to be very time-consuming, so I'm sure I'm Doing It Wrong. Hoping to > >> > find > >> > out the proper way of doing this. > >> > > >> > The input consists of 2 sections. There are about 3600 page elements > >> > with > >> > this structure: > >> > > >> > <page> > >> > [misc page content...] > >> > <list> > >> > <unit> > >> > [misc unit content 1...] > >> > <partinfo> > >> > <number>54321</number> > >> > <manuf>A321</manuf> > >> > </partinfo> > >> > <partinfo> > >> > <number>12345</number> > >> > <manuf>B123</manuf> > >> > </partinfo> > >> > [misc unit content 2...] > >> > </unit> > >> > [multiple units...] > >> > </list> > >> > </page> > >> > > >> > Each unit can have 1 or 2 partinfo elements. The other section has > about > >> > 82000 part elements like this: > >> > > >> > <part> > >> > <partinfo> > >> > <number>54321</number> > >> > <manuf>A321</manuf> > >> > </partinfo> > >> > [misc part content 1] > >> > </part> > >> > [...] > >> > <part> > >> > <partinfo> > >> > <number>12345</number> > >> > <manuf>B123</manuf> > >> > </partinfo> > >> > [misc part content 2] > >> > </part> > >> > > >> > I want to replace each unit/partinfo with the correpsonding part, like > >> > this: > >> > > >> > <page> > >> > [misc page content...] > >> > <list> > >> > <unit> > >> > [misc unit content 1...] > >> > <part> > >> > <partinfo> > >> > <number>54321</number> > >> > <manuf>A321</manuf> > >> > </partinfo> > >> > [misc part content 1] > >> > </part> > >> > <part> > >> > <partinfo> > >> > <number>12345</number> > >> > <manuf>B123</manuf> > >> > </partinfo> > >> > [misc part content 2] > >> > </part> > >> > [misc unit content 2...] > >> > </unit> > >> > [multiple units...] > >> > </list> > >> > </page> > >> > > >> > Is BaseX a good tool for this task? If so, how does one go about it? > >> > > >> > Finally, it would help to be able to output each page element in a > >> > separate > >> > file. Would it be better to have BaseX do this, or to output the whole > >> > database and chunk it with another tool? > >> > > >> > Thanks, > >> > > >> > Michael > > > > >

Christian Grün25 May 2016 07:49 #

Usually, 8GB should be much more than sufficient for such a query. You could try to increase the memory, which is assigned to Java, in the start scripts [1]. Does this help? Christian [1] http://docs.basex.org/wiki/Start_Scripts

Quoted text · Michael Sanborn · 199 lines

On Tue, May 24, 2016 at 11:52 PM, Michael Sanborn <galethog at gmail.com> wrote: > Seems like this would be perfect. I do need both number and manuf. Using > your combination map, I'm now getting an "Out of Main Memory" error. Tried > on a second computer - same issue. Would it be more likely to work if I > tried it from the command line rather than the GUI? If so, I'll need to look > up how to create a database that way, but I'm sure it's close to hand. Or is > there a better workaround (besides buying a computer with more than 8GB of > RAM)? > > Thanks again, > > Michael > > On Tue, May 24, 2016 at 2:10 PM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> Maybe you need something like this: >> >> for $partinfo in //unit/partinfo >> for $part in //part[deep-equal(partinfo, $partinfo)] >> return replace node $partinfo with $part/node() >> >> The deep-equal will be pretty slow. If the value of the number element >> is unique, you could do something like this: >> >> for $partinfo in //unit/partinfo >> let $number := $partinfo/number >> let $part := //part[partinfo/number, $number] >> return replace node $partinfo with $part/node() >> >> Using a map will even be faster: >> >> let $map := map:merge(//part/map:entry(partinfo/number/text(), .)) >> for $partinfo in //unit/partinfo >> let $part := $map($partinfo/number) >> return replace node $partinfo with $part/node() >> >> If you need to consider both number and manuf, you could e.g. combine >> these two in the map: >> >> let $map := map:merge( >> for $part in //part >> return map:entry(string-join($part/partinfo/*, '/'), $part) >> ) >> for $partinfo in //unit/partinfo >> let $part := $map(string-join($partinfo/*, '/')) >> return replace node $partinfo with $part/node() >> >> Does this help? >> Christian >> >> >> >> >> On Tue, May 24, 2016 at 10:54 PM, Michael Sanborn <galethog at gmail.com> >> wrote: >> > Thanks for that. The trouble in step 2 is, just wrapping partinfo with >> > the >> > part element doesn't get me what I've labelled "misc part content 1" and >> > "misc part content 2". It's not sufficient to have just the tags - I >> > need >> > all the content of the corresponding part elements in the later part of >> > the >> > file. Is that something that can be done without too much difficulty? >> > >> > Thanks, >> > >> > Michael >> > >> > On Tue, May 24, 2016 at 12:16 PM, Christian Grün >> > <christian.gruen at gmail.com> >> > wrote: >> >> >> >> Hi Michael, >> >> >> >> Yes, this can easily be done with XQuery. There are many ways to do >> >> this; here is just one: >> >> >> >> 1. First, create a database from your input file (e.g. with the BaseX >> >> GUI) >> >> >> >> 2. Second, run the following query to replace wrap your partinfo >> >> elements with part elements: >> >> >> >> //unit/partinfo/(replace node . with <part>{ . }</part>) >> >> >> >> 3. Third, write all page elements to disk: >> >> >> >> for $page at $c in //page >> >> return file:write($c || '.xml', $page) >> >> >> >> Hope this helps, >> >> Christian >> >> >> >> >> >> >> >> On Tue, May 24, 2016 at 8:54 PM, Michael Sanborn <galethog at gmail.com> >> >> wrote: >> >> > I need to perform a transformation that would be simple in XSLT, but >> >> > the >> >> > input is a file about 250 MBs in size. I'm wondering whether XQuery >> >> > and >> >> > BaseX in particular would be the most efficient way of doing it. I'm >> >> > new >> >> > to >> >> > XQuery, and I've come up with a couple of ways to do this, but they >> >> > turn >> >> > out >> >> > to be very time-consuming, so I'm sure I'm Doing It Wrong. Hoping to >> >> > find >> >> > out the proper way of doing this. >> >> > >> >> > The input consists of 2 sections. There are about 3600 page elements >> >> > with >> >> > this structure: >> >> > >> >> > <page> >> >> > [misc page content...] >> >> > <list> >> >> > <unit> >> >> > [misc unit content 1...] >> >> > <partinfo> >> >> > <number>54321</number> >> >> > <manuf>A321</manuf> >> >> > </partinfo> >> >> > <partinfo> >> >> > <number>12345</number> >> >> > <manuf>B123</manuf> >> >> > </partinfo> >> >> > [misc unit content 2...] >> >> > </unit> >> >> > [multiple units...] >> >> > </list> >> >> > </page> >> >> > >> >> > Each unit can have 1 or 2 partinfo elements. The other section has >> >> > about >> >> > 82000 part elements like this: >> >> > >> >> > <part> >> >> > <partinfo> >> >> > <number>54321</number> >> >> > <manuf>A321</manuf> >> >> > </partinfo> >> >> > [misc part content 1] >> >> > </part> >> >> > [...] >> >> > <part> >> >> > <partinfo> >> >> > <number>12345</number> >> >> > <manuf>B123</manuf> >> >> > </partinfo> >> >> > [misc part content 2] >> >> > </part> >> >> > >> >> > I want to replace each unit/partinfo with the correpsonding part, >> >> > like >> >> > this: >> >> > >> >> > <page> >> >> > [misc page content...] >> >> > <list> >> >> > <unit> >> >> > [misc unit content 1...] >> >> > <part> >> >> > <partinfo> >> >> > <number>54321</number> >> >> > <manuf>A321</manuf> >> >> > </partinfo> >> >> > [misc part content 1] >> >> > </part> >> >> > <part> >> >> > <partinfo> >> >> > <number>12345</number> >> >> > <manuf>B123</manuf> >> >> > </partinfo> >> >> > [misc part content 2] >> >> > </part> >> >> > [misc unit content 2...] >> >> > </unit> >> >> > [multiple units...] >> >> > </list> >> >> > </page> >> >> > >> >> > Is BaseX a good tool for this task? If so, how does one go about it? >> >> > >> >> > Finally, it would help to be able to output each page element in a >> >> > separate >> >> > file. Would it be better to have BaseX do this, or to output the >> >> > whole >> >> > database and chunk it with another tool? >> >> > >> >> > Thanks, >> >> > >> >> > Michael >> > >> > > >

Michael Sanborn26 May 2016 03:08 #

Sorry to say I still haven't been able to get it to work. Whether I edit basex.bat or basexgui.bat, changing Xmx512m to Xmx1024m, and launching them on two different computers, I get "Out of Main Memory" within a minute. I also tried Xmx2048m, but that gives me a "Could not reserve enough space" error. Now for basex.bat, in order to create a context, I started the script with 'declare context item := doc("input.xml");' which may not be the most efficient way to do this, I don't know. But on the command line or in the GUI, I haven't had any luck. Any other suggestions? Thanks, Michael

Quoted text · Christian Grün · 227 lines

On Tue, May 24, 2016 at 10:49 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Usually, 8GB should be much more than sufficient for such a query. You > could try to increase the memory, which is assigned to Java, in the > start scripts [1]. > > Does this help? > Christian > > [1] http://docs.basex.org/wiki/Start_Scripts > > > > On Tue, May 24, 2016 at 11:52 PM, Michael Sanborn <galethog at gmail.com> > wrote: > > Seems like this would be perfect. I do need both number and manuf. Using > > your combination map, I'm now getting an "Out of Main Memory" error. > Tried > > on a second computer - same issue. Would it be more likely to work if I > > tried it from the command line rather than the GUI? If so, I'll need to > look > > up how to create a database that way, but I'm sure it's close to hand. > Or is > > there a better workaround (besides buying a computer with more than 8GB > of > > RAM)? > > > > Thanks again, > > > > Michael > > > > On Tue, May 24, 2016 at 2:10 PM, Christian Grün < > christian.gruen at gmail.com> > > wrote: > >> > >> Maybe you need something like this: > >> > >> for $partinfo in //unit/partinfo > >> for $part in //part[deep-equal(partinfo, $partinfo)] > >> return replace node $partinfo with $part/node() > >> > >> The deep-equal will be pretty slow. If the value of the number element > >> is unique, you could do something like this: > >> > >> for $partinfo in //unit/partinfo > >> let $number := $partinfo/number > >> let $part := //part[partinfo/number, $number] > >> return replace node $partinfo with $part/node() > >> > >> Using a map will even be faster: > >> > >> let $map := map:merge(//part/map:entry(partinfo/number/text(), .)) > >> for $partinfo in //unit/partinfo > >> let $part := $map($partinfo/number) > >> return replace node $partinfo with $part/node() > >> > >> If you need to consider both number and manuf, you could e.g. combine > >> these two in the map: > >> > >> let $map := map:merge( > >> for $part in //part > >> return map:entry(string-join($part/partinfo/*, '/'), $part) > >> ) > >> for $partinfo in //unit/partinfo > >> let $part := $map(string-join($partinfo/*, '/')) > >> return replace node $partinfo with $part/node() > >> > >> Does this help? > >> Christian > >> > >> > >> > >> > >> On Tue, May 24, 2016 at 10:54 PM, Michael Sanborn <galethog at gmail.com> > >> wrote: > >> > Thanks for that. The trouble in step 2 is, just wrapping partinfo with > >> > the > >> > part element doesn't get me what I've labelled "misc part content 1" > and > >> > "misc part content 2". It's not sufficient to have just the tags - I > >> > need > >> > all the content of the corresponding part elements in the later part > of > >> > the > >> > file. Is that something that can be done without too much difficulty? > >> > > >> > Thanks, > >> > > >> > Michael > >> > > >> > On Tue, May 24, 2016 at 12:16 PM, Christian Grün > >> > <christian.gruen at gmail.com> > >> > wrote: > >> >> > >> >> Hi Michael, > >> >> > >> >> Yes, this can easily be done with XQuery. There are many ways to do > >> >> this; here is just one: > >> >> > >> >> 1. First, create a database from your input file (e.g. with the BaseX > >> >> GUI) > >> >> > >> >> 2. Second, run the following query to replace wrap your partinfo > >> >> elements with part elements: > >> >> > >> >> //unit/partinfo/(replace node . with <part>{ . }</part>) > >> >> > >> >> 3. Third, write all page elements to disk: > >> >> > >> >> for $page at $c in //page > >> >> return file:write($c || '.xml', $page) > >> >> > >> >> Hope this helps, > >> >> Christian > >> >> > >> >> > >> >> > >> >> On Tue, May 24, 2016 at 8:54 PM, Michael Sanborn <galethog at gmail.com > > > >> >> wrote: > >> >> > I need to perform a transformation that would be simple in XSLT, > but > >> >> > the > >> >> > input is a file about 250 MBs in size. I'm wondering whether XQuery > >> >> > and > >> >> > BaseX in particular would be the most efficient way of doing it. > I'm > >> >> > new > >> >> > to > >> >> > XQuery, and I've come up with a couple of ways to do this, but they > >> >> > turn > >> >> > out > >> >> > to be very time-consuming, so I'm sure I'm Doing It Wrong. Hoping > to > >> >> > find > >> >> > out the proper way of doing this. > >> >> > > >> >> > The input consists of 2 sections. There are about 3600 page > elements > >> >> > with > >> >> > this structure: > >> >> > > >> >> > <page> > >> >> > [misc page content...] > >> >> > <list> > >> >> > <unit> > >> >> > [misc unit content 1...] > >> >> > <partinfo> > >> >> > <number>54321</number> > >> >> > <manuf>A321</manuf> > >> >> > </partinfo> > >> >> > <partinfo> > >> >> > <number>12345</number> > >> >> > <manuf>B123</manuf> > >> >> > </partinfo> > >> >> > [misc unit content 2...] > >> >> > </unit> > >> >> > [multiple units...] > >> >> > </list> > >> >> > </page> > >> >> > > >> >> > Each unit can have 1 or 2 partinfo elements. The other section has > >> >> > about > >> >> > 82000 part elements like this: > >> >> > > >> >> > <part> > >> >> > <partinfo> > >> >> > <number>54321</number> > >> >> > <manuf>A321</manuf> > >> >> > </partinfo> > >> >> > [misc part content 1] > >> >> > </part> > >> >> > [...] > >> >> > <part> > >> >> > <partinfo> > >> >> > <number>12345</number> > >> >> > <manuf>B123</manuf> > >> >> > </partinfo> > >> >> > [misc part content 2] > >> >> > </part> > >> >> > > >> >> > I want to replace each unit/partinfo with the correpsonding part, > >> >> > like > >> >> > this: > >> >> > > >> >> > <page> > >> >> > [misc page content...] > >> >> > <list> > >> >> > <unit> > >> >> > [misc unit content 1...] > >> >> > <part> > >> >> > <partinfo> > >> >> > <number>54321</number> > >> >> > <manuf>A321</manuf> > >> >> > </partinfo> > >> >> > [misc part content 1] > >> >> > </part> > >> >> > <part> > >> >> > <partinfo> > >> >> > <number>12345</number> > >> >> > <manuf>B123</manuf> > >> >> > </partinfo> > >> >> > [misc part content 2] > >> >> > </part> > >> >> > [misc unit content 2...] > >> >> > </unit> > >> >> > [multiple units...] > >> >> > </list> > >> >> > </page> > >> >> > > >> >> > Is BaseX a good tool for this task? If so, how does one go about > it? > >> >> > > >> >> > Finally, it would help to be able to output each page element in a > >> >> > separate > >> >> > file. Would it be better to have BaseX do this, or to output the > >> >> > whole > >> >> > database and chunk it with another tool? > >> >> > > >> >> > Thanks, > >> >> > > >> >> > Michael > >> > > >> > > > > > >

Lizzi, Vincent26 May 2016 03:26 #

Hi Michael, Which Java are you using? If you are using 32 bit Java and have set a high memory value in Xmx Java might fail to start. Check that you are using a 64 bit version of Java. If you need to have more than one version of Java on your system, you can edit basex.bat or basexgui.bat to include the full path to the java that you want BaseX to use. Hope this helps. Vincent

Quoted text · basex-talk-bounces at mailman.uni-konstanz.de · 216 lines

From: basex-talk-bounces at mailman.uni-konstanz.de [mailto:basex-talk-bounces at mailman.uni-konstanz.de] On Behalf Of Michael Sanborn Sent: Wednesday, May 25, 2016 7:07 PM To: Christian Grün <christian.gruen at gmail.com> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Replacing node sets in a large file Sorry to say I still haven't been able to get it to work. Whether I edit basex.bat or basexgui.bat, changing Xmx512m to Xmx1024m, and launching them on two different computers, I get "Out of Main Memory" within a minute. I also tried Xmx2048m, but that gives me a "Could not reserve enough space" error. Now for basex.bat, in order to create a context, I started the script with 'declare context item := doc("input.xml");' which may not be the most efficient way to do this, I don't know. But on the command line or in the GUI, I haven't had any luck. Any other suggestions? Thanks, Michael On Tue, May 24, 2016 at 10:49 PM, Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> wrote: Usually, 8GB should be much more than sufficient for such a query. You could try to increase the memory, which is assigned to Java, in the start scripts [1]. Does this help? Christian [1] http://docs.basex.org/wiki/Start_Scripts<http://docs.basex.org/wiki/Start_Scripts> On Tue, May 24, 2016 at 11:52 PM, Michael Sanborn <galethog at gmail.com<mailto:galethog at gmail.com>> wrote: > Seems like this would be perfect. I do need both number and manuf. Using > your combination map, I'm now getting an "Out of Main Memory" error. Tried > on a second computer - same issue. Would it be more likely to work if I > tried it from the command line rather than the GUI? If so, I'll need to look > up how to create a database that way, but I'm sure it's close to hand. Or is > there a better workaround (besides buying a computer with more than 8GB of > RAM)? > > Thanks again, > > Michael > > On Tue, May 24, 2016 at 2:10 PM, Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> > wrote: >> >> Maybe you need something like this: >> >> for $partinfo in //unit/partinfo >> for $part in //part[deep-equal(partinfo, $partinfo)] >> return replace node $partinfo with $part/node() >> >> The deep-equal will be pretty slow. If the value of the number element >> is unique, you could do something like this: >> >> for $partinfo in //unit/partinfo >> let $number := $partinfo/number >> let $part := //part[partinfo/number, $number] >> return replace node $partinfo with $part/node() >> >> Using a map will even be faster: >> >> let $map := map:merge(//part/map:entry(partinfo/number/text(), .)) >> for $partinfo in //unit/partinfo >> let $part := $map($partinfo/number) >> return replace node $partinfo with $part/node() >> >> If you need to consider both number and manuf, you could e.g. combine >> these two in the map: >> >> let $map := map:merge( >> for $part in //part >> return map:entry(string-join($part/partinfo/*, '/'), $part) >> ) >> for $partinfo in //unit/partinfo >> let $part := $map(string-join($partinfo/*, '/')) >> return replace node $partinfo with $part/node() >> >> Does this help? >> Christian >> >> >> >> >> On Tue, May 24, 2016 at 10:54 PM, Michael Sanborn <galethog at gmail.com<mailto:galethog at gmail.com>> >> wrote: >> > Thanks for that. The trouble in step 2 is, just wrapping partinfo with >> > the >> > part element doesn't get me what I've labelled "misc part content 1" and >> > "misc part content 2". It's not sufficient to have just the tags - I >> > need >> > all the content of the corresponding part elements in the later part of >> > the >> > file. Is that something that can be done without too much difficulty? >> > >> > Thanks, >> > >> > Michael >> > >> > On Tue, May 24, 2016 at 12:16 PM, Christian Grün >> > <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> >> > wrote: >> >> >> >> Hi Michael, >> >> >> >> Yes, this can easily be done with XQuery. There are many ways to do >> >> this; here is just one: >> >> >> >> 1. First, create a database from your input file (e.g. with the BaseX >> >> GUI) >> >> >> >> 2. Second, run the following query to replace wrap your partinfo >> >> elements with part elements: >> >> >> >> //unit/partinfo/(replace node . with <part>{ . }</part>) >> >> >> >> 3. Third, write all page elements to disk: >> >> >> >> for $page at $c in //page >> >> return file:write($c || '.xml', $page) >> >> >> >> Hope this helps, >> >> Christian >> >> >> >> >> >> >> >> On Tue, May 24, 2016 at 8:54 PM, Michael Sanborn <galethog at gmail.com<mailto:galethog at gmail.com>> >> >> wrote: >> >> > I need to perform a transformation that would be simple in XSLT, but >> >> > the >> >> > input is a file about 250 MBs in size. I'm wondering whether XQuery >> >> > and >> >> > BaseX in particular would be the most efficient way of doing it. I'm >> >> > new >> >> > to >> >> > XQuery, and I've come up with a couple of ways to do this, but they >> >> > turn >> >> > out >> >> > to be very time-consuming, so I'm sure I'm Doing It Wrong. Hoping to >> >> > find >> >> > out the proper way of doing this. >> >> > >> >> > The input consists of 2 sections. There are about 3600 page elements >> >> > with >> >> > this structure: >> >> > >> >> > <page> >> >> > [misc page content...] >> >> > <list> >> >> > <unit> >> >> > [misc unit content 1...] >> >> > <partinfo> >> >> > <number>54321</number> >> >> > <manuf>A321</manuf> >> >> > </partinfo> >> >> > <partinfo> >> >> > <number>12345</number> >> >> > <manuf>B123</manuf> >> >> > </partinfo> >> >> > [misc unit content 2...] >> >> > </unit> >> >> > [multiple units...] >> >> > </list> >> >> > </page> >> >> > >> >> > Each unit can have 1 or 2 partinfo elements. The other section has >> >> > about >> >> > 82000 part elements like this: >> >> > >> >> > <part> >> >> > <partinfo> >> >> > <number>54321</number> >> >> > <manuf>A321</manuf> >> >> > </partinfo> >> >> > [misc part content 1] >> >> > </part> >> >> > [...] >> >> > <part> >> >> > <partinfo> >> >> > <number>12345</number> >> >> > <manuf>B123</manuf> >> >> > </partinfo> >> >> > [misc part content 2] >> >> > </part> >> >> > >> >> > I want to replace each unit/partinfo with the correpsonding part, >> >> > like >> >> > this: >> >> > >> >> > <page> >> >> > [misc page content...] >> >> > <list> >> >> > <unit> >> >> > [misc unit content 1...] >> >> > <part> >> >> > <partinfo> >> >> > <number>54321</number> >> >> > <manuf>A321</manuf> >> >> > </partinfo> >> >> > [misc part content 1] >> >> > </part> >> >> > <part> >> >> > <partinfo> >> >> > <number>12345</number> >> >> > <manuf>B123</manuf> >> >> > </partinfo> >> >> > [misc part content 2] >> >> > </part> >> >> > [misc unit content 2...] >> >> > </unit> >> >> > [multiple units...] >> >> > </list> >> >> > </page> >> >> > >> >> > Is BaseX a good tool for this task? If so, how does one go about it? >> >> > >> >> > Finally, it would help to be able to output each page element in a >> >> > separate >> >> > file. Would it be better to have BaseX do this, or to output the >> >> > whole >> >> > database and chunk it with another tool? >> >> > >> >> > Thanks, >> >> > >> >> > Michael >> > >> > > >

Christian Grünin reply to Michael Sanborn, 26 May 2016, 03:0826 May 2016 08:09 #

Hi Michael,

Quoted text · Michael Sanborn · 3 lines

> Now for basex.bat, in order to create a context, I started the script with > 'declare context item := doc("input.xml");' which may not be the most > efficient way to do this, I don't know.

If you have created a database, you can use the command-line flag -i:

  basex.bat -i input query.xq

The db:open function can be used as well:

  declare context item := db:open("input");

Is it only the query with the map constructor that requires more memory? Best, Christian But on the command line or in the

Quoted text · Michael Sanborn · 243 lines

> GUI, I haven't had any luck. > > Any other suggestions? > > Thanks, > > Michael > > > > On Tue, May 24, 2016 at 10:49 PM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> Usually, 8GB should be much more than sufficient for such a query. You >> could try to increase the memory, which is assigned to Java, in the >> start scripts [1]. >> >> Does this help? >> Christian >> >> [1] http://docs.basex.org/wiki/Start_Scripts >> >> >> >> On Tue, May 24, 2016 at 11:52 PM, Michael Sanborn <galethog at gmail.com> >> wrote: >> > Seems like this would be perfect. I do need both number and manuf. Using >> > your combination map, I'm now getting an "Out of Main Memory" error. >> > Tried >> > on a second computer - same issue. Would it be more likely to work if I >> > tried it from the command line rather than the GUI? If so, I'll need to >> > look >> > up how to create a database that way, but I'm sure it's close to hand. >> > Or is >> > there a better workaround (besides buying a computer with more than 8GB >> > of >> > RAM)? >> > >> > Thanks again, >> > >> > Michael >> > >> > On Tue, May 24, 2016 at 2:10 PM, Christian Grün >> > <christian.gruen at gmail.com> >> > wrote: >> >> >> >> Maybe you need something like this: >> >> >> >> for $partinfo in //unit/partinfo >> >> for $part in //part[deep-equal(partinfo, $partinfo)] >> >> return replace node $partinfo with $part/node() >> >> >> >> The deep-equal will be pretty slow. If the value of the number element >> >> is unique, you could do something like this: >> >> >> >> for $partinfo in //unit/partinfo >> >> let $number := $partinfo/number >> >> let $part := //part[partinfo/number, $number] >> >> return replace node $partinfo with $part/node() >> >> >> >> Using a map will even be faster: >> >> >> >> let $map := map:merge(//part/map:entry(partinfo/number/text(), .)) >> >> for $partinfo in //unit/partinfo >> >> let $part := $map($partinfo/number) >> >> return replace node $partinfo with $part/node() >> >> >> >> If you need to consider both number and manuf, you could e.g. combine >> >> these two in the map: >> >> >> >> let $map := map:merge( >> >> for $part in //part >> >> return map:entry(string-join($part/partinfo/*, '/'), $part) >> >> ) >> >> for $partinfo in //unit/partinfo >> >> let $part := $map(string-join($partinfo/*, '/')) >> >> return replace node $partinfo with $part/node() >> >> >> >> Does this help? >> >> Christian >> >> >> >> >> >> >> >> >> >> On Tue, May 24, 2016 at 10:54 PM, Michael Sanborn <galethog at gmail.com> >> >> wrote: >> >> > Thanks for that. The trouble in step 2 is, just wrapping partinfo >> >> > with >> >> > the >> >> > part element doesn't get me what I've labelled "misc part content 1" >> >> > and >> >> > "misc part content 2". It's not sufficient to have just the tags - I >> >> > need >> >> > all the content of the corresponding part elements in the later part >> >> > of >> >> > the >> >> > file. Is that something that can be done without too much difficulty? >> >> > >> >> > Thanks, >> >> > >> >> > Michael >> >> > >> >> > On Tue, May 24, 2016 at 12:16 PM, Christian Grün >> >> > <christian.gruen at gmail.com> >> >> > wrote: >> >> >> >> >> >> Hi Michael, >> >> >> >> >> >> Yes, this can easily be done with XQuery. There are many ways to do >> >> >> this; here is just one: >> >> >> >> >> >> 1. First, create a database from your input file (e.g. with the >> >> >> BaseX >> >> >> GUI) >> >> >> >> >> >> 2. Second, run the following query to replace wrap your partinfo >> >> >> elements with part elements: >> >> >> >> >> >> //unit/partinfo/(replace node . with <part>{ . }</part>) >> >> >> >> >> >> 3. Third, write all page elements to disk: >> >> >> >> >> >> for $page at $c in //page >> >> >> return file:write($c || '.xml', $page) >> >> >> >> >> >> Hope this helps, >> >> >> Christian >> >> >> >> >> >> >> >> >> >> >> >> On Tue, May 24, 2016 at 8:54 PM, Michael Sanborn >> >> >> <galethog at gmail.com> >> >> >> wrote: >> >> >> > I need to perform a transformation that would be simple in XSLT, >> >> >> > but >> >> >> > the >> >> >> > input is a file about 250 MBs in size. I'm wondering whether >> >> >> > XQuery >> >> >> > and >> >> >> > BaseX in particular would be the most efficient way of doing it. >> >> >> > I'm >> >> >> > new >> >> >> > to >> >> >> > XQuery, and I've come up with a couple of ways to do this, but >> >> >> > they >> >> >> > turn >> >> >> > out >> >> >> > to be very time-consuming, so I'm sure I'm Doing It Wrong. Hoping >> >> >> > to >> >> >> > find >> >> >> > out the proper way of doing this. >> >> >> > >> >> >> > The input consists of 2 sections. There are about 3600 page >> >> >> > elements >> >> >> > with >> >> >> > this structure: >> >> >> > >> >> >> > <page> >> >> >> > [misc page content...] >> >> >> > <list> >> >> >> > <unit> >> >> >> > [misc unit content 1...] >> >> >> > <partinfo> >> >> >> > <number>54321</number> >> >> >> > <manuf>A321</manuf> >> >> >> > </partinfo> >> >> >> > <partinfo> >> >> >> > <number>12345</number> >> >> >> > <manuf>B123</manuf> >> >> >> > </partinfo> >> >> >> > [misc unit content 2...] >> >> >> > </unit> >> >> >> > [multiple units...] >> >> >> > </list> >> >> >> > </page> >> >> >> > >> >> >> > Each unit can have 1 or 2 partinfo elements. The other section has >> >> >> > about >> >> >> > 82000 part elements like this: >> >> >> > >> >> >> > <part> >> >> >> > <partinfo> >> >> >> > <number>54321</number> >> >> >> > <manuf>A321</manuf> >> >> >> > </partinfo> >> >> >> > [misc part content 1] >> >> >> > </part> >> >> >> > [...] >> >> >> > <part> >> >> >> > <partinfo> >> >> >> > <number>12345</number> >> >> >> > <manuf>B123</manuf> >> >> >> > </partinfo> >> >> >> > [misc part content 2] >> >> >> > </part> >> >> >> > >> >> >> > I want to replace each unit/partinfo with the correpsonding part, >> >> >> > like >> >> >> > this: >> >> >> > >> >> >> > <page> >> >> >> > [misc page content...] >> >> >> > <list> >> >> >> > <unit> >> >> >> > [misc unit content 1...] >> >> >> > <part> >> >> >> > <partinfo> >> >> >> > <number>54321</number> >> >> >> > <manuf>A321</manuf> >> >> >> > </partinfo> >> >> >> > [misc part content 1] >> >> >> > </part> >> >> >> > <part> >> >> >> > <partinfo> >> >> >> > <number>12345</number> >> >> >> > <manuf>B123</manuf> >> >> >> > </partinfo> >> >> >> > [misc part content 2] >> >> >> > </part> >> >> >> > [misc unit content 2...] >> >> >> > </unit> >> >> >> > [multiple units...] >> >> >> > </list> >> >> >> > </page> >> >> >> > >> >> >> > Is BaseX a good tool for this task? If so, how does one go about >> >> >> > it? >> >> >> > >> >> >> > Finally, it would help to be able to output each page element in a >> >> >> > separate >> >> >> > file. Would it be better to have BaseX do this, or to output the >> >> >> > whole >> >> >> > database and chunk it with another tool? >> >> >> > >> >> >> > Thanks, >> >> >> > >> >> >> > Michael >> >> > >> >> > >> > >> > > >

Michael Sanborn26 May 2016 23:15 #

(Re: 32-bit vs 64-bit) Of the two machines I've been trying this on, one is my laptop, and it turned out that it was using 32-bit Java. So I upgraded to the 64-bit version, and I find that I can actually get it to work, more or less, with Xmx4096m. So thanks for that. The other machine has had 64-bit Java all along, and I still get an out of memory error even with Xmx4096m, but it's a VM so there might be resource-sharing issues - I'll come back to that later. So, about running this on the command line, here's the current state of the code:

let $map := map:merge(
    for $part in //part
    return map:entry(string-join($part/partinfo/*, '/'), $part)
)
for $partinfo in //unit/partinfo
let $part := $map(string-join($partinfo/*, '/'))
return replace node $partinfo with <part>{$part/node()}</part>
for $page in //page
    return file:write('data/'|| $page/@key || '.xml', $page,
map{"omit-xml-declaration":"no"})

1) There'll be a new version of the input file arriving periodically, so I'd prefer to be able to do everything at the command line without having to create the database inside the GUI. Not sure of the best way to go about that. 2) In the GUI for now, I can perform the replace with the map, or I can run the for loop that writes out all the files. But I get an "Unexpected end of query" error when I try to do both, so what's the problem with my syntax? Thanks so much, Michael

Quoted text · Christian Grün · 284 lines

On Wed, May 25, 2016 at 11:09 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Michael, > > > Now for basex.bat, in order to create a context, I started the script > with > > 'declare context item := doc("input.xml");' which may not be the most > > efficient way to do this, I don't know. > > If you have created a database, you can use the command-line flag -i: > > basex.bat -i input query.xq > > The db:open function can be used as well: > > declare context item := db:open("input"); > > Is it only the query with the map constructor that requires more memory? > > Best, > Christian > > > But on the command line or in the > > GUI, I haven't had any luck. > > > > Any other suggestions? > > > > Thanks, > > > > Michael > > > > > > > > On Tue, May 24, 2016 at 10:49 PM, Christian Grün < > christian.gruen at gmail.com> > > wrote: > >> > >> Usually, 8GB should be much more than sufficient for such a query. You > >> could try to increase the memory, which is assigned to Java, in the > >> start scripts [1]. > >> > >> Does this help? > >> Christian > >> > >> [1] http://docs.basex.org/wiki/Start_Scripts > >> > >> > >> > >> On Tue, May 24, 2016 at 11:52 PM, Michael Sanborn <galethog at gmail.com> > >> wrote: > >> > Seems like this would be perfect. I do need both number and manuf. > Using > >> > your combination map, I'm now getting an "Out of Main Memory" error. > >> > Tried > >> > on a second computer - same issue. Would it be more likely to work if > I > >> > tried it from the command line rather than the GUI? If so, I'll need > to > >> > look > >> > up how to create a database that way, but I'm sure it's close to hand. > >> > Or is > >> > there a better workaround (besides buying a computer with more than > 8GB > >> > of > >> > RAM)? > >> > > >> > Thanks again, > >> > > >> > Michael > >> > > >> > On Tue, May 24, 2016 at 2:10 PM, Christian Grün > >> > <christian.gruen at gmail.com> > >> > wrote: > >> >> > >> >> Maybe you need something like this: > >> >> > >> >> for $partinfo in //unit/partinfo > >> >> for $part in //part[deep-equal(partinfo, $partinfo)] > >> >> return replace node $partinfo with $part/node() > >> >> > >> >> The deep-equal will be pretty slow. If the value of the number > element > >> >> is unique, you could do something like this: > >> >> > >> >> for $partinfo in //unit/partinfo > >> >> let $number := $partinfo/number > >> >> let $part := //part[partinfo/number, $number] > >> >> return replace node $partinfo with $part/node() > >> >> > >> >> Using a map will even be faster: > >> >> > >> >> let $map := map:merge(//part/map:entry(partinfo/number/text(), .)) > >> >> for $partinfo in //unit/partinfo > >> >> let $part := $map($partinfo/number) > >> >> return replace node $partinfo with $part/node() > >> >> > >> >> If you need to consider both number and manuf, you could e.g. combine > >> >> these two in the map: > >> >> > >> >> let $map := map:merge( > >> >> for $part in //part > >> >> return map:entry(string-join($part/partinfo/*, '/'), $part) > >> >> ) > >> >> for $partinfo in //unit/partinfo > >> >> let $part := $map(string-join($partinfo/*, '/')) > >> >> return replace node $partinfo with $part/node() > >> >> > >> >> Does this help? > >> >> Christian > >> >> > >> >> > >> >> > >> >> > >> >> On Tue, May 24, 2016 at 10:54 PM, Michael Sanborn < > galethog at gmail.com> > >> >> wrote: > >> >> > Thanks for that. The trouble in step 2 is, just wrapping partinfo > >> >> > with > >> >> > the > >> >> > part element doesn't get me what I've labelled "misc part content > 1" > >> >> > and > >> >> > "misc part content 2". It's not sufficient to have just the tags - > I > >> >> > need > >> >> > all the content of the corresponding part elements in the later > part > >> >> > of > >> >> > the > >> >> > file. Is that something that can be done without too much > difficulty? > >> >> > > >> >> > Thanks, > >> >> > > >> >> > Michael > >> >> > > >> >> > On Tue, May 24, 2016 at 12:16 PM, Christian Grün > >> >> > <christian.gruen at gmail.com> > >> >> > wrote: > >> >> >> > >> >> >> Hi Michael, > >> >> >> > >> >> >> Yes, this can easily be done with XQuery. There are many ways to > do > >> >> >> this; here is just one: > >> >> >> > >> >> >> 1. First, create a database from your input file (e.g. with the > >> >> >> BaseX > >> >> >> GUI) > >> >> >> > >> >> >> 2. Second, run the following query to replace wrap your partinfo > >> >> >> elements with part elements: > >> >> >> > >> >> >> //unit/partinfo/(replace node . with <part>{ . }</part>) > >> >> >> > >> >> >> 3. Third, write all page elements to disk: > >> >> >> > >> >> >> for $page at $c in //page > >> >> >> return file:write($c || '.xml', $page) > >> >> >> > >> >> >> Hope this helps, > >> >> >> Christian > >> >> >> > >> >> >> > >> >> >> > >> >> >> On Tue, May 24, 2016 at 8:54 PM, Michael Sanborn > >> >> >> <galethog at gmail.com> > >> >> >> wrote: > >> >> >> > I need to perform a transformation that would be simple in XSLT, > >> >> >> > but > >> >> >> > the > >> >> >> > input is a file about 250 MBs in size. I'm wondering whether > >> >> >> > XQuery > >> >> >> > and > >> >> >> > BaseX in particular would be the most efficient way of doing it. > >> >> >> > I'm > >> >> >> > new > >> >> >> > to > >> >> >> > XQuery, and I've come up with a couple of ways to do this, but > >> >> >> > they > >> >> >> > turn > >> >> >> > out > >> >> >> > to be very time-consuming, so I'm sure I'm Doing It Wrong. > Hoping > >> >> >> > to > >> >> >> > find > >> >> >> > out the proper way of doing this. > >> >> >> > > >> >> >> > The input consists of 2 sections. There are about 3600 page > >> >> >> > elements > >> >> >> > with > >> >> >> > this structure: > >> >> >> > > >> >> >> > <page> > >> >> >> > [misc page content...] > >> >> >> > <list> > >> >> >> > <unit> > >> >> >> > [misc unit content 1...] > >> >> >> > <partinfo> > >> >> >> > <number>54321</number> > >> >> >> > <manuf>A321</manuf> > >> >> >> > </partinfo> > >> >> >> > <partinfo> > >> >> >> > <number>12345</number> > >> >> >> > <manuf>B123</manuf> > >> >> >> > </partinfo> > >> >> >> > [misc unit content 2...] > >> >> >> > </unit> > >> >> >> > [multiple units...] > >> >> >> > </list> > >> >> >> > </page> > >> >> >> > > >> >> >> > Each unit can have 1 or 2 partinfo elements. The other section > has > >> >> >> > about > >> >> >> > 82000 part elements like this: > >> >> >> > > >> >> >> > <part> > >> >> >> > <partinfo> > >> >> >> > <number>54321</number> > >> >> >> > <manuf>A321</manuf> > >> >> >> > </partinfo> > >> >> >> > [misc part content 1] > >> >> >> > </part> > >> >> >> > [...] > >> >> >> > <part> > >> >> >> > <partinfo> > >> >> >> > <number>12345</number> > >> >> >> > <manuf>B123</manuf> > >> >> >> > </partinfo> > >> >> >> > [misc part content 2] > >> >> >> > </part> > >> >> >> > > >> >> >> > I want to replace each unit/partinfo with the correpsonding > part, > >> >> >> > like > >> >> >> > this: > >> >> >> > > >> >> >> > <page> > >> >> >> > [misc page content...] > >> >> >> > <list> > >> >> >> > <unit> > >> >> >> > [misc unit content 1...] > >> >> >> > <part> > >> >> >> > <partinfo> > >> >> >> > <number>54321</number> > >> >> >> > <manuf>A321</manuf> > >> >> >> > </partinfo> > >> >> >> > [misc part content 1] > >> >> >> > </part> > >> >> >> > <part> > >> >> >> > <partinfo> > >> >> >> > <number>12345</number> > >> >> >> > <manuf>B123</manuf> > >> >> >> > </partinfo> > >> >> >> > [misc part content 2] > >> >> >> > </part> > >> >> >> > [misc unit content 2...] > >> >> >> > </unit> > >> >> >> > [multiple units...] > >> >> >> > </list> > >> >> >> > </page> > >> >> >> > > >> >> >> > Is BaseX a good tool for this task? If so, how does one go about > >> >> >> > it? > >> >> >> > > >> >> >> > Finally, it would help to be able to output each page element > in a > >> >> >> > separate > >> >> >> > file. Would it be better to have BaseX do this, or to output the > >> >> >> > whole > >> >> >> > database and chunk it with another tool? > >> >> >> > > >> >> >> > Thanks, > >> >> >> > > >> >> >> > Michael > >> >> > > >> >> > > >> > > >> > > > > > >

Christian Grün27 May 2016 08:21 #

Hi Michael,

Quoted text · Michael Sanborn · 4 lines

> 1) There'll be a new version of the input file arriving periodically, so I'd > prefer to be able to do everything at the command line without having to > create the database inside the GUI. Not sure of the best way to go about > that.

Using the GUI was just an example. You can create new databases via commands (CREATE), XQuery (db:create) or our APIs. Please check out our documentation for more hints.

Quoted text · Michael Sanborn · 3 lines

> 2) In the GUI for now, I can perform the replace with the map, or I can run > the for loop that writes out all the files. But I get an "Unexpected end of > query" error when I try to do both, so what's the problem with my syntax?

In XQuery, multiple expressions can be separated with commas. Note, however, that XQuery is a functional language; as such, it is not possible to first update items and then access them in the same query that easily. There are various alternatives to get around this limitation: 1. Use a BaseX command script to run all operations [1]:

  <commands>
    <create-db name='input'>...path/to/input.xml</create-db>
    <xquery><![CDATA[
      let $map := ...
    ]]></xquery>
    <xquery>
      for $page in ...
    <xquery>
  </commands>

2. Use copy/transform/return or update to do all updates in main-memory [2] and pass them on to your file:write function. This could e.g. look as follows:

  let $input := doc('input.xml')
  let $map := map:merge(
    for $part in $input//part
    return map:entry(string-join($part/partinfo/*, '/'), $part)
  )
  for $page in $input//page
  let $new-part := $page update {
    for $partinfo in .//unit/partinfo
    let $part := $map(string-join($partinfo/*, '/'))
    return replace node $partinfo with element part { $part/node() }
  }
  return file:write('data/'|| $page/@key || '.xml', $page,
    map{ "omit-xml-declaration": "no" })

The good thing here is that your replace operations won’t need to be cached until the very end; they will directly be run on each page element (and not persisted on disk; but this is something you don’t need anyway it seems) Christian [1] http://docs.basex.org/wiki/Commands#Command_Scripts [2] http://docs.basex.org/wiki/XQuery_Update#Non-Updating_Expressions

Michael Sanborn30 May 2016 23:07 #

Your example 2 does the job, thank you! Although I had to change the second-to-last line from return file:write('data/'|| $page/@key || '.xml', $page, to return file:write('data/'|| $page/@key || '.xml', $new-part, One last thing: if I want to specify the path to the input file on the command line, how do I then refer to it in the script? From the Command-Line Options page, it seems I can say -i/my/path/to/input.xml

But what do I use in place of $input in that case? If I just remove
"$input" from the script I get a "no context value bound" error if I say

-i/my/path/to/input.xml or -i/my/path/to/input.xml "/" Thanks, Michael

Quoted text · Christian Grün · 67 lines

On Thu, May 26, 2016 at 11:21 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Michael, > > > 1) There'll be a new version of the input file arriving periodically, so > I'd > > prefer to be able to do everything at the command line without having to > > create the database inside the GUI. Not sure of the best way to go about > > that. > > Using the GUI was just an example. You can create new databases via > commands (CREATE), XQuery (db:create) or our APIs. Please check out > our documentation for more hints. > > > 2) In the GUI for now, I can perform the replace with the map, or I can > run > > the for loop that writes out all the files. But I get an "Unexpected end > of > > query" error when I try to do both, so what's the problem with my syntax? > > In XQuery, multiple expressions can be separated with commas. > > Note, however, that XQuery is a functional language; as such, it is > not possible to first update items and then access them in the same > query that easily. There are various alternatives to get around this > limitation: > > 1. Use a BaseX command script to run all operations [1]: > > <commands> > <create-db name='input'>...path/to/input.xml</create-db> > <xquery><![CDATA[ > let $map := ... > ]]></xquery> > <xquery> > for $page in ... > <xquery> > </commands> > > 2. Use copy/transform/return or update to do all updates in > main-memory [2] and pass them on to your file:write function. This > could e.g. look as follows: > > let $input := doc('input.xml') > let $map := map:merge( > for $part in $input//part > return map:entry(string-join($part/partinfo/*, '/'), $part) > ) > for $page in $input//page > let $new-part := $page update { > for $partinfo in .//unit/partinfo > let $part := $map(string-join($partinfo/*, '/')) > return replace node $partinfo with element part { $part/node() } > } > return file:write('data/'|| $page/@key || '.xml', $page, > map{ "omit-xml-declaration": "no" }) > > The good thing here is that your replace operations won’t need to be > cached until the very end; they will directly be run on each page > element (and not persisted on disk; but this is something you don’t > need anyway it seems) > > Christian > > [1] http://docs.basex.org/wiki/Commands#Command_Scripts > [2] http://docs.basex.org/wiki/XQuery_Update#Non-Updating_Expressions >

Christian Grün30 May 2016 23:20 #

Quoted text · Michael Sanborn · 4 lines

> If I just remove "$input" > from the script I get a "no context value bound" error if I say > > -i/my/path/to/input.xml

It should work as described (I frequently use it by myself). Could you try again, or provide me with a little self-contained example? Cheers, Christian

Quoted text · Christian Grün · 70 lines

> On Thu, May 26, 2016 at 11:21 PM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> Hi Michael, >> >> > 1) There'll be a new version of the input file arriving periodically, so >> > I'd >> > prefer to be able to do everything at the command line without having to >> > create the database inside the GUI. Not sure of the best way to go about >> > that. >> >> Using the GUI was just an example. You can create new databases via >> commands (CREATE), XQuery (db:create) or our APIs. Please check out >> our documentation for more hints. >> >> > 2) In the GUI for now, I can perform the replace with the map, or I can >> > run >> > the for loop that writes out all the files. But I get an "Unexpected end >> > of >> > query" error when I try to do both, so what's the problem with my >> > syntax? >> >> In XQuery, multiple expressions can be separated with commas. >> >> Note, however, that XQuery is a functional language; as such, it is >> not possible to first update items and then access them in the same >> query that easily. There are various alternatives to get around this >> limitation: >> >> 1. Use a BaseX command script to run all operations [1]: >> >> <commands> >> <create-db name='input'>...path/to/input.xml</create-db> >> <xquery><![CDATA[ >> let $map := ... >> ]]></xquery> >> <xquery> >> for $page in ... >> <xquery> >> </commands> >> >> 2. Use copy/transform/return or update to do all updates in >> main-memory [2] and pass them on to your file:write function. This >> could e.g. look as follows: >> >> let $input := doc('input.xml') >> let $map := map:merge( >> for $part in $input//part >> return map:entry(string-join($part/partinfo/*, '/'), $part) >> ) >> for $page in $input//page >> let $new-part := $page update { >> for $partinfo in .//unit/partinfo >> let $part := $map(string-join($partinfo/*, '/')) >> return replace node $partinfo with element part { $part/node() } >> } >> return file:write('data/'|| $page/@key || '.xml', $page, >> map{ "omit-xml-declaration": "no" }) >> >> The good thing here is that your replace operations won’t need to be >> cached until the very end; they will directly be run on each page >> element (and not persisted on disk; but this is something you don’t >> need anyway it seems) >> >> Christian >> >> [1] http://docs.basex.org/wiki/Commands#Command_Scripts >> [2] http://docs.basex.org/wiki/XQuery_Update#Non-Updating_Expressions > >

Christian Grünin reply to Michael Sanborn, 31 May 2016, 03:4031 May 2016 01:42 #

Quoted text · Michael Sanborn · 1 line

> "\Program Files (x86)\BaseX\bin\basex" query.xq -iC:/data/input.xml

This should work:

Quoted text · Michael Sanborn · 1 line

> "\Program Files (x86)\BaseX\bin\basex" -iC:/data/input.xml query.xq

In BaseX, the order of arguments is important.

Quoted text · Michael Sanborn · 111 lines

> I get in response: > > Stopped at C:/scripts/query.xq, 2/19: > [XPDY0002] root(): no context value bound. > > It's the same whether I say > > -iC:/data/input.xml > -i/data/input.xml > -iC:\data\input.xml > -i\data\input.xml > > Thanks, > > Michael > > > On Mon, May 30, 2016 at 2:20 PM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> > If I just remove "$input" >> > from the script I get a "no context value bound" error if I say >> > >> > -i/my/path/to/input.xml >> >> It should work as described (I frequently use it by myself). Could you >> try again, or provide me with a little self-contained example? >> >> Cheers, >> Christian >> >> >> >> > On Thu, May 26, 2016 at 11:21 PM, Christian Grün >> > <christian.gruen at gmail.com> >> > wrote: >> >> >> >> Hi Michael, >> >> >> >> > 1) There'll be a new version of the input file arriving periodically, >> >> > so >> >> > I'd >> >> > prefer to be able to do everything at the command line without having >> >> > to >> >> > create the database inside the GUI. Not sure of the best way to go >> >> > about >> >> > that. >> >> >> >> Using the GUI was just an example. You can create new databases via >> >> commands (CREATE), XQuery (db:create) or our APIs. Please check out >> >> our documentation for more hints. >> >> >> >> > 2) In the GUI for now, I can perform the replace with the map, or I >> >> > can >> >> > run >> >> > the for loop that writes out all the files. But I get an "Unexpected >> >> > end >> >> > of >> >> > query" error when I try to do both, so what's the problem with my >> >> > syntax? >> >> >> >> In XQuery, multiple expressions can be separated with commas. >> >> >> >> Note, however, that XQuery is a functional language; as such, it is >> >> not possible to first update items and then access them in the same >> >> query that easily. There are various alternatives to get around this >> >> limitation: >> >> >> >> 1. Use a BaseX command script to run all operations [1]: >> >> >> >> <commands> >> >> <create-db name='input'>...path/to/input.xml</create-db> >> >> <xquery><![CDATA[ >> >> let $map := ... >> >> ]]></xquery> >> >> <xquery> >> >> for $page in ... >> >> <xquery> >> >> </commands> >> >> >> >> 2. Use copy/transform/return or update to do all updates in >> >> main-memory [2] and pass them on to your file:write function. This >> >> could e.g. look as follows: >> >> >> >> let $input := doc('input.xml') >> >> let $map := map:merge( >> >> for $part in $input//part >> >> return map:entry(string-join($part/partinfo/*, '/'), $part) >> >> ) >> >> for $page in $input//page >> >> let $new-part := $page update { >> >> for $partinfo in .//unit/partinfo >> >> let $part := $map(string-join($partinfo/*, '/')) >> >> return replace node $partinfo with element part { $part/node() } >> >> } >> >> return file:write('data/'|| $page/@key || '.xml', $page, >> >> map{ "omit-xml-declaration": "no" }) >> >> >> >> The good thing here is that your replace operations won’t need to be >> >> cached until the very end; they will directly be run on each page >> >> element (and not persisted on disk; but this is something you don’t >> >> need anyway it seems) >> >> >> >> Christian >> >> >> >> [1] http://docs.basex.org/wiki/Commands#Command_Scripts >> >> [2] http://docs.basex.org/wiki/XQuery_Update#Non-Updating_Expressions >> > >> > > >

Michael Sanbornin reply to Christian Grün, 30 May 2016, 23:2031 May 2016 03:40 #

I've tried with a number of variations. As far as a self-contained example, I'm not exactly sure what you're looking for, but suppose I have in C:\data\input.xml:

<?xml version="1.0" encoding="UTF-8"?>
<input>
<pages>
<page>
<lbl>Sample Page</lbl>
<list>
<unlt>
<lbl>Summer Unit</lbl>
<partinfo>
<number>54321</number>
<manuf>A321</manuf>
</partinfo>
<partinfo>
<number>12345</number>
<manuf>B123</manuf>
</partinfo>
<offer>25% off!</offer>
</unlt>
</list>
</page>
</pages>
<parts>
<part>
<partinfo>
<number>54321</number>
<manuf>A321</manuf>
</partinfo>
<color>Orange</color>
</part>
<part>
<partinfo>
<number>12345</number>
<manuf>B123</manuf>
</partinfo>
<color>Pink</color>
</part>
</parts>
</input>

And suppose I have in C:\scripts\query.xq:

let $map := map:merge(
    for $part in //part
    return map:entry(string-join($part/partinfo/*, '/'), $part)
)
for $page in //page
let $new-part := $page update {
    for $partinfo in .//unit/partinfo
    let $part := $map(string-join($partinfo/*, '/'))
    return replace node $partinfo with element part {$part/node()}
}
return file:write('/data/'|| $page/@key || '.xml', $new-part,
    map{"omit-xml-declaration":"no"})

And suppose my current directory is C:\scripts. If I launch the command: "\Program Files (x86)\BaseX\bin\basex" query.xq -iC:/data/input.xml I get in response: Stopped at C:/scripts/query.xq, 2/19: [XPDY0002] root(): no context value bound. It's the same whether I say -iC:/data/input.xml -i/data/input.xml -iC:\data\input.xml -i\data\input.xml Thanks, Michael

Quoted text · Christian Grün · 92 lines

On Mon, May 30, 2016 at 2:20 PM, Christian Grün <christian.gruen at gmail.com> wrote: > > If I just remove "$input" > > from the script I get a "no context value bound" error if I say > > > > -i/my/path/to/input.xml > > It should work as described (I frequently use it by myself). Could you > try again, or provide me with a little self-contained example? > > Cheers, > Christian > > > > > On Thu, May 26, 2016 at 11:21 PM, Christian Grün < > christian.gruen at gmail.com> > > wrote: > >> > >> Hi Michael, > >> > >> > 1) There'll be a new version of the input file arriving periodically, > so > >> > I'd > >> > prefer to be able to do everything at the command line without having > to > >> > create the database inside the GUI. Not sure of the best way to go > about > >> > that. > >> > >> Using the GUI was just an example. You can create new databases via > >> commands (CREATE), XQuery (db:create) or our APIs. Please check out > >> our documentation for more hints. > >> > >> > 2) In the GUI for now, I can perform the replace with the map, or I > can > >> > run > >> > the for loop that writes out all the files. But I get an "Unexpected > end > >> > of > >> > query" error when I try to do both, so what's the problem with my > >> > syntax? > >> > >> In XQuery, multiple expressions can be separated with commas. > >> > >> Note, however, that XQuery is a functional language; as such, it is > >> not possible to first update items and then access them in the same > >> query that easily. There are various alternatives to get around this > >> limitation: > >> > >> 1. Use a BaseX command script to run all operations [1]: > >> > >> <commands> > >> <create-db name='input'>...path/to/input.xml</create-db> > >> <xquery><![CDATA[ > >> let $map := ... > >> ]]></xquery> > >> <xquery> > >> for $page in ... > >> <xquery> > >> </commands> > >> > >> 2. Use copy/transform/return or update to do all updates in > >> main-memory [2] and pass them on to your file:write function. This > >> could e.g. look as follows: > >> > >> let $input := doc('input.xml') > >> let $map := map:merge( > >> for $part in $input//part > >> return map:entry(string-join($part/partinfo/*, '/'), $part) > >> ) > >> for $page in $input//page > >> let $new-part := $page update { > >> for $partinfo in .//unit/partinfo > >> let $part := $map(string-join($partinfo/*, '/')) > >> return replace node $partinfo with element part { $part/node() } > >> } > >> return file:write('data/'|| $page/@key || '.xml', $page, > >> map{ "omit-xml-declaration": "no" }) > >> > >> The good thing here is that your replace operations won’t need to be > >> cached until the very end; they will directly be run on each page > >> element (and not persisted on disk; but this is something you don’t > >> need anyway it seems) > >> > >> Christian > >> > >> [1] http://docs.basex.org/wiki/Commands#Command_Scripts > >> [2] http://docs.basex.org/wiki/XQuery_Update#Non-Updating_Expressions > > > > >

Michael Sanbornin reply to Christian Grün, 31 May 2016, 01:4231 May 2016 04:05 #

Indeed it does! Christian, thank you so much for all your help. I've got everything I need, not only for this particular task, but I can probably extend what you've shown me to other related issues where BaseX can prove invaluable for saving time in my work processes. This has been a thoroughly satisfying experience. Best regards, Michael

Quoted text · Christian Grün · 129 lines

On Mon, May 30, 2016 at 4:42 PM, Christian Grün <christian.gruen at gmail.com> wrote: > > "\Program Files (x86)\BaseX\bin\basex" query.xq -iC:/data/input.xml > > This should work: > > > "\Program Files (x86)\BaseX\bin\basex" -iC:/data/input.xml query.xq > > In BaseX, the order of arguments is important. > > > > > I get in response: > > > > Stopped at C:/scripts/query.xq, 2/19: > > [XPDY0002] root(): no context value bound. > > > > It's the same whether I say > > > > -iC:/data/input.xml > > -i/data/input.xml > > -iC:\data\input.xml > > -i\data\input.xml > > > > Thanks, > > > > Michael > > > > > > On Mon, May 30, 2016 at 2:20 PM, Christian Grün < > christian.gruen at gmail.com> > > wrote: > >> > >> > If I just remove "$input" > >> > from the script I get a "no context value bound" error if I say > >> > > >> > -i/my/path/to/input.xml > >> > >> It should work as described (I frequently use it by myself). Could you > >> try again, or provide me with a little self-contained example? > >> > >> Cheers, > >> Christian > >> > >> > >> > >> > On Thu, May 26, 2016 at 11:21 PM, Christian Grün > >> > <christian.gruen at gmail.com> > >> > wrote: > >> >> > >> >> Hi Michael, > >> >> > >> >> > 1) There'll be a new version of the input file arriving > periodically, > >> >> > so > >> >> > I'd > >> >> > prefer to be able to do everything at the command line without > having > >> >> > to > >> >> > create the database inside the GUI. Not sure of the best way to go > >> >> > about > >> >> > that. > >> >> > >> >> Using the GUI was just an example. You can create new databases via > >> >> commands (CREATE), XQuery (db:create) or our APIs. Please check out > >> >> our documentation for more hints. > >> >> > >> >> > 2) In the GUI for now, I can perform the replace with the map, or I > >> >> > can > >> >> > run > >> >> > the for loop that writes out all the files. But I get an > "Unexpected > >> >> > end > >> >> > of > >> >> > query" error when I try to do both, so what's the problem with my > >> >> > syntax? > >> >> > >> >> In XQuery, multiple expressions can be separated with commas. > >> >> > >> >> Note, however, that XQuery is a functional language; as such, it is > >> >> not possible to first update items and then access them in the same > >> >> query that easily. There are various alternatives to get around this > >> >> limitation: > >> >> > >> >> 1. Use a BaseX command script to run all operations [1]: > >> >> > >> >> <commands> > >> >> <create-db name='input'>...path/to/input.xml</create-db> > >> >> <xquery><![CDATA[ > >> >> let $map := ... > >> >> ]]></xquery> > >> >> <xquery> > >> >> for $page in ... > >> >> <xquery> > >> >> </commands> > >> >> > >> >> 2. Use copy/transform/return or update to do all updates in > >> >> main-memory [2] and pass them on to your file:write function. This > >> >> could e.g. look as follows: > >> >> > >> >> let $input := doc('input.xml') > >> >> let $map := map:merge( > >> >> for $part in $input//part > >> >> return map:entry(string-join($part/partinfo/*, '/'), $part) > >> >> ) > >> >> for $page in $input//page > >> >> let $new-part := $page update { > >> >> for $partinfo in .//unit/partinfo > >> >> let $part := $map(string-join($partinfo/*, '/')) > >> >> return replace node $partinfo with element part { $part/node() } > >> >> } > >> >> return file:write('data/'|| $page/@key || '.xml', $page, > >> >> map{ "omit-xml-declaration": "no" }) > >> >> > >> >> The good thing here is that your replace operations won’t need to be > >> >> cached until the very end; they will directly be run on each page > >> >> element (and not persisted on disk; but this is something you don’t > >> >> need anyway it seems) > >> >> > >> >> Christian > >> >> > >> >> [1] http://docs.basex.org/wiki/Commands#Command_Scripts > >> >> [2] > http://docs.basex.org/wiki/XQuery_Update#Non-Updating_Expressions > >> > > >> > > > > > >

Written for BaseX 8.4. BaseX may have changed since: please check the current documentation.

⚡Generated with XQuery

`J`	Select the next message
`K`	Select the previous message
`U`	Go up one level
`/`	Jump to the search field