Outputting large output
Monday, 21 September 2015 · 5 messages
B
buddyonweb-software21 Sep 2015 17:41 #
I am using BaseX to transform some an XML documents into another XML layout.When the incoming document gets large, run out of memory. I've bumped up memory in the JVM and that worked.
However, what if there was a really large document? Is there a way to redirect the outputso that instead of the results stored in memory they are sent to disk somewhere (whether to a fileon the file system or even back into BaseX - destination doesn't quite matter).
Thanks.
CG
Christian Grün21 Sep 2015 17:45 #
Once again, please provide us with more specific information, such as e.g. self-contained code snippets; otherwise we cannot help. There are simply too many ways to transform documents, query data, etc.
Quoted text · buddyonweb-software · 15 lines
On Mon, Sep 21, 2015 at 5:41 PM, <buddyonweb-software at yahoo.com> wrote: > I am using BaseX to transform some an XML documents into another XML layout. > When the incoming document gets large, run out of memory. I've bumped up > memory in the > JVM and that worked. > > However, what if there was a really large document? Is there a way to > redirect the output > so that instead of the results stored in memory they are sent to disk > somewhere (whether to a file > on the file system or even back into BaseX - destination doesn't quite > matter). > > Thanks. >
B
buddyonweb-software21 Sep 2015 17:58 #
Sure, let me provide some more details - and sorry for the original vagueness.
The main query that "kicks" the process off looks something like this (simplified a bit):
let $src := collection("someDataBase")[1] - illustrating with just 1 document
mynsp:mytransform(args.....)
The mynsp:mytransform function would then look something as follows
declare function spec:mytransform(args....) as element()*{<SomeTag>(
...code that then produces other sub elements. This is primarilyachieved by calling other functions similar to this one. For example.
spec:someTransform1(...),spec:someTransform2(...),etc.
)
<SomeTag>}Let me know if there is any other details you'd like or that I left out.Once again, I appreciate your efforts.
Quoted text · Christian Grün · 1 line
On Monday, September 21, 2015 11:45 AM, Christian Grün <christian.gruen at gmail.com> wrote:
Once again, please provide us with more specific information, such as e.g. self-contained code snippets; otherwise we cannot help. There are simply too many ways to transform documents, query data, etc.
Quoted text · Christian Grün · 15 lines
On Mon, Sep 21, 2015 at 5:41 PM, <buddyonweb-software at yahoo.com> wrote: > I am using BaseX to transform some an XML documents into another XML layout. > When the incoming document gets large, run out of memory. I've bumped up > memory in the > JVM and that worked. > > However, what if there was a really large document? Is there a way to > redirect the output > so that instead of the results stored in memory they are sent to disk > somewhere (whether to a file > on the file system or even back into BaseX - destination doesn't quite > matter). > > Thanks. >
CG
Christian Grün21 Sep 2015 18:05 #
Quoted text · buddyonweb-software · 4 lines
> let $src := collection("someDataBase")[1] - illustrating with just 1 > document > > mynsp:mytransform(args.....)
So args is an XQuery variable containing a document? What do you with the results? Do you output them as result, or do you store them somewhere?
Once again, please provide us something that's both complete and self-contained (this may help: http://sscce.org), or be more specific when it comes to the problem description. Take your time.
Quoted text · buddyonweb-software · 49 lines
> > The mynsp:mytransform function would then look something as follows > > declare function spec:mytransform(args....) as element()* > { > <SomeTag> > ( > ...code that then produces other sub elements. This is primarily > achieved by calling other functions similar to this one. For example. > > spec:someTransform1(...), > spec:someTransform2(...), > etc. > ) > <SomeTag> > } > > Let me know if there is any other details you'd like or that I left out. > Once again, I appreciate your efforts. > > > > On Monday, September 21, 2015 11:45 AM, Christian Grün > <christian.gruen at gmail.com> wrote: > > > Once again, please provide us with more specific information, such as > e.g. self-contained code snippets; otherwise we cannot help. There are > simply too many ways to transform documents, query data, etc. > > > On Mon, Sep 21, 2015 at 5:41 PM, <buddyonweb-software at yahoo.com> wrote: >> I am using BaseX to transform some an XML documents into another XML >> layout. >> When the incoming document gets large, run out of memory. I've bumped up >> memory in the >> JVM and that worked. >> >> However, what if there was a really large document? Is there a way to >> redirect the output >> so that instead of the results stored in memory they are sent to disk >> somewhere (whether to a file >> on the file system or even back into BaseX - destination doesn't quite >> matter). >> >> Thanks. >> > >
B
buddyonweb-software21 Sep 2015 18:14 #
The arg basically is a map that contains elements the other functions may need.So it can not only contain the root element, but any other sub elements. Since this map is beingpassed around, we are free to put in that map anything you suggest. Write now we just have the output being written to disk, but that is just because of where we are in the development right now.We could most certainly store it in BaseX as well. We are sort of open to options with respect to the storage. Let me know if there is any other details I can provide you with or that you may need.
Quoted text · Christian Grün · 5 lines
On Monday, September 21, 2015 12:06 PM, Christian Grün <christian.gruen at gmail.com> wrote: > let $src := collection("someDataBase")[1] - illustrating with just 1 > document > > mynsp:mytransform(args.....)
So args is an XQuery variable containing a document? What do you with the results? Do you output them as result, or do you store them somewhere?
Once again, please provide us something that's both complete and self-contained (this may help: http://sscce.org), or be more specific when it comes to the problem description. Take your time.
Quoted text · Christian Grün · 49 lines
> > The mynsp:mytransform function would then look something as follows > > declare function spec:mytransform(args....) as element()* > { > <SomeTag> > ( > ...code that then produces other sub elements. This is primarily > achieved by calling other functions similar to this one. For example. > > spec:someTransform1(...), > spec:someTransform2(...), > etc. > ) > <SomeTag> > } > > Let me know if there is any other details you'd like or that I left out. > Once again, I appreciate your efforts. > > > > On Monday, September 21, 2015 11:45 AM, Christian Grün > <christian.gruen at gmail.com> wrote: > > > Once again, please provide us with more specific information, such as > e.g. self-contained code snippets; otherwise we cannot help. There are > simply too many ways to transform documents, query data, etc. > > > On Mon, Sep 21, 2015 at 5:41 PM, <buddyonweb-software at yahoo.com> wrote: >> I am using BaseX to transform some an XML documents into another XML >> layout. >> When the incoming document gets large, run out of memory. I've bumped up >> memory in the >> JVM and that worked. >> >> However, what if there was a really large document? Is there a way to >> redirect the output >> so that instead of the results stored in memory they are sent to disk >> somewhere (whether to a file >> on the file system or even back into BaseX - destination doesn't quite >> matter). >> >> Thanks. >> > >
Written for BaseX 8.2. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery