processing large file and saving results in GUI

Sunday, 15 August 2010 · 4 messages
GD
Gérard Dupont15 Aug 2010 19:27 #
Hi all I'm trying to process some large XML file (~6.5Go). It goes quite fine thanks to the stateless approach of Xquery. But then when I want to save the results, I face some difficulty. The process freeze for a time (few minutes) but the final file is empty and I got no error on the console. The results it self should be quite small (~50 to 70Mo max) but I can't get it saved... That's a pity since the process has been achieved (note that I'm using the GUI). Is there any way to specify in the query to save the result in a file ? Should I use the CLI ? Any advices are welcome, I just start with BaseX. cheers
-- Gérard Dupont Information Processing Control and Cognition (IPCC) - EADS DS http://weblab.ow2.org Document & Learning team - LITIS Laboratory
AW
Andreas Weiler15 Aug 2010 19:34 #
Hi Gérard, thanks for your report. As a quick workaround you can try the following: java -cp BaseX61.jar org.basex.BaseX -o outputFile.txt -q query with this command the result of your query will be executed with the console version of BaseX and the result will be saved to the outputFile.txt. We will have a look at your reported error soon. I hope this helps, dont hesitate to ask for more. Kind regards, Andreas
Quoted text · Gérard Dupont · 30 lines
Gérard Dupont schrieb: > Hi all > > I'm trying to process some large XML file (~6.5Go). It goes quite fine > thanks to the stateless approach of Xquery. But then when I want to > save the results, I face some difficulty. The process freeze for a > time (few minutes) but the final file is empty and I got no error on > the console. The results it self should be quite small (~50 to 70Mo > max) but I can't get it saved... That's a pity since the process has > been achieved (note that I'm using the GUI). Is there any way to > specify in the query to save the result in a file ? Should I use the CLI ? > > Any advices are welcome, I just start with BaseX. > > cheers > > > -- > Gérard Dupont > Information Processing Control and Cognition (IPCC) - EADS DS > http://weblab.ow2.org > > Document & Learning team - LITIS Laboratory > ------------------------------------------------------------------------ > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
GD
Gérard Dupont15 Aug 2010 19:59 #
Thanks for your reactivity. Some more information about my saving problem. The file is constructed like this :
<session>4 ; 2006-05-23T13:33:43.000+02:00 ;
2006-05-23T13:48:14.000+02:00</session>
<session>1 ; 2006-05-18T10:56:36.000+02:00 ;
2006-05-18T10:56:36.000+02:00</session>
<session>1 ; 2006-05-22T12:59:57.000+02:00 ;
2006-05-22T12:59:57.000+02:00</session>
<session>2 ; 2006-05-08T12:31:18.000+02:00 ;
2006-05-08T12:31:41.000+02:00</session>
<session>6 ; 2006-05-25T11:43:33.000+02:00 ;
2006-05-25T11:56:39.000+02:00</session>
...
And I should have roughly 7M lines (thus the estimate is more like 10Mo). I tested your idea from CLI but can't get BaseX to work with my database. It's certainly a syntax problem, but I can't find how to select a database in the Xquery. Here is what I tried :
for $session in doc('xmlSessions')/logSessionArgs/logSession
let $nbAction := count($session/search_log_entry)
let $start := data($session/search_log_entry[1]/@date)
let $end := data($session/search_log_entry[$nbAction]/@date)
return (
<session>{$nbAction} ; {$start} ; {$end}</session>
)
Given that 'xmlSessions' is the name of the database. I also tried "basex:db(..." and "collection(..." instead of 'doc(...' but it leads to the same error : [XPDY0002] No context item set for 'nbActionStartEndPerSession.xq'. Any clue ? 2010/8/15 Andreas Weiler <andreas.weiler at uni-konstanz.de>
Quoted text · Andreas Weiler · 52 lines
> Hi Gérard, > > thanks for your report. > As a quick workaround you can try the following: > java -cp BaseX61.jar org.basex.BaseX -o outputFile.txt -q query > > with this command the result of your query will be executed with the > console version of > BaseX and the result will be saved to the outputFile.txt. > > We will have a look at your reported error soon. > > I hope this helps, > dont hesitate to ask for more. > Kind regards, > Andreas > > > Gérard Dupont schrieb: > >> Hi all >> >> I'm trying to process some large XML file (~6.5Go). It goes quite fine >> thanks to the stateless approach of Xquery. But then when I want to save the >> results, I face some difficulty. The process freeze for a time (few minutes) >> but the final file is empty and I got no error on the console. The results >> it self should be quite small (~50 to 70Mo max) but I can't get it saved... >> That's a pity since the process has been achieved (note that I'm using the >> GUI). Is there any way to specify in the query to save the result in a file >> ? Should I use the CLI ? >> >> Any advices are welcome, I just start with BaseX. >> >> cheers >> >> >> -- >> Gérard Dupont >> Information Processing Control and Cognition (IPCC) - EADS DS >> http://weblab.ow2.org >> >> Document & Learning team - LITIS Laboratory >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >> > >
-- Gérard Dupont Information Processing Control and Cognition (IPCC) - EADS DS http://weblab.ow2.org Document & Learning team - LITIS Laboratory
AW
Andreas Weiler15 Aug 2010 20:15 #
ok i see you have defined your query in a file, so you have to call it like this:
java -cp basex.jar org.basex.BaseX -o outputFile.txt query.xq
Quoted text · Gérard Dupont · 103 lines
Gérard Dupont schrieb: > Thanks for your reactivity. > > Some more information about my saving problem. The file is constructed > like this : > > <session>4 ; 2006-05-23T13:33:43.000+02:00 ; > 2006-05-23T13:48:14.000+02:00</session> > <session>1 ; 2006-05-18T10:56:36.000+02:00 ; > 2006-05-18T10:56:36.000+02:00</session> > <session>1 ; 2006-05-22T12:59:57.000+02:00 ; > 2006-05-22T12:59:57.000+02:00</session> > <session>2 ; 2006-05-08T12:31:18.000+02:00 ; > 2006-05-08T12:31:41.000+02:00</session> > <session>6 ; 2006-05-25T11:43:33.000+02:00 ; > 2006-05-25T11:56:39.000+02:00</session> > ... > > And I should have roughly 7M lines (thus the estimate is more like > 10Mo). I tested your idea from CLI but can't get BaseX to work with my > database. It's certainly a syntax problem, but I can't find how to > select a database in the Xquery. Here is what I tried : > > for $session in doc('xmlSessions')/logSessionArgs/logSession > let $nbAction := count($session/search_log_entry) > let $start := data($session/search_log_entry[1]/@date) > let $end := data($session/search_log_entry[$nbAction]/@date) > return ( > <session>{$nbAction} ; {$start} ; {$end}</session> > ) > > Given that 'xmlSessions' is the name of the database. I also tried > "basex:db(..." and "collection(..." instead of 'doc(...' but it leads > to the same error : > > [XPDY0002] No context item set for 'nbActionStartEndPerSession.xq'. > > Any clue ? > 2010/8/15 Andreas Weiler <andreas.weiler at uni-konstanz.de > <mailto:andreas.weiler at uni-konstanz.de>> > > Hi Gérard, > > thanks for your report. > As a quick workaround you can try the following: > java -cp BaseX61.jar org.basex.BaseX -o outputFile.txt -q query > > with this command the result of your query will be executed with > the console version of > BaseX and the result will be saved to the outputFile.txt. > > We will have a look at your reported error soon. > > I hope this helps, > dont hesitate to ask for more. > Kind regards, > Andreas > > > Gérard Dupont schrieb: > > Hi all > > I'm trying to process some large XML file (~6.5Go). It goes > quite fine thanks to the stateless approach of Xquery. But > then when I want to save the results, I face some difficulty. > The process freeze for a time (few minutes) but the final file > is empty and I got no error on the console. The results it > self should be quite small (~50 to 70Mo max) but I can't get > it saved... That's a pity since the process has been achieved > (note that I'm using the GUI). Is there any way to specify in > the query to save the result in a file ? Should I use the CLI ? > > Any advices are welcome, I just start with BaseX. > > cheers > > > -- > Gérard Dupont > Information Processing Control and Cognition (IPCC) - EADS DS > http://weblab.ow2.org > > Document & Learning team - LITIS Laboratory > ------------------------------------------------------------------------ > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > <mailto:BaseX-Talk at mailman.uni-konstanz.de> > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > > > > -- > Gérard Dupont > Information Processing Control and Cognition (IPCC) - EADS DS > http://weblab.ow2.org > > Document & Learning team - LITIS Laboratory

⚡Generated with XQuery