Attempting to write bulk import of files is failing
Wednesday, 22 May 2019 · 9 messages · last reply 27 May 2019
JK
Joshua Kordani22 May 2019 19:40 #
https://pastebin.com/BEA6XiE5
I write the function in lib.xqm as a standalone query. All it does it add some attributes to each testcase before importing the content into the database. Forgive me if this is hard to read, it was my first attempt at making a function which updates some records and then imports in the same query.
I decided to break that query out into its own function, where I then call it from another query, import-and-process.xq, which works as expected.
I am now trying to write a bulk import, which is import-all.xq and this is not working. I receive a message about updates not being written back. My first attempt to write that function inside of a for-each also failed.
I've read all the documentation I can get my hands on regarding how to write the kinds of queries I need but there still seems to be gotchas I can't seem to see in advance. Any help would be greatly appreciated.
Joshua Kordani
DISCLAIMER: This email communication and any attached files are Speedcast Wireless proprietary and may be legally privileged. Export-controlled information shall not be disseminated without proper authorization and proper export-control markings, per Speedcast Wireless export policy. If you have received this transmission in error please notify the sender immediately and then delete this email and all its attachments. If you are not the addressee, any disclosure, reproduction, copying, distribution, or any other dissemination or use of this communication is strictly prohibited. Thank you.
CG
Christian Grün22 May 2019 20:06 #
Hi Joshua,
Quoted text · Joshua Kordani · 1 line
> I write the function in lib.xqm as a standalone query. All it does it add some attributes to each testcase before importing the content into the database. Forgive me if this is hard to read, it was my first attempt at making a function which updates some records and then imports in the same query.
Looks fine. Thanks for sending code that can be parsed out-of-the-box.
Quoted text · Joshua Kordani · 1 line
> I am now trying to write a bulk import, which is import-all.xq and this is not working. I receive a message about updates not being written back. My first attempt to write that function inside of a for-each also failed.
This warning is only shown on command line. It simply indicates that your updates won’t be propagated back to the original files.
If that’s what you would like to achieve, you can use the command-line argument -u (see [1]). If something else is not working as expected, feel free to provide us with more details.
Best,
Christian
[1] http://docs.basex.org/wiki/Options#WRITEBACK
JK
Joshua Kordani22 May 2019 20:12 #
CC group, my mistake.
Quoted text · Christian Grün · 24 lines
-----Original Message----- From: Joshua Kordani Sent: Wednesday, May 22, 2019 14:12 PM To: Christian Grün <christian.gruen at gmail.com> Subject: RE: [basex-talk] Attempting to write bulk import of files is failing Christian, Thank you for your reply. I don't understand why import-all.xq would throw this warning and import-and-process.xq would not. In the case of the latter, the nodes imported with that call are in fact updated properly, while in the former, no updates are seen. Can you tell me why -u would change the behavior of the one and not the other, or in other words, what is different about the two queries that makes this distinction important? Josh -----Original Message----- From: Christian Grün <christian.gruen at gmail.com> Sent: Wednesday, May 22, 2019 14:07 PM To: Joshua Kordani <joshua.kordani at speedcastwireless.com> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Attempting to write bulk import of files is failing Hi Joshua, > I write the function in lib.xqm as a standalone query. All it does it add some attributes to each testcase before importing the content into the database. Forgive me if this is hard to read, it was my first attempt at making a function which updates some records and then imports in the same query. Looks fine. Thanks for sending code that can be parsed out-of-the-box. > I am now trying to write a bulk import, which is import-all.xq and this is not working. I receive a message about updates not being written back. My first attempt to write that function inside of a for-each also failed. This warning is only shown on command line. It simply indicates that your updates won’t be propagated back to the original files. If that’s what you would like to achieve, you can use the command-line argument -u (see [1]). If something else is not working as expected, feel free to provide us with more details. Best, Christian [1] http://docs.basex.org/wiki/Options#WRITEBACK DISCLAIMER: This email communication and any attached files are Speedcast Wireless proprietary and may be legally privileged. Export-controlled information shall not be disseminated without proper authorization and proper export-control markings, per Speedcast Wireless export policy. If you have received this transmission in error please notify the sender immediately and then delete this email and all its attachments. If you are not the addressee, any disclosure, reproduction, copying, distribution, or any other dissemination or use of this communication is strictly prohibited. Thank you.
CG
Christian Grün22 May 2019 20:22 #
Quoted text · Joshua Kordani · 1 line
> Thank you for your reply. I don't understand why import-all.xq would throw this warning and import-and-process.xq would not. In the case of the latter, the nodes imported with that call are in fact updated properly, while in the former, no updates are seen.
To be sure: You mean that…
• import-all.xq throws the warning, but peforms no updates, and
• import-and-process.xq doesn’t raise a warning, but works as expected?
So there may be two questions to solve?
In order to reproduce this behavior, you might need to provide me with 1, 2 files that you are trying to import.
Quoted text · Joshua Kordani · 1 line
> Can you tell me why -u would change the behavior of the one and not the other, or in other words, what is different about the two queries that makes this distinction important?
The -u argument should actually work consistently for all files.
Quoted text · Joshua Kordani · 25 lines
> -----Original Message----- > From: Christian Grün <christian.gruen at gmail.com> > Sent: Wednesday, May 22, 2019 14:07 PM > To: Joshua Kordani <joshua.kordani at speedcastwireless.com> > Cc: BaseX <basex-talk at mailman.uni-konstanz.de> > Subject: Re: [basex-talk] Attempting to write bulk import of files is failing > > Hi Joshua, > > > I write the function in lib.xqm as a standalone query. All it does it add some attributes to each testcase before importing the content into the database. Forgive me if this is hard to read, it was my first attempt at making a function which updates some records and then imports in the same query. > > Looks fine. Thanks for sending code that can be parsed out-of-the-box. > > > I am now trying to write a bulk import, which is import-all.xq and this is not working. I receive a message about updates not being written back. My first attempt to write that function inside of a for-each also failed. > > This warning is only shown on command line. It simply indicates that your updates won’t be propagated back to the original files. > > If that’s what you would like to achieve, you can use the command-line argument -u (see [1]). If something else is not working as expected, feel free to provide us with more details. > > Best, > Christian > > [1] http://docs.basex.org/wiki/Options#WRITEBACK > > DISCLAIMER: This email communication and any attached files are Speedcast Wireless proprietary and may be legally privileged. Export-controlled information shall not be disseminated without proper authorization and proper export-control markings, per Speedcast Wireless export policy. If you have received this transmission in error please notify the sender immediately and then delete this email and all its attachments. If you are not the addressee, any disclosure, reproduction, copying, distribution, or any other dissemination or use of this communication is strictly prohibited. Thank you.
JK
Joshua Kordani22 May 2019 20:25 #
Attached are two sample files.
Correct, in that import-all.xq throws the warning and performs no updates, and import and process doesn't raise a warning but works as expected.
Along those same lines, the -i flag seems to provide the database context for command line queries, but I was unable to figure out how to leverage this inside lib.xqm in order to avoid hardcoding the database name.
Josh
Quoted text · Christian Grün · 40 lines
-----Original Message----- From: Christian Grün <christian.gruen at gmail.com> Sent: Wednesday, May 22, 2019 14:22 PM To: Joshua Kordani <joshua.kordani at speedcastwireless.com> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] FW: Attempting to write bulk import of files is failing > Thank you for your reply. I don't understand why import-all.xq would throw this warning and import-and-process.xq would not. In the case of the latter, the nodes imported with that call are in fact updated properly, while in the former, no updates are seen. To be sure: You mean that… • import-all.xq throws the warning, but peforms no updates, and • import-and-process.xq doesn’t raise a warning, but works as expected? So there may be two questions to solve? In order to reproduce this behavior, you might need to provide me with 1, 2 files that you are trying to import. > Can you tell me why -u would change the behavior of the one and not the other, or in other words, what is different about the two queries that makes this distinction important? The -u argument should actually work consistently for all files. > -----Original Message----- > From: Christian Grün <christian.gruen at gmail.com> > Sent: Wednesday, May 22, 2019 14:07 PM > To: Joshua Kordani <joshua.kordani at speedcastwireless.com> > Cc: BaseX <basex-talk at mailman.uni-konstanz.de> > Subject: Re: [basex-talk] Attempting to write bulk import of files is > failing > > Hi Joshua, > > > I write the function in lib.xqm as a standalone query. All it does it add some attributes to each testcase before importing the content into the database. Forgive me if this is hard to read, it was my first attempt at making a function which updates some records and then imports in the same query. > > Looks fine. Thanks for sending code that can be parsed out-of-the-box. > > > I am now trying to write a bulk import, which is import-all.xq and this is not working. I receive a message about updates not being written back. My first attempt to write that function inside of a for-each also failed. > > This warning is only shown on command line. It simply indicates that your updates won’t be propagated back to the original files. > > If that’s what you would like to achieve, you can use the command-line argument -u (see [1]). If something else is not working as expected, feel free to provide us with more details. > > Best, > Christian > > [1] http://docs.basex.org/wiki/Options#WRITEBACK > > DISCLAIMER: This email communication and any attached files are Speedcast Wireless proprietary and may be legally privileged. Export-controlled information shall not be disseminated without proper authorization and proper export-control markings, per Speedcast Wireless export policy. If you have received this transmission in error please notify the sender immediately and then delete this email and all its attachments. If you are not the addressee, any disclosure, reproduction, copying, distribution, or any other dissemination or use of this communication is strictly prohibited. Thank you. DISCLAIMER: This email communication and any attached files are Speedcast Wireless proprietary and may be legally privileged. Export-controlled information shall not be disseminated without proper authorization and proper export-control markings, per Speedcast Wireless export policy. If you have received this transmission in error please notify the sender immediately and then delete this email and all its attachments. If you are not the addressee, any disclosure, reproduction, copying, distribution, or any other dissemination or use of this communication is strictly prohibited. Thank you.
CG
Christian Grün24 May 2019 13:38 #
Hi Joshua,
I need your input. I copied your XML files into the directory with your XQuery modules. Then created a basex command script in the same directory (test.bxs):
create db TestResults
set bindings filename=MultiParty 20190508-182649.xml
run import-and-process.xq
list TestResults
set bindings dir=./
run import-all.xq
list TestResults
I called this script and I got the attached output, which looks ok to me. What did you do differently? Could you create an updated command script for me?
Best,
Christian
Quoted text · Joshua Kordani · 1 line
> basex -v test.bxs
Database 'TestResults' created in 34.62 ms.
BINDINGS: filename=MultiParty 20190508-182649.xml
Query "import-and-process.xq" executed in 641.18 ms.
Input Path Type Content-Type Size
-----------------------------------------------------------
MultiParty 20190508-182649.xml xml application/xml 1291
1 Resource(s).
BINDINGS: dir=./
Query "import-all.xq" executed in 115.87 ms.
Input Path Type Content-Type Size
-----------------------------------------------------------
MultiParty 20190508-182649.xml xml application/xml 1291
TwoParty 20190508-192109.xml xml application/xml 754
2 Resource(s).JK
Joshua Kordani24 May 2019 16:04 #
Christian,
I had not created a bxs script. I was running things from the command line like so. I have discovered a new mode however. I originally started using -I because I was writing scripts that were ignorant of the database to use in order to facilitate using these scripts on a test database later. I found though that for the scripts that have to update the database, an explicit callout to the database in question inside the script was necessary. I accidentally typeo'd the -I TestRestults (note the spelling error) when I called import-and-process.xq, which was successful in updating the database with one file. Below is a test, when I issue a bogus -I, the update is successful. I don't understand what is going on here and I'd like to make my scripts either fully "database name" agnostic or else all of the scripts will have baked in references to TestResults.
C:\Users\joshua.kordani\Documents\ACS>basex
BaseX 9.2.2 [Standalone]
Try 'help' to get more information.
Quoted text · Christian Grün · 1 line
> list
Name Resources Size Input Path
---------------------------------
0 database(s).
Quoted text · Christian Grün · 1 line
> create db TestResults
Database 'TestResults' created in 220.32 ms.
Quoted text · Christian Grün · 1 line
> list
Name Resources Size Input Path
----------------------------------------
TestResults 0 4570
1 database(s).
Quoted text · Christian Grün · 1 line
> quit
See you.
C:\Users\joshua.kordani\Documents\ACS>basex -i TestResults -b dir=TestResults import-all.xq C:/Users/joshua.kordani/Documents/ACS/TestResults/: Updates are not written back.
C:\Users\joshua.kordani\Documents\ACS>basex
BaseX 9.2.2 [Standalone]
Try 'help' to get more information.
Quoted text · Christian Grün · 1 line
> list
Name Resources Size Input Path
----------------------------------------
TestResults 0 4570
1 database(s).
Quoted text · Christian Grün · 1 line
> quit
Have a nice day.
C:\Users\joshua.kordani\Documents\ACS>basex -i bogus -b dir=TestResults import-all.xq
C:\Users\joshua.kordani\Documents\ACS>basex
BaseX 9.2.2 [Standalone]
Try 'help' to get more information.
Quoted text · Christian Grün · 1 line
> list
Name Resources Size Input Path
------------------------------------------
TestResults 8 251589
1 database(s).
Quoted text · Christian Grün · 30 lines
> -----Original Message----- From: Christian Grün <christian.gruen at gmail.com> Sent: Friday, May 24, 2019 7:39 AM To: Joshua Kordani <joshua.kordani at speedcastwireless.com> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] FW: Attempting to write bulk import of files is failing Hi Joshua, I need your input. I copied your XML files into the directory with your XQuery modules. Then created a basex command script in the same directory (test.bxs): create db TestResults set bindings filename=MultiParty 20190508-182649.xml run import-and-process.xq list TestResults set bindings dir=./ run import-all.xq list TestResults I called this script and I got the attached output, which looks ok to me. What did you do differently? Could you create an updated command script for me? Best, Christian > basex -v test.bxs Database 'TestResults' created in 34.62 ms. BINDINGS: filename=MultiParty 20190508-182649.xml Query "import-and-process.xq" executed in 641.18 ms. Input Path Type Content-Type Size ----------------------------------------------------------- MultiParty 20190508-182649.xml xml application/xml 1291 1 Resource(s). BINDINGS: dir=./ Query "import-all.xq" executed in 115.87 ms. Input Path Type Content-Type Size ----------------------------------------------------------- MultiParty 20190508-182649.xml xml application/xml 1291 TwoParty 20190508-192109.xml xml application/xml 754 2 Resource(s). DISCLAIMER: This email communication and any attached files are Speedcast Wireless proprietary and may be legally privileged. Export-controlled information shall not be disseminated without proper authorization and proper export-control markings, per Speedcast Wireless export policy. If you have received this transmission in error please notify the sender immediately and then delete this email and all its attachments. If you are not the addressee, any disclosure, reproduction, copying, distribution, or any other dissemination or use of this communication is strictly prohibited. Thank you.
JK
Furthermore, there seems to be something about the -I flag that I don't understand. In my path, I have a folder also named TestResults. When I pass -I TestResults, it seems like queries are issued against the document set in this folder instead of the database named TestResults. I could have sworn that I had tested this before by renaming the folder to something else, but it seems that at this time, when I rename that folder, -I refers to an installed database and not a set of files in a folder in my path. I suppose that is the cause of the error with updates, since some machinery gets confused that calls to db now refer to a static document set instead of a database.
c:/Users/joshua.kordani/Documents/ACS $ basex -i TestResults BaseX 9.2.2 [Standalone]
Try 'help' to get more information.
Quoted text · Christian Grün · 1 line
> xquery distinct-values(//testcase/@testrun)
Query executed in 298.9 ms.
Quoted text · Christian Grün · 1 line
> quit
Have fun.
c:/Users/joshua.kordani/Documents/ACS $ basex
BaseX 9.2.2 [Standalone]
Try 'help' to get more information.
Quoted text · Christian Grün · 1 line
> open TestResults
Database 'TestResults' was opened in 159.15 ms.
Quoted text · Christian Grün · 1 line
> xquery distinct-values(//testcase/@testrun)
MultiParty 20190508-182649.xml
Range 20190506-142945.xml
Range.Aterm(RepetitionInfo) 20190506-144018.xml
Range.Aterm(RepetitionInfo) 20190506-144510.xml
Range.Bterm(RepetitionInfo) 20190506-144400.xml
Range.Bterm(RepetitionInfo) 20190506-144430.xml
test.xml
TwoParty 20190508-192109.xml
Query executed in 329.09 ms.
Quoted text · Christian Grün · 30 lines
> -----Original Message----- From: Christian Grün <christian.gruen at gmail.com> Sent: Friday, May 24, 2019 7:39 AM To: Joshua Kordani <joshua.kordani at speedcastwireless.com> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] FW: Attempting to write bulk import of files is failing Hi Joshua, I need your input. I copied your XML files into the directory with your XQuery modules. Then created a basex command script in the same directory (test.bxs): create db TestResults set bindings filename=MultiParty 20190508-182649.xml run import-and-process.xq list TestResults set bindings dir=./ run import-all.xq list TestResults I called this script and I got the attached output, which looks ok to me. What did you do differently? Could you create an updated command script for me? Best, Christian > basex -v test.bxs Database 'TestResults' created in 34.62 ms. BINDINGS: filename=MultiParty 20190508-182649.xml Query "import-and-process.xq" executed in 641.18 ms. Input Path Type Content-Type Size ----------------------------------------------------------- MultiParty 20190508-182649.xml xml application/xml 1291 1 Resource(s). BINDINGS: dir=./ Query "import-all.xq" executed in 115.87 ms. Input Path Type Content-Type Size ----------------------------------------------------------- MultiParty 20190508-182649.xml xml application/xml 1291 TwoParty 20190508-192109.xml xml application/xml 754 2 Resource(s). DISCLAIMER: This email communication and any attached files are Speedcast Wireless proprietary and may be legally privileged. Export-controlled information shall not be disseminated without proper authorization and proper export-control markings, per Speedcast Wireless export policy. If you have received this transmission in error please notify the sender immediately and then delete this email and all its attachments. If you are not the addressee, any disclosure, reproduction, copying, distribution, or any other dissemination or use of this communication is strictly prohibited. Thank you.
CG
Christian Grün27 May 2019 19:17 #
Hi Joshua,
Sorry for the late reply. I think you've already found out what's happening. The -i flag does different things:
• If an XML file or directory with the specified name exists, it will be set as context.
• A database with the specified name will be set as context unless a file with the same name exists, and if this file has a different timestamp.
If you want to be sure that a database is referenced, you can assign the name of the database to a variable (see -b for more details).
Best,
Christian
Joshua Kordani <joshua.kordani at speedcastwireless.com> schrieb am Fr., 24. Mai 2019, 16:55:
Quoted text · Joshua Kordani · 94 lines
> Furthermore, there seems to be something about the -I flag that I don't > understand. > In my path, I have a folder also named TestResults. When I pass -I > TestResults, it seems like queries are issued against the document set in > this folder instead of the database named TestResults. I could have sworn > that I had tested this before by renaming the folder to something else, but > it seems that at this time, when I rename that folder, -I refers to an > installed database and not a set of files in a folder in my path. I > suppose that is the cause of the error with updates, since some machinery > gets confused that calls to db now refer to a static document set instead > of a database. > > c:/Users/joshua.kordani/Documents/ACS $ basex -i TestResults > BaseX 9.2.2 [Standalone] > Try 'help' to get more information. > > xquery distinct-values(//testcase/@testrun) > > Query executed in 298.9 ms. > > quit > Have fun. > c:/Users/joshua.kordani/Documents/ACS $ basex > BaseX 9.2.2 [Standalone] > Try 'help' to get more information. > > open TestResults > Database 'TestResults' was opened in 159.15 ms. > > xquery distinct-values(//testcase/@testrun) > MultiParty 20190508-182649.xml > Range 20190506-142945.xml > Range.Aterm(RepetitionInfo) 20190506-144018.xml > Range.Aterm(RepetitionInfo) 20190506-144510.xml > Range.Bterm(RepetitionInfo) 20190506-144400.xml > Range.Bterm(RepetitionInfo) 20190506-144430.xml > test.xml > TwoParty 20190508-192109.xml > Query executed in 329.09 ms. > > > > -----Original Message----- > From: Christian Grün <christian.gruen at gmail.com> > Sent: Friday, May 24, 2019 7:39 AM > To: Joshua Kordani <joshua.kordani at speedcastwireless.com> > Cc: BaseX <basex-talk at mailman.uni-konstanz.de> > Subject: Re: [basex-talk] FW: Attempting to write bulk import of files is > failing > > Hi Joshua, > > I need your input. I copied your XML files into the directory with your > XQuery modules. Then created a basex command script in the same directory > (test.bxs): > > create db TestResults > set bindings filename=MultiParty 20190508-182649.xml run > import-and-process.xq list TestResults set bindings dir=./ run > import-all.xq list TestResults > > I called this script and I got the attached output, which looks ok to me. > What did you do differently? Could you create an updated command script for > me? > > Best, > Christian > > > basex -v test.bxs > > Database 'TestResults' created in 34.62 ms. > BINDINGS: filename=MultiParty 20190508-182649.xml > > Query "import-and-process.xq" executed in 641.18 ms. > Input Path Type Content-Type Size > ----------------------------------------------------------- > MultiParty 20190508-182649.xml xml application/xml 1291 > > 1 Resource(s). > BINDINGS: dir=./ > > Query "import-all.xq" executed in 115.87 ms. > Input Path Type Content-Type Size > ----------------------------------------------------------- > MultiParty 20190508-182649.xml xml application/xml 1291 > TwoParty 20190508-192109.xml xml application/xml 754 > > 2 Resource(s). > > DISCLAIMER: This email communication and any attached files are Speedcast > Wireless proprietary and may be legally privileged. Export-controlled > information shall not be disseminated without proper authorization and > proper export-control markings, per Speedcast Wireless export policy. If > you have received this transmission in error please notify the sender > immediately and then delete this email and all its attachments. If you are > not the addressee, any disclosure, reproduction, copying, distribution, or > any other dissemination or use of this communication is strictly > prohibited. Thank you. >
Written for BaseX 9.2. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery