adding zip contents – bug or feature?
When I add a directory to a database in 6.6.1 using the plain *.xml createfilter, and when there's a zip file below this directory, the zip file's contents will be indexed, too, as if they reside in the server's working directory. If this is a feature, how can I switch it off?
Gerrit
Dear Gerrit,
thanks for your observation. Yes, indeed you've come across a rather hidden feature, but I can understand pretty well that the behavior is not always desired. I've added a GitHub entry to document this:
https://github.com/BaseXdb/basex/issues/60
If you have some preferences how to resolve this issue, feel free to give more feedback.
Christian
Quoted text · le-tex · 13 lines
On Thu, Apr 7, 2011 at 10:01 PM, Imsieke, Gerrit, le-tex <gerrit.imsieke at le-tex.de> wrote: > When I add a directory to a database in 6.6.1 using the plain *.xml > createfilter, and when there's a zip file below this directory, the zip > file's contents will be indexed, too, as if they reside in the server's > working directory. If this is a feature, how can I switch it off? > > Gerrit > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
SET ADDARCHIVES ON/OFF
(I'm not shouting at you, it's just that uppercase convention thing…) Default should be OFF.
Gerrit
Quoted text · Christian Grün · 28 lines
On 2011-04-07 22:31, Christian Grün wrote: > Dear Gerrit, > > thanks for your observation. Yes, indeed you've come across a rather > hidden feature, but I can understand pretty well that the behavior is > not always desired. I've added a GitHub entry to document this: > > https://github.com/BaseXdb/basex/issues/60 > > If you have some preferences how to resolve this issue, feel free to > give more feedback. > > Christian > > > On Thu, Apr 7, 2011 at 10:01 PM, Imsieke, Gerrit, le-tex > <gerrit.imsieke at le-tex.de> wrote: >> When I add a directory to a database in 6.6.1 using the plain *.xml >> createfilter, and when there's a zip file below this directory, the zip >> file's contents will be indexed, too, as if they reside in the server's >> working directory. If this is a feature, how can I switch it off? >> >> Gerrit >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>
Quoted text · Imsieke, Gerrit, le-tex · 3 lines
> SET ADDARCHIVES ON/OFF > (I'm not shouting at you, it's just that uppercase convention thing…) > Default should be OFF.
;) Thanks; I've added this suggestion to the GitHub entry. – As BaseX is getting more and more options, we'd be even happier with a better default solution, but the option could be a preliminary solution as well.
Christian
Hi I like the feature, this is something, I was dreaming about, as we archive old xml files in zip files and importing them this way would easy the work quite well. It does not work perfecktly as my test is failing on some archives with Command: CREATE DB ZipTest D:\var\TICEReports\aws1\ziptest Error: "..." (Line 1): The processing instruction target matching "[xX][mM][lL]" is not allowed.
But I do not report this as bug until it is really published as supported feature (to know, what is really expected behaviour. Another wish for importing: sometime I have hundreds of xml files and some might be broken xml documents. Currently import always fails. It would be great to have an option "ignore invalid documents" which would allow quick import and which would print out file names of those invalid documents.
PS: We use for archiving command line python utility Dalimil<https://bitbucket.org/vlcinsky/dalimil>, which we wrote and published it under BSD Lincense. Still in beta phase, but is it considered safe and usable.
2011/4/7 Imsieke, Gerrit, le-tex <gerrit.imsieke at le-tex.de>
Quoted text · Imsieke, Gerrit, le-tex · 11 lines
> When I add a directory to a database in 6.6.1 using the plain *.xml > createfilter, and when there's a zip file below this directory, the zip > file's contents will be indexed, too, as if they reside in the server's > working directory. If this is a feature, how can I switch it off? > > Gerrit > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
Hi Jan,
Op 8 apr 2011, om 08:26 heeft Jan Vlčinský (CAD) het volgende geschreven:
Quoted text · Jan Vlčinský (CAD) · 4 lines
> I like the feature, this is something, I was dreaming about, as we archive old xml files in zip files and importing them this way would easy the work quite well. > It does not work perfecktly as my test is failing on some archives with > Command: CREATE DB ZipTest D:\var\TICEReports\aws1\ziptest > Error: "..." (Line 1): The processing instruction target matching "[xX][mM][lL]" is not allowed.
Looks like there's an error in one of your XML files, there's probably some whitespace before the XML declaration, that would trigger this error.
Quoted text · Jan Vlčinský (CAD) · 2 lines
> > Another wish for importing: sometime I have hundreds of xml files and some might be broken xml documents. Currently import always fails. It would be great to have an option "ignore invalid documents" which would allow quick import
Oooo yes! I want that too. Usecase: Sometimes during development some crud ends up in the folder I'm trying to import, f.i. from a bad export from another system. Then I have to clean everything first before importing into BaseX. But really, I don't care if a couple of documents fail to import, it's development and I don't need all of the documents in the DB.
Quoted text · Jan Vlčinský (CAD) · 1 line
> and which would print out file names of those invalid documents.
So for cleaning I use a shell script with a for loop and 'xmllint -noout "$f"'. That automates it, but it would be a "nice to have" anyway if I can tell BaseX to just ignore files with errors, that would also work with .zip files, a shell script doesn't.
Hartelijke groet,
Huib.
Hi Huib I added the request "*Import xml files - add option to allow ignoring invalid files* <https://github.com/BaseXdb/basex/issues/61#issue/61>" to Issues: https://github.com/BaseXdb/basex/issues/61
<https://github.com/BaseXdb/basex/issues/61>Vote for it, if you like.Jan
2011/4/8 Huib Verweij <Huib.Verwey at mpi.nl>
Quoted text · Huib Verweij · 54 lines
> Hi Jan, > > > Op 8 apr 2011, om 08:26 heeft Jan Vlčinský (CAD) het volgende geschreven: > > I like the feature, this is something, I was dreaming about, as we archive > old xml files in zip files and importing them this way would easy the work > quite well. > It does not work perfecktly as my test is failing on some archives with > Command: CREATE DB ZipTest D:\var\TICEReports\aws1\ziptest > Error: "..." (Line 1): The processing instruction target matching > "[xX][mM][lL]" is not allowed. > > > Looks like there's an error in one of your XML files, there's probably some > whitespace before the XML declaration, that would trigger this error. > > > Another wish for importing: sometime I have hundreds of xml files and some > might be broken xml documents. Currently import always fails. It would be > great to have an option "ignore invalid documents" which would allow quick > import > > > Oooo yes! I want that too. Usecase: Sometimes during development some crud > ends up in the folder I'm trying to import, f.i. from a bad export from > another system. Then I have to clean everything first before importing into > BaseX. But really, I don't care if a couple of documents fail to import, > it's development and I don't need all of the documents in the DB. > > and which would print out file names of those invalid documents. > > > So for cleaning I use a shell script with a for loop and 'xmllint -noout > "$f"'. That automates it, but it would be a "nice to have" anyway if I can > tell BaseX to just ignore files with errors, that would also work with .zip > files, a shell script doesn't. > > Hartelijke groet, > > Huib. > > -- > Drs. Huib Verweij > Senior software developer - The Language Archive > Max Planck Institute for Psycholinguistics > P.O. Box 310 > 6500 AH Nijmegen > The Netherlands > t +31-24-3521911 > e huib.verwey at mpi.nl > w http://www.mpi.nl/ > >
To add some more items to the wish list:
Currently basexclient doesn’t exit with an error code (≠0) when it encounters a file that throws an exception on the server side during import. This is unfortunate in our use case where we issue several svn checkout and BaseX ADD statements in a Makefile. Since there is no non-null exit status, make will always continue, and the errors will not be dealt with unless someone watches the process closely.
In addition, the server process will lock the file that it was unsuccessfully trying to import. I'll have to restart the server in order to make it (=server) unlock the file so that it (=file) can be deleted. So there should be some exception handler that at least closes the file handles.
And, although you (=Christian) expressed your feeling that there are already too many options around, it should be controllable by an option whether an error should be issued or whether corrupt files should be ignored (with a warning) during import, as Jan was asking for.
I don’t think it’s a bad thing to have many options, as long as they a) make sense b) are documented.
Gerrit
Quoted text · Huib Verweij · 58 lines
On 2011-04-08 08:50, Huib Verweij wrote: > Hi Jan, > > > Op 8 apr 2011, om 08:26 heeft Jan Vlčinský (CAD) het volgende geschreven: > >> I like the feature, this is something, I was dreaming about, as we >> archive old xml files in zip files and importing them this way would >> easy the work quite well. >> It does not work perfecktly as my test is failing on some archives with >> Command: CREATE DB ZipTest D:\var\TICEReports\aws1\ziptest >> Error: "..." (Line 1): The processing instruction target matching >> "[xX][mM][lL]" is not allowed. > > Looks like there's an error in one of your XML files, there's probably > some whitespace before the XML declaration, that would trigger this error. > >> >> Another wish for importing: sometime I have hundreds of xml files and >> some might be broken xml documents. Currently import always fails. It >> would be great to have an option "ignore invalid documents" which >> would allow quick import > > Oooo yes! I want that too. Usecase: Sometimes during development some > crud ends up in the folder I'm trying to import, f.i. from a bad export > from another system. Then I have to clean everything first before > importing into BaseX. But really, I don't care if a couple of documents > fail to import, it's development and I don't need all of the documents > in the DB. > >> and which would print out file names of those invalid documents. > > So for cleaning I use a shell script with a for loop and 'xmllint -noout > "$f"'. That automates it, but it would be a "nice to have" anyway if I > can tell BaseX to just ignore files with errors, that would also work > with .zip files, a shell script doesn't. > > Hartelijke groet, > > Huib. > > -- > Drs. Huib Verweij > Senior software developer - The Language Archive > Max Planck Institute for Psycholinguistics > P.O. Box 310 > 6500 AH Nijmegen > The Netherlands > t +31-24-3521911 > e huib.verwey at mpi.nl <mailto:huib.verwey at mpi.nl> > w http://www.mpi.nl/ > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
I voted, but isn't it the same issue as #12?
Hartelijke groet,
Huib Verweij.
Quoted text · Jan Vlčinský (CAD) · 6 lines
> Hi Huib > I added the request "Import xml files - add option to allow ignoring invalid files" to Issues: https://github.com/BaseXdb/basex/issues/61 > > Vote for it, if you like. > > Jan
I voted for #61 instead of #12 (yeah, gimme options)
Quoted text · Huib Verweij · 38 lines
On 2011-04-08 09:31, Huib Verweij wrote: > I voted, but isn't it the same issue as #12? > > Hartelijke groet, > > Huib Verweij. > > > > -- > Drs. Huib Verweij > Senior software developer - The Language Archive > Max Planck Institute for Psycholinguistics > P.O. Box 310 > 6500 AH Nijmegen > The Netherlands > t +31-24-3521911 > e huib.verwey at mpi.nl <mailto:huib.verwey at mpi.nl> > w http://www.mpi.nl/ > > > Op 8 apr 2011, om 09:10 heeft Jan Vlčinský (CAD) het volgende geschreven: > >> Hi Huib >> I added the request "*Import xml files - add option to allow ignoring >> invalid files* <https://github.com/BaseXdb/basex/issues/61#issue/61>" >> to Issues: https://github.com/BaseXdb/basex/issues/61 >> >> <https://github.com/BaseXdb/basex/issues/61>Vote for it, if you like. >> >> Jan > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Quoted text · Huib Verweij · 1 line
> I voted, but isn't it the same issue as #12?
…dito! As so many have already voted for #61, I've now closed issue #12, and added some comments.
Quoted text · Huib Verweij · 31 lines
> Hartelijke groet, > Huib Verweij. > > > -- > Drs. Huib Verweij > Senior software developer - The Language Archive > Max Planck Institute for Psycholinguistics > P.O. Box 310 > 6500 AH Nijmegen > The Netherlands > t +31-24-3521911 > e huib.verwey at mpi.nl > w http://www.mpi.nl/ > > > Op 8 apr 2011, om 09:10 heeft Jan Vlčinský (CAD) het volgende geschreven: > > Hi Huib > I added the request "Import xml files - add option to allow ignoring invalid > files" to Issues: https://github.com/BaseXdb/basex/issues/61 > Vote for it, if you like. > Jan > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >
Fine, so it quite looks as we don't get bored.. ;) I've added two issues in the bug tracker to memorize your observations.
Quoted text · Imsieke, Gerrit, le-tex · 104 lines
> To add some more items to the wish list: > > Currently basexclient doesn’t exit with an error code (≠0) when it > encounters a file that throws an exception on the server side during import. > This is unfortunate in our use case where we issue several svn checkout and > BaseX ADD statements in a Makefile. Since there is no non-null exit status, > make will always continue, and the errors will not be dealt with unless > someone watches the process closely. > > In addition, the server process will lock the file that it was > unsuccessfully trying to import. I'll have to restart the server in order to > make it (=server) unlock the file so that it (=file) can be deleted. So > there should be some exception handler that at least closes the file > handles. > > And, although you (=Christian) expressed your feeling that there are already > too many options around, it should be controllable by an option whether an > error should be issued or whether corrupt files should be ignored (with a > warning) during import, as Jan was asking for. > > I don’t think it’s a bad thing to have many options, as long as they a) make > sense b) are documented. > > Gerrit > > > On 2011-04-08 08:50, Huib Verweij wrote: >> >> Hi Jan, >> >> >> Op 8 apr 2011, om 08:26 heeft Jan Vlčinský (CAD) het volgende geschreven: >> >>> I like the feature, this is something, I was dreaming about, as we >>> archive old xml files in zip files and importing them this way would >>> easy the work quite well. >>> It does not work perfecktly as my test is failing on some archives with >>> Command: CREATE DB ZipTest D:\var\TICEReports\aws1\ziptest >>> Error: "..." (Line 1): The processing instruction target matching >>> "[xX][mM][lL]" is not allowed. >> >> Looks like there's an error in one of your XML files, there's probably >> some whitespace before the XML declaration, that would trigger this error. >> >>> >>> Another wish for importing: sometime I have hundreds of xml files and >>> some might be broken xml documents. Currently import always fails. It >>> would be great to have an option "ignore invalid documents" which >>> would allow quick import >> >> Oooo yes! I want that too. Usecase: Sometimes during development some >> crud ends up in the folder I'm trying to import, f.i. from a bad export >> from another system. Then I have to clean everything first before >> importing into BaseX. But really, I don't care if a couple of documents >> fail to import, it's development and I don't need all of the documents >> in the DB. >> >>> and which would print out file names of those invalid documents. >> >> So for cleaning I use a shell script with a for loop and 'xmllint -noout >> "$f"'. That automates it, but it would be a "nice to have" anyway if I >> can tell BaseX to just ignore files with errors, that would also work >> with .zip files, a shell script doesn't. >> >> Hartelijke groet, >> >> Huib. >> >> -- >> Drs. Huib Verweij >> Senior software developer - The Language Archive >> Max Planck Institute for Psycholinguistics >> P.O. Box 310 >> 6500 AH Nijmegen >> The Netherlands >> t +31-24-3521911 >> e huib.verwey at mpi.nl <mailto:huib.verwey at mpi.nl> >> w http://www.mpi.nl/ >> >> >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > -- > Gerrit Imsieke > Geschäftsführer / Managing Director > le-tex publishing services GmbH > Weissenfelser Str. 84, 04229 Leipzig, Germany > Phone +49 341 355356 110, Fax +49 341 355356 510 > gerrit.imsieke at le-tex.de, http://www.le-tex.de > > Registergericht / Commercial Register: Amtsgericht Leipzig > Registernummer / Registration Number: HRB 24930 > > Geschäftsführer: Gerrit Imsieke, Svea Jelonek, > Thomas Schmidt, Dr. Reinhard Vöckler > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
Quoted text · Imsieke, Gerrit, le-tex · 2 lines
> I don’t think it’s a bad thing to have many options, as long as they a) make > sense b) are documented.
In our code history, we experienced that it's very easy to add new options, but much more difficult to get them out again. Next, we regularly had options which conflicted with other options at some stage. This is why we're generally happy if we can avoid adding new options to the list, and find a consistent solution that eventually satisfies even more users. I agree, however, that they are a pragmatic solution if they are well documented, and that it's usually the fastest way of solving open issues. – Christian
Quoted text · Imsieke, Gerrit, le-tex · 6 lines
> Currently basexclient doesn’t exit with an error code (≠0) when it > encounters a file that throws an exception on the server side during import. > This is unfortunate in our use case where we issue several svn checkout and > BaseX ADD statements in a Makefile. Since there is no non-null exit status, > make will always continue, and the errors will not be dealt with unless > someone watches the process closely.
This has now been resolved for all command-line APIs; note that the Windows batch scripts (basex.bat, etc) were updated as well.
Christian
Quoted text · Imsieke, Gerrit, le-tex · 3 lines
> SET ADDARCHIVES ON/OFF > (I'm not shouting at you, it's just that uppercase convention thing…) > Default should be OFF.
...got it ;) It's resolved as proposed (well, pretty close, as we decided to stick with ON as default):
http://files.basex.org/releases/latest/ http://docs.basex.org/wiki/Options
Christian
Quoted text · Christian Grün · 49 lines
> On 2011-04-07 22:31, Christian Grün wrote: >> >> Dear Gerrit, >> >> thanks for your observation. Yes, indeed you've come across a rather >> hidden feature, but I can understand pretty well that the behavior is >> not always desired. I've added a GitHub entry to document this: >> >> https://github.com/BaseXdb/basex/issues/60 >> >> If you have some preferences how to resolve this issue, feel free to >> give more feedback. >> >> Christian >> >> >> On Thu, Apr 7, 2011 at 10:01 PM, Imsieke, Gerrit, le-tex >> <gerrit.imsieke at le-tex.de> wrote: >>> >>> When I add a directory to a database in 6.6.1 using the plain *.xml >>> createfilter, and when there's a zip file below this directory, the zip >>> file's contents will be indexed, too, as if they reside in the server's >>> working directory. If this is a feature, how can I switch it off? >>> >>> Gerrit >>> _______________________________________________ >>> BaseX-Talk mailing list >>> BaseX-Talk at mailman.uni-konstanz.de >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>> > > -- > Gerrit Imsieke > Geschäftsführer / Managing Director > le-tex publishing services GmbH > Weissenfelser Str. 84, 04229 Leipzig, Germany > Phone +49 341 355356 110, Fax +49 341 355356 510 > gerrit.imsieke at le-tex.de, http://www.le-tex.de > > Registergericht / Commercial Register: Amtsgericht Leipzig > Registernummer / Registration Number: HRB 24930 > > Geschäftsführer: Gerrit Imsieke, Svea Jelonek, > Thomas Schmidt, Dr. Reinhard Vöckler > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
Written for BaseX 6.6. BaseX may have changed since: please check the current documentation.