adding zip contents – bug or feature?

On Thu, Apr 7, 2011 at 10:01 PM, Imsieke, Gerrit, le-tex <gerrit.imsieke at le-tex.de> wrote: > When I add a directory to a database in 6.6.1 using the plain *.xml > createfilter, and when there's a zip file below this directory, the zip > file's contents will be indexed, too, as if they reside in the server's > working directory. If this is a feature, how can I switch it off? > > Gerrit > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >

On 2011-04-07 22:31, Christian Grün wrote: > Dear Gerrit, > > thanks for your observation. Yes, indeed you've come across a rather > hidden feature, but I can understand pretty well that the behavior is > not always desired. I've added a GitHub entry to document this: > > https://github.com/BaseXdb/basex/issues/60 > > If you have some preferences how to resolve this issue, feel free to > give more feedback. > > Christian > > > On Thu, Apr 7, 2011 at 10:01 PM, Imsieke, Gerrit, le-tex > <gerrit.imsieke at le-tex.de> wrote: >> When I add a directory to a database in 6.6.1 using the plain *.xml >> createfilter, and when there's a zip file below this directory, the zip >> file's contents will be indexed, too, as if they reside in the server's >> working directory. If this is a feature, how can I switch it off? >> >> Gerrit >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>

> SET ADDARCHIVES ON/OFF > (I'm not shouting at you, it's just that uppercase convention thing…) > Default should be OFF.

> When I add a directory to a database in 6.6.1 using the plain *.xml > createfilter, and when there's a zip file below this directory, the zip > file's contents will be indexed, too, as if they reside in the server's > working directory. If this is a feature, how can I switch it off? > > Gerrit > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >

> I like the feature, this is something, I was dreaming about, as we archive old xml files in zip files and importing them this way would easy the work quite well. > It does not work perfecktly as my test is failing on some archives with > Command: CREATE DB ZipTest D:\var\TICEReports\aws1\ziptest > Error: "..." (Line 1): The processing instruction target matching "[xX][mM][lL]" is not allowed.

> > Another wish for importing: sometime I have hundreds of xml files and some might be broken xml documents. Currently import always fails. It would be great to have an option "ignore invalid documents" which would allow quick import

> and which would print out file names of those invalid documents.

> Hi Jan, > > > Op 8 apr 2011, om 08:26 heeft Jan Vlčinský (CAD) het volgende geschreven: > > I like the feature, this is something, I was dreaming about, as we archive > old xml files in zip files and importing them this way would easy the work > quite well. > It does not work perfecktly as my test is failing on some archives with > Command: CREATE DB ZipTest D:\var\TICEReports\aws1\ziptest > Error: "..." (Line 1): The processing instruction target matching > "[xX][mM][lL]" is not allowed. > > > Looks like there's an error in one of your XML files, there's probably some > whitespace before the XML declaration, that would trigger this error. > > > Another wish for importing: sometime I have hundreds of xml files and some > might be broken xml documents. Currently import always fails. It would be > great to have an option "ignore invalid documents" which would allow quick > import > > > Oooo yes! I want that too. Usecase: Sometimes during development some crud > ends up in the folder I'm trying to import, f.i. from a bad export from > another system. Then I have to clean everything first before importing into > BaseX. But really, I don't care if a couple of documents fail to import, > it's development and I don't need all of the documents in the DB. > > and which would print out file names of those invalid documents. > > > So for cleaning I use a shell script with a for loop and 'xmllint -noout > "$f"'. That automates it, but it would be a "nice to have" anyway if I can > tell BaseX to just ignore files with errors, that would also work with .zip > files, a shell script doesn't. > > Hartelijke groet, > > Huib. > > -- > Drs. Huib Verweij > Senior software developer - The Language Archive > Max Planck Institute for Psycholinguistics > P.O. Box 310 > 6500 AH Nijmegen > The Netherlands > t +31-24-3521911 > e huib.verwey at mpi.nl > w http://www.mpi.nl/ > >

On 2011-04-08 08:50, Huib Verweij wrote: > Hi Jan, > > > Op 8 apr 2011, om 08:26 heeft Jan Vlčinský (CAD) het volgende geschreven: > >> I like the feature, this is something, I was dreaming about, as we >> archive old xml files in zip files and importing them this way would >> easy the work quite well. >> It does not work perfecktly as my test is failing on some archives with >> Command: CREATE DB ZipTest D:\var\TICEReports\aws1\ziptest >> Error: "..." (Line 1): The processing instruction target matching >> "[xX][mM][lL]" is not allowed. > > Looks like there's an error in one of your XML files, there's probably > some whitespace before the XML declaration, that would trigger this error. > >> >> Another wish for importing: sometime I have hundreds of xml files and >> some might be broken xml documents. Currently import always fails. It >> would be great to have an option "ignore invalid documents" which >> would allow quick import > > Oooo yes! I want that too. Usecase: Sometimes during development some > crud ends up in the folder I'm trying to import, f.i. from a bad export > from another system. Then I have to clean everything first before > importing into BaseX. But really, I don't care if a couple of documents > fail to import, it's development and I don't need all of the documents > in the DB. > >> and which would print out file names of those invalid documents. > > So for cleaning I use a shell script with a for loop and 'xmllint -noout > "$f"'. That automates it, but it would be a "nice to have" anyway if I > can tell BaseX to just ignore files with errors, that would also work > with .zip files, a shell script doesn't. > > Hartelijke groet, > > Huib. > > -- > Drs. Huib Verweij > Senior software developer - The Language Archive > Max Planck Institute for Psycholinguistics > P.O. Box 310 > 6500 AH Nijmegen > The Netherlands > t +31-24-3521911 > e huib.verwey at mpi.nl <mailto:huib.verwey at mpi.nl> > w http://www.mpi.nl/ > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

> Hi Huib > I added the request "Import xml files - add option to allow ignoring invalid files" to Issues: https://github.com/BaseXdb/basex/issues/61 > > Vote for it, if you like. > > Jan

On 2011-04-08 09:31, Huib Verweij wrote: > I voted, but isn't it the same issue as #12? > > Hartelijke groet, > > Huib Verweij. > > > > -- > Drs. Huib Verweij > Senior software developer - The Language Archive > Max Planck Institute for Psycholinguistics > P.O. Box 310 > 6500 AH Nijmegen > The Netherlands > t +31-24-3521911 > e huib.verwey at mpi.nl <mailto:huib.verwey at mpi.nl> > w http://www.mpi.nl/ > > > Op 8 apr 2011, om 09:10 heeft Jan Vlčinský (CAD) het volgende geschreven: > >> Hi Huib >> I added the request "*Import xml files - add option to allow ignoring >> invalid files* <https://github.com/BaseXdb/basex/issues/61#issue/61>" >> to Issues: https://github.com/BaseXdb/basex/issues/61 >> >> <https://github.com/BaseXdb/basex/issues/61>Vote for it, if you like. >> >> Jan > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

> I voted, but isn't it the same issue as #12?

> Hartelijke groet, > Huib Verweij. > > > -- > Drs. Huib Verweij > Senior software developer - The Language Archive > Max Planck Institute for Psycholinguistics > P.O. Box 310 > 6500 AH Nijmegen > The Netherlands > t +31-24-3521911 > e huib.verwey at mpi.nl > w http://www.mpi.nl/ > > > Op 8 apr 2011, om 09:10 heeft Jan Vlčinský (CAD) het volgende geschreven: > > Hi Huib > I added the request "Import xml files - add option to allow ignoring invalid > files" to Issues: https://github.com/BaseXdb/basex/issues/61 > Vote for it, if you like. > Jan > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >

> To add some more items to the wish list: > > Currently basexclient doesn’t exit with an error code (≠0) when it > encounters a file that throws an exception on the server side during import. > This is unfortunate in our use case where we issue several svn checkout and > BaseX ADD statements in a Makefile. Since there is no non-null exit status, > make will always continue, and the errors will not be dealt with unless > someone watches the process closely. > > In addition, the server process will lock the file that it was > unsuccessfully trying to import. I'll have to restart the server in order to > make it (=server) unlock the file so that it (=file) can be deleted. So > there should be some exception handler that at least closes the file > handles. > > And, although you (=Christian) expressed your feeling that there are already > too many options around, it should be controllable by an option whether an > error should be issued or whether corrupt files should be ignored (with a > warning) during import, as Jan was asking for. > > I don’t think it’s a bad thing to have many options, as long as they a) make > sense b) are documented. > > Gerrit > > > On 2011-04-08 08:50, Huib Verweij wrote: >> >> Hi Jan, >> >> >> Op 8 apr 2011, om 08:26 heeft Jan Vlčinský (CAD) het volgende geschreven: >> >>> I like the feature, this is something, I was dreaming about, as we >>> archive old xml files in zip files and importing them this way would >>> easy the work quite well. >>> It does not work perfecktly as my test is failing on some archives with >>> Command: CREATE DB ZipTest D:\var\TICEReports\aws1\ziptest >>> Error: "..." (Line 1): The processing instruction target matching >>> "[xX][mM][lL]" is not allowed. >> >> Looks like there's an error in one of your XML files, there's probably >> some whitespace before the XML declaration, that would trigger this error. >> >>> >>> Another wish for importing: sometime I have hundreds of xml files and >>> some might be broken xml documents. Currently import always fails. It >>> would be great to have an option "ignore invalid documents" which >>> would allow quick import >> >> Oooo yes! I want that too. Usecase: Sometimes during development some >> crud ends up in the folder I'm trying to import, f.i. from a bad export >> from another system. Then I have to clean everything first before >> importing into BaseX. But really, I don't care if a couple of documents >> fail to import, it's development and I don't need all of the documents >> in the DB. >> >>> and which would print out file names of those invalid documents. >> >> So for cleaning I use a shell script with a for loop and 'xmllint -noout >> "$f"'. That automates it, but it would be a "nice to have" anyway if I >> can tell BaseX to just ignore files with errors, that would also work >> with .zip files, a shell script doesn't. >> >> Hartelijke groet, >> >> Huib. >> >> -- >> Drs. Huib Verweij >> Senior software developer - The Language Archive >> Max Planck Institute for Psycholinguistics >> P.O. Box 310 >> 6500 AH Nijmegen >> The Netherlands >> t +31-24-3521911 >> e huib.verwey at mpi.nl <mailto:huib.verwey at mpi.nl> >> w http://www.mpi.nl/ >> >> >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > -- > Gerrit Imsieke > Geschäftsführer / Managing Director > le-tex publishing services GmbH > Weissenfelser Str. 84, 04229 Leipzig, Germany > Phone +49 341 355356 110, Fax +49 341 355356 510 > gerrit.imsieke at le-tex.de, http://www.le-tex.de > > Registergericht / Commercial Register: Amtsgericht Leipzig > Registernummer / Registration Number: HRB 24930 > > Geschäftsführer: Gerrit Imsieke, Svea Jelonek, > Thomas Schmidt, Dr. Reinhard Vöckler > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >

> I don’t think it’s a bad thing to have many options, as long as they a) make > sense b) are documented.

> Currently basexclient doesn’t exit with an error code (≠0) when it > encounters a file that throws an exception on the server side during import. > This is unfortunate in our use case where we issue several svn checkout and > BaseX ADD statements in a Makefile. Since there is no non-null exit status, > make will always continue, and the errors will not be dealt with unless > someone watches the process closely.

> SET ADDARCHIVES ON/OFF > (I'm not shouting at you, it's just that uppercase convention thing…) > Default should be OFF.

> On 2011-04-07 22:31, Christian Grün wrote: >> >> Dear Gerrit, >> >> thanks for your observation. Yes, indeed you've come across a rather >> hidden feature, but I can understand pretty well that the behavior is >> not always desired. I've added a GitHub entry to document this: >> >> https://github.com/BaseXdb/basex/issues/60 >> >> If you have some preferences how to resolve this issue, feel free to >> give more feedback. >> >> Christian >> >> >> On Thu, Apr 7, 2011 at 10:01 PM, Imsieke, Gerrit, le-tex >> <gerrit.imsieke at le-tex.de> wrote: >>> >>> When I add a directory to a database in 6.6.1 using the plain *.xml >>> createfilter, and when there's a zip file below this directory, the zip >>> file's contents will be indexed, too, as if they reside in the server's >>> working directory. If this is a feature, how can I switch it off? >>> >>> Gerrit >>> _______________________________________________ >>> BaseX-Talk mailing list >>> BaseX-Talk at mailman.uni-konstanz.de >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>> > > -- > Gerrit Imsieke > Geschäftsführer / Managing Director > le-tex publishing services GmbH > Weissenfelser Str. 84, 04229 Leipzig, Germany > Phone +49 341 355356 110, Fax +49 341 355356 510 > gerrit.imsieke at le-tex.de, http://www.le-tex.de > > Registergericht / Commercial Register: Amtsgericht Leipzig > Registernummer / Registration Number: HRB 24930 > > Geschäftsführer: Gerrit Imsieke, Svea Jelonek, > Thomas Schmidt, Dr. Reinhard Vöckler > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >