Whitespace handling on ingest
Wednesday, 20 February 2013 · 4 messages · last reply 22 Feb 2013
WP
Wendell Piez20 Feb 2013 18:35 #
Hi,
I see the 'CHOP' option, turned on by default, for trimming leading and trailing whitespace and eliminating empty text nodes.
What about going further? Is there a good way to normalize whitespace entirely, collapsing any runs of tab-LF-space into single spaces in my data?
I think I mentioned earlier the idea of specifying an XSLT
transformation to filter data on ingest (for a similar requirement, namely removing all comments and PIs). That might be going too far but any hints you can give me (or pointers to docs) about functionality to address this sort of thing in general would be welcome.
Thanks!
Wendell
--
Wendell Piez | http://www.wendellpiez.com
XML | XSLT | electronic publishing
Eat Your Vegetables
_____oo_________o_o___ooooo____ooooooo_^
CG
Christian Grün22 Feb 2013 11:34 #
Hi Wendell,
the CHOP option has been introduced at a verly stage of BaseX, and I’m not sure if we had added it today. We could add one or more additional options to normalize whitespaces or removing PIs/comments from the input, but the wish list, and the exception list, would probably continue to grow, so I believe that it would be more convenient to have a general pre-processing step that takes care of all the normalization steps. I’m not sure, however, what’s the best approach to do this within BaseX. If it’s possible to cache files on disk before adding them to the database, I would recommend XQuery or BaseX command scripts, XProc or anything else to prepare the data and delete it afterwards.
Comments are welcome,
Christan
Quoted text · Wendell Piez · 28 lines
On Wed, Feb 20, 2013 at 5:35 PM, Wendell Piez <wapiez at wendellpiez.com> wrote: > Hi, > > I see the 'CHOP' option, turned on by default, for trimming leading > and trailing whitespace and eliminating empty text nodes. > > What about going further? Is there a good way to normalize whitespace > entirely, collapsing any runs of tab-LF-space into single spaces in my > data? > > I think I mentioned earlier the idea of specifying an XSLT > transformation to filter data on ingest (for a similar requirement, > namely removing all comments and PIs). That might be going too far but > any hints you can give me (or pointers to docs) about functionality to > address this sort of thing in general would be welcome. > > Thanks! > Wendell > > -- > Wendell Piez | http://www.wendellpiez.com > XML | XSLT | electronic publishing > Eat Your Vegetables > _____oo_________o_o___ooooo____ooooooo_^ > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
CG
Quoted text · Wendell Piez · 1 line
> (I'm pretty new to XQuery update. I suppose I could always just try it. :-)
Feel free… ;) This should work:
for $x in //*[empty(../*)]
return replace value of node $x
with normalize-space($x)Quoted text · Wendell Piez · 56 lines
> On Fri, Feb 22, 2013 at 5:34 AM, Christian Grün > <christian.gruen at gmail.com> wrote: >> Hi Wendell, >> >> the CHOP option has been introduced at a verly stage of BaseX, and I’m >> not sure if we had added it today. We could add one or more additional >> options to normalize whitespaces or removing PIs/comments from the >> input, but the wish list, and the exception list, would probably >> continue to grow, so I believe that it would be more convenient to >> have a general pre-processing step that takes care of all the >> normalization steps. I’m not sure, however, what’s the best approach >> to do this within BaseX. If it’s possible to cache files on disk >> before adding them to the database, I would recommend XQuery or BaseX >> command scripts, XProc or anything else to prepare the data and delete >> it afterwards. >> >> Comments are welcome, >> Christan >> ___________________________ >> >> On Wed, Feb 20, 2013 at 5:35 PM, Wendell Piez <wapiez at wendellpiez.com> wrote: >>> Hi, >>> >>> I see the 'CHOP' option, turned on by default, for trimming leading >>> and trailing whitespace and eliminating empty text nodes. >>> >>> What about going further? Is there a good way to normalize whitespace >>> entirely, collapsing any runs of tab-LF-space into single spaces in my >>> data? >>> >>> I think I mentioned earlier the idea of specifying an XSLT >>> transformation to filter data on ingest (for a similar requirement, >>> namely removing all comments and PIs). That might be going too far but >>> any hints you can give me (or pointers to docs) about functionality to >>> address this sort of thing in general would be welcome. >>> >>> Thanks! >>> Wendell >>> >>> -- >>> Wendell Piez | http://www.wendellpiez.com >>> XML | XSLT | electronic publishing >>> Eat Your Vegetables >>> _____oo_________o_o___ooooo____ooooooo_^ >>> _______________________________________________ >>> BaseX-Talk mailing list >>> BaseX-Talk at mailman.uni-konstanz.de >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > -- > Wendell Piez | http://www.wendellpiez.com > XML | XSLT | electronic publishing > Eat Your Vegetables > _____oo_________o_o___ooooo____ooooooo_^
WP
Christian,
Indeed, I concur that the wish list would grow; a generalized approach is what we need. I'll let you think about that. :-)
In the meantime, as you suggest, if I'm willing to cache the data first, I have many options. Certainly it's possible in my testing framework but as we build out, it'll be another issue.
Alternatively, once I'm in BaseX -- I'm already deleting unwanted nodes including comments and PIs using a command script. Could I similarly do something like this?
replace value of node //text()[empty(../*)] with
normalize-space(//text()[empty(../*)])
?
(I'm pretty new to XQuery update. I suppose I could always just try it. :-)
Thanks as always,
Wendell
Quoted text · Christian Grün · 48 lines
On Fri, Feb 22, 2013 at 5:34 AM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Wendell, > > the CHOP option has been introduced at a verly stage of BaseX, and I’m > not sure if we had added it today. We could add one or more additional > options to normalize whitespaces or removing PIs/comments from the > input, but the wish list, and the exception list, would probably > continue to grow, so I believe that it would be more convenient to > have a general pre-processing step that takes care of all the > normalization steps. I’m not sure, however, what’s the best approach > to do this within BaseX. If it’s possible to cache files on disk > before adding them to the database, I would recommend XQuery or BaseX > command scripts, XProc or anything else to prepare the data and delete > it afterwards. > > Comments are welcome, > Christan > ___________________________ > > On Wed, Feb 20, 2013 at 5:35 PM, Wendell Piez <wapiez at wendellpiez.com> wrote: >> Hi, >> >> I see the 'CHOP' option, turned on by default, for trimming leading >> and trailing whitespace and eliminating empty text nodes. >> >> What about going further? Is there a good way to normalize whitespace >> entirely, collapsing any runs of tab-LF-space into single spaces in my >> data? >> >> I think I mentioned earlier the idea of specifying an XSLT >> transformation to filter data on ingest (for a similar requirement, >> namely removing all comments and PIs). That might be going too far but >> any hints you can give me (or pointers to docs) about functionality to >> address this sort of thing in general would be welcome. >> >> Thanks! >> Wendell >> >> -- >> Wendell Piez | http://www.wendellpiez.com >> XML | XSLT | electronic publishing >> Eat Your Vegetables >> _____oo_________o_o___ooooo____ooooooo_^ >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
--
Wendell Piez | http://www.wendellpiez.com
XML | XSLT | electronic publishing
Eat Your Vegetables
_____oo_________o_o___ooooo____ooooooo_^
Written for BaseX 7.6. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery