Bug in parse-xml-fragment() and ampersand entity?
Tuesday, 21 November 2023 · 7 messages · last reply 22 Nov 2023
ZD
Zimmel, Daniel21 Nov 2023 15:17 #
Hi,
is this a bug?
Query:
parse-xml-fragment('Tom & Jerry')
Result:
Tom ? Jerry
Same result with:
parse-xml-fragment('Tom &DUMMY; Jerry')
BaseX 10.7
Saxon complains correctly that the resulting document node is not well-formed. BaseX should also return an error, shouldn't it?
Best, Daniel
CG
Christian Grün21 Nov 2023 16:59 #
Hi Daniel,
Yes, I assume we’ll need to call it a bug… Although what BaseX is currently doing is known to us to be out of spec behavior. The function fn:parse-xml-fragments is based on our internal XML parser, which is much faster than the standard XML parser (in particular for small input), and it tolerates input that’s not perfectly well-formed. In addition, it accepts HTML entities without a linked DTD:
parse-xml-fragment(`ä`)We should at least document the behavior or (better) introduce a custom BaseX function for it.
Hope this helps (for now),
Christian
Quoted text · Daniel · 23 lines
On Tue, Nov 21, 2023 at 3:17 PM Zimmel, Daniel <D.Zimmel at esvmedien.de> wrote: > Hi, > > is this a bug? > > Query: > parse-xml-fragment('Tom & Jerry') > > Result: > Tom ? Jerry > > Same result with: > parse-xml-fragment('Tom &DUMMY; Jerry') > > BaseX 10.7 > > Saxon complains correctly that the resulting document node is not > well-formed. > BaseX should also return an error, shouldn't it? > > Best, Daniel >
ZD
Zimmel, Daniel21 Nov 2023 18:34 #
Thanks for the insight!
I can see the benefit with your example – if you look at my example, it is clearly eating the text (“DUMMY”) which might be an edge case, but is obviously a problem when you think the function will give you an error in case of non-wellformedness – some text has silently been deleted.
Daniel
Quoted text · Christian Grün · 24 lines
Von: Christian Grün <christian.gruen at gmail.com> Gesendet: Dienstag, 21. November 2023 16:59 An: Zimmel, Daniel <D.Zimmel at ESVmedien.de> Cc: basex-talk at mailman.uni-konstanz.de Betreff: Re: [basex-talk] Bug in parse-xml-fragment() and ampersand entity? Hi Daniel, Yes, I assume we’ll need to call it a bug… Although what BaseX is currently doing is known to us to be out of spec behavior. The function fn:parse-xml-fragments is based on our internal XML parser, which is much faster than the standard XML parser (in particular for small input), and it tolerates input that’s not perfectly well-formed. In addition, it accepts HTML entities without a linked DTD: parse-xml-fragment(`ä`) We should at least document the behavior or (better) introduce a custom BaseX function for it. Hope this helps (for now), Christian On Tue, Nov 21, 2023 at 3:17 PM Zimmel, Daniel <D.Zimmel at esvmedien.de<mailto:D.Zimmel at esvmedien.de>> wrote: Hi, is this a bug? Query: parse-xml-fragment('Tom & Jerry') Result: Tom ? Jerry Same result with: parse-xml-fragment('Tom &DUMMY; Jerry') BaseX 10.7 Saxon complains correctly that the resulting document node is not well-formed. BaseX should also return an error, shouldn't it? Best, Daniel
CG
Christian Grün21 Nov 2023 18:54 #
Yes, I can see the problem: &DUMMY; ist interpreted as unknown entity and thus replaced with a question mark (a better choice would be the Unicode Replacement Character xFFFD anyway, from today's perspective). We'll keep that in mind and think about alternatives.
If your input is supposed to be interpreted as a single text fragment, one fallback solution (for now) would be
data(parse-xml('<x>' || $string || '</x>'))
Quoted text · Daniel · 82 lines
Zimmel, Daniel <D.Zimmel at esvmedien.de> schrieb am Di., 21. Nov. 2023, 18:34: > Thanks for the insight! > > > > I can see the benefit with your example – if you look at my example, it is > clearly eating the text (“DUMMY”) which might be an edge case, but is > obviously a problem when you think the function will give you an error in > case of non-wellformedness – some text has silently been deleted. > > > > Daniel > > > > *Von:* Christian Grün <christian.gruen at gmail.com> > *Gesendet:* Dienstag, 21. November 2023 16:59 > *An:* Zimmel, Daniel <D.Zimmel at ESVmedien.de> > *Cc:* basex-talk at mailman.uni-konstanz.de > *Betreff:* Re: [basex-talk] Bug in parse-xml-fragment() and ampersand > entity? > > > > Hi Daniel, > > > > Yes, I assume we’ll need to call it a bug… Although what BaseX is > currently doing is known to us to be out of spec behavior. The function > fn:parse-xml-fragments is based on our internal XML parser, which is much > faster than the standard XML parser (in particular for small input), and it > tolerates input that’s not perfectly well-formed. In addition, it accepts > HTML entities without a linked DTD: > > > > parse-xml-fragment(`ä`) > > > > We should at least document the behavior or (better) introduce a custom > BaseX function for it. > > > > Hope this helps (for now), > > Christian > > > > > > > > On Tue, Nov 21, 2023 at 3:17 PM Zimmel, Daniel <D.Zimmel at esvmedien.de> > wrote: > > Hi, > > is this a bug? > > Query: > parse-xml-fragment('Tom & Jerry') > > Result: > Tom ? Jerry > > Same result with: > parse-xml-fragment('Tom &DUMMY; Jerry') > > BaseX 10.7 > > Saxon complains correctly that the resulting document node is not > well-formed. > BaseX should also return an error, shouldn't it? > > Best, Daniel > >
ZD
Zimmel, Daniel22 Nov 2023 12:15 #
Thanks for the example, this makes sense.
In my case, I think I can safely add a function to ensure that any “&” string will be passed to parse-xml-fragment() as “&amp;” because the source of the unparsed xml will always be well-formed.
For background:
I need to wrap a given list of string matches in a textnode to elements and because I need to to this recursively (eg the string is found three times in the textnode) and also need to pass a list of different matches with different replacements, I find it easier to do it with string replacements and then parsing the thenfragmented text node back to XML rather than ending up splitting the text node to multiple nodes directly.
declare function my:replaceText(
$word as xs:string,
$search as xs:string*,
$replace as xs:string*) as xs:string {
(: $search = ("String1", "String2", "String3")
$replace = ("<Replace1/>", "<Replace2/>", "<Replace3/>")
:)
if (empty($search)) then $word
else
replace(my:replaceText($word, tail($search), tail($replace)), '(?<![">])'||my:escapeText(head($search)), head($replace),'j')
};perhaps there is a better way to do this... how would one do this sort of replacement on mixed content without having to parse this back to XML? But this works for me when being careful to add some handling for &, > and <.
+1 for internal parser velocity!
Quoted text · Christian Grün · 36 lines
Von: Christian Grün <christian.gruen at gmail.com> Gesendet: Dienstag, 21. November 2023 18:54 An: Zimmel, Daniel <D.Zimmel at ESVmedien.de> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Betreff: Re: [basex-talk] Bug in parse-xml-fragment() and ampersand entity? Yes, I can see the problem: &DUMMY; ist interpreted as unknown entity and thus replaced with a question mark (a better choice would be the Unicode Replacement Character xFFFD anyway, from today's perspective). We'll keep that in mind and think about alternatives. If your input is supposed to be interpreted as a single text fragment, one fallback solution (for now) would be data(parse-xml('<x>' || $string || '</x>')) Zimmel, Daniel <D.Zimmel at esvmedien.de<mailto:D.Zimmel at esvmedien.de>> schrieb am Di., 21. Nov. 2023, 18:34: Thanks for the insight! I can see the benefit with your example – if you look at my example, it is clearly eating the text (“DUMMY”) which might be an edge case, but is obviously a problem when you think the function will give you an error in case of non-wellformedness – some text has silently been deleted. Daniel Von: Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> Gesendet: Dienstag, 21. November 2023 16:59 An: Zimmel, Daniel <D.Zimmel at ESVmedien.de<mailto:D.Zimmel at ESVmedien.de>> Cc: basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de> Betreff: Re: [basex-talk] Bug in parse-xml-fragment() and ampersand entity? Hi Daniel, Yes, I assume we’ll need to call it a bug… Although what BaseX is currently doing is known to us to be out of spec behavior. The function fn:parse-xml-fragments is based on our internal XML parser, which is much faster than the standard XML parser (in particular for small input), and it tolerates input that’s not perfectly well-formed. In addition, it accepts HTML entities without a linked DTD: parse-xml-fragment(`ä`) We should at least document the behavior or (better) introduce a custom BaseX function for it. Hope this helps (for now), Christian On Tue, Nov 21, 2023 at 3:17 PM Zimmel, Daniel <D.Zimmel at esvmedien.de<mailto:D.Zimmel at esvmedien.de>> wrote: Hi, is this a bug? Query: parse-xml-fragment('Tom & Jerry') Result: Tom ? Jerry Same result with: parse-xml-fragment('Tom &DUMMY; Jerry') BaseX 10.7 Saxon complains correctly that the resulting document node is not well-formed. BaseX should also return an error, shouldn't it? Best, Daniel
CG
Christian Grün22 Nov 2023 12:41 #
You could try this (at least for basic use cases):
declare function local:replaceText(
$input as text(),
$search as xs:string*,
$replace as xs:string*
) as node()* {
let $pattern := string-join($search ! ('\b' || . || '\b'), '|')
for $m in analyze-string($input, $pattern, 'j')/*
return if($m/self::fn:match) then (
element { $replace[index-of($search, $m)] } {}
) else (
text { data($m) }
)
};
local:replaceText(
text { 'a String1 b String2 c' },
("String1", "String2", "String3"),
("Replace1", "Replace2", "Replace3")
)Quoted text · Daniel · 178 lines
On Wed, Nov 22, 2023 at 12:15 PM Zimmel, Daniel <D.Zimmel at esvmedien.de> wrote: > Thanks for the example, this makes sense. > > In my case, I think I can safely add a function to ensure that any “&” > string will be passed to parse-xml-fragment() as “&amp;” because the > source of the unparsed xml will always be well-formed. > > > > For background: > > > > I need to wrap a given list of string matches in a textnode to elements > and because I need to to this recursively (eg the string is found three > times in the textnode) and also need to pass a list of different matches > with different replacements, I find it easier to do it with string > replacements and then parsing the thenfragmented text node back to XML > rather than ending up splitting the text node to multiple nodes directly. > > > > declare function my:replaceText( > > $word as xs:string, > > $search as xs:string*, > > $replace as xs:string*) as xs:string { > > (: $search = ("String1", "String2", "String3") > > $replace = ("<Replace1/>", "<Replace2/>", > "<Replace3/>") > > :) > > if (empty($search)) then $word > > else > > replace(my:replaceText($word, tail($search), tail($replace)), > '(?<![">])'||my:escapeText(head($search)), head($replace),'j') > > }; > > > > perhaps there is a better way to do this... how would one do this sort of > replacement on mixed content without having to parse this back to XML? > > But this works for me when being careful to add some handling for &, > > and <. > > > > +1 for internal parser velocity! > > > > *Von:* Christian Grün <christian.gruen at gmail.com> > *Gesendet:* Dienstag, 21. November 2023 18:54 > *An:* Zimmel, Daniel <D.Zimmel at ESVmedien.de> > *Cc:* BaseX <basex-talk at mailman.uni-konstanz.de> > *Betreff:* Re: [basex-talk] Bug in parse-xml-fragment() and ampersand > entity? > > > > Yes, I can see the problem: &DUMMY; ist interpreted as unknown entity and > thus replaced with a question mark (a better choice would be the Unicode > Replacement Character xFFFD anyway, from today's perspective). We'll keep > that in mind and think about alternatives. > > > > If your input is supposed to be interpreted as a single text fragment, one > fallback solution (for now) would be > > > > data(parse-xml('<x>' || $string || '</x>')) > > > > > > > > > > > > Zimmel, Daniel <D.Zimmel at esvmedien.de> schrieb am Di., 21. Nov. 2023, > 18:34: > > Thanks for the insight! > > > > I can see the benefit with your example – if you look at my example, it is > clearly eating the text (“DUMMY”) which might be an edge case, but is > obviously a problem when you think the function will give you an error in > case of non-wellformedness – some text has silently been deleted. > > > > Daniel > > > > *Von:* Christian Grün <christian.gruen at gmail.com> > *Gesendet:* Dienstag, 21. November 2023 16:59 > *An:* Zimmel, Daniel <D.Zimmel at ESVmedien.de> > *Cc:* basex-talk at mailman.uni-konstanz.de > *Betreff:* Re: [basex-talk] Bug in parse-xml-fragment() and ampersand > entity? > > > > Hi Daniel, > > > > Yes, I assume we’ll need to call it a bug… Although what BaseX is > currently doing is known to us to be out of spec behavior. The function > fn:parse-xml-fragments is based on our internal XML parser, which is much > faster than the standard XML parser (in particular for small input), and it > tolerates input that’s not perfectly well-formed. In addition, it accepts > HTML entities without a linked DTD: > > > > parse-xml-fragment(`ä`) > > > > We should at least document the behavior or (better) introduce a custom > BaseX function for it. > > > > Hope this helps (for now), > > Christian > > > > > > > > On Tue, Nov 21, 2023 at 3:17 PM Zimmel, Daniel <D.Zimmel at esvmedien.de> > wrote: > > Hi, > > is this a bug? > > Query: > parse-xml-fragment('Tom & Jerry') > > Result: > Tom ? Jerry > > Same result with: > parse-xml-fragment('Tom &DUMMY; Jerry') > > BaseX 10.7 > > Saxon complains correctly that the resulting document node is not > well-formed. > BaseX should also return an error, shouldn't it? > > Best, Daniel > >
ZD
Zimmel, Daniel22 Nov 2023 13:40 #
Cool – this looks smart with using index-of on the $replace sequence. I got a feeling that analyze-string might help out with it returning text mixed with elements, but never thought of this pattern.
Thanks for the XQuery enlightenment, I think I can very much build on this example and avoid the entity problems.
Quoted text · Christian Grün · 79 lines
Von: Christian Grün <christian.gruen at gmail.com> Gesendet: Mittwoch, 22. November 2023 12:41 An: Zimmel, Daniel <D.Zimmel at ESVmedien.de> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Betreff: Re: [basex-talk] Bug in parse-xml-fragment() and ampersand entity? You could try this (at least for basic use cases): declare function local:replaceText( $input as text(), $search as xs:string*, $replace as xs:string* ) as node()* { let $pattern := string-join($search ! ('\b' || . || '\b'), '|') for $m in analyze-string($input, $pattern, 'j')/* return if($m/self::fn:match) then ( element { $replace[index-of($search, $m)] } {} ) else ( text { data($m) } ) }; local:replaceText( text { 'a String1 b String2 c' }, ("String1", "String2", "String3"), ("Replace1", "Replace2", "Replace3") ) On Wed, Nov 22, 2023 at 12:15 PM Zimmel, Daniel <D.Zimmel at esvmedien.de<mailto:D.Zimmel at esvmedien.de>> wrote: Thanks for the example, this makes sense. In my case, I think I can safely add a function to ensure that any “&” string will be passed to parse-xml-fragment() as “&amp;” because the source of the unparsed xml will always be well-formed. For background: I need to wrap a given list of string matches in a textnode to elements and because I need to to this recursively (eg the string is found three times in the textnode) and also need to pass a list of different matches with different replacements, I find it easier to do it with string replacements and then parsing the thenfragmented text node back to XML rather than ending up splitting the text node to multiple nodes directly. declare function my:replaceText( $word as xs:string, $search as xs:string*, $replace as xs:string*) as xs:string { (: $search = ("String1", "String2", "String3") $replace = ("<Replace1/>", "<Replace2/>", "<Replace3/>") :) if (empty($search)) then $word else replace(my:replaceText($word, tail($search), tail($replace)), '(?<![">])'||my:escapeText(head($search)), head($replace),'j') }; perhaps there is a better way to do this... how would one do this sort of replacement on mixed content without having to parse this back to XML? But this works for me when being careful to add some handling for &, > and <. +1 for internal parser velocity! Von: Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> Gesendet: Dienstag, 21. November 2023 18:54 An: Zimmel, Daniel <D.Zimmel at ESVmedien.de<mailto:D.Zimmel at ESVmedien.de>> Cc: BaseX <basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de>> Betreff: Re: [basex-talk] Bug in parse-xml-fragment() and ampersand entity? Yes, I can see the problem: &DUMMY; ist interpreted as unknown entity and thus replaced with a question mark (a better choice would be the Unicode Replacement Character xFFFD anyway, from today's perspective). We'll keep that in mind and think about alternatives. If your input is supposed to be interpreted as a single text fragment, one fallback solution (for now) would be data(parse-xml('<x>' || $string || '</x>')) Zimmel, Daniel <D.Zimmel at esvmedien.de<mailto:D.Zimmel at esvmedien.de>> schrieb am Di., 21. Nov. 2023, 18:34: Thanks for the insight! I can see the benefit with your example – if you look at my example, it is clearly eating the text (“DUMMY”) which might be an edge case, but is obviously a problem when you think the function will give you an error in case of non-wellformedness – some text has silently been deleted. Daniel Von: Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> Gesendet: Dienstag, 21. November 2023 16:59 An: Zimmel, Daniel <D.Zimmel at ESVmedien.de<mailto:D.Zimmel at ESVmedien.de>> Cc: basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de> Betreff: Re: [basex-talk] Bug in parse-xml-fragment() and ampersand entity? Hi Daniel, Yes, I assume we’ll need to call it a bug… Although what BaseX is currently doing is known to us to be out of spec behavior. The function fn:parse-xml-fragments is based on our internal XML parser, which is much faster than the standard XML parser (in particular for small input), and it tolerates input that’s not perfectly well-formed. In addition, it accepts HTML entities without a linked DTD: parse-xml-fragment(`ä`) We should at least document the behavior or (better) introduce a custom BaseX function for it. Hope this helps (for now), Christian On Tue, Nov 21, 2023 at 3:17 PM Zimmel, Daniel <D.Zimmel at esvmedien.de<mailto:D.Zimmel at esvmedien.de>> wrote: Hi, is this a bug? Query: parse-xml-fragment('Tom & Jerry') Result: Tom ? Jerry Same result with: parse-xml-fragment('Tom &DUMMY; Jerry') BaseX 10.7 Saxon complains correctly that the resulting document node is not well-formed. BaseX should also return an error, shouldn't it? Best, Daniel
Written for BaseX 10.7. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery