Some string peculiarities
Friday, 17 November 2017 · 2 messages
KK
Kristian Kankainen17 Nov 2017 10:51 #
Hello
I encountered some strange things when tokenizing text. Sample runnable code is added below. Here is my list of problems:
1) the regular expression "(\.){3}" doesn't match the same as "(\.\.\.)". Shouldn't they be equal?
2) a very annoying whitespace is placed text to the newline of out:nl(). It is placed before out:nl() if it is called in the beginning of an element, or it is placed after the newline if out:nl() is called in the end of an element.
E.g the serialized output is either "<s> this" or ". </s>".
Sample runnable code:
for $text in ("this one... is the first.", "this one is second.")
return
<s>{
out:nl(),
string-join(
analyze-string($text, '(\.){3}|[\W]')//text()[not(.=" ")],
out:nl()
),
out:nl()
}</s>
,
"--------- VERSUS --------- "
,
for $text in ("this one... is the first.", "this one is second.")
return
<s>{
out:nl(),
string-join(
analyze-string($text, '(\.\.\.)|[\W]')//text()[not(.=" ")],
out:nl()
),
out:nl()
}</s>CG
Christian Grün17 Nov 2017 15:44 #
Hi Kristian,
Quoted text · Kristian Kankainen · 2 lines
> 1) the regular expression "(\.){3}" doesn't match the same as "(\.\.\.)". > Shouldn't they be equal?
They look similar indeed, but are not equivalent. In the first expression, the repeated dots will be part of the resulting match, but not of the subordinate match group. "(\.{3})" is probably what you are looking for.
Quoted text · Kristian Kankainen · 4 lines
> 2) a very annoying whitespace is placed text to the newline of out:nl(). It > is placed before out:nl() if it is called in the beginning of an element, or > it is placed after the newline if out:nl() is called in the end of an > element.
This is not related to out:nl(), but to the way how XQuery node construction works (“The individual strings resulting from the previous step are merged into a single string by concatenating them with a single space character between each pair.” [1]). Simply use string-join() for concatenating the results, as you do anyway:
let $regex := '(\.{3})|[\W]'
let $text := "this one... is the first."
return <s>{ string-join(
(
out:nl(),
analyze-string($text, $regex)//text()[not(.=" ")],
out:nl()
),
out:nl()
)}</s>Best,
Christian
[1] https://www.w3.org/TR/xquery-31/#id-textConstructors
Written for BaseX 8.6. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery