Main Page » Mailing List » January 2017

Losing some space entities

Friday, 13 January 2017 · 12 messages · last reply 20 Jan 2017

France Baril13 Jan 2017 20:46 #

Hi, When I serialize content to HTML5, I lose some entities. xquery: adds '‍' in front of some content and outputs html html: should have '&zwj;', but doesn't. I've tried with 8204 and even the half space 8201. The only special space that seems to work is  , but it won't do for what I need right now. Code sample:

*let* *$target-table* :=
      *copy* *$copy* := *$base-table*
      *modify*(
         *for **$td* *in **$copy*//*tr*/*td*[*position*()=
*$column-to-filter-by*]
         *let* *$new-value* := ('&#8205;', *for **$node* *in **$td*/*node*()
*return **$node*)
         *return replace* *value* *of* *node* *$td* *with **$new-value*
      )
      *return **$copy*

*return * *$target-table* Is there any way to solve this?

-- France Baril Architecte documentaire / Documentation architect france.baril at architextus.com

Christian Grün14 Jan 2017 10:34 #

Hi France, In the database, all Unicode characters will be stored in their standard (decoded) representation. As a result, it is not possible to preserve entities from an original document. For XML serialization via WebDAV, we have one special rule for converting non-breaking spaces (xA0) to entities. Which other Unicode characters would you like to have converted to entities? Cheers, Christian

Quoted text · France Baril · 35 lines

On Fri, Jan 13, 2017 at 7:45 PM, France Baril <france.baril at architextus.com> wrote: > Hi, > > When I serialize content to HTML5, I lose some entities. > > xquery: adds '‍' in front of some content and outputs html > html: should have '&zwj;', but doesn't. I've tried with 8204 and even the > half space 8201. > > The only special space that seems to work is  , but it won't do for > what I need right now. > > Code sample: > > let $target-table := > copy $copy := $base-table > modify( > for $td in $copy//tr/td[position()=$column-to-filter-by] > let $new-value := ('‍', for $node in $td/node() return $node) > return replace value of node $td with $new-value > ) > return $copy > > return > $target-table > > > Is there any way to solve this? > > > -- > France Baril > Architecte documentaire / Documentation architect > france.baril at architextus.com

France Baril15 Jan 2017 02:51 #

I use   - non-breakable space a lot. It's used widely in French. I wanted:

&#8204; - &zwnj;
&#8205; - &zwj;

I've had clients who have asked for the thin space in the past &#8201... I nicely told them 'no'. I've resolved my joiner issue with an alternate solution for now. But if you are opening up the can of worms, and it's not much more work, why not implement all the space and joiner entities   to ‍? Regards, France

Quoted text · Christian Grün · 52 lines

On Sat, Jan 14, 2017 at 4:34 AM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi France, > > In the database, all Unicode characters will be stored in their > standard (decoded) representation. As a result, it is not possible to > preserve entities from an original document. For XML serialization via > WebDAV, we have one special rule for converting non-breaking spaces > (xA0) to entities. Which other Unicode characters would you like to > have converted to entities? > > Cheers, > Christian > > > On Fri, Jan 13, 2017 at 7:45 PM, France Baril > <france.baril at architextus.com> wrote: > > Hi, > > > > When I serialize content to HTML5, I lose some entities. > > > > xquery: adds '‍' in front of some content and outputs html > > html: should have '&zwj;', but doesn't. I've tried with 8204 and even the > > half space 8201. > > > > The only special space that seems to work is  , but it won't do for > > what I need right now. > > > > Code sample: > > > > let $target-table := > > copy $copy := $base-table > > modify( > > for $td in $copy//tr/td[position()=$column-to-filter-by] > > let $new-value := ('‍', for $node in $td/node() return > $node) > > return replace value of node $td with $new-value > > ) > > return $copy > > > > return > > $target-table > > > > > > Is there any way to solve this? > > > > > > -- > > France Baril > > Architecte documentaire / Documentation architect > > france.baril at architextus.com >

-- France Baril Architecte documentaire / Documentation architect france.baril at architextus.com

Christian Grün15 Jan 2017 17:17 #

Hi France,

Quoted text · France Baril · 1 line

> why not implement all the space and joiner entities   to ‍?

Good idea. I have added even some more characters (see [1]): 8192 – 8207 8232 – 8239 8287 – 8303 A new snapshot (close to the final version 8.6) is online [2]; I’be grateful for some testing feedback! Christian [1] http://docs.basex.org/wiki/WebDAV#Resources [2] http://files.basex.org/releases/latest/

Quoted text · Christian Grün · 61 lines

> On Sat, Jan 14, 2017 at 4:34 AM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> Hi France, >> >> In the database, all Unicode characters will be stored in their >> standard (decoded) representation. As a result, it is not possible to >> preserve entities from an original document. For XML serialization via >> WebDAV, we have one special rule for converting non-breaking spaces >> (xA0) to entities. Which other Unicode characters would you like to >> have converted to entities? >> >> Cheers, >> Christian >> >> >> On Fri, Jan 13, 2017 at 7:45 PM, France Baril >> <france.baril at architextus.com> wrote: >> > Hi, >> > >> > When I serialize content to HTML5, I lose some entities. >> > >> > xquery: adds '‍' in front of some content and outputs html >> > html: should have '&zwj;', but doesn't. I've tried with 8204 and even >> > the >> > half space 8201. >> > >> > The only special space that seems to work is  , but it won't do for >> > what I need right now. >> > >> > Code sample: >> > >> > let $target-table := >> > copy $copy := $base-table >> > modify( >> > for $td in $copy//tr/td[position()=$column-to-filter-by] >> > let $new-value := ('‍', for $node in $td/node() return >> > $node) >> > return replace value of node $td with $new-value >> > ) >> > return $copy >> > >> > return >> > $target-table >> > >> > >> > Is there any way to solve this? >> > >> > >> > -- >> > France Baril >> > Architecte documentaire / Documentation architect >> > france.baril at architextus.com > > > > > -- > France Baril > Architecte documentaire / Documentation architect > france.baril at architextus.com

Christian Grünin reply to France Baril, 19 Jan 2017, 01:0019 Jan 2017 00:02 #

Entities will only be serialized when using WebDAV. Did you try that as well? Am 19.01.2017 12:00 vorm. schrieb "France Baril" < france.baril at architextus.com>: I haven't run many tests yet, but importing the file and exporting the files again from the gui only retains   Also, when the entity is in decimal (&8332;), it gets transformed to hex (not a big deal in my book). Other entities are 'applied', but not retained. I'm on BaseX 8.6 beta 4c3daeb

Quoted text · Christian Grün · 86 lines

On Sun, Jan 15, 2017 at 11:17 AM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi France, > > > why not implement all the space and joiner entities   to ‍? > > Good idea. I have added even some more characters (see [1]): > > 8192 – 8207 > 8232 – 8239 > 8287 – 8303 > > A new snapshot (close to the final version 8.6) is online [2]; I’be > grateful for some testing feedback! > > Christian > > [1] http://docs.basex.org/wiki/WebDAV#Resources > [2] http://files.basex.org/releases/latest/ > > > > > On Sat, Jan 14, 2017 at 4:34 AM, Christian Grün < > christian.gruen at gmail.com> > > wrote: > >> > >> Hi France, > >> > >> In the database, all Unicode characters will be stored in their > >> standard (decoded) representation. As a result, it is not possible to > >> preserve entities from an original document. For XML serialization via > >> WebDAV, we have one special rule for converting non-breaking spaces > >> (xA0) to entities. Which other Unicode characters would you like to > >> have converted to entities? > >> > >> Cheers, > >> Christian > >> > >> > >> On Fri, Jan 13, 2017 at 7:45 PM, France Baril > >> <france.baril at architextus.com> wrote: > >> > Hi, > >> > > >> > When I serialize content to HTML5, I lose some entities. > >> > > >> > xquery: adds '‍' in front of some content and outputs html > >> > html: should have '&zwj;', but doesn't. I've tried with 8204 and even > >> > the > >> > half space 8201. > >> > > >> > The only special space that seems to work is  , but it won't do > for > >> > what I need right now. > >> > > >> > Code sample: > >> > > >> > let $target-table := > >> > copy $copy := $base-table > >> > modify( > >> > for $td in $copy//tr/td[position()=$column-to-filter-by] > >> > let $new-value := ('‍', for $node in $td/node() return > >> > $node) > >> > return replace value of node $td with $new-value > >> > ) > >> > return $copy > >> > > >> > return > >> > $target-table > >> > > >> > > >> > Is there any way to solve this? > >> > > >> > > >> > -- > >> > France Baril > >> > Architecte documentaire / Documentation architect > >> > france.baril at architextus.com > > > > > > > > > > -- > > France Baril > > Architecte documentaire / Documentation architect > > france.baril at architextus.com >

-- France Baril Architecte documentaire / Documentation architect france.baril at architextus.com

France Barilin reply to Christian Grün, 15 Jan 2017, 17:1719 Jan 2017 01:00 #

I haven't run many tests yet, but importing the file and exporting the files again from the gui only retains   Also, when the entity is in decimal (&8332;), it gets transformed to hex (not a big deal in my book). Other entities are 'applied', but not retained. I'm on BaseX 8.6 beta 4c3daeb

Quoted text · Christian Grün · 86 lines

On Sun, Jan 15, 2017 at 11:17 AM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi France, > > > why not implement all the space and joiner entities   to ‍? > > Good idea. I have added even some more characters (see [1]): > > 8192 – 8207 > 8232 – 8239 > 8287 – 8303 > > A new snapshot (close to the final version 8.6) is online [2]; I’be > grateful for some testing feedback! > > Christian > > [1] http://docs.basex.org/wiki/WebDAV#Resources > [2] http://files.basex.org/releases/latest/ > > > > > On Sat, Jan 14, 2017 at 4:34 AM, Christian Grün < > christian.gruen at gmail.com> > > wrote: > >> > >> Hi France, > >> > >> In the database, all Unicode characters will be stored in their > >> standard (decoded) representation. As a result, it is not possible to > >> preserve entities from an original document. For XML serialization via > >> WebDAV, we have one special rule for converting non-breaking spaces > >> (xA0) to entities. Which other Unicode characters would you like to > >> have converted to entities? > >> > >> Cheers, > >> Christian > >> > >> > >> On Fri, Jan 13, 2017 at 7:45 PM, France Baril > >> <france.baril at architextus.com> wrote: > >> > Hi, > >> > > >> > When I serialize content to HTML5, I lose some entities. > >> > > >> > xquery: adds '‍' in front of some content and outputs html > >> > html: should have '&zwj;', but doesn't. I've tried with 8204 and even > >> > the > >> > half space 8201. > >> > > >> > The only special space that seems to work is  , but it won't do > for > >> > what I need right now. > >> > > >> > Code sample: > >> > > >> > let $target-table := > >> > copy $copy := $base-table > >> > modify( > >> > for $td in $copy//tr/td[position()=$column-to-filter-by] > >> > let $new-value := ('‍', for $node in $td/node() return > >> > $node) > >> > return replace value of node $td with $new-value > >> > ) > >> > return $copy > >> > > >> > return > >> > $target-table > >> > > >> > > >> > Is there any way to solve this? > >> > > >> > > >> > -- > >> > France Baril > >> > Architecte documentaire / Documentation architect > >> > france.baril at architextus.com > > > > > > > > > > -- > > France Baril > > Architecte documentaire / Documentation architect > > france.baril at architextus.com >

-- France Baril Architecte documentaire / Documentation architect france.baril at architextus.com

France Barilin reply to Christian Grün, 19 Jan 2017, 00:0219 Jan 2017 02:00 #

Import works, playing with webdav is giving good results so far. I still want to see how it outputs html through file:write with html5 serialization and if I can retain the entites in my pdfs. That's my next step.

Quoted text · Christian Grün · 113 lines

On Wed, Jan 18, 2017 at 6:02 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Entities will only be serialized when using WebDAV. Did you try that as > well? > > > Am 19.01.2017 12:00 vorm. schrieb "France Baril" < > france.baril at architextus.com>: > > I haven't run many tests yet, but importing the file and exporting the > files again from the gui only retains   Also, when the entity is in > decimal (&8332;), it gets transformed to hex (not a big deal in my book). > Other entities are 'applied', but not retained. > > I'm on BaseX 8.6 beta 4c3daeb > > On Sun, Jan 15, 2017 at 11:17 AM, Christian Grün < > christian.gruen at gmail.com> wrote: > >> Hi France, >> >> > why not implement all the space and joiner entities   to ‍? >> >> Good idea. I have added even some more characters (see [1]): >> >> 8192 – 8207 >> 8232 – 8239 >> 8287 – 8303 >> >> A new snapshot (close to the final version 8.6) is online [2]; I’be >> grateful for some testing feedback! >> >> Christian >> >> [1] http://docs.basex.org/wiki/WebDAV#Resources >> [2] http://files.basex.org/releases/latest/ >> >> >> >> > On Sat, Jan 14, 2017 at 4:34 AM, Christian Grün < >> christian.gruen at gmail.com> >> > wrote: >> >> >> >> Hi France, >> >> >> >> In the database, all Unicode characters will be stored in their >> >> standard (decoded) representation. As a result, it is not possible to >> >> preserve entities from an original document. For XML serialization via >> >> WebDAV, we have one special rule for converting non-breaking spaces >> >> (xA0) to entities. Which other Unicode characters would you like to >> >> have converted to entities? >> >> >> >> Cheers, >> >> Christian >> >> >> >> >> >> On Fri, Jan 13, 2017 at 7:45 PM, France Baril >> >> <france.baril at architextus.com> wrote: >> >> > Hi, >> >> > >> >> > When I serialize content to HTML5, I lose some entities. >> >> > >> >> > xquery: adds '‍' in front of some content and outputs html >> >> > html: should have '&zwj;', but doesn't. I've tried with 8204 and even >> >> > the >> >> > half space 8201. >> >> > >> >> > The only special space that seems to work is  , but it won't do >> for >> >> > what I need right now. >> >> > >> >> > Code sample: >> >> > >> >> > let $target-table := >> >> > copy $copy := $base-table >> >> > modify( >> >> > for $td in $copy//tr/td[position()=$column-to-filter-by] >> >> > let $new-value := ('‍', for $node in $td/node() return >> >> > $node) >> >> > return replace value of node $td with $new-value >> >> > ) >> >> > return $copy >> >> > >> >> > return >> >> > $target-table >> >> > >> >> > >> >> > Is there any way to solve this? >> >> > >> >> > >> >> > -- >> >> > France Baril >> >> > Architecte documentaire / Documentation architect >> >> > france.baril at architextus.com >> > >> > >> > >> > >> > -- >> > France Baril >> > Architecte documentaire / Documentation architect >> > france.baril at architextus.com >> > > > > -- > France Baril > Architecte documentaire / Documentation architect > france.baril at architextus.com > > >

-- France Baril Architecte documentaire / Documentation architect france.baril at architextus.com

France Baril19 Jan 2017 02:45 #

When I serialize these from a restxq to file with file:write I get the   as   and   as   but I get none of the other entities.

Quoted text · France Baril · 129 lines

On Wed, Jan 18, 2017 at 6:59 PM, France Baril <france.baril at architextus.com> wrote: > Import works, playing with webdav is giving good results so far. I still > want to see how it outputs html through file:write with html5 serialization > and if I can retain the entites in my pdfs. That's my next step. > > On Wed, Jan 18, 2017 at 6:02 PM, Christian Grün <christian.gruen at gmail.com > > wrote: > >> Entities will only be serialized when using WebDAV. Did you try that as >> well? >> >> >> Am 19.01.2017 12:00 vorm. schrieb "France Baril" < >> france.baril at architextus.com>: >> >> I haven't run many tests yet, but importing the file and exporting the >> files again from the gui only retains   Also, when the entity is in >> decimal (&8332;), it gets transformed to hex (not a big deal in my book). >> Other entities are 'applied', but not retained. >> >> I'm on BaseX 8.6 beta 4c3daeb >> >> On Sun, Jan 15, 2017 at 11:17 AM, Christian Grün < >> christian.gruen at gmail.com> wrote: >> >>> Hi France, >>> >>> > why not implement all the space and joiner entities   to ‍? >>> >>> Good idea. I have added even some more characters (see [1]): >>> >>> 8192 – 8207 >>> 8232 – 8239 >>> 8287 – 8303 >>> >>> A new snapshot (close to the final version 8.6) is online [2]; I’be >>> grateful for some testing feedback! >>> >>> Christian >>> >>> [1] http://docs.basex.org/wiki/WebDAV#Resources >>> [2] http://files.basex.org/releases/latest/ >>> >>> >>> >>> > On Sat, Jan 14, 2017 at 4:34 AM, Christian Grün < >>> christian.gruen at gmail.com> >>> > wrote: >>> >> >>> >> Hi France, >>> >> >>> >> In the database, all Unicode characters will be stored in their >>> >> standard (decoded) representation. As a result, it is not possible to >>> >> preserve entities from an original document. For XML serialization via >>> >> WebDAV, we have one special rule for converting non-breaking spaces >>> >> (xA0) to entities. Which other Unicode characters would you like to >>> >> have converted to entities? >>> >> >>> >> Cheers, >>> >> Christian >>> >> >>> >> >>> >> On Fri, Jan 13, 2017 at 7:45 PM, France Baril >>> >> <france.baril at architextus.com> wrote: >>> >> > Hi, >>> >> > >>> >> > When I serialize content to HTML5, I lose some entities. >>> >> > >>> >> > xquery: adds '‍' in front of some content and outputs html >>> >> > html: should have '&zwj;', but doesn't. I've tried with 8204 and >>> even >>> >> > the >>> >> > half space 8201. >>> >> > >>> >> > The only special space that seems to work is  , but it won't >>> do for >>> >> > what I need right now. >>> >> > >>> >> > Code sample: >>> >> > >>> >> > let $target-table := >>> >> > copy $copy := $base-table >>> >> > modify( >>> >> > for $td in $copy//tr/td[position()=$column-to-filter-by] >>> >> > let $new-value := ('‍', for $node in $td/node() >>> return >>> >> > $node) >>> >> > return replace value of node $td with $new-value >>> >> > ) >>> >> > return $copy >>> >> > >>> >> > return >>> >> > $target-table >>> >> > >>> >> > >>> >> > Is there any way to solve this? >>> >> > >>> >> > >>> >> > -- >>> >> > France Baril >>> >> > Architecte documentaire / Documentation architect >>> >> > france.baril at architextus.com >>> > >>> > >>> > >>> > >>> > -- >>> > France Baril >>> > Architecte documentaire / Documentation architect >>> > france.baril at architextus.com >>> >> >> >> >> -- >> France Baril >> Architecte documentaire / Documentation architect >> france.baril at architextus.com >> >> >> > > > -- > France Baril > Architecte documentaire / Documentation architect > france.baril at architextus.com >

-- France Baril Architecte documentaire / Documentation architect france.baril at architextus.com

Christian Grün19 Jan 2017 16:04 #

In XQuery, you can specify entities via output parameters; see the attached example.

Quoted text · France Baril · 17 lines

> When I serialize these from a restxq to file with file:write > > I get the   as   and   as   but I get none of the > other entities. > > On Wed, Jan 18, 2017 at 6:59 PM, France Baril <france.baril at architextus.com> > wrote: >> >> Import works, playing with webdav is giving good results so far. I still >> want to see how it outputs html through file:write with html5 serialization >> and if I can retain the entites in my pdfs. That's my next step. >> >> On Wed, Jan 18, 2017 at 6:02 PM, Christian Grün >> <christian.gruen at gmail.com> wrote: >>> >>> Entities will only be serialized when using WebDAV. Did you try that as >>> well?

Kristian Kankainen20 Jan 2017 17:53 #

Hi! This was informative and I'd like to add this example to the BaseX wiki. Could I add it somewhere on the page about Serialization? http://docs.basex.org/wiki/Serialization Thanks Kristian K 19.01.2017 17:04 Christian Grün kirjutas:

Quoted text · Christian Grün · 19 lines

> In XQuery, you can specify entities via output parameters; see the > attached example. > > >> When I serialize these from a restxq to file with file:write >> >> I get the   as   and   as   but I get none of the >> other entities. >> >> On Wed, Jan 18, 2017 at 6:59 PM, France Baril <france.baril at architextus.com> >> wrote: >>> Import works, playing with webdav is giving good results so far. I still >>> want to see how it outputs html through file:write with html5 serialization >>> and if I can retain the entites in my pdfs. That's my next step. >>> >>> On Wed, Jan 18, 2017 at 6:02 PM, Christian Grün >>> <christian.gruen at gmail.com> wrote: >>>> Entities will only be serialized when using WebDAV. Did you try that as >>>> well?

Christian Grün20 Jan 2017 17:54 #

Hi Kristian,

Quoted text · Kristian Kankainen · 2 lines

> This was informative and I'd like to add this example to the BaseX wiki. > Could I add it somewhere on the page about Serialization?

Edits are always welcome! Thanks in advance, Christian

Quoted text · Kristian Kankainen · 35 lines

> > http://docs.basex.org/wiki/Serialization > > Thanks > Kristian K > > > > 19.01.2017 17:04 Christian Grün kirjutas: >> >> In XQuery, you can specify entities via output parameters; see the >> attached example. >> >> >>> When I serialize these from a restxq to file with file:write >>> >>> I get the   as   and   as   but I get none of the >>> other entities. >>> >>> On Wed, Jan 18, 2017 at 6:59 PM, France Baril >>> <france.baril at architextus.com> >>> wrote: >>>> >>>> Import works, playing with webdav is giving good results so far. I still >>>> want to see how it outputs html through file:write with html5 >>>> serialization >>>> and if I can retain the entites in my pdfs. That's my next step. >>>> >>>> On Wed, Jan 18, 2017 at 6:02 PM, Christian Grün >>>> <christian.gruen at gmail.com> wrote: >>>>> >>>>> Entities will only be serialized when using WebDAV. Did you try that as >>>>> well? > >

Kristian Kankainen20 Jan 2017 21:42 #

I tried my best at http://docs.basex.org/wiki/Serialization#Character_mappings Cheers Kristian 20.01.2017 18:54 Christian Grün kirjutas:

Quoted text · Christian Grün · 39 lines

> Hi Kristian, > >> This was informative and I'd like to add this example to the BaseX wiki. >> Could I add it somewhere on the page about Serialization? > Edits are always welcome! > > Thanks in advance, > Christian > >> http://docs.basex.org/wiki/Serialization >> >> Thanks >> Kristian K >> >> >> >> 19.01.2017 17:04 Christian Grün kirjutas: >>> In XQuery, you can specify entities via output parameters; see the >>> attached example. >>> >>> >>>> When I serialize these from a restxq to file with file:write >>>> >>>> I get the   as   and   as   but I get none of the >>>> other entities. >>>> >>>> On Wed, Jan 18, 2017 at 6:59 PM, France Baril >>>> <france.baril at architextus.com> >>>> wrote: >>>>> Import works, playing with webdav is giving good results so far. I still >>>>> want to see how it outputs html through file:write with html5 >>>>> serialization >>>>> and if I can retain the entites in my pdfs. That's my next step. >>>>> >>>>> On Wed, Jan 18, 2017 at 6:02 PM, Christian Grün >>>>> <christian.gruen at gmail.com> wrote: >>>>>> Entities will only be serialized when using WebDAV. Did you try that as >>>>>> well? >>

Written for BaseX 8.5. BaseX may have changed since: please check the current documentation.

⚡Generated with XQuery

`J`	Select the next message
`K`	Select the previous message
`N`	Next thread
`P`	Previous thread
`U`	Go up one level
`/`	Jump to the search field