creating epub and odf with bases

On Tue, Sep 8, 2020 at 9:29 AM Jos van den Oever <jos at vandenoever.info> wrote: > > Hello all, > > As you might know, epub files and ODF files are zip files with specific > contents. BaseX supports the expath zip module and could in theory be used for > creating these files if it were not for a missing simple feature. > > There is one rule for epub and ODF files that cannot be followed by BaseX at > the moment: the first file in the zip container should be named 'mimetype' and > is a plain test file that contains the mimetype string. This is meant to allow > applications to read the mimetype at a fixed offset in the file and without > doing decompression. > > In unzip -vl it looks like this: > > Length Method Size Cmpr Date Time CRC-32 Name > -------- ------ ------- ---- ---------- ----- -------- ---- > 20 Stored 20 0% 10-14-2018 05:57 2cab616f mimetype > > Here is an XQuery to create a file with just that entry: > > ````xquery > declare namespace zip = "http://expath.org/ns/zip"; > > let $zip := > <zip:file href="new.epub"> > <zip:entry name="mimetype" compressed="no" method="text"> > {"application/epub+zip"} > </zip:entry> > </zip:file> > return zip:zip-file($zip) > ``` > > BaseX does not support the 'compressed' option. Without that option the file > 'mimetype' is stored in compressed form and cannot be used by applications to > quickly determine the mimetype of the file. > > Modifying the xml in an exisiting epub or ODF with zip:update-entries is also > not possible because the mimetype file is still compressed. > > An additional issue: when reading a zip file, the entries in <zip:file> are > not in the same order as they are in the zip file. So when modifying an > existing file, the mimetype entry has to moved to the front of the list > explicitly. > > In short: to make BaseX support the creation of epub en ODF files it should: > - support the 'compressed' attribute > - retain the order of files in the zip file in the <zip:file> element. > > Best regards, > Jos

On dinsdag 8 september 2020 09:57:50 CEST Christian Grün wrote: > Hi Jos, > > While the ZIP Module is still part of our distribution, it’s not > actively maintained anymore, and we generally recommend our users to > switch to the Archive Module [1]. Providing custom compression levels > for each archive entry is one of the features that is provided by this > newer module.

> > Hope this helps, > Christian > > [1] https://docs.basex.org/wiki/Archive_Module > > On Tue, Sep 8, 2020 at 9:29 AM Jos van den Oever <jos at vandenoever.info> wrote: > > Hello all, > > > > As you might know, epub files and ODF files are zip files with specific > > contents. BaseX supports the expath zip module and could in theory be used > > for creating these files if it were not for a missing simple feature. > > > > There is one rule for epub and ODF files that cannot be followed by BaseX > > at the moment: the first file in the zip container should be named > > 'mimetype' and is a plain test file that contains the mimetype string. > > This is meant to allow applications to read the mimetype at a fixed > > offset in the file and without doing decompression. > > > > In unzip -vl it looks like this: > > Length Method Size Cmpr Date Time CRC-32 Name > > > > -------- ------ ------- ---- ---------- ----- -------- ---- > > > > 20 Stored 20 0% 10-14-2018 05:57 2cab616f mimetype > > > > Here is an XQuery to create a file with just that entry: > > > > ````xquery > > declare namespace zip = "http://expath.org/ns/zip"; > > > > let $zip := > > <zip:file href="new.epub"> > > > > <zip:entry name="mimetype" compressed="no" method="text"> > > > > {"application/epub+zip"} > > > > </zip:entry> > > > > </zip:file> > > return zip:zip-file($zip) > > ``` > > > > BaseX does not support the 'compressed' option. Without that option the > > file 'mimetype' is stored in compressed form and cannot be used by > > applications to quickly determine the mimetype of the file. > > > > Modifying the xml in an exisiting epub or ODF with zip:update-entries is > > also not possible because the mimetype file is still compressed. > > > > An additional issue: when reading a zip file, the entries in <zip:file> > > are > > not in the same order as they are in the zip file. So when modifying an > > existing file, the mimetype entry has to moved to the front of the list > > explicitly. > > > > In short: to make BaseX support the creation of epub en ODF files it

> > - support the 'compressed' attribute > > - retain the order of files in the zip file in the <zip:file> element. > > > > Best regards, > > Jos

> Oh, a shame that the cross-implementation module is not maintained.

> The archive module also compresses the 'mimetype' file with this code:

> > > let $file := "test.ods" > let $archive := file:read-binary($file) > let $content := parse-xml(archive:extract-text($archive, "content.xml")) > let $content := local:change($content, local:add_number_value_type#1) > let $updated := archive:update($archive, "content.xml", $content) > return file:write-binary($file, $updated) > > Cheers, > Jos > > > > > Hope this helps, > > Christian > > > > [1] https://docs.basex.org/wiki/Archive_Module > > > > On Tue, Sep 8, 2020 at 9:29 AM Jos van den Oever <jos at vandenoever.info> > wrote: > > > Hello all, > > > > > > As you might know, epub files and ODF files are zip files with specific > > > contents. BaseX supports the expath zip module and could in theory be used > > > for creating these files if it were not for a missing simple feature. > > > > > > There is one rule for epub and ODF files that cannot be followed by BaseX > > > at the moment: the first file in the zip container should be named > > > 'mimetype' and is a plain test file that contains the mimetype string. > > > This is meant to allow applications to read the mimetype at a fixed > > > offset in the file and without doing decompression. > > > > > > In unzip -vl it looks like this: > > > Length Method Size Cmpr Date Time CRC-32 Name > > > > > > -------- ------ ------- ---- ---------- ----- -------- ---- > > > > > > 20 Stored 20 0% 10-14-2018 05:57 2cab616f mimetype > > > > > > Here is an XQuery to create a file with just that entry: > > > > > > ````xquery > > > declare namespace zip = "http://expath.org/ns/zip"; > > > > > > let $zip := > > > <zip:file href="new.epub"> > > > > > > <zip:entry name="mimetype" compressed="no" method="text"> > > > > > > {"application/epub+zip"} > > > > > > </zip:entry> > > > > > > </zip:file> > > > return zip:zip-file($zip) > > > ``` > > > > > > BaseX does not support the 'compressed' option. Without that option the > > > file 'mimetype' is stored in compressed form and cannot be used by > > > applications to quickly determine the mimetype of the file. > > > > > > Modifying the xml in an exisiting epub or ODF with zip:update-entries is > > > also not possible because the mimetype file is still compressed. > > > > > > An additional issue: when reading a zip file, the entries in <zip:file> > > > are > > > not in the same order as they are in the zip file. So when modifying an > > > existing file, the mimetype entry has to moved to the front of the list > > > explicitly. > > > > > > In short: to make BaseX support the creation of epub en ODF files it > should: > > > - support the 'compressed' attribute > > > - retain the order of files in the zip file in the <zip:file> element. > > > > > > Best regards, > > > Jos >

On dinsdag 8 september 2020 10:59:37 CEST Christian Grün wrote: > > Oh, a shame that the cross-implementation module is not maintained. > > The Archive Module was supposed to become the new EXPath standard. > Unfortunately, different versions of that module were specified one > after another such that the spec that’s currently publicly available > doesn’t reflect our implementation anymore [1]. > > I didn’t know that the ZIP Module is still maintained in other > implementations of XQuery. Is it still popular e.g. in eXist-db?

> > The archive module also compresses the 'mimetype' file with this code: > When calling archive:update, you can supply more properties with an > archive:entry element: > > <archive:entry last-modified='2011-11-11T11:11:11' > compression-level='8' > encoding='US-ASCII'>hello.txt</archive:entry>

> > Best, > Christian > > [1] http://expath.org/spec/archive/20130930 > > > let $file := "test.ods" > > let $archive := file:read-binary($file) > > let $content := parse-xml(archive:extract-text($archive, "content.xml")) > > let $content := local:change($content, local:add_number_value_type#1) > > let $updated := archive:update($archive, "content.xml", $content) > > return file:write-binary($file, $updated) > > > > Cheers, > > Jos > > > > > Hope this helps, > > > Christian > > > > > > [1] https://docs.basex.org/wiki/Archive_Module > > > > > > On Tue, Sep 8, 2020 at 9:29 AM Jos van den Oever <jos at vandenoever.info> > > > > wrote: > > > > Hello all, > > > > > > > > As you might know, epub files and ODF files are zip files with > > > > specific > > > > contents. BaseX supports the expath zip module and could in theory be > > > > used > > > > for creating these files if it were not for a missing simple feature. > > > > > > > > There is one rule for epub and ODF files that cannot be followed by > > > > BaseX > > > > at the moment: the first file in the zip container should be named > > > > 'mimetype' and is a plain test file that contains the mimetype string. > > > > This is meant to allow applications to read the mimetype at a fixed > > > > offset in the file and without doing decompression. > > > > > > > > In unzip -vl it looks like this: > > > > Length Method Size Cmpr Date Time CRC-32 Name > > > > > > > > -------- ------ ------- ---- ---------- ----- -------- ---- > > > > > > > > 20 Stored 20 0% 10-14-2018 05:57 2cab616f mimetype > > > > > > > > Here is an XQuery to create a file with just that entry: > > > > > > > > ````xquery > > > > declare namespace zip = "http://expath.org/ns/zip"; > > > > > > > > let $zip := > > > > <zip:file href="new.epub"> > > > > > > > > <zip:entry name="mimetype" compressed="no" method="text"> > > > > > > > > {"application/epub+zip"} > > > > > > > > </zip:entry> > > > > > > > > </zip:file> > > > > return zip:zip-file($zip) > > > > ``` > > > > > > > > BaseX does not support the 'compressed' option. Without that option > > > > the > > > > file 'mimetype' is stored in compressed form and cannot be used by > > > > applications to quickly determine the mimetype of the file. > > > > > > > > Modifying the xml in an exisiting epub or ODF with zip:update-entries > > > > is > > > > also not possible because the mimetype file is still compressed. > > > > > > > > An additional issue: when reading a zip file, the entries in > > > > <zip:file> > > > > are > > > > not in the same order as they are in the zip file. So when modifying > > > > an > > > > existing file, the mimetype entry has to moved to the front of the > > > > list > > > > explicitly. > > > > > > > > In short: to make BaseX support the creation of epub en ODF files it > > > > should: > > > > - support the 'compressed' attribute > > > > - retain the order of files in the zip file in the <zip:file> > > > > element. > > > > > > > > Best regards, > > > > Jos

On dinsdag 8 september 2020 11:05:45 CEST Jos van den Oever wrote: > On dinsdag 8 september 2020 10:59:37 CEST Christian Grün wrote: > > > Oh, a shame that the cross-implementation module is not maintained. > > > > The Archive Module was supposed to become the new EXPath standard. > > Unfortunately, different versions of that module were specified one > > after another such that the spec that’s currently publicly available > > doesn’t reflect our implementation anymore [1]. > > > > I didn’t know that the ZIP Module is still maintained in other > > implementations of XQuery. Is it still popular e.g. in eXist-db? > > I've used it in production to create governemnt epub files (law bundles). > > > > The archive module also compresses the 'mimetype' file with this code: > > When calling archive:update, you can supply more properties with an > > archive:entry element: > > > > <archive:entry last-modified='2011-11-11T11:11:11' > > > > compression-level='8' > > encoding='US-ASCII'>hello.txt</archive:entry> > > I assumed that files that are not mentioned in the archive:update call or > zip:update-entries call would not be touched. > > I'll see if this way works.

> > [1] http://expath.org/spec/archive/20130930 > > > > > let $file := "test.ods" > > > let $archive := file:read-binary($file) > > > let $content := parse-xml(archive:extract-text($archive, "content.xml")) > > > let $content := local:change($content, local:add_number_value_type#1) > > > let $updated := archive:update($archive, "content.xml", $content) > > > return file:write-binary($file, $updated) > > > > > > Cheers, > > > Jos > > > > > > > Hope this helps, > > > > Christian > > > > > > > > [1] https://docs.basex.org/wiki/Archive_Module > > > > > > > > On Tue, Sep 8, 2020 at 9:29 AM Jos van den Oever > > > > <jos at vandenoever.info> > > > > > > wrote: > > > > > Hello all, > > > > > > > > > > As you might know, epub files and ODF files are zip files with > > > > > specific > > > > > contents. BaseX supports the expath zip module and could in theory > > > > > be > > > > > used > > > > > for creating these files if it were not for a missing simple > > > > > feature. > > > > > > > > > > There is one rule for epub and ODF files that cannot be followed by > > > > > BaseX > > > > > at the moment: the first file in the zip container should be named > > > > > 'mimetype' and is a plain test file that contains the mimetype > > > > > string. > > > > > This is meant to allow applications to read the mimetype at a fixed > > > > > offset in the file and without doing decompression. > > > > > > > > > > In unzip -vl it looks like this: > > > > > Length Method Size Cmpr Date Time CRC-32 Name > > > > > > > > > > -------- ------ ------- ---- ---------- ----- -------- ---- > > > > > > > > > > 20 Stored 20 0% 10-14-2018 05:57 2cab616f mimetype > > > > > > > > > > Here is an XQuery to create a file with just that entry: > > > > > > > > > > ````xquery > > > > > declare namespace zip = "http://expath.org/ns/zip"; > > > > > > > > > > let $zip := > > > > > <zip:file href="new.epub"> > > > > > > > > > > <zip:entry name="mimetype" compressed="no" method="text"> > > > > > > > > > > {"application/epub+zip"} > > > > > > > > > > </zip:entry> > > > > > > > > > > </zip:file> > > > > > return zip:zip-file($zip) > > > > > ``` > > > > > > > > > > BaseX does not support the 'compressed' option. Without that option > > > > > the > > > > > file 'mimetype' is stored in compressed form and cannot be used by > > > > > applications to quickly determine the mimetype of the file. > > > > > > > > > > Modifying the xml in an exisiting epub or ODF with > > > > > zip:update-entries > > > > > is > > > > > also not possible because the mimetype file is still compressed. > > > > > > > > > > An additional issue: when reading a zip file, the entries in > > > > > <zip:file> > > > > > are > > > > > not in the same order as they are in the zip file. So when modifying > > > > > an > > > > > existing file, the mimetype entry has to moved to the front of the > > > > > list > > > > > explicitly. > > > > > > > > > > In short: to make BaseX support the creation of epub en ODF files it > > > > > > should: > > > > > - support the 'compressed' attribute > > > > > - retain the order of files in the zip file in the <zip:file> > > > > > element. > > > > > > > > > > Best regards, > > > > > Jos

> This example demonstrates that compression-level="0" does do what > the api promises:

On dinsdag 8 september 2020 11:57:16 CEST Christian Grün wrote: > > This example demonstrates that compression-level="0" does do what > > > the api promises: > I can have a closer look into that. Could you possibly provide me with > a little self-contained example that I can run out of the box?

> Here is an example that creates a new archive that uses compression-level="0" > and algorithm="stored" and still compresses that entry. > > Note that the archive level option 'algorithm' is unfortumate because often it > is only single entries such as 'mimetype' or images that should not be > compressed.

On dinsdag 8 september 2020 13:06:19 CEST Christian Grün wrote: > > Here is an example that creates a new archive that uses > > compression-level="0" and algorithm="stored" and still compresses that > > entry. > > > > Note that the archive level option 'algorithm' is unfortumate because > > often it is only single entries such as 'mimetype' or images that should > > not be compressed. > > Thanks for the example. – My observation is that the entry is indeed > archived uncompressed if you choose compression-level="0"; but I think > what you are saying is that an uncompressed DEFLATE entry is not the > same as an uncompressed STORED entry, right, and that ODS and ePub > files require certain files to be stored with the STORED algorithm, is > that right?

> The Archive Module has a long history, and was initially based on a > proposal for the Zorba XQuery Processor back in 2012. I don’t actually > remember why the algorithm option was not adopted for the single > archive entries; maybe that would have been more reasonable. As we > seem to be the only implementation left today, we could think about > changing that. I doubt anyway that people will use different > compression levels for single archive entries (apart from archiving > them uncompressed), so it might be a better solution to define one > global compression level for the whole archive.

On dinsdag 8 september 2020 14:06:20 CEST Jos van den Oever wrote: > On dinsdag 8 september 2020 13:06:19 CEST Christian Grün wrote: > > > Here is an example that creates a new archive that uses > > > compression-level="0" and algorithm="stored" and still compresses that > > > entry. > > > > > > Note that the archive level option 'algorithm' is unfortumate because > > > often it is only single entries such as 'mimetype' or images that should > > > not be compressed. > > > > Thanks for the example. – My observation is that the entry is indeed > > archived uncompressed if you choose compression-level="0"; but I think > > what you are saying is that an uncompressed DEFLATE entry is not the > > same as an uncompressed STORED entry, right, and that ODS and ePub > > files require certain files to be stored with the STORED algorithm, is > > that right? > > The thing that counts is that you can read the mimetype enty name and > contents without decompression starting from byte 30. That way tools such > as 'find' can report the mimetype. > > The file generated with the attached script in BaseX 9.4.3 beta gives this: > > $ file -i test.epub > test.epub: application/octet-stream; charset=binary > $ unzip -vl test.epub > Archive: test.epub > Length Method Size Cmpr Date Time CRC-32 Name > -------- ------ ------- ---- ---------- ----- -------- ---- > 20 Defl:N 25 -25% 09-08-2020 13:54 2cab616f mimetype > -------- ------- --- ------- > 20 25 -25% 1 file > $ hexdump -C test.epub | head -4 > 00000000 50 4b 03 04 14 00 08 08 08 00 d9 6e 28 51 00 00 > |PK.........n(Q..| 00000010 00 00 00 00 00 00 00 00 00 00 08 00 00 00 6d > 69 |..............mi| 00000020 6d 65 74 79 70 65 01 14 00 eb ff 61 70 70 > 6c 69 |metype.....appli| 00000030 63 61 74 69 6f 6e 2f 65 70 75 62 2b 7a > 69 70 50 |cation/epub+zipP| > > There are 5 bytes between 'mimetype' and 'applicatino/epub+zip'. These are > deflate information. If the entry is 'stored' there are no bytes between the > entry name and the contents and the zip will be recognized by the epub and > ODF applications (and use less space) than when it is deflated with > compression- level 0. > > > The Archive Module has a long history, and was initially based on a > > proposal for the Zorba XQuery Processor back in 2012. I don’t actually > > remember why the algorithm option was not adopted for the single > > archive entries; maybe that would have been more reasonable. As we > > seem to be the only implementation left today, we could think about > > changing that. I doubt anyway that people will use different > > compression levels for single archive entries (apart from archiving > > them uncompressed), so it might be a better solution to define one > > global compression level for the whole archive. > > From a practical point of view (regardless of what is in the specification) > it makes sense to store 'mimetype' uncompressed and also store files such > as png and jpg that are already compressed in the 'stored' way. If that can > be achieved easily: great, but at least it should be possible. I think the > simplest solution is to save compression-level=0 as stored. > > Best regards, > Jos

> There are 5 bytes between 'mimetype' and 'applicatino/epub+zip'. These are > deflate information. If the entry is 'stored' there are no bytes between the > entry name […]

> I think the simplest solution is to save compression-level=0 as stored.

> Operation failed: STORED entry missing size, compressed size, or crc-32.

On Tue, Sep 8, 2020 at 2:06 PM Jos van den Oever <jos at vandenoever.info> wrote: > > On dinsdag 8 september 2020 13:06:19 CEST Christian Grün wrote: > > > Here is an example that creates a new archive that uses > > > compression-level="0" and algorithm="stored" and still compresses that > > > entry. > > > > > > Note that the archive level option 'algorithm' is unfortumate because > > > often it is only single entries such as 'mimetype' or images that should > > > not be compressed. > > > > Thanks for the example. – My observation is that the entry is indeed > > archived uncompressed if you choose compression-level="0"; but I think > > what you are saying is that an uncompressed DEFLATE entry is not the > > same as an uncompressed STORED entry, right, and that ODS and ePub > > files require certain files to be stored with the STORED algorithm, is > > that right? > > The thing that counts is that you can read the mimetype enty name and contents > without decompression starting from byte 30. That way tools such as 'find' can > report the mimetype. > > The file generated with the attached script in BaseX 9.4.3 beta gives this: > > $ file -i test.epub > test.epub: application/octet-stream; charset=binary > $ unzip -vl test.epub > Archive: test.epub > Length Method Size Cmpr Date Time CRC-32 Name > -------- ------ ------- ---- ---------- ----- -------- ---- > 20 Defl:N 25 -25% 09-08-2020 13:54 2cab616f mimetype > -------- ------- --- ------- > 20 25 -25% 1 file > $ hexdump -C test.epub | head -4 > 00000000 50 4b 03 04 14 00 08 08 08 00 d9 6e 28 51 00 00 |PK.........n(Q..| > 00000010 00 00 00 00 00 00 00 00 00 00 08 00 00 00 6d 69 |..............mi| > 00000020 6d 65 74 79 70 65 01 14 00 eb ff 61 70 70 6c 69 |metype.....appli| > 00000030 63 61 74 69 6f 6e 2f 65 70 75 62 2b 7a 69 70 50 |cation/epub+zipP| > > There are 5 bytes between 'mimetype' and 'applicatino/epub+zip'. These are > deflate information. If the entry is 'stored' there are no bytes between the > entry name and the contents and the zip will be recognized by the epub and ODF > applications (and use less space) than when it is deflated with compression- > level 0. > > > The Archive Module has a long history, and was initially based on a > > proposal for the Zorba XQuery Processor back in 2012. I don’t actually > > remember why the algorithm option was not adopted for the single > > archive entries; maybe that would have been more reasonable. As we > > seem to be the only implementation left today, we could think about > > changing that. I doubt anyway that people will use different > > compression levels for single archive entries (apart from archiving > > them uncompressed), so it might be a better solution to define one > > global compression level for the whole archive. > > From a practical point of view (regardless of what is in the specification) it > makes sense to store 'mimetype' uncompressed and also store files such as png > and jpg that are already compressed in the 'stored' way. If that can be > > Best regards, > Jos

On Tue, Sep 8, 2020 at 2:27 PM Christian Grün <christian.gruen at gmail.com> wrote: > > Hi Jos, > > > There are 5 bytes between 'mimetype' and 'applicatino/epub+zip'. These are > > deflate information. If the entry is 'stored' there are no bytes between the > > entry name […] > > Great, so we are talking about the same thing. > > > I think the simplest solution is to save compression-level=0 as stored. > > That was also my thought. A quick fix caused the following error > message (similar to what is described here [1])… > > > Operation failed: STORED entry missing size, compressed size, or crc-32. > > …which means we’ll probably need to set additional values before > writing the actual byte array. I’ll see what we can do. > > I was surprised to learn more about the deficiencies of the Archive > Module. The module was already used many times in the past to create > ePub files, so my guess would be that these files could be opened by > many readers, but were not 100% valid. How do you usually proceed to > check the validity of ePub files? > > Best, > Christian > > [1] https://stackoverflow.com/questions/1206970/how-to-create-uncompressed-zip-archive-in-java > > > > > > On Tue, Sep 8, 2020 at 2:06 PM Jos van den Oever <jos at vandenoever.info> wrote: > > > > On dinsdag 8 september 2020 13:06:19 CEST Christian Grün wrote: > > > > Here is an example that creates a new archive that uses > > > > compression-level="0" and algorithm="stored" and still compresses that > > > > entry. > > > > > > > > Note that the archive level option 'algorithm' is unfortumate because > > > > often it is only single entries such as 'mimetype' or images that should > > > > not be compressed. > > > > > > Thanks for the example. – My observation is that the entry is indeed > > > archived uncompressed if you choose compression-level="0"; but I think > > > what you are saying is that an uncompressed DEFLATE entry is not the > > > same as an uncompressed STORED entry, right, and that ODS and ePub > > > files require certain files to be stored with the STORED algorithm, is > > > that right? > > > > The thing that counts is that you can read the mimetype enty name and contents > > without decompression starting from byte 30. That way tools such as 'find' can > > report the mimetype. > > > > The file generated with the attached script in BaseX 9.4.3 beta gives this: > > > > $ file -i test.epub > > test.epub: application/octet-stream; charset=binary > > $ unzip -vl test.epub > > Archive: test.epub > > Length Method Size Cmpr Date Time CRC-32 Name > > -------- ------ ------- ---- ---------- ----- -------- ---- > > 20 Defl:N 25 -25% 09-08-2020 13:54 2cab616f mimetype > > -------- ------- --- ------- > > 20 25 -25% 1 file > > $ hexdump -C test.epub | head -4 > > 00000000 50 4b 03 04 14 00 08 08 08 00 d9 6e 28 51 00 00 |PK.........n(Q..| > > 00000010 00 00 00 00 00 00 00 00 00 00 08 00 00 00 6d 69 |..............mi| > > 00000020 6d 65 74 79 70 65 01 14 00 eb ff 61 70 70 6c 69 |metype.....appli| > > 00000030 63 61 74 69 6f 6e 2f 65 70 75 62 2b 7a 69 70 50 |cation/epub+zipP| > > > > There are 5 bytes between 'mimetype' and 'applicatino/epub+zip'. These are > > deflate information. If the entry is 'stored' there are no bytes between the > > entry name and the contents and the zip will be recognized by the epub and ODF > > applications (and use less space) than when it is deflated with compression- > > level 0. > > > > > The Archive Module has a long history, and was initially based on a > > > proposal for the Zorba XQuery Processor back in 2012. I don’t actually > > > remember why the algorithm option was not adopted for the single > > > archive entries; maybe that would have been more reasonable. As we > > > seem to be the only implementation left today, we could think about > > > changing that. I doubt anyway that people will use different > > > compression levels for single archive entries (apart from archiving > > > them uncompressed), so it might be a better solution to define one > > > global compression level for the whole archive. > > > > From a practical point of view (regardless of what is in the specification) it > > makes sense to store 'mimetype' uncompressed and also store files such as png > > and jpg that are already compressed in the 'stored' way. If that can be > > > > Best regards, > > Jos

On dinsdag 8 september 2020 14:27:55 CEST Christian Grün wrote: > Hi Jos, > > > There are 5 bytes between 'mimetype' and 'applicatino/epub+zip'. These are > > deflate information. If the entry is 'stored' there are no bytes between > > the entry name […] > > Great, so we are talking about the same thing. > > > I think the simplest solution is to save compression-level=0 as stored. > > That was also my thought. A quick fix caused the following error > message (similar to what is described here [1])… > > > Operation failed: STORED entry missing size, compressed size, or crc-32. > > …which means we’ll probably need to set additional values before > writing the actual byte array. I’ll see what we can do. > > I was surprised to learn more about the deficiencies of the Archive > Module. The module was already used many times in the past to create > ePub files, so my guess would be that these files could be opened by > many readers, but were not 100% valid. How do you usually proceed to > check the validity of ePub files?

On dinsdag 8 september 2020 14:40:34 CEST Christian Grün wrote: > I’ve updated the code; if the compression level is set to 0, entries > will be STORED [1]. Feel free to check out the latest snapshot [2].

> > [1] > https://github.com/BaseXdb/basex/commit/67ad584a85e0848432e19b4f587fbabfc2f > c38e5 [2] https://files.basex.org/releases/latest/ > > On Tue, Sep 8, 2020 at 2:27 PM Christian Grün <christian.gruen at gmail.com> wrote: > > Hi Jos, > > > > > There are 5 bytes between 'mimetype' and 'applicatino/epub+zip'. These > > > are > > > deflate information. If the entry is 'stored' there are no bytes between > > > the entry name […] > > > > Great, so we are talking about the same thing. > > > > > I think the simplest solution is to save compression-level=0 as stored. > > > > That was also my thought. A quick fix caused the following error > > message (similar to what is described here [1])… > > > > > Operation failed: STORED entry missing size, compressed size, or crc-32. > > > > …which means we’ll probably need to set additional values before > > writing the actual byte array. I’ll see what we can do. > > > > I was surprised to learn more about the deficiencies of the Archive > > Module. The module was already used many times in the past to create > > ePub files, so my guess would be that these files could be opened by > > many readers, but were not 100% valid. How do you usually proceed to > > check the validity of ePub files? > > > > Best, > > Christian > > > > [1] > > https://stackoverflow.com/questions/1206970/how-to-create-uncompressed-zi > > p-archive-in-java> > > On Tue, Sep 8, 2020 at 2:06 PM Jos van den Oever <jos at vandenoever.info> wrote: > > > On dinsdag 8 september 2020 13:06:19 CEST Christian Grün wrote: > > > > > Here is an example that creates a new archive that uses > > > > > compression-level="0" and algorithm="stored" and still compresses > > > > > that > > > > > entry. > > > > > > > > > > Note that the archive level option 'algorithm' is unfortumate > > > > > because > > > > > often it is only single entries such as 'mimetype' or images that > > > > > should > > > > > not be compressed. > > > > > > > > Thanks for the example. – My observation is that the entry is indeed > > > > archived uncompressed if you choose compression-level="0"; but I think > > > > what you are saying is that an uncompressed DEFLATE entry is not the > > > > same as an uncompressed STORED entry, right, and that ODS and ePub > > > > files require certain files to be stored with the STORED algorithm, is > > > > that right? > > > > > > The thing that counts is that you can read the mimetype enty name and > > > contents without decompression starting from byte 30. That way tools > > > such as 'find' can report the mimetype. > > > > > > The file generated with the attached script in BaseX 9.4.3 beta gives > > > this: > > > > > > $ file -i test.epub > > > test.epub: application/octet-stream; charset=binary > > > $ unzip -vl test.epub > > > Archive: test.epub > > > > > > Length Method Size Cmpr Date Time CRC-32 Name > > > > > > -------- ------ ------- ---- ---------- ----- -------- ---- > > > > > > 20 Defl:N 25 -25% 09-08-2020 13:54 2cab616f mimetype > > > > > > -------- ------- --- ------- > > > > > > 20 25 -25% 1 file > > > > > > $ hexdump -C test.epub | head -4 > > > 00000000 50 4b 03 04 14 00 08 08 08 00 d9 6e 28 51 00 00 > > > |PK.........n(Q..| 00000010 00 00 00 00 00 00 00 00 00 00 08 00 00 00 > > > 6d 69 |..............mi| 00000020 6d 65 74 79 70 65 01 14 00 eb ff > > > 61 70 70 6c 69 |metype.....appli| 00000030 63 61 74 69 6f 6e 2f 65 > > > 70 75 62 2b 7a 69 70 50 |cation/epub+zipP| > > > > > > There are 5 bytes between 'mimetype' and 'applicatino/epub+zip'. These > > > are > > > deflate information. If the entry is 'stored' there are no bytes between > > > the entry name and the contents and the zip will be recognized by the > > > epub and ODF applications (and use less space) than when it is deflated > > > with compression- level 0. > > > > > > > The Archive Module has a long history, and was initially based on a > > > > proposal for the Zorba XQuery Processor back in 2012. I don’t actually > > > > remember why the algorithm option was not adopted for the single > > > > archive entries; maybe that would have been more reasonable. As we > > > > seem to be the only implementation left today, we could think about > > > > changing that. I doubt anyway that people will use different > > > > compression levels for single archive entries (apart from archiving > > > > them uncompressed), so it might be a better solution to define one > > > > global compression level for the whole archive. > > > > > > From a practical point of view (regardless of what is in the > > > specification) it makes sense to store 'mimetype' uncompressed and also > > > store files such as png and jpg that are already compressed in the > > > 'stored' way. If that can be > > > > > > Best regards, > > > Jos

> Creating a new file epub or odf file works correctly now but archive:update() > does not retain the 'stored' property for the manifest file. (It does retain > the order of the entries).

> EPub actually adopted the practice of a stored mimetype from ODF.

On dinsdag 8 september 2020 16:59:14 CEST Christian Grün wrote: > > Creating a new file epub or odf file works correctly now but > > archive:update() does not retain the 'stored' property for the manifest > > file. (It does retain the order of the entries). > > The stored property will now be retained if an archive is updated. > > Up to now, archive:update removed existing update candidates from the > archive and added new entries at the end. I changed this as well: If > existing files are updated, the original order will be preserved, and > new files will be added in the order in which they were supplied by > the user. > > > EPub actually adopted the practice of a stored mimetype from ODF. > > I didn’t know that. And thanks for the link to the W3C epub checker > (which I actually used by myself a long time ago) and the ODF checker.

> BaseX 9.4.3 is scheduled to be released later this week.

On dinsdag 8 september 2020 19:31:41 CEST Liam R. E. Quin wrote: > On Tue, 2020-09-08 at 17:07 +0200, Jos van den Oever wrote: > > Thank you for making the improvements. This is much cleaner imho than > > bash + > > zip + xsltproc. :-) > > A minor addition - i've sometimes started with a base zip file with the > uncompressed "mimetype" entry in it, and just added the rest to that > file fromXQuery or XSLT, without problems.

On Tue, 2020-09-08 at 17:07 +0200, Jos van den Oever wrote: > Thank you for making the improvements. This is much cleaner imho than > bash + > zip + xsltproc. :-)