Extracting binary stored zip into database as binary stored elements not working
Friday, 11 December 2015 · 10 messages · last reply 12 Dec 2015
CG
Quoted text · E. Wray Johnson · 4 lines
> First I used db:store to store delta.zip into the database. > My first problem was I got an error "invalid entry CRC: expected 0x0 but > got..." > The raw stored file was slightly larger than the original zip.
Could you possibly give us an example to reproduce this?
Quoted text · E. Wray Johnson · 1 line
> What is wrong with this code?
The info view shows…
Result:
- Hit(s): 0 Items
- Updated: 0 Items
- Printed: 0 Bytes
- Read Locking: global
- Write Locking: none
…so my current assumption is that the higher-order functions are not prepared yet to do updates. I will look at this. For now, the following code should work.
for $entry at $pos in $entries
let $content := $contents[$pos]
let $target := $basePath || '/' || $entry/text()
return db:store($database, $target, $content)Notice, btw, that try/catch won’t catch anything in your query, because the update itself will be run after query evaluation has been completed (look for Pending Update List in our documentation for more information).
EJ
E. Wray Johnson11 Dec 2015 13:03 #
My zip file has files nested in subfolders.
First I used db:store to store delta.zip into the database.
My first problem was I got an error "
*invalid entry CRC: expected 0x0 but got..."*
The raw stored file was slightly larger
*than the original zip.*
So I replaced the zip file in the raw folder with the original one and the first error went away.
Now I don't get any error bu the extracted file are not stored as binaries
What is wrong with this code?
I removed the updating annotation and returned $target instead of the try/store/catch inside the loop and got the expected result path name
let $database := 'db'
let $extractTo := 'target.zip'
(: path to raw stored zip :)
let $archivePath := 'alpha/bravo/charlie/delta.zip'
let $archive := db:retrieve($database, $archivePath)
let $basePath := replace($archivePath, '[^/]+$', '') || $extractTo
let $entries := archive:entries($archive)
let $contents := archive:extract-binary($archive)
return for-each-pair($entries, $contents, %updating function($entry,
$content) {
let $target := $basePath || '/' || $entry/text()
return try {
db:store($database, $target, $content)
} catch * {
()
}
})CG
Christian Grün11 Dec 2015 16:10 #
Quoted text · E. Wray Johnson · 2 lines
> In the meantime can you shed some light into the > [FODC0002] Resource > 'delta.zip' does not exist.' error?
(cc to the list): It simply says that a database resource does not exist.
Maybe you have more than one BaseX database directory in your environment? Could you try
db:list('...db...') and check what results are returned in the GUI and via the client?Quoted text · Christian Grün · 17 lines
> On Fri, Dec 11, 2015 at 9:53 AM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> > The raw stored file was slightly larger than the original zip. >> >> Do you have an example for that? >> >> > But when I submit the exact same query using the Java client I get the >> > following error: >> > >> > Stopped at ., 1/216: >> > [FODC0002] Resource 'delta.zip' does not exist. >> >> It shouldn’t make a difference. I’ll probably need more info to track this >> down. > >
CG
Quoted text · E. Wray Johnson · 2 lines
> This is happening in the call to db:store where the path is > 'alpha/bravo/charlie/delta.zip/subfolder/subfile.ext'
Hm, so this is the relative path to the local resource you want to store? Does it work if you specify the absolute path?
Quoted text · E. Wray Johnson · 2 lines
> Could it have something to do with a default archive parser for archives in > the Java Client ?
The Java client (if it’s the one I believe it is) is very dumb; it just passes on everything to the server.
Quoted text · E. Wray Johnson · 38 lines
> > > > On Fri, Dec 11, 2015 at 10:10 AM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> > In the meantime can you shed some light into the > [FODC0002] Resource >> > 'delta.zip' does not exist.' error? >> >> (cc to the list): It simply says that a database resource does not exist. >> >> Maybe you have more than one BaseX database directory in your >> environment? Could you try db:list('...db...') and check what results >> are returned in the GUI and via the client? >> >> >> > On Fri, Dec 11, 2015 at 9:53 AM, Christian Grün >> > <christian.gruen at gmail.com> >> > wrote: >> >> >> >> > The raw stored file was slightly larger than the original zip. >> >> >> >> Do you have an example for that? >> >> >> >> > But when I submit the exact same query using the Java client I get >> >> > the >> >> > following error: >> >> > >> >> > Stopped at ., 1/216: >> >> > [FODC0002] Resource 'delta.zip' does not exist. >> >> >> >> It shouldn’t make a difference. I’ll probably need more info to track >> >> this >> >> down. >> > >> > > >
EJ
This is happening in the call to db:store where the path is
'alpha/bravo/charlie/delta.zip/subfolder/subfile.ext'
The error is:
Stopped at ., 1/216:
Quoted text · Christian Grün · 1 line
>> > [FODC0002] Resource 'delta.zip' does not exist.
Could it have something to do with a default archive parser for archives in the Java Client ?
Quoted text · Christian Grün · 32 lines
On Fri, Dec 11, 2015 at 10:10 AM, Christian Grün <christian.gruen at gmail.com> wrote: > > In the meantime can you shed some light into the > [FODC0002] Resource > > 'delta.zip' does not exist.' error? > > (cc to the list): It simply says that a database resource does not exist. > > Maybe you have more than one BaseX database directory in your > environment? Could you try db:list('...db...') and check what results > are returned in the GUI and via the client? > > > > On Fri, Dec 11, 2015 at 9:53 AM, Christian Grün < > christian.gruen at gmail.com> > > wrote: > >> > >> > The raw stored file was slightly larger than the original zip. > >> > >> Do you have an example for that? > >> > >> > But when I submit the exact same query using the Java client I get the > >> > following error: > >> > > >> > Stopped at ., 1/216: > >> > [FODC0002] Resource 'delta.zip' does not exist. > >> > >> It shouldn’t make a difference. I’ll probably need more info to track > this > >> down. > > > > >
CG
Quoted text · E. Wray Johnson · 1 line
> return for-each-pair($entries, $contents, %updating function($entry,
In the latest snapshot [1], the built-in higher-order functions (for-each-pair, apply, etc.) are not allowed anymore to evaluate updating functions. As a result, your original query will now yield an error (XPTY0004). As I didn’t see this covered in the specs, I have added a W3 bug entry [2].
Cheers,
Christian
[1] http://files.basex.org/releases/latest/
[2] https://www.w3.org/Bugs/Public/show_bug.cgi?id=29333
Quoted text · E. Wray Johnson · 9 lines
> $content) { > let $target := $basePath || '/' || $entry/text() > return try { > db:store($database, $target, $content) > } catch * { > () > } > }) >
EJ
What is the format for an absolute path for db:store?
Neither of the following works:
- '/db/alpha/bravo/charlie/delta.zip/subfolder/subfile.ext'
- and '/'alpha/bravo/charlie/delta.zip/subfolder/subfile.ext'
However my original code works in the BaseX GUI, just not through the Java Client.
Could it have something to the with the period or .zip in the middle of the path?
Quoted text · Christian Grün · 58 lines
On Fri, Dec 11, 2015 at 11:11 AM, Christian Grün <christian.gruen at gmail.com> wrote: > > This is happening in the call to db:store where the path is > > 'alpha/bravo/charlie/delta.zip/subfolder/subfile.ext' > > Hm, so this is the relative path to the local resource you want to > store? Does it work if you specify the absolute path? > > > Could it have something to do with a default archive parser for archives > in > > the Java Client ? > > The Java client (if it’s the one I believe it is) is very dumb; it > just passes on everything to the server. > > > > > > > > > > On Fri, Dec 11, 2015 at 10:10 AM, Christian Grün < > christian.gruen at gmail.com> > > wrote: > >> > >> > In the meantime can you shed some light into the > [FODC0002] > Resource > >> > 'delta.zip' does not exist.' error? > >> > >> (cc to the list): It simply says that a database resource does not > exist. > >> > >> Maybe you have more than one BaseX database directory in your > >> environment? Could you try db:list('...db...') and check what results > >> are returned in the GUI and via the client? > >> > >> > >> > On Fri, Dec 11, 2015 at 9:53 AM, Christian Grün > >> > <christian.gruen at gmail.com> > >> > wrote: > >> >> > >> >> > The raw stored file was slightly larger than the original zip. > >> >> > >> >> Do you have an example for that? > >> >> > >> >> > But when I submit the exact same query using the Java client I get > >> >> > the > >> >> > following error: > >> >> > > >> >> > Stopped at ., 1/216: > >> >> > [FODC0002] Resource 'delta.zip' does not exist. > >> >> > >> >> It shouldn’t make a difference. I’ll probably need more info to track > >> >> this > >> >> down. > >> > > >> > > > > > >
CG
Christian Grün11 Dec 2015 19:28 #
Could you please provide me with a minimized example that shows the problem, and which I can run both in the GUI and via the Java client?
Quoted text · E. Wray Johnson · 76 lines
On Fri, Dec 11, 2015 at 6:24 PM, E. Wray Johnson <wray.johnson at gmail.com> wrote: > What is the format for an absolute path for db:store? > Neither of the following works: > > '/db/alpha/bravo/charlie/delta.zip/subfolder/subfile.ext' > and '/'alpha/bravo/charlie/delta.zip/subfolder/subfile.ext' > > However my original code works in the BaseX GUI, just not through the Java > Client. > > Could it have something to the with the period or .zip in the middle of the > path? > > > > On Fri, Dec 11, 2015 at 11:11 AM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> > This is happening in the call to db:store where the path is >> > 'alpha/bravo/charlie/delta.zip/subfolder/subfile.ext' >> >> Hm, so this is the relative path to the local resource you want to >> store? Does it work if you specify the absolute path? >> >> > Could it have something to do with a default archive parser for archives >> > in >> > the Java Client ? >> >> The Java client (if it’s the one I believe it is) is very dumb; it >> just passes on everything to the server. >> >> >> > >> > >> > >> > On Fri, Dec 11, 2015 at 10:10 AM, Christian Grün >> > <christian.gruen at gmail.com> >> > wrote: >> >> >> >> > In the meantime can you shed some light into the > [FODC0002] >> >> > Resource >> >> > 'delta.zip' does not exist.' error? >> >> >> >> (cc to the list): It simply says that a database resource does not >> >> exist. >> >> >> >> Maybe you have more than one BaseX database directory in your >> >> environment? Could you try db:list('...db...') and check what results >> >> are returned in the GUI and via the client? >> >> >> >> >> >> > On Fri, Dec 11, 2015 at 9:53 AM, Christian Grün >> >> > <christian.gruen at gmail.com> >> >> > wrote: >> >> >> >> >> >> > The raw stored file was slightly larger than the original zip. >> >> >> >> >> >> Do you have an example for that? >> >> >> >> >> >> > But when I submit the exact same query using the Java client I get >> >> >> > the >> >> >> > following error: >> >> >> > >> >> >> > Stopped at ., 1/216: >> >> >> > [FODC0002] Resource 'delta.zip' does not exist. >> >> >> >> >> >> It shouldn’t make a difference. I’ll probably need more info to >> >> >> track >> >> >> this >> >> >> down. >> >> > >> >> > >> > >> > > >
EJ
E. Wray Johnson11 Dec 2015 21:05 #
Steps:
1. Create or choose a database (e.g. named 'db')
2. Store in BaseX a raw zip file (e.g.
'alpha/bravo/charlie/delta.zip'). My zip file has files in folders.
3. Choose a target name that also has a ".zip" extension to similar to 'include name of archive in path' behavior. Note, it should not be the same as the raw zip file stored (e.g. 'delta.zip')
4. Run the following in the BaseX GUI client - it will work
5. Repeat 1 and 2 using the BaseX Java Client example - it does not work
GUI CLIENT - DOES WORK
========================================
let $database := 'db'
let $extractTo := 'target.zip'
let $archivePath := 'alpha/bravo/charlie/delta.zip'
let $archive := db:retrieve($database, $archivePath)
let $basePath := replace($archivePath, '[^/]+$', '') || $extractTo
let $entries := archive:entries($archive)
let $contents := archive:extract-binary($archive)
return for-each-pair($entries, $contents, %updating function($entry,
$content) {
let $target := $basePath || '/' || $entry/text()
return try {
db:store($database, $target, $content)
} catch * {
()
}
})JAVA - DOES NOT WORK:
===============================================
// Usage: Create a BaseXClient object and call this method.
// For example:
// ExplodeArchive("db", "target.zip", "alpha/bravo/charlie/delta.zip", ..)
private static void ExplodeArchive(final String database, String extractTo,
String archivePath, BaseXClient session) throws IOException {
try {
String queryString =
"let $database := '" + database +"' " +
"let $extractTo := '" + extractTo + "' " +
"let $archivePath := '"+ archivePath + "' " +
"let $archive := db:retrieve($database, $archivePath)
" +
"let $basePath := replace($archivePath, '[^/]+$', '')
|| $extractTo " +
"let $entries := archive:entries($archive) " +
"let $contents := archive:extract-binary($archive) " +
"for $entry at $pos in $entries " +
"let $content := $contents[$pos] " +
"let $target := $basePath || '/' || $entry/text() " +
"return db:store($database, "+ "$target, $content)";
final BaseXClient.Query query = session.query(queryString);
query.execute();
} finally {
session.close();
}
}Quoted text · Christian Grün · 89 lines
On Fri, Dec 11, 2015 at 1:28 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Could you please provide me with a minimized example that shows the > problem, and which I can run both in the GUI and via the Java client? > > > On Fri, Dec 11, 2015 at 6:24 PM, E. Wray Johnson <wray.johnson at gmail.com> > wrote: > > What is the format for an absolute path for db:store? > > Neither of the following works: > > > > '/db/alpha/bravo/charlie/delta.zip/subfolder/subfile.ext' > > and '/'alpha/bravo/charlie/delta.zip/subfolder/subfile.ext' > > > > However my original code works in the BaseX GUI, just not through the > Java > > Client. > > > > Could it have something to the with the period or .zip in the middle of > the > > path? > > > > > > > > On Fri, Dec 11, 2015 at 11:11 AM, Christian Grün < > christian.gruen at gmail.com> > > wrote: > >> > >> > This is happening in the call to db:store where the path is > >> > 'alpha/bravo/charlie/delta.zip/subfolder/subfile.ext' > >> > >> Hm, so this is the relative path to the local resource you want to > >> store? Does it work if you specify the absolute path? > >> > >> > Could it have something to do with a default archive parser for > archives > >> > in > >> > the Java Client ? > >> > >> The Java client (if it’s the one I believe it is) is very dumb; it > >> just passes on everything to the server. > >> > >> > >> > > >> > > >> > > >> > On Fri, Dec 11, 2015 at 10:10 AM, Christian Grün > >> > <christian.gruen at gmail.com> > >> > wrote: > >> >> > >> >> > In the meantime can you shed some light into the > [FODC0002] > >> >> > Resource > >> >> > 'delta.zip' does not exist.' error? > >> >> > >> >> (cc to the list): It simply says that a database resource does not > >> >> exist. > >> >> > >> >> Maybe you have more than one BaseX database directory in your > >> >> environment? Could you try db:list('...db...') and check what results > >> >> are returned in the GUI and via the client? > >> >> > >> >> > >> >> > On Fri, Dec 11, 2015 at 9:53 AM, Christian Grün > >> >> > <christian.gruen at gmail.com> > >> >> > wrote: > >> >> >> > >> >> >> > The raw stored file was slightly larger than the original zip. > >> >> >> > >> >> >> Do you have an example for that? > >> >> >> > >> >> >> > But when I submit the exact same query using the Java client I > get > >> >> >> > the > >> >> >> > following error: > >> >> >> > > >> >> >> > Stopped at ., 1/216: > >> >> >> > [FODC0002] Resource 'delta.zip' does not exist. > >> >> >> > >> >> >> It shouldn’t make a difference. I’ll probably need more info to > >> >> >> track > >> >> >> this > >> >> >> down. > >> >> > > >> >> > > >> > > >> > > > > > >
EJ
E. Wray Johnson12 Dec 2015 05:06 #
Both problems fixed. Here is how:
- I had transposed the 2 parameters in my Java method call.
- I unzipped and the re-zipped the defective archive using WIndows' built-in zip and it no longer generates the error. My only guess is that it was created using some non-standard zip utility.
Quoted text · E. Wray Johnson · 162 lines
On Fri, Dec 11, 2015 at 2:05 PM, E. Wray Johnson <wray.johnson at gmail.com> wrote: > Steps: > > 1. Create or choose a database (e.g. named 'db') > 2. Store in BaseX a raw zip file (e.g. > 'alpha/bravo/charlie/delta.zip'). My zip file has files in folders. > 3. Choose a target name that also has a ".zip" extension to similar to > 'include name of archive in path' behavior. Note, it should not be the > same as the raw zip file stored (e.g. 'delta.zip') > 4. Run the following in the BaseX GUI client - it will work > 5. Repeat 1 and 2 using the BaseX Java Client example - it does not > work > > > GUI CLIENT - DOES WORK > ======================================== > let $database := 'db' > let $extractTo := 'target.zip' > let $archivePath := 'alpha/bravo/charlie/delta.zip' > > let $archive := db:retrieve($database, $archivePath) > let $basePath := replace($archivePath, '[^/]+$', '') || $extractTo > let $entries := archive:entries($archive) > let $contents := archive:extract-binary($archive) > return for-each-pair($entries, $contents, %updating function($entry, > $content) { > let $target := $basePath || '/' || $entry/text() > return try { > db:store($database, $target, $content) > } catch * { > () > } > }) > > > JAVA - DOES NOT WORK: > =============================================== > // Usage: Create a BaseXClient object and call this method. > // For example: > // ExplodeArchive("db", "target.zip", "alpha/bravo/charlie/delta.zip", ..) > > private static void ExplodeArchive(final String database, String > extractTo, String archivePath, BaseXClient session) throws IOException { > try { > String queryString = > "let $database := '" + database +"' " + > "let $extractTo := '" + extractTo + "' " + > "let $archivePath := '"+ archivePath + "' " + > "let $archive := db:retrieve($database, $archivePath) > " + > "let $basePath := replace($archivePath, '[^/]+$', '') > || $extractTo " + > "let $entries := archive:entries($archive) " + > "let $contents := archive:extract-binary($archive) " + > "for $entry at $pos in $entries " + > "let $content := $contents[$pos] " + > "let $target := $basePath || '/' || $entry/text() " + > "return db:store($database, "+ "$target, $content)"; > final BaseXClient.Query query = session.query(queryString); > query.execute(); > } finally { > session.close(); > } > } > > > > > On Fri, Dec 11, 2015 at 1:28 PM, Christian Grün <christian.gruen at gmail.com > > wrote: > >> Could you please provide me with a minimized example that shows the >> problem, and which I can run both in the GUI and via the Java client? >> >> >> On Fri, Dec 11, 2015 at 6:24 PM, E. Wray Johnson <wray.johnson at gmail.com> >> wrote: >> > What is the format for an absolute path for db:store? >> > Neither of the following works: >> > >> > '/db/alpha/bravo/charlie/delta.zip/subfolder/subfile.ext' >> > and '/'alpha/bravo/charlie/delta.zip/subfolder/subfile.ext' >> > >> > However my original code works in the BaseX GUI, just not through the >> Java >> > Client. >> > >> > Could it have something to the with the period or .zip in the middle of >> the >> > path? >> > >> > >> > >> > On Fri, Dec 11, 2015 at 11:11 AM, Christian Grün < >> christian.gruen at gmail.com> >> > wrote: >> >> >> >> > This is happening in the call to db:store where the path is >> >> > 'alpha/bravo/charlie/delta.zip/subfolder/subfile.ext' >> >> >> >> Hm, so this is the relative path to the local resource you want to >> >> store? Does it work if you specify the absolute path? >> >> >> >> > Could it have something to do with a default archive parser for >> archives >> >> > in >> >> > the Java Client ? >> >> >> >> The Java client (if it’s the one I believe it is) is very dumb; it >> >> just passes on everything to the server. >> >> >> >> >> >> > >> >> > >> >> > >> >> > On Fri, Dec 11, 2015 at 10:10 AM, Christian Grün >> >> > <christian.gruen at gmail.com> >> >> > wrote: >> >> >> >> >> >> > In the meantime can you shed some light into the > [FODC0002] >> >> >> > Resource >> >> >> > 'delta.zip' does not exist.' error? >> >> >> >> >> >> (cc to the list): It simply says that a database resource does not >> >> >> exist. >> >> >> >> >> >> Maybe you have more than one BaseX database directory in your >> >> >> environment? Could you try db:list('...db...') and check what >> results >> >> >> are returned in the GUI and via the client? >> >> >> >> >> >> >> >> >> > On Fri, Dec 11, 2015 at 9:53 AM, Christian Grün >> >> >> > <christian.gruen at gmail.com> >> >> >> > wrote: >> >> >> >> >> >> >> >> > The raw stored file was slightly larger than the original zip. >> >> >> >> >> >> >> >> Do you have an example for that? >> >> >> >> >> >> >> >> > But when I submit the exact same query using the Java client I >> get >> >> >> >> > the >> >> >> >> > following error: >> >> >> >> > >> >> >> >> > Stopped at ., 1/216: >> >> >> >> > [FODC0002] Resource 'delta.zip' does not exist. >> >> >> >> >> >> >> >> It shouldn’t make a difference. I’ll probably need more info to >> >> >> >> track >> >> >> >> this >> >> >> >> down. >> >> >> > >> >> >> > >> >> > >> >> > >> > >> > >> > >
Written for BaseX 8.3. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery