XQuery performance degradation as number of json files grows

Wednesday, 25 January 2017 · 4 messages · last reply 31 Jan 2017
BJ
Baysdon, Joseph25 Jan 2017 23:19 #
The performance of XQuery updates degrades as the number of JSON resources in my BaseX instance grows. In what ways might the number of JSON resources affect the performance of an XQuery update for a particular XML resource? I was under the impression that JSON resource were not indexed. (Note: this particular performance degradation is most dramatic when the BaseX server points to an NFS mount for the BaseX database.) Thanks, Joe
CG
Christian Grün27 Jan 2017 12:11 #
Quoted text · Baysdon, Joseph · 2 lines
> In what ways might the number of JSON resources > affect the performance of an XQuery update for a particular XML resource?
What kind of updates are you performing? How do you store your JSON resources?
BJ
Baysdon, Joseph31 Jan 2017 20:31 #
Below is a representative XQuery expression, wherein the JSON file is added via db:rename, and that addition is journaled to an XML log file via the write-log-event function. *I’ve noticed that the update to the xml log is adversely affected as the number of JSON files in the db grows.* Here’s the XQuery:
declare namespace _="http://com.sap.ariba.metadata.event/log";
declare namespace log="http://com.sap.ariba.metadata.event/log";
declare variable $logFilename external := "2017-01-31.xml";
declare variable $logDirectory external := "log";
declare variable $dbSchema external := "schemaRepo";
declare function _:write-log-event() as empty-sequence() {
  let $event :=    <entry id="6669" timestamp="2017-01-31T14:24:49.83-05:00">      <command toPath="meta/services/demo/serviceversions/v1/docs/demo/biggerform.json/1" isBinary="true" dbSchema="schemaRepo" type="ADD_BINARY" ></command></entry>
  let $logPath := $logDirectory || '/' || $logFilename
  let $fullPath := $dbSchema || '/' || $logPath
  return insert node $event as last into
           doc($fullPath)//log:log
};
declare variable $tempFileUri external := "temp/7b04aab6-6ee6-47c7-89e0-e5132e505daf/meta/services/demo/serviceversions/v1/docs/demo/biggerform.json/1";
declare variable $toPath external := "meta/services/demo/serviceversions/v1/docs/demo/biggerform.json/1";
if (not(db:exists($dbSchema, $logDirectory || '/' || $logFilename))) then (
    error(QName('http://com.sap.ariba.metadata/err', 'LogFileReq'),
	"write operation logFile not exist.")
)
else (
    try{
        db:rename(
           $dbSchema,
           $tempFileUri,
           $toPath),
        _:write-log-event(),
        db:output("SUCCESSFULLY")

    }
    catch * {
       fn:string("FAILED" || ": " || $err:description)
    }
)
Thanks, Joe
Quoted text · Christian Grün · 3 lines
On 1/27/17, 6:11 AM, "Christian Grün" <christian.gruen at gmail.com> wrote: > In what ways might the number of JSON resources > affect the performance of an XQuery update for a particular XML resource?
What kind of updates are you performing? How do you store your JSON resources?
CG
Christian Grün31 Jan 2017 21:05 #
Hi Joe,
Quoted text · Baysdon, Joseph · 1 line
> Below is a representative XQuery expression, wherein the JSON file is added via db:rename, and that addition is journaled to an XML log file via the write-log-event function.
Sorry, I can’t follow. How/why do you use db:rename to add resources? Do you store all JSON files as binaries (raw files)? In that case, it’s probably your file system that does not support large numbers of files in the same directory. Did you check out our Wiki article on binary files [1]? Christian [1] http://docs.basex.org/wiki/Binary_Data

⚡Generated with XQuery