Paging takes time.
Monday, 20 January 2014 · 2 messages
GG
Geet Gangwar20 Jan 2014 11:06 #
Hi,
I have huge XML document containing around 70000000 child nodes. I want to implement paging on this, so that I can fetch a chunk of nodes at time, but as the start position value increases from 10000000 the execution time takes more than a second to execute, is there a any optimized way I can follow.
I have written following code:
public class XQJ
{
public static void main(String[] args) throws XQException
{
XQDataSource xqs = new BaseXXQDataSource();
xqs.setProperty("serverName", "localhost");
xqs.setProperty("port", "1984");
// Change USERNAME and PASSWORD values
XQConnection conn = xqs.getConnection("admin", "admin");
XQPreparedExpression xqpe =
conn.prepareExpression("declare namespace xbrli='
http://www.xbrl.org/2003/instance';" +
" declare variable $doc as xs:string external;" +
" declare variable $start as xs:integer external;" +
" declare variable $pageSize as xs:integer external;" +
" let $allMatches := doc($doc)/xbrli:xbrl/*" +
" return subsequence($allMatches,$start,$pageSize)");
// " return $matches");
xqpe.bindString(new QName("doc"),
"1389962424906/1389962424906/facts.xml", null);
int totalRecords = 70000000;
int pageSize = 10000;
int noOfPages = totalRecords/pageSize;
long startTime = 0;
int start = 0;
for(int i=0; i<noOfPages; i++){
startTime = System.currentTimeMillis();
start = i*pageSize;
xqpe.bindInt(new QName("start"), start, null);
xqpe.bindInt(new QName("pageSize"), pageSize, null); XQResultSequence rs = xqpe.executeQuery();
// while(rs.next()){}
// System.out.println(rs.getItemAsString(null));
// System.out.println(start + " : " +
(System.currentTimeMillis()-startTime));
}
System.out.println(start + " : " +
(System.currentTimeMillis()-startTime));
conn.close();
}CG
Christian Grün20 Jan 2014 18:42 #
Hi Geet,
I can’t think of an easy way to page results with the current database/query architecture. One spontaneous idea: You could request all node ids in a first query [1] and requests the hits in a second query [2]. When requesting 70 million nodes, however, the list would get pretty huge.
Christian
[1] http://docs.basex.org/wiki/Database_Module#db:node-id
[2] http://docs.basex.org/wiki/Database_Module#db:node-pre
Quoted text · Geet Gangwar · 63 lines
On Mon, Jan 20, 2014 at 11:06 AM, Geet Gangwar <geetgangwar at gmail.com> wrote: > Hi, > > I have huge XML document containing around 70000000 child nodes. I want to > implement paging on this, so that I can fetch a chunk of nodes at time, but > as the start position value increases from 10000000 the execution time takes > more than a second to execute, is there a any optimized way I can follow. > > I have written following code: > > public class XQJ > { > public static void main(String[] args) throws XQException > { > XQDataSource xqs = new BaseXXQDataSource(); > xqs.setProperty("serverName", "localhost"); > xqs.setProperty("port", "1984"); > > // Change USERNAME and PASSWORD values > XQConnection conn = xqs.getConnection("admin", "admin"); > > XQPreparedExpression xqpe = > conn.prepareExpression("declare namespace > xbrli='http://www.xbrl.org/2003/instance';" + > " declare variable $doc as xs:string external;" + > " declare variable $start as xs:integer external;" + > " declare variable $pageSize as xs:integer external;" + > " let $allMatches := doc($doc)/xbrli:xbrl/*" + > " return subsequence($allMatches,$start,$pageSize)"); > // " return $matches"); > > xqpe.bindString(new QName("doc"), > "1389962424906/1389962424906/facts.xml", null); > int totalRecords = 70000000; > int pageSize = 10000; > int noOfPages = totalRecords/pageSize; > long startTime = 0; > int start = 0; > for(int i=0; i<noOfPages; i++){ > startTime = System.currentTimeMillis(); > start = i*pageSize; > > xqpe.bindInt(new QName("start"), start, null); > xqpe.bindInt(new QName("pageSize"), pageSize, null); > > XQResultSequence rs = xqpe.executeQuery(); > > // while(rs.next()){} > // System.out.println(rs.getItemAsString(null)); > > // System.out.println(start + " : " + > (System.currentTimeMillis()-startTime)); > } > System.out.println(start + " : " + > (System.currentTimeMillis()-startTime)); > conn.close(); > } > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
Written for BaseX 7.7. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery