"Input is too large for a single database" error
Friday, 17 April 2015 · 3 messages · last reply 23 May 2015
HV
Huib Verweij17 Apr 2015 15:32 #
Hi Christian,
congrats on the new release. I’ve downloaded it yesterday and while importing new data BaseX ran into a “Input is too large for a single database” error. It has been reported before and you mentioned Basex will be using 2^63 bits instead of 2^31 bits for counting nodes. Is there a branch that I could check out that already supports importing up to 2^63 nodes? The “node-counter” branch perhaps? For now I’ll just split the data and see if that works.
Hartelijke groet,
Huib Verweij
CG
Christian Grün17 Apr 2015 16:49 #
Hi Huib,
Quoted text · Huib Verweij · 7 lines
> congrats on the new release. I’ve downloaded it yesterday and while > importing new data BaseX ran into a “Input is too large for a single > database” error. It has been reported before and you mentioned Basex will be > using 2^63 bits instead of 2^31 bits for counting nodes. Is there a branch > that I could check out that already supports importing up to 2^63 nodes? The > “node-counter” branch perhaps? For now I’ll just split the data and see if > that works.
Not really; as it has effects on numerous parts of the architecture, and as it's not a pressing issue in our own projects, we haven't seriously begun implementing it.
Did you think about distributing your data to multiple databases, or is this no option?
Best,
Christian
CG
Christian Grün23 May 2015 17:39 #
Hi Huib,
Quoted text · Christian Grün · 2 lines
> However, using 8.2 I managed to load the entire Dutch legislation documents > into 5 separate databases. Yay!
Good to hear!
Quoted text · Christian Grün · 4 lines
> Anyway, and I must confess I am a bit lazy here, how do I run a query over > multiple databases/collections in BaseX? Of course I would like to run the > same dumb count query in BaseX too, over these 5 databases containing all my > data, see if it works...
This could be a solution:
count(
for $name in ('laws1', 'laws2')
return db:open($name)//law
)This could be another:
count(
for $i in 1 to 5
let $name := 'laws' || $i
return db:open($name)//law
)This is a third (it may be the fastest one, because the dynamic binding of database names in the first 2 queries prevents some optimizations to be triggered):
sum(
for $i in 1 to 5
let $query := 'count(db:open("db' || $i || '")//law)'
return xquery:eval($query)
)Hope this helps,
Christian
Written for BaseX 8.1, 8.2. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery