Real time DB population using Nifi
Thursday, 21 December 2017 · 3 messages · last reply 22 Dec 2017
JG
Juan Pablo Gardella21 Dec 2017 20:45 #
Hello team,
First of all, congratulations for the awesome tool! BaseX is amazing. We are starting using it for querying some XML data that we are receiving from a queue.
I've developed a Nifi processor based on
https://github.com/BaseXdb/basex/blob/master/basex-examples/src/main/java/org/basex/examples/server/ServerCommands.java class.
The problem that I'm facing is that write throughput is very slow. Do you know any way to increase the write throughput? All the XML files are stored in the same DB. So basically I'm doing:
1) open db
2) replace (xml id path, inputstream)
Thanks,
Juan
CG
Christian Grün22 Dec 2017 15:41 #
Hi Juan,
Welcome to the list and sorry for the late reply.
Quoted text · Juan Pablo Gardella · 2 lines
> 1) open db > 2) replace (xml id path, inputstream)
Some suggestions:
1. If you can use ADD instead of REPLACE, you can usually safe a lot of time.
2. If you need to replace docs, it may depend on what the "xml id path" is:
a) If it’s an attribute in the document, you may need to enable the updatable index (e.g. via the GUI, DBA, or the
UPDINDEX option).
b) If it’s the database path, you could try to disable AUTOFLUSH, or provide us with more information on what you do exactly.
Best,
ChristianJG
Juan Pablo Gardella22 Dec 2017 16:39 #
Thanks Christian,
We are receiving from a repayable queue (Kafka topic), so the idea is if we replay the messages, the XML data is updated in the DB. We are taking the message ID as the file path. Basically:
clientSession.replace(messageId, XML file).
Juan
Quoted text · Christian Grün · 25 lines
On Fri, 22 Dec 2017 at 11:41 Christian Grün <christian.gruen at gmail.com> wrote: > Hi Juan, > > Welcome to the list and sorry for the late reply. > > > 1) open db > > 2) replace (xml id path, inputstream) > > Some suggestions: > > 1. If you can use ADD instead of REPLACE, you can usually safe a lot of > time. > > 2. If you need to replace docs, it may depend on what the "xml id path" is: > > a) If it’s an attribute in the document, you may need to enable the > updatable index (e.g. via the GUI, DBA, or the UPDINDEX option). > > b) If it’s the database path, you could try to disable AUTOFLUSH, or > provide us with more information on what you do exactly. > > Best, > Christian >
Written for BaseX 8.6. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery