Main Page » Mailing List » December 2010

Multiple producers/writers for one collection and concurrency

Friday, 31 December 2010 · 5 messages

Jan Vlčinský (CAD)31 Dec 2010 10:40 #

Hi This topic was mentioned few times before, but it is key aspect of my application, so I will try anyway. *Question*: What is recommended approach for application, where exists multiple document producers which shall store/modify documents in one, share collection. *Background* - Having *feeds*, which might deliver some new messages. - *Multiple preprocessing processes* - preprocessing the messages before storing them in collection (Python, celery and RabbitMQ to be ready for processing bigger amount of incoming messages in parallel) - Having *single process for write access to my BaseX collection*(arranging this is not very easy for me so far) - There are some read only queries for stored messages, there are no problems with this as it shall be easy to run multiple read-only queries concurrently. *Assumptions* If I am wrong here, please, correct me. - one collection can be written/modified only by one process at given moment, other attempts to write are rejected and these attempts will fail. In other words - one collection can be opened only by one process (or are multiple opening clients allowed as long as the do not modify database concurrently?) - read only queries are easy as they can be run concurrently without blocking/rejecting *Current approach* 1. Multiple preprocessing processes are concurrently sending documents into single basex-storage queue in RabbitMQ 2. There is single one-process worker, consuming documents from queue basex-storage which does all the work with adding documents for given collection. *Wishes* Managing single write process adds quite complexity to whole solution. It would be nice, if I can write to single collection by multiple processes at once and could expect success. Any comments are welcome Best regards Jan

Andreas Weiler31 Dec 2010 10:57 #

Hi Jan, if a writing or reading transaction is active, all new incoming transactions are stored in a waiting queue. So all writing transactions will be processed in first come - first server order. Reading transactions are processed concurrently. Transactions will no be rejected at any time. Kind regards, Andreas

Quoted text · Jan Vlčinský (CAD) · 6 lines

Am 31.12.10 10:40, schrieb Jan Vlčinský (CAD): > one collection can be written/modified only by one process at given > moment, other attempts to write are rejected and these attempts will fail. > In other words - one collection can be opened only by one process (or > are multiple opening clients allowed as long as the do not modify > database concurrently?)

Jan Vlčinský (CAD)31 Dec 2010 11:06 #

Hi Andreas It seems, like I was too smart (and learned a lot about concurrency in AMQP this very long night) :-) What are starting/ending points of one transaction 1. creating connection - closing it. probably not 2. open database - close ?? 3. performing one command (like xquery, add, delete etc.) This would be the best Jan 2010/12/31 Andreas Weiler <andreas.weiler at uni-konstanz.de>

Quoted text · Andreas Weiler · 24 lines

> Hi Jan, > > if a writing or reading transaction is active, all new incoming > transactions are stored in > a waiting queue. > So all writing transactions will be processed in first come - first server > order. > Reading transactions are processed concurrently. > Transactions will no be rejected at any time. > > Kind regards, > Andreas > > > Am 31.12.10 10:40, schrieb Jan Vlčinský (CAD): > > one collection can be written/modified only by one process at given >> moment, other attempts to write are rejected and these attempts will fail. >> In other words - one collection can be opened only by one process (or are >> multiple opening clients allowed as long as the do not modify database >> concurrently?) >> > >

-- *Ing. Jan Vlčinský* CAD programy Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic tel: +420-597 602 024; mob: +420-608 979 040 skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com http://cz.linkedin.com/in/vlcinsky

Andreas Weiler31 Dec 2010 11:47 #

Quoted text · Jan Vlčinský (CAD) · 2 lines

> performing one command (like xquery, add, delete etc.) > This would be the best

Thats the way it is. Note that all commands are atomic transactions, like a alter database -> closes the database, renames it and reopens it...but you havent care about that. Kind regards and happy new year (with BaseX), Andreas

Quoted text · Jan Vlčinský (CAD) · 55 lines

Am 31.12.10 11:06, schrieb Jan Vlčinský (CAD): > Hi Andreas > It seems, like I was too smart (and learned a lot about concurrency in > AMQP this very long night) :-) > > What are starting/ending points of one transaction > > 1. creating connection - closing it. > probably not > 2. open database - close > ?? > 3. performing one command (like xquery, add, delete etc.) > This would be the best > > Jan > > 2010/12/31 Andreas Weiler <andreas.weiler at uni-konstanz.de > <mailto:andreas.weiler at uni-konstanz.de>> > > Hi Jan, > > if a writing or reading transaction is active, all new incoming > transactions are stored in > a waiting queue. > So all writing transactions will be processed in first come - > first server order. > Reading transactions are processed concurrently. > Transactions will no be rejected at any time. > > Kind regards, > Andreas > > > Am 31.12.10 10:40, schrieb Jan Vlčinský (CAD): > > one collection can be written/modified only by one process at > given moment, other attempts to write are rejected and these > attempts will fail. > In other words - one collection can be opened only by one > process (or are multiple opening clients allowed as long as > the do not modify database concurrently?) > > > > > > -- > *Ing. Jan Vlčinský* > CAD programy > Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic > tel: +420-597 602 024; mob: +420-608 979 040 > skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com > <mailto:jan.vlcinsky at gmail.com> > http://cz.linkedin.com/in/vlcinsky >

Jan Vlčinský (CAD)31 Dec 2010 11:54 #

Thanks "for the best". I tested it and storing 100 documents by 3 client processes in parallel works very well. BaseX makes me happy. Happy New Year to all of you. Jan 2010/12/31 Andreas Weiler <andreas.weiler at uni-konstanz.de>

Quoted text · Andreas Weiler · 66 lines

> performing one command (like xquery, add, delete etc.) > This would be the best > > Thats the way it is. > Note that all commands are atomic transactions, like a alter database -> > closes the database, > renames it and reopens it...but you havent care about that. > > Kind regards and happy new year (with BaseX), > Andreas > > Am 31.12.10 11:06, schrieb Jan Vlčinský (CAD): > > Hi Andreas > It seems, like I was too smart (and learned a lot about concurrency in AMQP > this very long night) :-) > > What are starting/ending points of one transaction > > 1. creating connection - closing it. > probably not > 2. open database - close > ?? > 3. performing one command (like xquery, add, delete etc.) > This would be the best > > Jan > > 2010/12/31 Andreas Weiler <andreas.weiler at uni-konstanz.de> > >> Hi Jan, >> >> if a writing or reading transaction is active, all new incoming >> transactions are stored in >> a waiting queue. >> So all writing transactions will be processed in first come - first server >> order. >> Reading transactions are processed concurrently. >> Transactions will no be rejected at any time. >> >> Kind regards, >> Andreas >> >> >> Am 31.12.10 10:40, schrieb Jan Vlčinský (CAD): >> >> one collection can be written/modified only by one process at given >>> moment, other attempts to write are rejected and these attempts will fail. >>> In other words - one collection can be opened only by one process (or are >>> multiple opening clients allowed as long as the do not modify database >>> concurrently?) >>> >> >> > > > -- > *Ing. Jan Vlčinský* > CAD programy > Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic > tel: +420-597 602 024; mob: +420-608 979 040 > skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com > http://cz.linkedin.com/in/vlcinsky > > >

Written for BaseX 6. BaseX may have changed since: please check the current documentation.

⚡Generated with XQuery

`J`	Select the next message
`K`	Select the previous message
`N`	Next thread
`P`	Previous thread
`U`	Go up one level
`/`	Jump to the search field