Database files across multiple directories

Hello everybody out there using BaseX, Is there a possibility to have xml files in multiple subdirectories? I would like to sort binary files according to date into different directories and write the corresponding metadata next to them in an xml file. Alternative solutions are also appreciated. Kind regards, Julia

Dear Julia,

I am not 100% sure what you would like to achieve.. * Do you talk about XML documents that are stored in a BaseX database? In that case, BaseX should do all the work. You can store millions of documents in a single database, and the document index will give you quick access to single resources. * Or would you like to use BaseX to organise your files in real directory structures? In that case, I may need more information on what you have already done so far. Hope this helps, Christian

Hi Julia, the Xml files I'm using are also in different directories, one for each day. I create one database on each of these directories, each containing between several hundred and up to 6000 documents. Usually, queries need to take in account only one day, so the search space is considerably smaller than for one large database. If access to documents of more than one day is required, you can create the range of database identifiers (supposing there is an algorithm to do so) and then open each database in a for loop, thus getting a sequence of top nodes. Also, for more complicated queries, I can choose to (re)create some the databases in-memory in order to improve performance. Since creating a database of this size takes just a couple of seconds, this is an approach which would possibly not be feasible if all the documents were in one huge database. Kind regards, Goetz

Small correction; I should have said "reference each database with doc()" instead of "open each database". My apologies. Kind regards, Goetz

Thanks a lot for your answers so far. What I was thinking of as a use case are many small xml-Documents in many different (sub)directories that should be used together as one database, like for example one xml file for one address and address arranged into folders according to towns so one would have 9 xml files in a folder named "berlin" and 5 ones in another folder named "munich" and so on. Does BaseX create an index of these files so that I could for example query for example for persons with the same name? If I want to change something in these files, will the changes be also made in the xml documents? Kind regards, Julia

Hi Julia,

for persons with the same name? BaseX will create a database from these files. This can e.g. be done by starting the GUI, choosing "Database" → "New…", selecting your XML directory, and pressing OK. All documents will then be stored, and the textual contents will be indexed. Next, you can use XQuery, which is not only a query language, but a powerful programming language, to do all kinds of things with the data. See e.g. [1] or [2] for some examples. You probably don't want to do that by yourself, but XQuery can be used to write full web applications.

Currently, there is no automatic synchronisation between databases and local file directories, so you'll either have to export the database after updates, or create a new database after files have been changed. Cheers, Christian [1] http://docs.basex.org/wiki/XQuery_3.0 [2] http://www.w3schools.com/xquery/ If I want to change something in these files, will the changes be also made in the xml documents?

Hi Julia, this looks to me as if you had hybrid data - some relational (tabular) data (addresses etc.) and other semi-structured data. If so, then think about storing the tabular data in a relational database management system (RDBMS) like Maria SQL or MS SQL Server and accessing it from within your xQueries. This is something which can be achieved very easily since the BaseX implementation accommodates adding Java code to your queries simply by dropping your jar files into the lib directory and adding a namespace declaration to the prolog of your query - it's just a snap. You can do anything you want with the relational data - insert, update, delete, and, of course, select, you can run stored procedures, and still access your XML data as easily, combine results from both sources, create great reports, and so on. Think of xQuery as a domain-specific language (DSL) for querying XML data, just as SQL is a DSL for dealing with relational data, which you can enhance according to your needs. If you need to modify only the relational part of the data, you can do this easily via the RDBMS since this is standard, and BaseX still remains the perfect choice for handling the XML part of your data. Although many RDBMSs offer also some XML and even xQuery functionality, compared to a system like BaseX this is only rudimentary. E.g. Oracle and SQL Server implement only xQuery 1.0, and even this only partially. I did something similar with BaseX and SQL Server, and - just as a proof of concept - also with Oracle. E.g., you can create SQL code with xQuery on the fly so it incorporates current context data, and pass it to some generic database access function. With JDBC, you've got the glue to connect to a wide range of database servers. I did not figure out whether you can make a database server and BaseX participate in the same distributed transaction because I had no necessity to do so; maybe Christian can tell you something more about it. Kind regards, Goetz