Load files to database
Thursday, 23 February 2012 · 2 messages
AT
Anh Thuấn23 Feb 2012 10:11 #
Hi, baseX team. I have some problems of loading xml files into the database of BaseX. I am comparing the time to load xml files into database, there are 2 ways to load: Loading all the folder into the database and Loading each file into the database. The time to load all the folder and the time to load each file have a big difference. The time to load all the folder into the datebase is 84ms, whereas the time to load each file into the database is 912 ms. I have written the 2 ways to load in C#. The following code is to load all the folder into the datebase:
private void CreateDBAndLoadDirectory(string directoryPath)
{
this.contextDirectory = new Context();
this.serverDirectory = new
org.basex.BaseXServer(this.contextDirectory);
Set.execute("createfilter", "*" + this.fileExtension,
this.contextDirectory);
Set.execute("CACHEQUERY", true, this.contextDirectory);
Set.execute("SERIALIZE", false, this.contextDirectory);
Set.execute("CHOP", true, this.contextDirectory);
Set.execute("DBPATH", this.dbPath, this.contextDirectory);
new CreateDB(this.dbDirectory,
directoryPath).execute(this.contextDirectory);
}
And the following code is to load each file into the database:
private void CreateDBAndLoadFiles(string directoryPath)
{
this.contextFile = new Context();
this.serverFile = new org.basex.BaseXServer(this.contextFile);
Set.execute("CACHEQUERY", true, this.contextFile);
Set.execute("SERIALIZE", false, this.contextFile);
Set.execute("CHOP", true, this.contextFile);
Set.execute("DBPATH", this.dbPath, this.contextFile);
new CreateDB(this.dbFile).execute(this.contextFile);
string[] fileList = Directory.GetFiles(directoryPath, "*" +
this.fileExtension,SearchOption.AllDirectories);
foreach (string fileName in fileList)
{
string path = fileName.Substring(this.otxPath.Length + 1);
new Add(path, fileName).execute(this.contextFile);
new Optimize().execute(this.contextFile);
}
}
I don't know if I have made mistakes in these codes or not? I want to
ask about the fastest way to load each file into the database.CG
Christian Grün23 Feb 2012 13:03 #
Dear Anh,
bulk operations (your first solution) will always be faster than multiple single updates. However, you can speed up single updates by temporarily setting the
AUTOFLUSH option to true [1].
Hope this helps,
Christian
[1] http://docs.basex.org/wiki/Options#AUTOFLUSHQuoted text · Anh Thuấn · 45 lines
On Thu, Feb 23, 2012 at 10:11 AM, Anh Thuấn <thuan.ly at emotive.de> wrote: > > Hi, baseX team. I have some problems of loading xml files into the database > of BaseX. I am comparing the time to load xml files into database, there are > 2 ways to load: Loading all the folder into the database and Loading each > file into the database. The time to load all the folder and the time to load > each file have a big difference. The time to load all the folder into the > datebase is 84ms, whereas the time to load each file into the database is > 912 ms. I have written the 2 ways to load in C#. The following code is to > load all the folder into the datebase: > private void CreateDBAndLoadDirectory(string directoryPath) > { > this.contextDirectory = new Context(); > this.serverDirectory = new > org.basex.BaseXServer(this.contextDirectory); > Set.execute("createfilter", "*" + this.fileExtension, > this.contextDirectory); > Set.execute("CACHEQUERY", true, this.contextDirectory); > Set.execute("SERIALIZE", false, this.contextDirectory); > Set.execute("CHOP", true, this.contextDirectory); > Set.execute("DBPATH", this.dbPath, this.contextDirectory); > new CreateDB(this.dbDirectory, > directoryPath).execute(this.contextDirectory); > } > And the following code is to load each file into the database: > private void CreateDBAndLoadFiles(string directoryPath) > { > this.contextFile = new Context(); > this.serverFile = new org.basex.BaseXServer(this.contextFile); > Set.execute("CACHEQUERY", true, this.contextFile); > Set.execute("SERIALIZE", false, this.contextFile); > Set.execute("CHOP", true, this.contextFile); > Set.execute("DBPATH", this.dbPath, this.contextFile); > new CreateDB(this.dbFile).execute(this.contextFile); > string[] fileList = Directory.GetFiles(directoryPath, "*" + > this.fileExtension,SearchOption.AllDirectories); > foreach (string fileName in fileList) > { > string path = fileName.Substring(this.otxPath.Length + 1); > new Add(path, fileName).execute(this.contextFile); > new Optimize().execute(this.contextFile); > } > } > I don't know if I have made mistakes in these codes or not? I want to ask > about the fastest way to load each file into the database.
Written for BaseX 7.1. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery