[Gambas-user] using a "file system database"

[Rolf-Werner Eilert]

Am 15.04.2011 09:22, schrieb Kevin Fishburne:
> I'm in the early phases of creating a "database" that uses the file
> system for data organization rather than a traditional software database
> such as MySQL, etc. I'm hoping that this could be faster since my
> requirements are very specific and (I think) don't need a general
> purpose database.
>
> I will have 4,194,304 "cells", each of which has about three datafiles
> that will need to be opened, read from, written to and closed regularly.
> I could consolidate the three files into one, reducing the number of
> files, but this would increase the amount of time taken to parse the
> files and slow the program significantly.
>
> I'm considering dividing them into hierarchies of directories to avoid
> having four to 16 million data files in the same directory. Initial
> tests hit file system (or file space, not sure yet) limits.
>
> Does anyone have any insights into the ext3/4 filesystem, possible
> alternate file systems, and databases to know what the best solution
> would be for this type of problem? Any insights into the most efficient
> way to create hierarchies of directories and files would be appreciated
> as well.
>

Spontaneously, I know that when creating a filesystem you have the 
possibility of determining the sector size. As far as I read, people who 
know there will be a lot of very small files (especially when the file 
size is predictable) prefer to set a smaller sector size to avoid 
wasting space (may have been one reason in your test) and vice-versa. 
When you set up a system with default values, these are a compromise 
between efficiency and waste (if your sector size is too small and the 
files regularly are longer than that, you will waste space again and it 
costs a lot sector jumping time).

My experience is that file systems are extremely efficient and fast 
under Linux. And they are intelligently cached which might be of help in 
your case. The more RAM you have the better here.

Just my 2 cents ;-)

Rolf