[Gambas-user] using a "file system database"

[Kevin Fishburne]

On 04/15/2011 03:50 AM, Rolf-Werner Eilert wrote:
> Am 15.04.2011 09:22, schrieb Kevin Fishburne:
>> I'm in the early phases of creating a "database" that uses the file
>> system for data organization rather than a traditional software database
>> such as MySQL, etc. I'm hoping that this could be faster since my
>> requirements are very specific and (I think) don't need a general
>> purpose database.
>>
>> I will have 4,194,304 "cells", each of which has about three datafiles
>> that will need to be opened, read from, written to and closed regularly.
>> I could consolidate the three files into one, reducing the number of
>> files, but this would increase the amount of time taken to parse the
>> files and slow the program significantly.
>>
>> I'm considering dividing them into hierarchies of directories to avoid
>> having four to 16 million data files in the same directory. Initial
>> tests hit file system (or file space, not sure yet) limits.
>>
>> Does anyone have any insights into the ext3/4 filesystem, possible
>> alternate file systems, and databases to know what the best solution
>> would be for this type of problem? Any insights into the most efficient
>> way to create hierarchies of directories and files would be appreciated
>> as well.
>
> Spontaneously, I know that when creating a filesystem you have the
> possibility of determining the sector size. As far as I read, people who
> know there will be a lot of very small files (especially when the file
> size is predictable) prefer to set a smaller sector size to avoid
> wasting space (may have been one reason in your test) and vice-versa.
> When you set up a system with default values, these are a compromise
> between efficiency and waste (if your sector size is too small and the
> files regularly are longer than that, you will waste space again and it
> costs a lot sector jumping time).
>
> My experience is that file systems are extremely efficient and fast
> under Linux. And they are intelligently cached which might be of help in
> your case. The more RAM you have the better here.

Hi Rolf. Excellent information. I refined the routine that was creating 
the directories and files and will be performing tests this week to see 
what filesystem and filesystem characteristics are the fastest for this. 
Do you know if there is a program like gparted that allows you to 
specify these details? I'm not averse to using software RAID if necessary.

-- 
Kevin Fishburne
Eight Virtues
www: http://sales.eightvirtues.com
e-mail: sales at ...1887...
phone: (770) 853-6271