[Gambas-user] using a "file system database"

[Doriano Blengino]

Kevin Fishburne ha scritto:
> On 04/15/2011 03:56 AM, Doriano Blengino wrote:
>    
>> Kevin Fishburne ha scritto:
>>      
>>> I'm in the early phases of creating a "database" that uses the file
>>> system for data organization rather than a traditional software database
>>> such as MySQL,...
>>>
>>> I will have 4,194,304 "cells", each of which has about three datafiles
>>> that will need to be opened, read from, written to and closed regularly.
>>> ...
>>> I'm considering dividing them into hierarchies of directories to avoid
>>> having four to 16 million data files in the same directory. Initial
>>> tests hit file system (or file space, not sure yet) limits.
>>>
>>>
>>>        
>> Very interesting problem.
>>
>> For the file system, the more sophisticated is, the more it is expensive
>> to modify it; etx3/4 or reiserfs are journaled, so they are slower and
>> heavier to manage. But it could be that a well-planned journaled fs is
>> faster than a bad-planned non-journaled fs...
>>      
> Hi Doriano. Good advice about functionality versus speed. I'm going to
> be testing both ext2 and xfs this week to see which is superior for my
> purposes. As far as data integrity, I'll probably have some sort of
> local RAID as a backup target with slow, incremental writes to it. If
> the server dies, then at least most of the game data will be preserved
> without harming the performance of the server app. So the weakness of
> the filesystem with regard to crash recovery is irrelevant.
>    
Please let us know about your tests...
>
> My current plan is to create a directory for each region
> ([65536/32/32]^2). Each region directory contains 32^2 data files
> (1024). Hopefully this won't stress any particular file system as far as
> how many directories and files are contained within a single directory.
>    
But... I am missing something... the number of files was 4M, right? And 
64 directories with 1024 files does not sum up to 4M...

Regards,
Doriano