[Gambas-user] using a "file system database"

[Kevin Fishburne]

On 04/18/2011 04:32 AM, Doriano Blengino wrote:
> Kevin Fishburne ha scritto:
>> On 04/15/2011 03:56 AM, Doriano Blengino wrote:
>>
>>> Kevin Fishburne ha scritto:
>>>
>>>> I'm in the early phases of creating a "database" that uses the file
>>>> system for data organization rather than a traditional software database
>>>> such as MySQL,...
>>>>
>>>> I will have 4,194,304 "cells", each of which has about three datafiles
>>>> that will need to be opened, read from, written to and closed regularly.
>>>> ...
>>>> I'm considering dividing them into hierarchies of directories to avoid
>>>> having four to 16 million data files in the same directory. Initial
>>>> tests hit file system (or file space, not sure yet) limits.
>>>>
>>>>
>>>>
>>> Very interesting problem.
>>>
>>> For the file system, the more sophisticated is, the more it is expensive
>>> to modify it; etx3/4 or reiserfs are journaled, so they are slower and
>>> heavier to manage. But it could be that a well-planned journaled fs is
>>> faster than a bad-planned non-journaled fs...
>>>
>> Hi Doriano. Good advice about functionality versus speed. I'm going to
>> be testing both ext2 and xfs this week to see which is superior for my
>> purposes. As far as data integrity, I'll probably have some sort of
>> local RAID as a backup target with slow, incremental writes to it. If
>> the server dies, then at least most of the game data will be preserved
>> without harming the performance of the server app. So the weakness of
>> the filesystem with regard to crash recovery is irrelevant.
>>
> Please let us know about your tests...

I will.

>>
>> My current plan is to create a directory for each region
>> ([65536/32/32]^2). Each region directory contains 32^2 data files
>> (1024). Hopefully this won't stress any particular file system as far as
>> how many directories and files are contained within a single directory.
>>
> But... I am missing something... the number of files was 4M, right? And
> 64 directories with 1024 files does not sum up to 4M...

A cell is 32x32 tiles (bytes), and the map is 65536^2 tiles, so there 
are 2048^2 cells. Each of these are organized into directories of 32^2 
cells (regions). The data files alone are 4.2 million in number, the 
directories that organize them hopefully add a layer of filesystem 
efficiency. Right now a separate directory is created for each region 
and empty data files are created for each cell within that region.

-- 
Kevin Fishburne
Eight Virtues
www: http://sales.eightvirtues.com
e-mail: sales at ...1887...
phone: (770) 853-6271