[Gambas-user] using a "file system database"

[Kevin Fishburne]

On 04/15/2011 04:33 AM, Caveat wrote:
> Hi Kevin
> If you can find a way to talk to this database, it could provide just
> what you're looking for in terms of speed:
>
> http://hsqldb.org/
>
> It's a SQL database, but it can be run entirely in RAM.  I've used it
> before (although I don't recall using it from Gambas...yet!) and it's an
> amazing piece of work.
>
> 4M files sounds like a LOT of files, so it worries me that it'll prove
> too slow.  I'm not convinced that using the file system would be any
> faster than using a 'proper' database.
>
> If you do go the fs route, there's some discussion here:
> http://fixunix.com/ubuntu/356538-filesystem-lots-small-files.html which
> may help.

Hi Caveat, thanks for the info. I found this quote in the link:

"In any case, the key here is that he has a _lot_ of _small_ files; this 
just
cries out for a filesystem with a small blocksize and a large number of
inodes. I believe that the EXT2 filesystem should handle his requirements,
given a small (say 1k or 2k) blocksize and a suitable ratio of bytes/inode.
You might even find a predefined fstype in /etc/mke2fs.conf that suites the
requirements."

The server almost exclusively deals with millions of very small files, 
so I'll start at the bottom of what is allowable and benchmark 
subsequently larger settings for comparison. The files will start pretty 
small (many empty), and their number of records will change over time. 
I'll probably not even code pruning routines, as there will only be so 
many objects stored in a single file due to game mechanics. A 
sufficiently large RAID should be able to accommodate for raw storage, 
though seeks will take longer over time as the files slowly increase 
with dead data. I'll at least allow overwriting dead fields with new 
ones to keep things sane.

-- 
Kevin Fishburne
Eight Virtues
www: http://sales.eightvirtues.com
e-mail: sales at ...1887...
phone: (770) 853-6271