[Gambas-user] using a "file system database"

[Caveat]

Hi Kevin
If you can find a way to talk to this database, it could provide just
what you're looking for in terms of speed:

http://hsqldb.org/

It's a SQL database, but it can be run entirely in RAM.  I've used it
before (although I don't recall using it from Gambas...yet!) and it's an
amazing piece of work.

4M files sounds like a LOT of files, so it worries me that it'll prove
too slow.  I'm not convinced that using the file system would be any
faster than using a 'proper' database.

If you do go the fs route, there's some discussion here:
http://fixunix.com/ubuntu/356538-filesystem-lots-small-files.html which
may help.  

Regards,
Caveat

On Fri, 2011-04-15 at 03:22 -0400, Kevin Fishburne wrote:
> I'm in the early phases of creating a "database" that uses the file 
> system for data organization rather than a traditional software database 
> such as MySQL, etc. I'm hoping that this could be faster since my 
> requirements are very specific and (I think) don't need a general 
> purpose database.
> 
> I will have 4,194,304 "cells", each of which has about three datafiles 
> that will need to be opened, read from, written to and closed regularly. 
> I could consolidate the three files into one, reducing the number of 
> files, but this would increase the amount of time taken to parse the 
> files and slow the program significantly.
> 
> I'm considering dividing them into hierarchies of directories to avoid 
> having four to 16 million data files in the same directory. Initial 
> tests hit file system (or file space, not sure yet) limits.
> 
> Does anyone have any insights into the ext3/4 filesystem, possible 
> alternate file systems, and databases to know what the best solution 
> would be for this type of problem? Any insights into the most efficient 
> way to create hierarchies of directories and files would be appreciated 
> as well.
>