[Gambas-user] Need Gambas3 Made Apps.

[Tobias Boege]

On Thu, 07 Feb 2013, Rolf-Werner Eilert wrote:
> Hi Tobias,
> 
> following your thread, I found
> 
>  > About soft(ware?) for media (streams?), that's a point I would 
> support: I'm
>  > not aware of easy methods for bit-fiddling or using SIMD, etc. like 
> one can
>  > do it with gcc intrinsics. If there a library for it (I don't know) this
>  > could be a target for a component.
> 
> But one could write the functions in C or C++ and include them as a 
> library. At the end of the day, there is an end user who wants to use 
> the functionality, not the functions.
> 
> 
> I simply wondered about what you said here, so I'm curious:
> 
> > That clearly depends on _how_ you use the files. This is the same with all
> > programming languages: Never read a file into memory for processing it but
> > use a Stream instead. The number of files opened and their size should not
> > matter then.
> >
> 
> I must confess, this is the first time I've heared about this. Could you 
> explain, where is the difference and how do I manage the data in the 
> file then? A pointer to the bytes? And how do I read/write while working 
> on the data? (I'm thinking of picture manipulation or a video cutting 
> program, real big data, not a text file of course, or did I get you 
> wrong here?)

You can apply this concept to text files, too. A common example is to parse
a configuration file. Normally, these contain line-oriented, human readable
data on Unices. You can equally easily use Split(File.Load("config"), "\n")
and LINE INPUT to process lines individually but the stream thing is much
superior from a memory point of view:

The first methods loads the entire file at once and splits it into an array
of lines, which temporarily leaves the entire file in memory twice. Once in
the array, once in the return value of File.Load() which is subsequently
freed. Using LINE INPUT you keep a line at a time in memory. Less memory
usage means less swap usage, etc..

It is a different thing when the underlying mechanisms use mmap() to map a
file into memory which does not appear to be the case for File.Load().

How you use streams to process your data depends on the data. gbx3 uses
mmap() to load executable Gambas archives[0]. These are uncompressed
archives and so the interpreter can examine the header of that archive to
find the files it needs. The file is perhaps not completely loaded in one
particular execution of the program but only the needed parts. (I only
glanced at the sources. Benoit knows this for sure.)

Regards,
Tobi

[0] http://gambasdoc.org/help/doc/arch?v3