[Gambas-user] Runaway memory consumption not released until program exit

Richard richard.j.walker at ...247...
Sat May 8 02:39:16 CEST 2010


On Saturday 08 May 2010 00:50:04 Benoît Minisini wrote:
> > On Friday 07 May 2010 23:11:02 Benoît Minisini wrote:
> > > > On Friday 07 May 2010 20:31:25 Fabien Bodard wrote:
> > > > > frame.Load(SourceImage).Stretch(NewWidth,
> > > > > NewHeight).Save(DestinationImage)
> > > > >
> > > > > 2010/5/7 Richard <richard.j.walker at ...247...>:
> > > > > > On Friday 07 May 2010 19:08:49 Fabien Bodard wrote:
> > > > > >> frame = frame.Stretch(NewWidth, NewHeight)
> > > > > >>
> > > > > >> if you do not re assign frame stretching returned image ... the
> > > > > >> image will not be changed
> > > > > >>
> > > > > >> http://gambasdoc.org/help/comp/gb.image.imlib/image/stretch?v3
> > > > > >>
> > > > > >> 2010/5/7 Richard <richard.j.walker at ...247...>:
> > > > > >> > Hi everybody,
> > > > > >> >
> > > > > >> > I am using Gambas 2.20.0 on Mandriva 2010.0. The desktop is
> > > > > >> > LXDE and the gui toolkit might possibly be GTK.
> > > > > >> >
> > > > > >> > I have written a button event handler to process hundreds or
> > > > > >> > thousands of .png images in a directory. The main memory
> > > > > >> > available when the program starts is around 1.8 GBytes. After
> > > > > >> > processing just 744 images (in testing) this falls to about 74
> > > > > >> > MBytes. The memory is not freed on exit from the event
> > > > > >> > handler, only when I exit from the program.
> > > > > >> >
> > > > > >> > I use only 1 Image object; a variable called "frame". I have
> > > > > >> > tried declaring this as a local in the event handler and as a
> > > > > >> > class-visible variable with no apparent change in the way
> > > > > >> > memory is eaten and never disgorged.
> > > > > >> >
> > > > > >> > I do three things with my frame variable in a loop which
> > > > > >> > iterates for all frames in a directory;
> > > > > >> >
> > > > > >> > 1. I load it from an image on disc
> > > > > >> > 2. I "stretch" it to make its width smaller(!)
> > > > > >> > 3. I save the altered frame to a different directory
> > > > > >> >
> > > > > >> > Specifically the significant lines of code look like;
> > > > > >> >
> > > > > >> > frame = Image.Load(SourceImage)
> > > > > >> > frame.Stretch(NewWidth, NewHeight)
> > > > > >> > frame.Save(DestinationImage)
> > > > > >> >
> > > > > >> > Outside the loop I now do frame = NULL, but this has no
> > > > > >> > effect.
> > > > > >> >
> > > > > >> > Is it possible that my code is allocating multiple "anonymous"
> > > > > >> > frame objects and not releasing them? I have read Benoit's
> > > > > >> > explanation in this list of Gambas memory management (Re:
> > > > > >> > [Gambas-user] Memory Management, on 08/09/2007 09:19).
> > > > > >> >
> > > > > >> > I turned to Gambas to program this function because my first
> > > > > >> > choice, imagemagick, would always run out of memory without
> > > > > >> > processing a single frame. The Gambas solution works, but no
> > > > > >> > more work can be done without killing the program and
> > > > > >> > restarting it - not very convenient. I anticipate that typical
> > > > > >> > uses of the program will have to handle 10-30 thousand png
> > > > > >> > images at a time.
> > > > > >> >
> > > > > >> > Can anyone suggest how I find out what is so hungry for my
> > > > > >> > memory, and hopefully how to suppress its appetite?
> > > > > >> >
> > > > > >> > Richard
> > > > > >> >
> > > > > >> > --------------------------------------------------------------
> > > > > >> >-- --
> > > > > >> >
> > > > > >> >-- --
> > > > > >> >
> > > > > >> >--- -----
> > > > > >> >
> > > > > >> > _______________________________________________
> > > > > >> > Gambas-user mailing list
> > > > > >> > Gambas-user at lists.sourceforge.net
> > > > > >> > https://lists.sourceforge.net/lists/listinfo/gambas-user
> > > > > >>
> > > > > >> ----------------------------------------------------------------
> > > > > >>-- --
> > > > > >>
> > > > > >>-- --
> > > > > >>
> > > > > >>--- ---
> > > > > >>
> > > > > >> _______________________________________________
> > > > > >> Gambas-user mailing list
> > > > > >> Gambas-user at lists.sourceforge.net
> > > > > >> https://lists.sourceforge.net/lists/listinfo/gambas-user
> > > > > >
> > > > > > Thanks for reminding me, Fabien, about a fascinating side issue.
> > > > > > Originally I had tried frame.Resize(NewWidth, NewHeight) but I
> > > > > > was disappointed to discover it doesn't resize the image at all;
> > > > > > it just re-frames it by cropping the right hand side.
> > > > > >
> > > > > > I changed to Stretch by just typing over Resize and was quite a
> > > > > > bit surprised when that didn't seem to work either! That was when
> > > > > > I realised that while Resize(W,H) is a procedure, Stretch(W,H)
> > > > > > is, as you said, a function! I was a bit hasty in providing my
> > > > > > example code lines so I repeated the mistake which I had
> > > > > > corrected in my event handler and which you have corrected above.
> > > > > >
> > > > > > The code which I copied for my example does not assign the result
> > > > > > of the Stretch function, but immediately writes it back to disc.
> > > > > > It reads:
> > > > > >
> > > > > > frame.Stretch(NewWidth,NewHeight).Save(DestinationImage)
> > > > > >
> > > > > > It works. This is what I do now, although I had originally used
> > > > > > the version you suggested and assigned the "stretched" image to
> > > > > > my frame variable before saving. I have been trying to nail down
> > > > > > the greedy memory culprit and the frame.Stretch().Save() version
> > > > > > made no difference, but I was too lazy to change it back to
> > > > > > frame=frame.Stretch() followed by frame.Save().
> > > > > >
> > > > > > I have since discovered that it is the frame=Image.Load() step
> > > > > > which is to blame. Each png is at most 1MiB, though most are much
> > > > > > less. 740 of these when loaded in turn by  frame=Image.Load()
> > > > > > uses 1.8GiB of main memory and nearly 2GiB of swap. This seems
> > > > > > excessive to me, especially since I don't seem able to get it
> > > > > > back without quitting from the program. Any ideas about that?
> > > > >
> > > > > frame=null ?
> > > > >
> > > > > > Richard
> > > > > >
> > > > > > -----------------------------------------------------------------
> > > > > >-- --
> > > > > >
> > > > > >-- --
> > > > > >
> > > > > >-----
> > > > > >
> > > > > > _______________________________________________
> > > > > > Gambas-user mailing list
> > > > > > Gambas-user at lists.sourceforge.net
> > > > > > https://lists.sourceforge.net/lists/listinfo/gambas-user
> > > > >
> > > > > -------------------------------------------------------------------
> > > > >-- --
> > > > >
> > > > >-- --
> > > > >
> > > > >---
> > > > >
> > > > > _______________________________________________
> > > > > Gambas-user mailing list
> > > > > Gambas-user at lists.sourceforge.net
> > > > > https://lists.sourceforge.net/lists/listinfo/gambas-user
> > > >
> > > > Yes. Right after the FOR loop which iterates through all the frames
> > > > for loading and processing.
> > > >
> > > > That is one of the reasons I thought I would ask for help on this. As
> > > > far as I can tell I have complied with all of Benoit's hints and tips
> > > > on managing memory. The frame variable is declared local to the event
> > > > handler: DIM frame AS NEW Image
> > > > The event handler is the only code which uses this variable. There is
> > > > a WAIT statement in the FOR loop so that the ProgressBar works, but
> > > > the memory is eaten up at the same ferocious rate either with or
> > > > without the WAIT.
> > > >
> > > > What I can tell for certain is that the memory is only allocated
> > > > while my code is executing the FOR loop. The frame variable is
> > > > created and destroyed outside this loop which strongly suggests that
> > > > I have not programmed the code which is using the memory. By deleting
> > > > all code within the loop EXCEPT the frame = Image.Load() line I can
> > > > run the code and watch as it runs through my memory at an estimated
> > > > rate of something like 4MiB per sub-1MiB frame. This just doesn't
> > > > look right. If it is then I cannot use Gambas to apply simple changes
> > > > to very large numbers of PNG images. That can't be right either.
> > > >
> > > > For completeness I should also mention that I have run the FOR loop
> > > > with both Image handling lines removed (load/stretch and save) and
> > > > there is no sign of memory loss; just blindness 'cos there are no
> > > > pictures:-)
> > > >
> > > > Richard
> > >
> > > Can you send your project, or better isolate the image processing that
> > > burns your memory in a little project?
> > >
> > > Regards,
> >
> > Glad to; I have been running a simple test project which reveals what I
> > now believe to be a bug of some long standing. I have only been able to
> > test 2.4, 2,16 and 2.20 so far. 2.4 works without consuming memory,
> > 2.16.and 2.20 have the bug.
> >
> > I put together a simple form with a progress bar and two buttons. I
> > observe memory and swap consumption while the progress bar moves towards
> > 100% using gkrellm - 'cos it's always there.
> >
> > The test program merely reloads a single image file many times so it is
> > quicker to run than the "real thing" where there may be hundreds  of
> > different files. Nevertheless its behaviour is just the same, suggesting
> > to me that there may be a problem in the Image.Load() method. The same
> > memory consumption problem occurs if the object is of type Picture and
> > Picture.Load() is used.
> >
> > Hope it's easy to find because I would really like to get this working
> > for my brother.
> >
> > Richard
> >
> > ' Gambas class file
> >
> > PUBLIC SUB _new()
> >
> > END
> >
> > PUBLIC SUB Form_Open()
> >
> > END
> >
> > PUBLIC SUB Quit_Click()
> >
> >   ME.Close
> >
> > END
> >
> > PUBLIC SUB Test_Click()
> > DIM n AS Integer
> > DIM Frame AS NEW Image
> >
> >     ProgressBar1.Visible = TRUE
> >     FOR n = 0 TO 799
> >         Frame = Image.Load("/home/richard/1024x576.png")
> >         ProgressBar1.Value = n / 499
> >         WAIT
> >     NEXT
> >     ProgressBar1.Visible = FALSE
> >
> > END
>
> Do you use gb.qt or gb.gtk?

Ah Benoit! Always the difficult questions :-p) According to the test project's 
components list it uses gb, gb.form, gb.gui and gb.image. The desktop/window 
manager is LXDE for the 2.16 and 2.20 tests. The 2.4 test was carried out in 
KDE.

I have just tried to select gb.qt for the test project but it said that gb.qt 
is incompatible with gb.gui. So I de-selected gb.gui and wasn't prevented 
from now selecting gb.qt. 

Recompiled and re-run and the test works perfectly. Thank you thank you thank 
you! I have never given any thought in the past about which toolkit I was 
using. Being a devotee of KDE, up to v3.5, I suppose I have always used gb.qt 
etc. Now since Mandriva 2010 I am forced to seek an alternative to KDE as 4.3 
is too much like Windows for my liking, certainly in terms of its poor 
performance. I have been using LXDE without realising that it might be using 
the gtk toolkit.

Now I have a solution I can easily deploy to my brother's PC (a clone of the 
one I develop on for him). Ooops, spoke too soon. The working project 
compiles ok but breaks now where I test the presence of an object in a 
collection. Looks like the place I get the collection key is not 
initialising. It is a ComboBox which I fill in one splash with DIr(path, 
pattern, gb.File). Must be a difference in qt and gtk combo boxes. Shouldn't 
take too long to fix though.

Thanks again for your help with this. 

Richard




More information about the User mailing list