Discussion:
Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
(too old to reply)
J. Gareth Moreton
2018-07-30 10:30:57 UTC
Permalink
On Mon 30/07/18 01:58 , Ben Grasset
I was even able to successfully actually
fix some of the leaks, but others
boiled down to a "node" being created as
a local variable in the middle of
some 1000+ line method, getting assigned
to the "left" or "right" property
of some other globally visible node, and
then never being freed.
To add to that, some of then actually
*could not* be freed even when I
tried to do so without raising an
untraceable access violation later in the
compilation process, also, which means
the compiler is basically knowingly
relying on undebuggable undefined
behaviour in multiple places that could
easily be affected or altered by
anything else anywhere else in the
codebase at any time.
The way-too-short, highly undescriptive
naming of variables doesn't
exactly help, either! "hp" and "p" are
not good variable names. They really
aren't! 
Overall, I'm not trying to put blame on
anyone in particular here, but as
someone who loves using FPC and would
like to see it continue to grow, it
would be nice if we could collectively
address some of these basic,
fundamental issues (which I'd be happy
to help with myself, of course.)
I too love Free Pascal and seek to improve
it. If you've found a fix, by all means
submit it as a patch in the bug tracker.

I've noticed that the compiler doesn't use
try...finally blocks to help with freeing
blocks. I'm not sure why this is the case,
but might be speed related.

More complex refactoring might require
deeper planning and approval from Florian.
For example, the node construction relies
a lot on global variables (especially
current_procinfo) that could possibly use
some improvement.

I think a big issue with refactoring is
the amount of testing required because of
the numerous target platforms.

Don't be disheartened though. With enough
eyes, all bugs are shallow!

Gareth aka. Kit

_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/li
Michael Van Canneyt
2018-07-30 11:36:50 UTC
Permalink
Post by J. Gareth Moreton
On Mon 30/07/18 01:58 , Ben Grasset
I was even able to successfully actually
fix some of the leaks, but others
boiled down to a "node" being created as
a local variable in the middle of
some 1000+ line method, getting assigned
to the "left" or "right" property
of some other globally visible node, and
then never being freed.
To add to that, some of then actually
*could not* be freed even when I
tried to do so without raising an
untraceable access violation later in the
compilation process, also, which means
the compiler is basically knowingly
relying on undebuggable undefined
behaviour in multiple places that could
easily be affected or altered by
anything else anywhere else in the
codebase at any time.
The way-too-short, highly undescriptive
naming of variables doesn't
exactly help, either! "hp" and "p" are
not good variable names. They really
aren't! 
Overall, I'm not trying to put blame on
anyone in particular here, but as
someone who loves using FPC and would
like to see it continue to grow, it
would be nice if we could collectively
address some of these basic,
fundamental issues (which I'd be happy
to help with myself, of course.)
I too love Free Pascal and seek to improve
it. If you've found a fix, by all means
submit it as a patch in the bug tracker.
I've noticed that the compiler doesn't use
try...finally blocks to help with freeing
blocks. I'm not sure why this is the case,
but might be speed related.
Yes. try/finally blocks cause *tremendous* slowdown.

Michael.
Sven Barth via fpc-devel
2018-07-30 12:14:39 UTC
Permalink
Post by J. Gareth Moreton
I've noticed that the compiler doesn't use
try...finally blocks to help with freeing
blocks. I'm not sure why this is the case,
but might be speed related.
Correct. Even implicit try-finally frame generation is disabled for the
compiler source.

Regards,
Sven
Marcos Douglas B. Santos
2018-07-30 12:24:48 UTC
Permalink
On Mon, Jul 30, 2018 at 9:14 AM, Sven Barth via fpc-devel <
Post by Sven Barth via fpc-devel
Post by J. Gareth Moreton
I've noticed that the compiler doesn't use
try...finally blocks to help with freeing
blocks. I'm not sure why this is the case,
but might be speed related.
Correct. Even implicit try-finally frame generation is disabled for the
compiler source.
Is performance more important than being correct? :|

regards,
Marcos Douglas
Martok
2018-07-30 12:32:01 UTC
Permalink
Is performance more important than being correct?  :|
In this project, the answer is always taken to be yes.
--
Regards,
Martok


_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/list
R0b0t1
2018-07-30 15:29:27 UTC
Permalink
Post by Martok
Post by Marcos Douglas B. Santos
Is performance more important than being correct? :|
In this project, the answer is always taken to be yes.
To hopefully offer some explanation for Mr. Santos - if I can compile
and run the program 5 times over the course of 20 minutes, that is
certainly better than running it and testing it once over the same
amount of time, no?

It might be hard to imagine FPC taking that much longer than it does
currently but ~30min for a large program is the standard with other
compilers. I very much enjoy the speed of FPC.
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/m
Michael Van Canneyt
2018-07-30 15:35:14 UTC
Permalink
Post by R0b0t1
Post by Martok
Post by Marcos Douglas B. Santos
Is performance more important than being correct? :|
In this project, the answer is always taken to be yes.
To hopefully offer some explanation for Mr. Santos - if I can compile
and run the program 5 times over the course of 20 minutes, that is
certainly better than running it and testing it once over the same
amount of time, no?
It might be hard to imagine FPC taking that much longer than it does
currently but ~30min for a large program is the standard with other
compilers. I very much enjoy the speed of FPC.
30 *Minutes*, is this real ?

Hm. I just complained to a component provider because Delphi takes 25 seconds to
compile my program with the way they set up their paths...

Michael.
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal
R0b0t1
2018-07-30 16:11:23 UTC
Permalink
On Mon, Jul 30, 2018 at 10:35 AM, Michael Van Canneyt
Post by Michael Van Canneyt
Post by R0b0t1
Post by Martok
Post by Marcos Douglas B. Santos
Is performance more important than being correct? :|
In this project, the answer is always taken to be yes.
To hopefully offer some explanation for Mr. Santos - if I can compile
and run the program 5 times over the course of 20 minutes, that is
certainly better than running it and testing it once over the same
amount of time, no?
It might be hard to imagine FPC taking that much longer than it does
currently but ~30min for a large program is the standard with other
compilers. I very much enjoy the speed of FPC.
30 *Minutes*, is this real ?
Hm. I just complained to a component provider because Delphi takes 25
seconds to compile my program with the way they set up their paths...
This is real.[1] I've experienced it myself and you can see some good
discussions on https://news.ycombinator.com with a search. FPC/Lazarus
has made some waves there in the last few years.

FPC does have quite a bit of a lead it could shed in an attempt to be
"more correct" but I would caution against this. If at every point in
a software stack someone says "modern computers are just so fast, we
have a little bit of wiggle room" you end up with something slower
than molasses. Perfection requires effort at every single step.

A good case study on this is keyboard latency in modern systems.[2]

Cheers,
R0b0t1

[1]: The most extreme comparison was closer to being something like a
couple of minutes vs. a couple of hours, but I can't source that right
now. Most people think it's a lie; granted, some of it is likely due
to the excellent incremental compilation FPC has by default.
[2]: https://danluu.com/keyboard-latency/
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinf
w***@windstream.net
2018-07-30 18:39:24 UTC
Permalink
Post by Michael Van Canneyt
Post by R0b0t1
It might be hard to imagine FPC taking that much longer than it does
currently but ~30min for a large program is the standard with other
compilers. I very much enjoy the speed of FPC. >
30 *Minutes*, is this real ?
yep, for sufficiently large projects... when we compile OSG (OpenSceneGraph)
over here (using -j 8) it takes 20+ minutes for the first time or if we
reconfigure from debug to release mode or the other way around...

now, after the first time we compile it, the next times takes mere 10's of
seconds... then we move on to SceneGear and FlightGear when we're building that
whole project which uses OSG... 30-45 minutes for a complete, from the ground up
builds are not uncommon... it takes me back to the days of 30 years ago when
you'd start the (borland/turbo pascal) compile process and go get some c0ffee ;)

note: the machines these builds are being done on have 16Gig RAM, 1Tb HD, and a
4Ghz 8-core AMD FX-8350 CPU...
--
NOTE: No off-list assistance is given without prior approval.
*Please keep mailing list traffic on the list unless*
*a signed and pre-paid contract is in effect with us.*
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.
Marcos Douglas B. Santos
2018-07-30 16:42:37 UTC
Permalink
Post by R0b0t1
Post by Martok
Post by Marcos Douglas B. Santos
Is performance more important than being correct? :|
In this project, the answer is always taken to be yes.
To hopefully offer some explanation for Mr. Santos - if I can compile
and run the program 5 times over the course of 20 minutes, that is
certainly better than running it and testing it once over the same
amount of time, no?
It might be hard to imagine FPC taking that much longer than it does
currently but ~30min for a large program is the standard with other
compilers. I very much enjoy the speed of FPC.
All us like fast compiling...
However, IMHO, a compiler should use the same good practices that
others programs must use.
If it cannot use them because it's very slow, what is the matter about
good practices?

"Do what I'm telling you to do, but don't do what I do" ? :)

best regards,
Marcos Douglas
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/l
R0b0t1
2018-07-30 18:31:45 UTC
Permalink
On Mon, Jul 30, 2018 at 11:42 AM, Marcos Douglas B. Santos
Post by Marcos Douglas B. Santos
Post by R0b0t1
Post by Martok
Post by Marcos Douglas B. Santos
Is performance more important than being correct? :|
In this project, the answer is always taken to be yes.
To hopefully offer some explanation for Mr. Santos - if I can compile
and run the program 5 times over the course of 20 minutes, that is
certainly better than running it and testing it once over the same
amount of time, no?
It might be hard to imagine FPC taking that much longer than it does
currently but ~30min for a large program is the standard with other
compilers. I very much enjoy the speed of FPC.
All us like fast compiling...
However, IMHO, a compiler should use the same good practices that
others programs must use.
If it cannot use them because it's very slow, what is the matter about
good practices?
"Do what I'm telling you to do, but don't do what I do" ? :)
If a program isn't long running I see programmers tend to not care
about memory management, nor do I make a habit of caring about it
myself. I find it hard to see anything wrong with this.

As someone has mentioned this can fail on smaller computers, but how
much RAM is too little? SBCs now typically come with 1-4GB of memory.

Cheers,
R0b0t1
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://li
Marcos Douglas B. Santos
2018-07-30 20:27:27 UTC
Permalink
Post by R0b0t1
On Mon, Jul 30, 2018 at 11:42 AM, Marcos Douglas B. Santos
[...]
If a program isn't long running I see programmers tend to not care
about memory management, nor do I make a habit of caring about it
myself. I find it hard to see anything wrong with this.
As someone has mentioned this can fail on smaller computers, but how
much RAM is too little? SBCs now typically come with 1-4GB of memory.
I'm sorry, but we work with a language that expected programmers
should manage their objects.
The language even provides a clean way (try-finall) to do that.
I use try-finally even to write a snippet to show an example of code...

IMO, that must be a habit in such languages.

regards,
Marcos Douglas
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/l

Michael Van Canneyt
2018-07-30 12:36:10 UTC
Permalink
Post by Marcos Douglas B. Santos
On Mon, Jul 30, 2018 at 9:14 AM, Sven Barth via fpc-devel <
Post by Sven Barth via fpc-devel
Post by J. Gareth Moreton
I've noticed that the compiler doesn't use
try...finally blocks to help with freeing
blocks. I'm not sure why this is the case,
but might be speed related.
Correct. Even implicit try-finally frame generation is disabled for the
compiler source.
Is performance more important than being correct? :|
You've been around long enough to know that complaints about
the speed of the compiler is something that pops up on a regular basis.

Also, and this is important: the compiler was developed for TP.
It had no concept of exceptions. Everything was done using records and
pointers.

Many people will probably be surprised, but the use of try/finally
is not required for an application:

Gradually, some parts of the compiler were switched to classes, but even
then, without exceptions: As long as you do not use the sysutils unit,
(which was the case for many years for the compiler) exceptions will
not be raised, and so there is simply no need for try/finally blocks.

Probably it would be better to remove the dependency on sysutils, to keep
the compiler free of try/finally blocks.

But the compiler devs will be able to provide a more correct&detailed picture than
this.

Michael.
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freep
Mattias Gaertner
2018-07-30 13:19:53 UTC
Permalink
On Mon, 30 Jul 2018 14:36:10 +0200 (CEST)
Post by Michael Van Canneyt
[...]
Many people will probably be surprised, but the use of try/finally
Gradually, some parts of the compiler were switched to classes, but even
then, without exceptions: As long as you do not use the sysutils unit,
(which was the case for many years for the compiler) exceptions will
not be raised, and so there is simply no need for try/finally blocks.
Many people will probably surprised, that try/finally/except works
without sysutils and try/except does not need "Exception". It
works with any TObject.
Post by Michael Van Canneyt
Probably it would be better to remove the dependency on sysutils, to keep
the compiler free of try/finally blocks.
Sysutils has nothing to do with creating exception frames.

Mattias
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.free
Michael Van Canneyt
2018-07-30 13:30:00 UTC
Permalink
Post by Mattias Gaertner
On Mon, 30 Jul 2018 14:36:10 +0200 (CEST)
Post by Michael Van Canneyt
[...]
Many people will probably be surprised, but the use of try/finally
Gradually, some parts of the compiler were switched to classes, but even
then, without exceptions: As long as you do not use the sysutils unit,
(which was the case for many years for the compiler) exceptions will
not be raised, and so there is simply no need for try/finally blocks.
Many people will probably surprised, that try/finally/except works
without sysutils and try/except does not need "Exception". It
works with any TObject.
Post by Michael Van Canneyt
Probably it would be better to remove the dependency on sysutils, to keep
the compiler free of try/finally blocks.
Sysutils has nothing to do with creating exception frames.
It does: if you know the program will never raise exceptions, it makes no
sense to install try/finally blocks.

Runtime errors are converted to exceptions by the sysutils unit.
By not including it, you know you will just get runtime errors, and hence
you don't need try..finally.

Obviously provided you don't install another mechanism that does this and
don't raise exceptions manually, which - AFAIK - is the case in the
compiler...

Michael.
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepa
Sven Barth via fpc-devel
2018-07-30 15:16:57 UTC
Permalink
Post by Michael Van Canneyt
Obviously provided you don't install another mechanism that does this and
don't raise exceptions manually, which - AFAIK - is the case in the
compiler...
The compiler does use exceptions when the compilation needs to be aborted
e.g. when too many errors occurred or an internal error is encountered.

Regards,
Sven
Michael Van Canneyt
2018-07-30 15:24:14 UTC
Permalink
Post by Sven Barth via fpc-devel
Post by Michael Van Canneyt
Obviously provided you don't install another mechanism that does this and
don't raise exceptions manually, which - AFAIK - is the case in the
compiler...
The compiler does use exceptions when the compilation needs to be aborted
e.g. when too many errors occurred or an internal error is encountered.
This is bad news, because it in effect means that the try/finally is necessary
if people want to fix memleaks :(

This will considerably slow down the compiler.

Michael.
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-deve
R0b0t1
2018-07-30 16:22:22 UTC
Permalink
On Mon, Jul 30, 2018 at 10:24 AM, Michael Van Canneyt
Post by Michael Van Canneyt
Post by Sven Barth via fpc-devel
Post by Michael Van Canneyt
Obviously provided you don't install another mechanism that does this and
don't raise exceptions manually, which - AFAIK - is the case in the
compiler...
The compiler does use exceptions when the compilation needs to be aborted
e.g. when too many errors occurred or an internal error is encountered.
This is bad news, because it in effect means that the try/finally is necessary
if people want to fix memleaks :(
This will considerably slow down the compiler.
I would very much like to see the compiler usable as a dynamic
library. Compartmentalizing the bulk of the code in this manner might
aid readability and comprehension even if it is not immediately wise
to actually use the shared object in a third party application due to
memory leaks.

I wouldn't mind running some static analysis on FPC in an attempt to
get rid of memory leaks. It's fairly mechanical and probably a good
introduction to the codebase. This page[1] is promising as it
indicates valgrind should work well on FPC binaries (I didn't have
much a reason to expect otherwise, but you never know).

Is there any reason this hasn't been done? What could raise statements
be replaced with if they must be replaced?

Cheers,
R0b0t1

[1]: http://wiki.lazarus.freepascal.org/Profiling
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists
Thorsten Engler
2018-07-30 12:40:27 UTC
Permalink
From: fpc-devel <fpc-devel-***@lists.freepascal.org> On Behalf Of Marcos Douglas B. Santos
Sent: Monday, 30 July 2018 22:25
To: FPC developers' list <fpc-***@lists.freepascal.org>
Subject: Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?



On Mon, Jul 30, 2018 at 9:14 AM, Sven Barth via fpc-devel <fpc-***@lists.freepascal.org <mailto:fpc-***@lists.freepascal.org> > wrote:

J. Gareth Moreton <***@moreton-family.com <mailto:***@moreton-family.com> > schrieb am Mo., 30. Juli 2018, 13:31:

I've noticed that the compiler doesn't use
try...finally blocks to help with freeing
blocks. I'm not sure why this is the case,
but might be speed related.



Correct. Even implicit try-finally frame generation is disabled for the compiler source.



Is performance more important than being correct? :|





Freeing memory is irrelevant if your process is going to exit anyway right after freeing it. The OS doesn’t care about any memory your process has allocated at that time. All pages go back into the available pool.



Depending on how the allocation/free behaviour of the compiler looks like, but it might even make sense to use a memory manager that doesn’t even know how to free memory.



(I’m using windows terminology here, but I’m sure equivalent can be found on other platforms)



Reserve a large block of address space.

Install a vectored exception handler that reacts to an access violation that falls inside the reserved block by committing the memory between start and error address + 1MB (or more or less).



Naïvely and without error checking, Alloc gets reduced to:



var

CurrentPtr: PByte;



function Alloc(Size: Integer):Pointer;

begin

Size := (Size + 3) and not 3; //adjust depending on what allocation alignment is desired;

If Size > 0 then begin

Result := CurrentPtr;

CurrentPtr := CurrentPtr + Size;

end else

Result := nil;

end;



procedure Free(p: Pointer);

begin

{do nothing}

end;
Loading...