Discussion:
Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
(too old to reply)
J. Gareth Moreton
2018-07-30 10:30:57 UTC
Permalink
Raw Message
On Mon 30/07/18 01:58 , Ben Grasset
***@gmail.com sent:
> I was even able to successfully actually
fix some of the leaks, but others
> boiled down to a "node" being created as
a local variable in the middle of
> some 1000+ line method, getting assigned
to the "left" or "right" property
> of some other globally visible node, and
then never being freed.
> To add to that, some of then actually
*could not* be freed even when I
> tried to do so without raising an
untraceable access violation later in the
> compilation process, also, which means
the compiler is basically knowingly
> relying on undebuggable undefined
behaviour in multiple places that could
> easily be affected or altered by
anything else anywhere else in the
> codebase at any time.
> The way-too-short, highly undescriptive
naming of variables doesn't
> exactly help, either! "hp" and "p" are
not good variable names. They really
> aren't! 
>
> Overall, I'm not trying to put blame on
anyone in particular here, but as
> someone who loves using FPC and would
like to see it continue to grow, it
> would be nice if we could collectively
address some of these basic,
> fundamental issues (which I'd be happy
to help with myself, of course.)
>

I too love Free Pascal and seek to improve
it. If you've found a fix, by all means
submit it as a patch in the bug tracker.

I've noticed that the compiler doesn't use
try...finally blocks to help with freeing
blocks. I'm not sure why this is the case,
but might be speed related.

More complex refactoring might require
deeper planning and approval from Florian.
For example, the node construction relies
a lot on global variables (especially
current_procinfo) that could possibly use
some improvement.

I think a big issue with refactoring is
the amount of testing required because of
the numerous target platforms.

Don't be disheartened though. With enough
eyes, all bugs are shallow!

Gareth aka. Kit

_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/li
Michael Van Canneyt
2018-07-30 11:36:50 UTC
Permalink
Raw Message
On Mon, 30 Jul 2018, J. Gareth Moreton wrote:

>
>
> On Mon 30/07/18 01:58 , Ben Grasset
> ***@gmail.com sent:
>> I was even able to successfully actually
> fix some of the leaks, but others
>> boiled down to a "node" being created as
> a local variable in the middle of
>> some 1000+ line method, getting assigned
> to the "left" or "right" property
>> of some other globally visible node, and
> then never being freed.
>> To add to that, some of then actually
> *could not* be freed even when I
>> tried to do so without raising an
> untraceable access violation later in the
>> compilation process, also, which means
> the compiler is basically knowingly
>> relying on undebuggable undefined
> behaviour in multiple places that could
>> easily be affected or altered by
> anything else anywhere else in the
>> codebase at any time.
>> The way-too-short, highly undescriptive
> naming of variables doesn't
>> exactly help, either! "hp" and "p" are
> not good variable names. They really
>> aren't! 
>>
>> Overall, I'm not trying to put blame on
> anyone in particular here, but as
>> someone who loves using FPC and would
> like to see it continue to grow, it
>> would be nice if we could collectively
> address some of these basic,
>> fundamental issues (which I'd be happy
> to help with myself, of course.)
>>
>
> I too love Free Pascal and seek to improve
> it. If you've found a fix, by all means
> submit it as a patch in the bug tracker.
>
> I've noticed that the compiler doesn't use
> try...finally blocks to help with freeing
> blocks. I'm not sure why this is the case,
> but might be speed related.

Yes. try/finally blocks cause *tremendous* slowdown.

Michael.
Sven Barth via fpc-devel
2018-07-30 12:14:39 UTC
Permalink
Raw Message
J. Gareth Moreton <***@moreton-family.com> schrieb am Mo., 30. Juli
2018, 13:31:

> I've noticed that the compiler doesn't use
> try...finally blocks to help with freeing
> blocks. I'm not sure why this is the case,
> but might be speed related.
>

Correct. Even implicit try-finally frame generation is disabled for the
compiler source.

Regards,
Sven

>
Marcos Douglas B. Santos
2018-07-30 12:24:48 UTC
Permalink
Raw Message
On Mon, Jul 30, 2018 at 9:14 AM, Sven Barth via fpc-devel <
fpc-***@lists.freepascal.org> wrote:

> J. Gareth Moreton <***@moreton-family.com> schrieb am Mo., 30. Juli
> 2018, 13:31:
>
>> I've noticed that the compiler doesn't use
>> try...finally blocks to help with freeing
>> blocks. I'm not sure why this is the case,
>> but might be speed related.
>>
>
> Correct. Even implicit try-finally frame generation is disabled for the
> compiler source.
>

Is performance more important than being correct? :|

regards,
Marcos Douglas
Martok
2018-07-30 12:32:01 UTC
Permalink
Raw Message
Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos:
> Is performance more important than being correct?  :|
In this project, the answer is always taken to be yes.

--
Regards,
Martok


_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/list
R0b0t1
2018-07-30 15:29:27 UTC
Permalink
Raw Message
On Mon, Jul 30, 2018 at 7:32 AM, Martok <***@martoks-place.de> wrote:
> Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos:
>> Is performance more important than being correct? :|
> In this project, the answer is always taken to be yes.
>

To hopefully offer some explanation for Mr. Santos - if I can compile
and run the program 5 times over the course of 20 minutes, that is
certainly better than running it and testing it once over the same
amount of time, no?

It might be hard to imagine FPC taking that much longer than it does
currently but ~30min for a large program is the standard with other
compilers. I very much enjoy the speed of FPC.
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/m
Michael Van Canneyt
2018-07-30 15:35:14 UTC
Permalink
Raw Message
On Mon, 30 Jul 2018, R0b0t1 wrote:

> On Mon, Jul 30, 2018 at 7:32 AM, Martok <***@martoks-place.de> wrote:
>> Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos:
>>> Is performance more important than being correct? :|
>> In this project, the answer is always taken to be yes.
>>
>
> To hopefully offer some explanation for Mr. Santos - if I can compile
> and run the program 5 times over the course of 20 minutes, that is
> certainly better than running it and testing it once over the same
> amount of time, no?
>
> It might be hard to imagine FPC taking that much longer than it does
> currently but ~30min for a large program is the standard with other
> compilers. I very much enjoy the speed of FPC.

30 *Minutes*, is this real ?

Hm. I just complained to a component provider because Delphi takes 25 seconds to
compile my program with the way they set up their paths...

Michael.
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal
R0b0t1
2018-07-30 16:11:23 UTC
Permalink
Raw Message
On Mon, Jul 30, 2018 at 10:35 AM, Michael Van Canneyt
<***@freepascal.org> wrote:
>
>
> On Mon, 30 Jul 2018, R0b0t1 wrote:
>
>> On Mon, Jul 30, 2018 at 7:32 AM, Martok <***@martoks-place.de> wrote:
>>>
>>> Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos:
>>>>
>>>> Is performance more important than being correct? :|
>>>
>>> In this project, the answer is always taken to be yes.
>>>
>>
>> To hopefully offer some explanation for Mr. Santos - if I can compile
>> and run the program 5 times over the course of 20 minutes, that is
>> certainly better than running it and testing it once over the same
>> amount of time, no?
>>
>> It might be hard to imagine FPC taking that much longer than it does
>> currently but ~30min for a large program is the standard with other
>> compilers. I very much enjoy the speed of FPC.
>
>
> 30 *Minutes*, is this real ?
>
> Hm. I just complained to a component provider because Delphi takes 25
> seconds to compile my program with the way they set up their paths...
>

This is real.[1] I've experienced it myself and you can see some good
discussions on https://news.ycombinator.com with a search. FPC/Lazarus
has made some waves there in the last few years.

FPC does have quite a bit of a lead it could shed in an attempt to be
"more correct" but I would caution against this. If at every point in
a software stack someone says "modern computers are just so fast, we
have a little bit of wiggle room" you end up with something slower
than molasses. Perfection requires effort at every single step.

A good case study on this is keyboard latency in modern systems.[2]

Cheers,
R0b0t1

[1]: The most extreme comparison was closer to being something like a
couple of minutes vs. a couple of hours, but I can't source that right
now. Most people think it's a lie; granted, some of it is likely due
to the excellent incremental compilation FPC has by default.
[2]: https://danluu.com/keyboard-latency/
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinf
w***@windstream.net
2018-07-30 18:39:24 UTC
Permalink
Raw Message
On 07/30/2018 11:35 AM, Michael Van Canneyt wrote:
> On Mon, 30 Jul 2018, R0b0t1 wrote:
>> It might be hard to imagine FPC taking that much longer than it does
>> currently but ~30min for a large program is the standard with other
>> compilers. I very much enjoy the speed of FPC. >
> 30 *Minutes*, is this real ?


yep, for sufficiently large projects... when we compile OSG (OpenSceneGraph)
over here (using -j 8) it takes 20+ minutes for the first time or if we
reconfigure from debug to release mode or the other way around...

now, after the first time we compile it, the next times takes mere 10's of
seconds... then we move on to SceneGear and FlightGear when we're building that
whole project which uses OSG... 30-45 minutes for a complete, from the ground up
builds are not uncommon... it takes me back to the days of 30 years ago when
you'd start the (borland/turbo pascal) compile process and go get some c0ffee ;)

note: the machines these builds are being done on have 16Gig RAM, 1Tb HD, and a
4Ghz 8-core AMD FX-8350 CPU...


--
NOTE: No off-list assistance is given without prior approval.
*Please keep mailing list traffic on the list unless*
*a signed and pre-paid contract is in effect with us.*
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.
Marcos Douglas B. Santos
2018-07-30 16:42:37 UTC
Permalink
Raw Message
On Mon, Jul 30, 2018 at 12:29 PM, R0b0t1 <***@gmail.com> wrote:
> On Mon, Jul 30, 2018 at 7:32 AM, Martok <***@martoks-place.de> wrote:
>> Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos:
>>> Is performance more important than being correct? :|
>> In this project, the answer is always taken to be yes.
>>
>
> To hopefully offer some explanation for Mr. Santos - if I can compile
> and run the program 5 times over the course of 20 minutes, that is
> certainly better than running it and testing it once over the same
> amount of time, no?
>
> It might be hard to imagine FPC taking that much longer than it does
> currently but ~30min for a large program is the standard with other
> compilers. I very much enjoy the speed of FPC.

All us like fast compiling...
However, IMHO, a compiler should use the same good practices that
others programs must use.
If it cannot use them because it's very slow, what is the matter about
good practices?

"Do what I'm telling you to do, but don't do what I do" ? :)

best regards,
Marcos Douglas
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/l
R0b0t1
2018-07-30 18:31:45 UTC
Permalink
Raw Message
On Mon, Jul 30, 2018 at 11:42 AM, Marcos Douglas B. Santos
<***@delfire.net> wrote:
> On Mon, Jul 30, 2018 at 12:29 PM, R0b0t1 <***@gmail.com> wrote:
>> On Mon, Jul 30, 2018 at 7:32 AM, Martok <***@martoks-place.de> wrote:
>>> Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos:
>>>> Is performance more important than being correct? :|
>>> In this project, the answer is always taken to be yes.
>>>
>>
>> To hopefully offer some explanation for Mr. Santos - if I can compile
>> and run the program 5 times over the course of 20 minutes, that is
>> certainly better than running it and testing it once over the same
>> amount of time, no?
>>
>> It might be hard to imagine FPC taking that much longer than it does
>> currently but ~30min for a large program is the standard with other
>> compilers. I very much enjoy the speed of FPC.
>
> All us like fast compiling...
> However, IMHO, a compiler should use the same good practices that
> others programs must use.
> If it cannot use them because it's very slow, what is the matter about
> good practices?
>
> "Do what I'm telling you to do, but don't do what I do" ? :)
>

If a program isn't long running I see programmers tend to not care
about memory management, nor do I make a habit of caring about it
myself. I find it hard to see anything wrong with this.

As someone has mentioned this can fail on smaller computers, but how
much RAM is too little? SBCs now typically come with 1-4GB of memory.

Cheers,
R0b0t1
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://li
Marcos Douglas B. Santos
2018-07-30 20:27:27 UTC
Permalink
Raw Message
On Mon, Jul 30, 2018 at 3:31 PM, R0b0t1 <***@gmail.com> wrote:
> On Mon, Jul 30, 2018 at 11:42 AM, Marcos Douglas B. Santos
> <***@delfire.net> wrote:
>> On Mon, Jul 30, 2018 at 12:29 PM, R0b0t1 <***@gmail.com> wrote:
>>
>> [...]
>
> If a program isn't long running I see programmers tend to not care
> about memory management, nor do I make a habit of caring about it
> myself. I find it hard to see anything wrong with this.
>
> As someone has mentioned this can fail on smaller computers, but how
> much RAM is too little? SBCs now typically come with 1-4GB of memory.

I'm sorry, but we work with a language that expected programmers
should manage their objects.
The language even provides a clean way (try-finall) to do that.
I use try-finally even to write a snippet to show an example of code...

IMO, that must be a habit in such languages.

regards,
Marcos Douglas
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/l
Michael Van Canneyt
2018-07-30 12:36:10 UTC
Permalink
Raw Message
On Mon, 30 Jul 2018, Marcos Douglas B. Santos wrote:

> On Mon, Jul 30, 2018 at 9:14 AM, Sven Barth via fpc-devel <
> fpc-***@lists.freepascal.org> wrote:
>
>> J. Gareth Moreton <***@moreton-family.com> schrieb am Mo., 30. Juli
>> 2018, 13:31:
>>
>>> I've noticed that the compiler doesn't use
>>> try...finally blocks to help with freeing
>>> blocks. I'm not sure why this is the case,
>>> but might be speed related.
>>>
>>
>> Correct. Even implicit try-finally frame generation is disabled for the
>> compiler source.
>>
>
> Is performance more important than being correct? :|

You've been around long enough to know that complaints about
the speed of the compiler is something that pops up on a regular basis.

Also, and this is important: the compiler was developed for TP.
It had no concept of exceptions. Everything was done using records and
pointers.

Many people will probably be surprised, but the use of try/finally
is not required for an application:

Gradually, some parts of the compiler were switched to classes, but even
then, without exceptions: As long as you do not use the sysutils unit,
(which was the case for many years for the compiler) exceptions will
not be raised, and so there is simply no need for try/finally blocks.

Probably it would be better to remove the dependency on sysutils, to keep
the compiler free of try/finally blocks.

But the compiler devs will be able to provide a more correct&detailed picture than
this.

Michael.
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freep
Mattias Gaertner
2018-07-30 13:19:53 UTC
Permalink
Raw Message
On Mon, 30 Jul 2018 14:36:10 +0200 (CEST)
Michael Van Canneyt <***@freepascal.org> wrote:

>[...]
> Many people will probably be surprised, but the use of try/finally
> is not required for an application:
>
> Gradually, some parts of the compiler were switched to classes, but even
> then, without exceptions: As long as you do not use the sysutils unit,
> (which was the case for many years for the compiler) exceptions will
> not be raised, and so there is simply no need for try/finally blocks.

Many people will probably surprised, that try/finally/except works
without sysutils and try/except does not need "Exception". It
works with any TObject.


> Probably it would be better to remove the dependency on sysutils, to keep
> the compiler free of try/finally blocks.

Sysutils has nothing to do with creating exception frames.

Mattias
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.free
Michael Van Canneyt
2018-07-30 13:30:00 UTC
Permalink
Raw Message
On Mon, 30 Jul 2018, Mattias Gaertner wrote:

> On Mon, 30 Jul 2018 14:36:10 +0200 (CEST)
> Michael Van Canneyt <***@freepascal.org> wrote:
>
>> [...]
>> Many people will probably be surprised, but the use of try/finally
>> is not required for an application:
>>
>> Gradually, some parts of the compiler were switched to classes, but even
>> then, without exceptions: As long as you do not use the sysutils unit,
>> (which was the case for many years for the compiler) exceptions will
>> not be raised, and so there is simply no need for try/finally blocks.
>
> Many people will probably surprised, that try/finally/except works
> without sysutils and try/except does not need "Exception". It
> works with any TObject.
>
>> Probably it would be better to remove the dependency on sysutils, to keep
>> the compiler free of try/finally blocks.
>
> Sysutils has nothing to do with creating exception frames.

It does: if you know the program will never raise exceptions, it makes no
sense to install try/finally blocks.

Runtime errors are converted to exceptions by the sysutils unit.
By not including it, you know you will just get runtime errors, and hence
you don't need try..finally.

Obviously provided you don't install another mechanism that does this and
don't raise exceptions manually, which - AFAIK - is the case in the
compiler...

Michael.
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepa
Sven Barth via fpc-devel
2018-07-30 15:16:57 UTC
Permalink
Raw Message
Michael Van Canneyt <***@freepascal.org> schrieb am Mo., 30. Juli 2018,
15:30:

> Obviously provided you don't install another mechanism that does this and
> don't raise exceptions manually, which - AFAIK - is the case in the
> compiler...
>

The compiler does use exceptions when the compilation needs to be aborted
e.g. when too many errors occurred or an internal error is encountered.

Regards,
Sven

>
Michael Van Canneyt
2018-07-30 15:24:14 UTC
Permalink
Raw Message
On Mon, 30 Jul 2018, Sven Barth via fpc-devel wrote:

> Michael Van Canneyt <***@freepascal.org> schrieb am Mo., 30. Juli 2018,
> 15:30:
>
>> Obviously provided you don't install another mechanism that does this and
>> don't raise exceptions manually, which - AFAIK - is the case in the
>> compiler...
>>
>
> The compiler does use exceptions when the compilation needs to be aborted
> e.g. when too many errors occurred or an internal error is encountered.

This is bad news, because it in effect means that the try/finally is necessary
if people want to fix memleaks :(

This will considerably slow down the compiler.

Michael.
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-deve
R0b0t1
2018-07-30 16:22:22 UTC
Permalink
Raw Message
On Mon, Jul 30, 2018 at 10:24 AM, Michael Van Canneyt
<***@freepascal.org> wrote:
>
>
> On Mon, 30 Jul 2018, Sven Barth via fpc-devel wrote:
>
>> Michael Van Canneyt <***@freepascal.org> schrieb am Mo., 30. Juli
>> 2018,
>> 15:30:
>>
>>> Obviously provided you don't install another mechanism that does this and
>>> don't raise exceptions manually, which - AFAIK - is the case in the
>>> compiler...
>>>
>>
>> The compiler does use exceptions when the compilation needs to be aborted
>> e.g. when too many errors occurred or an internal error is encountered.
>
>
> This is bad news, because it in effect means that the try/finally is
> necessary
> if people want to fix memleaks :(
>
> This will considerably slow down the compiler.
>

I would very much like to see the compiler usable as a dynamic
library. Compartmentalizing the bulk of the code in this manner might
aid readability and comprehension even if it is not immediately wise
to actually use the shared object in a third party application due to
memory leaks.

I wouldn't mind running some static analysis on FPC in an attempt to
get rid of memory leaks. It's fairly mechanical and probably a good
introduction to the codebase. This page[1] is promising as it
indicates valgrind should work well on FPC binaries (I didn't have
much a reason to expect otherwise, but you never know).

Is there any reason this hasn't been done? What could raise statements
be replaced with if they must be replaced?

Cheers,
R0b0t1

[1]: http://wiki.lazarus.freepascal.org/Profiling
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists
Thorsten Engler
2018-07-30 12:40:27 UTC
Permalink
Raw Message
From: fpc-devel <fpc-devel-***@lists.freepascal.org> On Behalf Of Marcos Douglas B. Santos
Sent: Monday, 30 July 2018 22:25
To: FPC developers' list <fpc-***@lists.freepascal.org>
Subject: Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?



On Mon, Jul 30, 2018 at 9:14 AM, Sven Barth via fpc-devel <fpc-***@lists.freepascal.org <mailto:fpc-***@lists.freepascal.org> > wrote:

J. Gareth Moreton <***@moreton-family.com <mailto:***@moreton-family.com> > schrieb am Mo., 30. Juli 2018, 13:31:

I've noticed that the compiler doesn't use
try...finally blocks to help with freeing
blocks. I'm not sure why this is the case,
but might be speed related.



Correct. Even implicit try-finally frame generation is disabled for the compiler source.



Is performance more important than being correct? :|





Freeing memory is irrelevant if your process is going to exit anyway right after freeing it. The OS doesn’t care about any memory your process has allocated at that time. All pages go back into the available pool.



Depending on how the allocation/free behaviour of the compiler looks like, but it might even make sense to use a memory manager that doesn’t even know how to free memory.



(I’m using windows terminology here, but I’m sure equivalent can be found on other platforms)



Reserve a large block of address space.

Install a vectored exception handler that reacts to an access violation that falls inside the reserved block by committing the memory between start and error address + 1MB (or more or less).



Naïvely and without error checking, Alloc gets reduced to:



var

CurrentPtr: PByte;



function Alloc(Size: Integer):Pointer;

begin

Size := (Size + 3) and not 3; //adjust depending on what allocation alignment is desired;

If Size > 0 then begin

Result := CurrentPtr;

CurrentPtr := CurrentPtr + Size;

end else

Result := nil;

end;



procedure Free(p: Pointer);

begin

{do nothing}

end;
Loading...