Discussion:
Why/how does the compiler have a non-trivial numberofmemory leaks after over two decades of development?
(too old to reply)
J. Gareth Moreton
2018-07-30 12:45:58 UTC
Permalink
Raw Message
I would say that that's a little naïve and dangerous to think like that.
Sure, Windows might have the means to clean up memory after an application
terminates, but not all platforms have such heap deallocation features
(e.g. pure DOS, where certain procedures and interrupts remain in memory
even after the application terminates... so-called memory-resident
programs).

Even if all blocks are freed when the compiler finishes its work, memory
leaks may still cause problems with sufficiently large projects that are
being compiled in one go.  The more leaks you have, the more likely you're
going to hit an out of memory situation, especially as you can't really
control the target platforms in regards to how they have their physical and
virtual memory and page files configured.
A lot of the issues do come about from the fact that Free Pascal has a lot
of legacy code and started life as a 16-bit Turbo Pascal project (it was
first made because Turbo Pascal was being discontinued so Borland could
focus on Windows development, and it didn't have 32-bit support).  Free
Pascal is advanced enough now that a lot of the features could be rewritten
to use more robust and self-contained features, but it's going to be a long
job.
I've made some suggestions to major improvements, especially to speed or
reducing the size of the compiler's binary (and the size of the binaries
that it generates), but Florian has turned down some of the patches either
because it's too experimental (my Deep Optimizer prototype) or
overcomplicates the codes (I replaced a large case block for the x86
peephole optimiser with a custom-made binary search system, which was
slightly faster but wasn't obvious in its function and would silently break
if it wasn't sorted properly).

I would say that trying to minimise the reliance on global variables would
be a big improvement to the node builder, because that would allow it to be
multi-threaded.  *makes note of a possible future area of research!*

Gareth aka. Kit

On Mon 30/07/18 13:40 , "Thorsten Engler" ***@gmx.net sent:...


Freeing memory is irrelevant if your process is going to exit anyway right
after freeing it. The OS doesn’t care about any memory your process has
allocated at that time. All pages go back into the available pool.

 ...
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org [1]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
[2]">http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel


Links:
------
[1] mailto:fpc-***@lists.freepascal.org
[2] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Michael Van Canneyt
2018-07-30 13:57:02 UTC
Permalink
Raw Message
On Mon, 30 Jul 2018, J. Gareth Moreton wrote:

> I would say that that's a little naïve and dangerous to think like that.
> Sure, Windows might have the means to clean up memory after an application
> terminates, but not all platforms have such heap deallocation features
> (e.g. pure DOS, where certain procedures and interrupts remain in memory
> even after the application terminates... so-called memory-resident
> programs).

The compiler is not such program. You obviously need to choose for which
programs you do this.

But having the memory freed properly would also enable the use of the compiler
as a DLL in e.g. Lazarus.

Michael.
Karoly Balogh (Charlie/SGR)
2018-07-30 14:05:43 UTC
Permalink
Raw Message
Hi,

On Mon, 30 Jul 2018, J. Gareth Moreton wrote:

> I would say that that's a little naïve and dangerous to think like that.
> Sure, Windows might have the means to clean up memory after an
> application terminates, but not all platforms have such heap
> deallocation features (e.g. pure DOS, where certain procedures and
> interrupts remain in memory even after the application terminates...
> so-called memory-resident programs).

While I generally agree, in practice it's usually a non-issue.

Pure DOS, as in the 16bit target is not supported to run the compiler,
only via GO32V2, and having DPMI TSRs is... well... :)

On Amiga and similar systems, AROS, MorphOS for example, which indeed does
not free up the memory after a program terminates, I implemented a memory
pool in the FPC RTL, so all memory is freed anyway during the program's
exit. Any application which wants to go beyond this behavior for whatever
of its own reasons is free to use the OS allocation functions directly, or
implement its own memory manager. The same with open files on this
platform, actually, which wouldn't be closed without the RTL taking care
of them. And I also have to manage our own stack, and lets not even
mention threading... :)

The sideeffect is, that this approach silently "fix" the compiler's own
leaks too, so FPC remains useable on these systems. In general it is much
safer anyway, because expecting all the broken code out there to get fixed
for such systems is sadly just naive.

Having this said, anyone who still fixes leaks win the compiler will be my
hero! :)

Charlie
J. Gareth Moreton
2018-07-30 14:38:48 UTC
Permalink
Raw Message
Not necessarily.  The Internal Errors are effectively assertions that
should never be triggered. True, cleaning up the memory is a bonus, but
here it's safe to assume that the internal state of the compiler is bad in
some way.  Throwing an exception upon too many errors might be problematic
because there's no telling where it might trigger, but otherwise, the
compiler abort tends to occur after it's finished attempting to compile its
current node trees, so most memory should be freed.  Still, it's something
one might want to research to see if the exception handler can check and
free some global structures.

Gareth aka. Kit

On Mon 30/07/18 16:24 , "Michael Van Canneyt" ***@freepascal.org sent:

On Mon, 30 Jul 2018, Sven Barth via fpc-devel wrote:

> Michael Van Canneyt schrieb am Mo., 30. Juli 2018,
> 15:30:
>
>> Obviously provided you don't install another mechanism that does this
and
>> don't raise exceptions manually, which - AFAIK - is the case in the
>> compiler...
>>
>
> The compiler does use exceptions when the compilation needs to be
aborted
> e.g. when too many errors occurred or an internal error is encountered.

This is bad news, because it in effect means that the try/finally is
necessary
if people want to fix memleaks :(

This will considerably slow down the compiler.

Michael.
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
[1]">http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel



Links:
------
[1] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
J. Gareth Moreton
2018-07-30 17:01:01 UTC
Permalink
Raw Message
Compilers are one of those programs that fall under a very special, very
exclusive set of projects where a lot of the standard rules may not apply,
but if they can be followed where possible, brilliant.
Sometimes, increasing complexity and maintainability in the name of speed
is not a solution, as I found out with my binary search implementation. 
However, a way around that is to see if we can program such a feature into
the compiler itself if it detects it can gain a speed boost from it (in
this situation, a case block with 36 values to choose from, but where each
branch calls an identically-structured procedure... some of the values call
the same routine, but otherwise the routines have the same parameters and
return types).

If you can find a way to fix a memory leak without sacrificing performance
too significantly, by all means submit a patch for review.
If I had my way, if I wanted to make the compiler as fast as possible, I'd
just drop into assembly language for the most critical parts, but that's a
big no-no in this particular project, not because of how hard it is to
maintain, debug and explain, but because of cross compilation issues.  You
should be able to create a compiler that targets any platform of your
choice.  I think the only time you can use assembly language is with some
of the internal compiler routines like the Frac function.

Gareth aka. Kit

On Mon 30/07/18 17:42 , "Marcos Douglas B. Santos" ***@delfire.net sent:
On Mon, Jul 30, 2018 at 12:29 PM, R0b0t1 wrote:
> On Mon, Jul 30, 2018 at 7:32 AM, Martok wrote:
>> Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos:
>>> Is performance more important than being correct? :|
>> In this project, the answer is always taken to be yes.
>>
>
> To hopefully offer some explanation for Mr. Santos - if I can compile
> and run the program 5 times over the course of 20 minutes, that is
> certainly better than running it and testing it once over the same
> amount of time, no?
>
> It might be hard to imagine FPC taking that much longer than it does
> currently but ~30min for a large program is the standard with other
> compilers. I very much enjoy the speed of FPC.

All us like fast compiling...
However, IMHO, a compiler should use the same good practices that
others programs must use.
If it cannot use them because it's very slow, what is the matter about
good practices?

"Do what I'm telling you to do, but don't do what I do" ? :)

best regards,
Marcos Douglas
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org [3]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
[4]">http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel



Links:
Anthony Walter
2018-07-30 18:24:04 UTC
Permalink
Raw Message
I thought I'd add a point about compiling and memory. On the Raspberry Pi,
I know for a fact that building the compiler can fail after a while due to
insufficient memory. When this happens the problem with no memory doesn't
present itself as such. Rather you're left with an error indicating a
problem with the source. I've gotten around this problem by making the
compiler without a desktop environment running and re configuring the swap
size.

Might this problem be related in some way to the discussion in this thread?
J. Gareth Moreton
2018-07-30 17:55:53 UTC
Permalink
Raw Message
Correct me if I'm mistaken, but I believe one shortcoming with the Free
Pascal Compiler, despite its speed, is that it isn't multi-threaded.  It
can't compile multiple independent units at once, and probably won't be
able to for a while due to the use of global variables in the node
builder.  When it comes to building Lazarus and the compiler, this is
gotten around by the Makefiles running several instances of the compiler on
different processes, building independent packages that can later be
linked.
I personally think that if the compiler can be made multi-threaded, then
projects like OpenSceneGraph might see a huge improvement in compile times
if the compiler is smart enough to identify units that don't depend on each
other and compile them in parallel.  I'm not sure how cross-compatible it
can be made, since different platforms approach multithreading in different
ways and I've noticed that the compiler tends to avoid using SysUtils and
Classes and the like (so no using TThread for now).

This is getting a bit off-subject from fixing memory leaks though.
Gareth aka. Kit

On Mon 30/07/18 19:39 , ***@windstream.net sent:
On 07/30/2018 11:35 AM, Michael Van Canneyt wrote:
> On Mon, 30 Jul 2018, R0b0t1 wrote:
>> It might be hard to imagine FPC taking that much longer than it does
>> currently but ~30min for a large program is the standard with other
>> compilers. I very much enjoy the speed of FPC. >
> 30 *Minutes*, is this real ?

yep, for sufficiently large projects... when we compile OSG
(OpenSceneGraph)
over here (using -j 8) it takes 20+ minutes for the first time or if we
reconfigure from debug to release mode or the other way around...

now, after the first time we compile it, the next times takes mere 10's of

seconds... then we move on to SceneGear and FlightGear when we're building
that
whole project which uses OSG... 30-45 minutes for a complete, from the
ground up
builds are not uncommon... it takes me back to the days of 30 years ago
when
you'd start the (borland/turbo pascal) compile process and go get some
c0ffee ;)

note: the machines these builds are being done on have 16Gig RAM, 1Tb HD,
and a
4Ghz 8-core AMD FX-8350 CPU...

--
NOTE: No off-list assistance is given without prior approval.
*Please keep mailing list traffic on the list unless*
*a signed and pre-paid contract is in effect with us.*
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org [1]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
[2]">http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel



Links:
------
[1] mailto:fpc-***@lists.freepascal.org
[2] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Sven Barth via fpc-devel
2018-07-30 21:08:24 UTC
Permalink
Raw Message
Am 30.07.2018 um 19:55 schrieb J. Gareth Moreton:
> Correct me if I'm mistaken, but I believe one shortcoming with the
> Free Pascal Compiler, despite its speed, is that it isn't
> multi-threaded.  It can't compile multiple independent units at once,
> and probably won't be able to for a while due to the use of global
> variables in the node builder.  When it comes to building Lazarus and
> the compiler, this is gotten around by the Makefiles running several
> instances of the compiler on different processes, building independent
> packages that can later be linked.
>
> I personally think that if the compiler can be made multi-threaded,
> then projects like OpenSceneGraph might see a huge improvement in
> compile times if the compiler is smart enough to identify units that
> don't depend on each other and compile them in parallel.  I'm not sure
> how cross-compatible it can be made, since different platforms
> approach multithreading in different ways and I've noticed that the
> compiler tends to avoid using SysUtils and Classes and the like (so no
> using TThread for now).
>
TThread is not a necessity. It's functionality and the synchronization
primitives are based on what TThreadManager provides. This manager
resides in the System unit and is accessible through functions like
BeginThread and such. Please note however that this might need an
external thread manager on certain platforms (e.g. Unix-like systems)
and the idea on those platforms is for the compiler *not* to rely on the
C-library.

Regards,
Sven
J. Gareth Moreton
2018-07-30 19:40:37 UTC
Permalink
Raw Message
As mentioned before, the main problem with try...finally blocks is that
they incur a massive speed penalty, so they're not things you want to enter
and exit on a frequent basis. Whether that means we need to work on the
compiler so it reduces this penalty on such blocks (such as internally
removing the construct and copying the code in the 'finally' section into a
local exception handler) I'm not sure.

I do agree though that we should manage memory with care, especially as
Pascal gives us such fine control over it (compared to, say, C++'s RAII
feature which is a little nebulous at times with its garbage collection). 
I think first though we need to fix any obvious memory leaks, namely
objects and blocks of memory that we forget to free, then build up from
there.  At least that will help reduce the risk of an out of memory
situation during normal operation.

Gareth aka. Kit

On Mon 30/07/18 21:27 , "Marcos Douglas B. Santos" ***@delfire.net sent:
On Mon, Jul 30, 2018 at 3:31 PM, R0b0t1 wrote:
> On Mon, Jul 30, 2018 at 11:42 AM, Marcos Douglas B. Santos
> wrote:
>> On Mon, Jul 30, 2018 at 12:29 PM, R0b0t1 wrote:
>>
>> [...]
>
> If a program isn't long running I see programmers tend to not care
> about memory management, nor do I make a habit of caring about it
> myself. I find it hard to see anything wrong with this.
>
> As someone has mentioned this can fail on smaller computers, but how
> much RAM is too little? SBCs now typically come with 1-4GB of memory.

I'm sorry, but we work with a language that expected programmers
should manage their objects.
The language even provides a clean way (try-finall) to do that.
I use try-finally even to write a snippet to show an example of code...

IMO, that must be a habit in such languages.

regards,
Marcos Douglas
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org [4]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
[5]">http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel



Links:
------
[1] mailto:***@gmail.com
[2] mailto:***@delfire.net
[3] mailto:***@gmail.com
[4] mailto:fpc-***@lists.freepascal.org
[5] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
R0b0t1
2018-07-30 21:35:58 UTC
Permalink
Raw Message
On Mon, Jul 30, 2018 at 2:40 PM, J. Gareth Moreton
<***@moreton-family.com> wrote:
> As mentioned before, the main problem with try...finally blocks is that they
> incur a massive speed penalty, so they're not things you want to enter and
> exit on a frequent basis. Whether that means we need to work on the compiler
> so it reduces this penalty on such blocks (such as internally removing the
> construct and copying the code in the 'finally' section into a local
> exception handler) I'm not sure.
>
> I do agree though that we should manage memory with care, especially as
> Pascal gives us such fine control over it (compared to, say, C++'s RAII
> feature which is a little nebulous at times with its garbage collection). I
> think first though we need to fix any obvious memory leaks, namely objects
> and blocks of memory that we forget to free, then build up from there. At
> least that will help reduce the risk of an out of memory situation during
> normal operation.
>
> Gareth aka. Kit
>

Adding locally allocated structures to a global list is a pretty
gnarly issue that will take some time to solve. The easier ones are
simply using labels to easily manage exit conditions. It looks a lot
nicer than nested if statements.

The issue with such code is it is fairly boring to write, so I
understand if someone didn't do it initially.
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/c
Loading...