Discussion:
Why/how does the compiler have a non-trivialnumberofmemory leaks after over two decades of development?
(too old to reply)
J. Gareth Moreton
2018-07-30 13:05:11 UTC
Permalink
Raw Message
Yeah, I realised after I said that that keeping programs resident is very
different from a memory leak!  Still, it's still very good practice to do
your own clean-up, especially as you described below.  Embedding a
compiler DLL into Lazarus would allow for much more interaction.

Gareth aka. Kit

On Mon 30/07/18 14:57 , Michael Van Canneyt ***@freepascal.org sent:

On Mon, 30 Jul 2018, J. Gareth Moreton wrote:

> I would say that that's a little naïve and dangerous to think like
that.
> Sure, Windows might have the means to clean up memory after an
application
> terminates, but not all platforms have such heap deallocation features
> (e.g. pure DOS, where certain procedures and interrupts remain in memory

> even after the application terminates... so-called memory-resident
> programs).

The compiler is not such program. You obviously need to choose for which
programs you do this.

But having the memory freed properly would also enable the use of the
compiler
as a DLL in e.g. Lazarus.

Michael.

_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org [1]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
[2]">http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel



Links:
------
[1] mailto:fpc-***@lists.freepascal.org
[2] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
J. Gareth Moreton
2018-07-30 13:41:35 UTC
Permalink
Raw Message
Being your hero is a good enough reason for me!  One little thing that's
driven me, even though it's not really an issue, is the size of the EXE
files that are produced, so I seek to find any little way to shrink them
down without compromising performance in the final binary.

What I like about this project is that we all have our little motivators
and objectives, and we're free to explore and research.  I had an idea to
potentially improve the compiler speed on modern systems, but requires a
lot of refactoring and I'm currently got my attention focused on
researching pure function support.
One drawback with any kind of memory management code is that it can put a
lot of space and speed overhead on your project, although it is very useful
from a debugging point of view.  I've programmed my own ones in the past
for C-based test projects that are meant to run for 24 hours sometimes,
mostly in response to frequent crashes.  It's proven very useful in
catching memory leaks and identifying where they are so the memory can be
freed properly.
So, long story short... if you find memory that isn't freed, free it (once
you check it's not used afterwards), even if the operating system or RTL
cleans it up for you.

Gareth aka. Kit

On Mon 30/07/18 15:05 , "Karoly Balogh (Charlie/SGR)"
***@scenergy.dfmk.hu sent:
Hi,

On Mon, 30 Jul 2018, J. Gareth Moreton wrote:

> I would say that that's a little naïve and dangerous to think like
that.
> Sure, Windows might have the means to clean up memory after an
> application terminates, but not all platforms have such heap
> deallocation features (e.g. pure DOS, where certain procedures and
> interrupts remain in memory even after the application terminates...
> so-called memory-resident programs).

While I generally agree, in practice it's usually a non-issue.

Pure DOS, as in the 16bit target is not supported to run the compiler,
only via GO32V2, and having DPMI TSRs is... well... :)

On Amiga and similar systems, AROS, MorphOS for example, which indeed does

not free up the memory after a program terminates, I implemented a memory
pool in the FPC RTL, so all memory is freed anyway during the program's
exit. Any application which wants to go beyond this behavior for whatever
of its own reasons is free to use the OS allocation functions directly, or

implement its own memory manager. The same with open files on this
platform, actually, which wouldn't be closed without the RTL taking care
of them. And I also have to manage our own stack, and lets not even
mention threading... :)

The sideeffect is, that this approach silently "fix" the compiler's own
leaks too, so FPC remains useable on these systems. In general it is much
safer anyway, because expecting all the broken code out there to get fixed

for such systems is sadly just naive.

Having this said, anyone who still fixes leaks win the compiler will be my

hero! :)

Charlie
J. Gareth Moreton
2018-07-30 17:49:26 UTC
Permalink
Raw Message
It might be a problem because it's a false error and not exactly clean. 
Granted, testing such things is not that straightforward.  Platforms such
as Windows do have a feature where you can tell it to randomly fail memory
allocations after about 8 minutes of start time (so the machine can fully
boot up) - the trouble is that graphical elements on the desktop stop being
rendered properly, among other failures, and you can't predict where an
allocation will fail in the program.  I think you can put in exceptions so
allocations don't fail for Lazarus or the Command Prompt, for example.
The trouble is, these allocation failures are designed for C programs,
where "malloc" returns NULL if memory isn't allocated.  Free Pascal's
GetMem raises an exception instead.  Granted, it shouldn't be too much of
a struggle to implement a custom memory manager for testing and to cause
random (or deterministic) allocation failures to see how cleanly the
situation is handled.  It could be a fun test suite!
It is times like these that give weight to being careful with memory
allocations. I started programming in the mid-90s where memory was still
quite tight (namely, a computer running Windows 95 with only 8 MB of
RAM!)  While computers are very fast and spacious today, the same
standards still apply in places, notably in tight loops and functions that
get executed very frequently.
Gareth aka. Kit

On Mon 30/07/18 19:24 , "Anthony Walter" ***@gmail.com sent:
I thought I'd add a point about compiling and memory. On the Raspberry Pi,
I know for a fact that building the compiler can fail after a
while due to insufficient memory. When this happens the problem with no
memory doesn't present itself as such. Rather you're left with an error
indicating a problem with the source. I've gotten around this problem by
making the compiler without a desktop environment running and re
configuring the swap size.
Might this problem be related in some way to the discussion in this
thread? _______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
[1]">http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel



Links:
------
[1] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Sven Barth via fpc-devel
2018-07-30 21:10:28 UTC
Permalink
Raw Message
Am 30.07.2018 um 19:49 schrieb J. Gareth Moreton:
> The trouble is, these allocation failures are designed for C programs,
> where "malloc" returns NULL if memory isn't allocated.  Free Pascal's
> GetMem raises an exception instead. Granted, it shouldn't be too much
> of a struggle to implement a custom memory manager for testing and to
> cause random (or deterministic) allocation failures to see how cleanly
> the situation is handled.  It could be a fun test suite!
The default heap of FPC's RTL respects the value of
ReturnNilIfGrowHeapFails (which by default is False).

Regards,
Sven
J. Gareth Moreton
2018-07-30 20:15:03 UTC
Permalink
Raw Message
Well, if a certain platform relies on an external library, multi-threading
could be disabled on that platform through a flag of some kind, at least
until something can be programmed for that platform (or more
conservatively, only enabled if it's known to work without any hitches). 
Just an idea.  I can't remember off-hand a good enumeration set to include
such a flag in though.  If multi-threading is not supported, then it can
just run single-threaded as it does now.

Gareth aka. Kit

On Mon 30/07/18 22:08 , Sven Barth via fpc-devel
fpc-***@lists.freepascal.org sent:
Am 30.07.2018 um 19:55 schrieb J. Gareth Moreton:
Correct me if I'm mistaken, but I believe one shortcoming with the
Free Pascal Compiler, despite its speed, is that it isn't multi-threaded. 
It can't compile multiple independent units at once, and probably won't be
able to for a while due to the use of global variables in the node
builder.  When it comes to building Lazarus and the compiler, this is
gotten around by the Makefiles running several instances of the compiler on
different processes, building independent packages that can later be
linked.
I personally think that if the compiler can be made multi-threaded, then
projects like OpenSceneGraph might see a huge improvement in compile times
if the compiler is smart enough to identify units that don't depend on each
other and compile them in parallel.  I'm not sure how cross-compatible it
can be made, since different platforms approach multithreading in different
ways and I've noticed that the compiler tends to avoid using SysUtils and
Classes and the like (so no using TThread for now).

TThread is not a necessity. It's functionality and the synchronization
primitives are based on what TThreadManager provides. This manager resides
in the System unit and is accessible through functions like BeginThread and
such. Please note however that this might need an external thread manager
on certain platforms (e.g. Unix-like systems) and the idea on those
platforms is for the compiler *not* to rely on the C-library.

Regards,
Sven
_______________________________________________
fpc-devel maillist - fpc-***@lists.freepascal.org [1]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
[2]">http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel



Links:
------
[1] mailto:fpc-***@lists.freepascal.org
[2] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Loading...