[fpc-devel] StdOut capture for FPC RTL

[Anton Kavalenka]

Dear All.

Currently to implement logger we use the following approach (the code
was implemented for Delphi/Kylix/FPC).
Logger is intended for displaying in the GUI outputs of several threads
and child processes and also for storage into internal bases.

Logic is the following:

Get stdout handle (duplicate it under linux),
create pipe,
replace the stdout (keeping the old stdout) for current process with
write handle of pipe,

There are following problems with FPC
Under Windows: I have to call *rewrite(output) *for every thread which
wants to use new (captured) stdout

So the questions:
How to force all the threads of process and all DLLs write into same
captured stdout?
How to revert stdout back (stop capturing)?

procedure TLogger.Start;
{$IFDEF LINUX}
{$ifdef fpc}
type TPipeDescriptors=TFilDes;
{$endif}
var
pds:TPipeDescriptors;
{$ENDIF}
begin
if fCaptureStdout then
{$IFDEF MSWINDOWS}
hConsole := GetStdHandle(STD_OUTPUT_HANDLE);
{$ELSE}
{$ifdef fpc}
hConsole := fpdup(StdOutputHandle);;//stdout;
{$else}
hConsole := dup(STDOUT_FILENO);;//stdout;
{$endif}
{$ENDIF}

{$IFDEF MSWINDOWS}
CreatePipe(hReadPipe, hWritePipe, nil, 0);
{$ENDIF}

{$IFDEF LINUX}
{$ifdef fpc}
fppipe(pds);
hReadPipe:=pds[0];
hWritePipe:=pds[1];
{$else}
pipe(pds);
hReadPipe:=pds.ReadDes;
hWritePipe:=pds.WriteDes;
{$endif}
{$ENDIF}

if fCaptureStdout then
begin
{$IFDEF MSWINDOWS}
if not SetStdHandle(STD_OUTPUT_HANDLE, hWritePipe)
then MessageBox(0,PChar(Format('Function: SetStdHandle(%x,%x),
Error: %d',[STD_OUTPUT_HANDLE, hWritePipe, GetLastError])),'Failed',0);
{$IFDEF fpc}
* StdOutputHandle:=hWritePipe; // modify global runtime handle*
{$ENDIF}
if GetStdHandle(STD_OUTPUT_HANDLE)<>hWritePipe then
begin
safeputs('Looks like SetStdHandle failed in Windows 7, will try
AllocConsole workaround'#13#10,hWritePipe);
if not fWinConsole then
begin
AllocConsole;
FreeConsole;
SetStdHandle(STD_OUTPUT_HANDLE, hWritePipe);
end;
if GetStdHandle(STD_OUTPUT_HANDLE)<>hWritePipe then
begin
safeputs('AllocConsole workaround failed, will use
GlobalWritePipe workaround'#13#10,hWritePipe);
GlobalWritePipe:=hWritePipe;
TTextRec(output).Handle:=hWritePipe;
end;
end;
*rewrite(output);*
{$ENDIF}
{$IFDEF LINUX}
{$ifdef fpc}
fpdup2(hWritePipe, StdOutputHandle);
{$else}
dup2(hWritePipe, STDOUT_FILENO);
{$endif}
{$ENDIF}
end;
end;


finalizing capture made via

procedure TLogger.Stop;
var dummy:integer;
{$IFDEF LINUX}
p:pointer;
{$ENDIF}
begin
if fExitFlag then Exit;

fExitFlag := True;

// make empty write to wake up the thread sleeping on read operation
dummy:=0;
FileWrite(hWritePipe,dummy,1);

if hThread<>0 then
begin
{$IFDEF MSWINDOWS}
if WaitForSingleObject(hThread,2000)=WAIT_TIMEOUT
then begin
TerminateThread(hThread,0);
MessageBeep(UINT(-1));
writeln('Thread was terminated abnormally');
end;
Closehandle(hThread);
{$ENDIF}
{$IFDEF LINUX}
{$ifndef FPC} // Kylix
pthread_join(hThread,nil); //Unfortunately, will wait INFINITEly
pthread_detach(hThread);
{$else}
WaitForThreadTerminate (hThread, 2000); // implies pthread_join
KillThread(hThread); // implies pthread_detach and pthread_kill
{$endif}
{$ENDIF}
hThread:=0;
end;

if fCaptureStdout then
begin
{$IFDEF MSWINDOWS}
SetStdHandle(STD_OUTPUT_HANDLE, hConsole);
if GlobalWritePipe = hWritePipe then
GlobalWritePipe:=INVALID_HANDLE_VALUE;
{$ENDIF}
{$IFDEF LINUX}
{$ifdef fpc}
fpdup2(hConsole, StdOutputHandle);
{$else}
dup2(hConsole, STDOUT_FILENO);
{$endif}
{$ENDIF}
rewrite(output);
end;

if hReadPipe <> 0 then
begin
FileClose(hReadPipe);
hReadPipe := 0;
end;

if hWritePipe <> 0 then
begin
FileClose(hWritePipe);
hWritePipe := 0;
end;


if IsLogFileOpened then
begin
FileClose(hLogFile);
hLogFile := 0;
end;
end;

[Michael Van Canneyt]

Post by Anton Kavalenka
Dear All.
Currently to implement logger we use the following approach (the code was implemented for Delphi/Kylix/FPC).
Logger is intended for displaying in the GUI outputs of several threads and child processes and also for storage into internal bases.
Get stdout handle (duplicate it under linux),
create pipe,
replace the stdout (keeping the old stdout) for current process with write handle of pipe,
There are following problems with FPC
Under Windows: I have to call rewrite(output) for every thread which wants to use  new (captured) stdout
How to force all the threads of process and all DLLs write into same captured stdout?

There is no way that I know of to avoid the rewrite(), since the file descriptor
records for Input, Output and StdErr are threadvars.
And the threadvars must be initialized on thread start.

Post by Anton Kavalenka
How to revert stdout back (stop capturing)?

Close and reopen using the standard filedescriptors ?

Michael.

[Thaddy]

On Wed, 24 Nov 2010, Anton Kavalenka wrote: Get stdout handle
(duplicate it under linux),

Post by Anton Kavalenka
create pipe,
replace the stdout (keeping the old stdout) for current process with
write handle of pipe,
There are following problems with FPC
Under Windows: I have to call rewrite(output) for every thread which
wants to use new (captured) stdout
How to force all the threads of process and all DLLs write into same captured stdout?

There is no way that I know of to avoid the rewrite(), since the file
descriptor records for Input, Output and StdErr are threadvars. And
the threadvars must be initialized on thread start.

Post by Anton Kavalenka
How to revert stdout back (stop capturing)?

Close and reopen using the standard filedescriptors ?
Michael.

You can write a "classic" pascal textfile device driver and assign
directly to the global TEXT variables Input, Output and ErrOutput.
Don't know is this is threadsafe, though.
I have some code by Peter Below on archive for that, that also works on
FPC win. The implementation part of that unit can be adapted for nixen.
the file is called streamio.pas and can be easily found on the codegear
website, but I will be happy to send it to you if you want by email.

[m***@wisa.be]

On Wed, 24 Nov 2010, Anton Kavalenka wrote: Get stdout handle (duplicate it
under linux),

Post by Anton Kavalenka
create pipe,
replace the stdout (keeping the old stdout) for current process with write
handle of pipe,
There are following problems with FPC
Under Windows: I have to call rewrite(output) for every thread which wants
to use new (captured) stdout
How to force all the threads of process and all DLLs write into same captured stdout?

There is no way that I know of to avoid the rewrite(), since the file
descriptor records for Input, Output and StdErr are threadvars. And the
threadvars must be initialized on thread start.

Post by Anton Kavalenka
How to revert stdout back (stop capturing)?

Close and reopen using the standard filedescriptors ?
Michael.

You can write a "classic" pascal textfile device driver and assign directly
to the global TEXT variables Input, Output and ErrOutput.
Don't know is this is threadsafe, though.

No, it will not be thread-safe. You'll have to do it over and over again for
each thread.

I have some code by Peter Below on archive for that, that also works on FPC
win. The implementation part of that unit can be adapted for nixen.
the file is called streamio.pas and can be easily found on the codegear
website, but I will be happy to send it to you if you want by email.

The same unit and implementation exists in FPC since years. You can assign
any classical pascal textfile to an arbitrary TStream descendent.

Michael.

[Thaddy]

Post by m***@wisa.be
No, it will not be thread-safe. You'll have to do it over and over again for
each thread.

Post by Thaddy
I have some code by Peter Below on archive for that, that also works
on FPC win. The implementation part of that unit can be adapted for
nixen.
the file is called streamio.pas and can be easily found on the
codegear website, but I will be happy to send it to you if you want
by email.

The same unit and implementation exists in FPC since years. You can
assign any classical pascal textfile to an arbitrary TStream descendent.
Michael.

It is not the same - it is just called the same - but can render the
same effect.
Which also means that it looks like it only takes some normal
precautions for threads, otherwise the whole console IO system wouldn't
be usable with threads.
I guess Marco is right: Just flush after thread operations.
On its own it is not threadsafe, I agree, but it might be a simple and
solid solution. *without* the rewrites :)

[Marco van de Voort]

Post by m***@wisa.be

Post by Michael Van Canneyt
Michael.

You can write a "classic" pascal textfile device driver and assign directly
to the global TEXT variables Input, Output and ErrOutput.
Don't know is this is threadsafe, though.

No, it will not be thread-safe. You'll have to do it over and over again for
each thread.

Just curious:Why not? I thought it was safe if you flushed before you created threads?

[m***@wisa.be]

Post by Marco van de Voort

Post by m***@wisa.be

Post by Michael Van Canneyt
Michael.

You can write a "classic" pascal textfile device driver and assign directly
to the global TEXT variables Input, Output and ErrOutput.
Don't know is this is threadsafe, though.

No, it will not be thread-safe. You'll have to do it over and over again for
each thread.

Just curious:Why not? I thought it was safe if you flushed before you created threads?

Writing to the files is thread-safe, but in new threads, the file variables
are initialized from their initial main thread values, i.e. the ones without
the recapturing.

Unless I missed something ?

Michael.

[Anton Kavalenka]

Post by m***@wisa.be
On
Writing to the files is thread-safe, but in new threads, the file variables
are initialized from their initial main thread values, i.e. the ones without
the recapturing.
Unless I missed something ?
Michael.
_______________________________________________
http://lists.freepascal.org/mailman/listinfo/fpc-devel

So the question is:
How to reinitialize RTL to have new threads started with working output
file (i.e. without need of rewrite).

regards,
Anton

[Thaddy]

Post by Anton Kavalenka
How to reinitialize RTL to have new threads started with working
output file (i.e. without need of rewrite).
regards,
Anton

In Delphi it works - with a critical section - without re-initialization
because it's var,
In FPC there's no easy work-around because it is threadvar.

It's a nice puzzle, though :)

[Marco van de Voort]

Post by m***@wisa.be

Post by Marco van de Voort

Post by m***@wisa.be
No, it will not be thread-safe. You'll have to do it over and over again for
each thread.

Just curious:Why not? I thought it was safe if you flushed before you created threads?

Writing to the files is thread-safe, but in new threads, the file variables
are initialized from their initial main thread values, i.e. the ones without
the recapturing.

Some stuff is dawning on me, never mind.

[Thaddy]

Post by m***@wisa.be

Post by Marco van de Voort

Post by m***@wisa.be
No, it will not be thread-safe. You'll have to do it over and over again for
each thread.

Just curious:Why not? I thought it was safe if you flushed before you created threads?

Writing to the files is thread-safe, but in new threads, the file variables
are initialized from their initial main thread values, i.e. the ones without

Sorry, Michael,

Ignore my last post.
I am testing it and "strange things happen" :0

[Marco van de Voort]

Post by Thaddy
I guess Marco is right: Just flush after thread operations.
On its own it is not threadsafe, I agree, but it might be a simple and
solid solution. *without* the rewrites :)

IIRC there is a pointer in the filetype that points into the buffer.
(bufptr). Just copying that means they still point into the same pointer.

[Thaddy]

Post by Thaddy
solid solution. *without* the rewrites :)
IIRC there is a pointer in the filetype that points into the buffer.
(bufptr). Just copying that means they still point into the same pointer.
_______________________________________________

Yes, They (input, output err) are vars, not threadvars in system. So
whatever thread, they point to the same.

[m***@wisa.be]

Post by Thaddy
solid solution. *without* the rewrites :)
IIRC there is a pointer in the filetype that points into the buffer.
(bufptr). Just copying that means they still point into the same pointer.
_______________________________________________

Yes, They (input, output err) are vars, not threadvars in system. So whatever
thread, they point to the same.

They are threadvars in trunk:

ThreadVar
ThreadID : TThreadID;
{ Standard In- and Output }
ErrOutput,
Output,
Input,
StdOut,
StdErr : Text;

Michael.

[Thaddy]

Post by m***@wisa.be
ThreadVar
ThreadID : TThreadID;
{ Standard In- and Output }
ErrOutput,
Output,
Input,
StdOut,
StdErr : Text;
Michael.
_______________________________________________

That's not Delphi compatible, btw: ;) : op.cit: system.pas
var
...........
Input: Text; { Standard input }
Output: Text; { Standard output }
ErrOutput: Text; { Standard error output }

It may be a good thing, though.
It also explains my strange results in testing D vs FPC this morning.

[Anton Kavalenka]

Post by Thaddy

Post by m***@wisa.be
ThreadVar
ThreadID : TThreadID;
{ Standard In- and Output }
ErrOutput,
Output,
Input,
StdOut,
StdErr : Text;
Michael.
_______________________________________________

That's not Delphi compatible, btw: ;) : op.cit: system.pas
var
...........
Input: Text; { Standard input }
Output: Text; { Standard output }
ErrOutput: Text; { Standard error output }
It may be a good thing, though.
It also explains my strange results in testing D vs FPC this morning.
_______________________________________________
http://lists.freepascal.org/mailman/listinfo/fpc-devel

That is!

Due to incompatibility of RTL between FPC and Delphi my capturing tricks
not work.
BTW what the real reason to make these files as threadvar?
Per-thread StdOut, StdIn, StdErr (ooooo! the application with multiple
standard outputs)?

regards,
Anton

[Jonas Maebe]

Post by Anton Kavalenka
Due to incompatibility of RTL between FPC and Delphi my capturing
tricks not work.
BTW what the real reason to make these files as threadvar?

To prevent multiple threads writing to stdin/stdout/sterr at the same
time from corrupting each other's data (all of these text files use an
internal buffer, even when the output is immediately flushed
afterwards).


Jonas

[Thaddy]

Post by Jonas Maebe

Post by Anton Kavalenka
Due to incompatibility of RTL between FPC and Delphi my capturing
tricks not work.
BTW what the real reason to make these files as threadvar?

To prevent multiple threads writing to stdin/stdout/sterr at the same
time from corrupting each other's data (all of these text files use an
internal buffer, even when the output is immediately flushed afterwards).
Jonas

That's not a task for a compiler or frame work, but a task for the
programmer.
Std is short for standard and should be standard. Someone screwed up his
semantics and thereby the code when this anomaly was invented for FPC.
Although the rationale about why this happened is obvious, I have to agree.

[Anton Kavalenka]

This post might be inappropriate. Click to display it.

[Jonas Maebe]

Post by Anton Kavalenka

Post by Jonas Maebe

Post by Anton Kavalenka
Due to incompatibility of RTL between FPC and Delphi my capturing
tricks not work.
BTW what the real reason to make these files as threadvar?

To prevent multiple threads writing to stdin/stdout/sterr at the
same time from corrupting each other's data (all of these text
files use an internal buffer, even when the output is immediately
flushed afterwards).

lock_output();
try
// do something nasty with the buffer
finally
flush(output);
unlock_output();
end;

It's obviously better for Delphi compatibility and for what you want
to do, but in general I'd say that's simply a different way. It's also
worse in other ways: performance (not just locking/unlocking, which
doesn't matter that much since stdio is never that fast, but mainly
creating the mutexes when the threads are started), backwards
compatibility with existing FPC code, resources required (mutexes
require more than threadvars).

I can't imagine that any of that matters much to you though, since it
does not help you solve your problem. I can't immediately think of a
way to make this programmatically changeable at run time either.

Post by Anton Kavalenka
There is NO other runtimes on this planet instead FPC RTL which take
care of per-thread buffer of output.

That would very much surprise me.

Post by Anton Kavalenka
It is programmers duty to lock/flush buffer in multithreaded
environments.

I don't think it's a programmer's duty to define a global lock that
they manually acquire/release whenever they use writeln() to print
something to stdout in a multi-threaded program using writeln(). Just
like the heap manager, ioresult, and the widestring manager are also
thread-safe by default. In general, I think that all "global" system
unit functionality is thread safe by default. The only part of the
standard RTL that isn't, is afaik the crt unit.


Jonas

[Thaddy]

Post by Jonas Maebe
It's obviously better for Delphi compatibility and for what you want
to do, but in general I'd say that's simply a different way. It's also
worse in other ways: performance (not just locking/unlocking, which
doesn't matter that much since stdio is never that fast, but mainly
creating the mutexes when the threads are started), backwards
compatibility with existing FPC code, resources required (mutexes
require more than threadvars).
I can't imagine that any of that matters much to you though, since it
does not help you solve your problem. I can't immediately think of a
way to make this programmatically changeable at run time either.

It is about semantics in this case, a "contract" if you want. This
contract hasn't been defined, except for implementation. It seems that
is not good enough in the context of what FPC is trying to achieve.

Post by Jonas Maebe

Post by Anton Kavalenka
There is NO other runtimes on this planet instead FPC RTL which take
care of per-thread buffer of output.

That would very much surprise me.

Example needed... In very bended Dutch: go your gang ;-)

Post by Jonas Maebe

Post by Anton Kavalenka
It is programmers duty to lock/flush buffer in multithreaded
environments.

I don't think it's a programmer's duty to define a global lock that
they manually acquire/release whenever they use writeln() to print
something to stdout in a multi-threaded program using writeln(). Just
like the heap manager, ioresult, and the widestring manager are also
thread-safe by default. In general, I think that all "global" system
unit functionality is thread safe by default. The only part of the
standard RTL that isn't, is afaik the crt unit.
Jonas

In my opinion it is. Furthermore it is a classic example of when and how
the programmer should not rely on what's given to him beforehand.
If "standard" is interpreted als globally valid (Unless...). Are we
wrong to assume this?
But I am also happy sending a bug report through Quality Central to ask
for FPC compliance.
AFAIK it should be a lock, and a lock written by the programmer.
It should definitely not be a threadvar in my opinion.

Hey, It's Friday!

Regards,

Thaddy

[Thaddy]

Yes, How "standard" is standard? ....when you make it a threadvar.
Should I submit a bug report? It seems like I should:
Because the file drivers allow for redirection anyway and with good reason.

[Thaddy]

To quote some fiction: "There can be only one."
In the context of stdxxx.

[Michael Schnell]

Of course if stdin, stdout, and stderr are not thredvars, their use is
not thread safe.

But if they are threadvars, how are they initialized ? Supposedly the
main thread instances are initialized as everybody expects and thus
connected to the appropriate pipes (or whatever this is in Windows). But
what are the instances for threads connected to ?

-Michael

[Thaddy]

On 25-11-2010 11:08, Michael Schnell wrote:
But what are the instances for threads connected to ?


This is - should be - opaque.


Regard,

Thaddy

[Sven Barth]

Post by Michael Schnell
Of course if stdin, stdout, and stderr are not thredvars, their use is
not thread safe.
But if they are threadvars, how are they initialized ? Supposedly the
main thread instances are initialized as everybody expects and thus
connected to the appropriate pipes (or whatever this is in Windows). But
what are the instances for threads connected to ?

They are initialised when the thread is started (before your own routine
is called) to the same values that are used for StdIO on application
startup.

Regards,
Sven

[Anton Kavalenka]

Post by Sven Barth

Post by Michael Schnell
Of course if stdin, stdout, and stderr are not thredvars, their use is
not thread safe.
But if they are threadvars, how are they initialized ? Supposedly the
main thread instances are initialized as everybody expects and thus
connected to the appropriate pipes (or whatever this is in Windows). But
what are the instances for threads connected to ?

They are initialised when the thread is started (before your own
routine is called) to the same values that are used for StdIO on
application startup.
Regards,
Sven
_______________________________________________
http://lists.freepascal.org/mailman/listinfo/fpc-devel

So the new question:
What I have to do to properly initialize these defaults for new threads
AFTER capturing StdOut?

regards,
Anton

[Jonas Maebe]

Post by Anton Kavalenka
What I have to do to properly initialize these defaults for new
threads AFTER capturing StdOut?

Store a copy of your stdout in a global variable, and after creating a
new thread

close(stdout);
stdout:=myglobalstdout;

(and maybe the same for "output").

One possible problem: the output handles are automatically flushed
when a thread exits (so that may cause synchronisation issues).


Jonas

[Jonas Maebe]

What I have to do to properly initialize these defaults for new threads AFTER capturing StdOut?

Store a copy of your stdout in a global variable, and after creating a new thread
close(stdout);
stdout:=myglobalstdout;
(and maybe the same for "output").

Actually, that won't work because the different threads will then work on a common buffer but with distinct pointers into it. A better solution is probably to do this in the intialisation code of each thread instead:

{$ifdef unix}
fpclose(ttextrec(stdout).handle);
{$elsif defined(MSWINDOWS)}
{ this is a copy of do_close() from the rtl, I don't know whether
a new handle from a thread can actually have any of these values }
if (handle <> StdInputHandle) and
(handle <> StdOutputHandle) and
(handle <> StdErrorHandle) then
CloseHandle(ttextrec(stdout).handle);
{$else}
{$error Add support for this platform}
{$endif}

ttextrec(stdout).handle:=myglobalstdouthandle;


Jonas

[Anton Kavalenka]

Post by Jonas Maebe

What I have to do to properly initialize these defaults for new threads AFTER capturing StdOut?

Store a copy of your stdout in a global variable, and after creating a new thread
close(stdout);
stdout:=myglobalstdout;
(and maybe the same for "output").

{$ifdef unix}
fpclose(ttextrec(stdout).handle);
{$elsif defined(MSWINDOWS)}
{ this is a copy of do_close() from the rtl, I don't know whether
a new handle from a thread can actually have any of these values }
if (handle<> StdInputHandle) and
(handle<> StdOutputHandle) and
(handle<> StdErrorHandle) then
CloseHandle(ttextrec(stdout).handle);
{$else}
{$error Add support for this platform}
{$endif}
ttextrec(stdout).handle:=myglobalstdouthandle;
Jonas_______________________________________________
http://lists.freepascal.org/mailman/listinfo/fpc-devel

That's unsuitable. I have lots of modules and lost of threads. Many
modules built with C++ runtime, others with C.
Output goes from many places and many threads.
The task is to do inside process what the UNIX tee command does outside.

regards,
Anton

[Thaddy]

the std streams should be on a per process basis (implementation as
var), not on a per thread basis (Iimplementation as threadvar).
Not only is this not Delphi compliant, it is also not Posix compliant.
Although I can see some neat advantages, it seems to me FPC should
better comply with what is expected by virtually every other language
since FORTRAN and up until and including all .NET.
Plz consider this behaveour to change.

[Jonas Maebe]

Post by Anton Kavalenka

Post by Jonas Maebe
Actually, that won't work because the different threads will then
work on a common buffer but with distinct pointers into it. A
better solution is probably to do this in the intialisation code of
{$ifdef unix}
fpclose(ttextrec(stdout).handle);
{$elsif defined(MSWINDOWS)}
{ this is a copy of do_close() from the rtl, I don't know whether
a new handle from a thread can actually have any of these
values }
if (handle<> StdInputHandle) and
(handle<> StdOutputHandle) and
(handle<> StdErrorHandle) then
CloseHandle(ttextrec(stdout).handle);
{$else}
{$error Add support for this platform}
{$endif}
ttextrec(stdout).handle:=myglobalstdouthandle;

That's unsuitable. I have lots of modules and lost of threads. Many
modules built with C++ runtime, others with C.

The C++ and C threads won't have their input/output handles replaced
by default, since they are not started via the FPC RTL. The only
exception is in case they call back into Pascal code and they are
subsequently hooked by the RTL because the Pascal code (indirectly)
accesses a threadvar.

Coincidentally, just the other day we were discussing introducing the
ability to install hooks that will be called automatically whenever an
external thread is hooked by the RTL (since the RTL has to initialise
a bunch of things at such a point, so may user code). You could do
whatever initialisation you need to do at that point.

I think we could extend such a callback mechanism to all threads,
possibly with a boolean parameter indicating whether the thread was
started via the FPC RTL or not in case the difference is important in
some use cases.


Jonas

[Michael Van Canneyt]

Post by Jonas Maebe
Actually, that won't work because the different threads will then work on
a common buffer but with distinct pointers into it. A better solution is
{$ifdef unix}
fpclose(ttextrec(stdout).handle);
{$elsif defined(MSWINDOWS)}
{ this is a copy of do_close() from the rtl, I don't know whether
a new handle from a thread can actually have any of these values }
if (handle<> StdInputHandle) and
(handle<> StdOutputHandle) and
(handle<> StdErrorHandle) then
CloseHandle(ttextrec(stdout).handle);
{$else}
{$error Add support for this platform}
{$endif}
ttextrec(stdout).handle:=myglobalstdouthandle;

That's unsuitable. I have lots of modules and lost of threads. Many modules
built with C++ runtime, others with C.

The C++ and C threads won't have their input/output handles replaced by
default, since they are not started via the FPC RTL. The only exception is in
case they call back into Pascal code and they are subsequently hooked by the
RTL because the Pascal code (indirectly) accesses a threadvar.
Coincidentally, just the other day we were discussing introducing the ability
to install hooks that will be called automatically whenever an external
thread is hooked by the RTL (since the RTL has to initialise a bunch of
things at such a point, so may user code). You could do whatever
initialisation you need to do at that point.
I think we could extend such a callback mechanism to all threads, possibly
with a boolean parameter indicating whether the thread was started via the
FPC RTL or not in case the difference is important in some use cases.

I think this is not bad; the question is whether we can do it on unix.
On Unix, we can currently only check when a threadvar is accessed.

Michael.

[Jonas Maebe]

Post by Michael Van Canneyt
I think this is not bad; the question is whether we can do it on unix.
On Unix, we can currently only check when a threadvar is accessed.

The semantics of those callbacks would be that they are called as soon
as the RTL is initialised for a thread. On Unix, this is indeed only
done on the first threadvar access.


Jonas

[Michael Van Canneyt]

Post by Michael Van Canneyt
I think this is not bad; the question is whether we can do it on unix.
On Unix, we can currently only check when a threadvar is accessed.

The semantics of those callbacks would be that they are called as soon as the
RTL is initialised for a thread. On Unix, this is indeed only done on the
first threadvar access.

If we decide to add this, maybe we should give the users a chance to test for this themselves ?

Procedure CheckThreadInitialize;

This way they can call this at the entry point of some externally called routines.
For windows, we'd need this 'check' procedure anyway, since there are 2 notification mechanisms.
(TLS callback and DLL_THREAD_ATTACH)
If the RTL is already initialized for this thread, it would do nothing.

Michael.

[Thaddy]

Post by Jonas Maebe
{$ifdef unix}
fpclose(ttextrec(stdout).handle);
{$elsif defined(MSWINDOWS)}
{ this is a copy of do_close() from the rtl, I don't know whether
a new handle from a thread can actually have any of these values }
if (handle<> StdInputHandle) and
(handle<> StdOutputHandle) and
(handle<> StdErrorHandle) then
CloseHandle(ttextrec(stdout).handle);
{$else}
{$error Add support for this platform}
{$endif}
ttextrec(stdout).handle:=myglobalstdouthandle;
Jonas_______________________________________________

This is the world upside down. Both for nixes and windows, you would
expect the streams on a per process basis, not per thread, and handle
possible contention by code.
i.e. it is - should be - expected that they work on a common
buffer/pointer by definition.

[Tomas Hajny]

Post by Jonas Maebe

Post by Anton Kavalenka
What I have to do to properly initialize these defaults for new threads
AFTER capturing StdOut?

Store a copy of your stdout in a global variable, and after creating a new thread
close(stdout);
stdout:=myglobalstdout;
(and maybe the same for "output").

Actually, that won't work because the different threads will then work on
a common buffer but with distinct pointers into it. A better solution is
{$ifdef unix}
fpclose(ttextrec(stdout).handle);
{$elsif defined(MSWINDOWS)}
{ this is a copy of do_close() from the rtl, I don't know whether
a new handle from a thread can actually have any of these values }
if (handle <> StdInputHandle) and
(handle <> StdOutputHandle) and
(handle <> StdErrorHandle) then
CloseHandle(ttextrec(stdout).handle);
{$else}
{$error Add support for this platform}
{$endif}
ttextrec(stdout).handle:=myglobalstdouthandle;

I'm not sure if closing the handle using the platform specific API is
actually a good idea here, because the RTL is probably not opening the
file in this case and trying to close it multiple times (in different
threads) may have unwanted effects (it would result in a failure return
code probably).

BTW, other alternative solutions for the original problem (not necessarily
fitting your needs, just for completeness sake):

- Perform the writes always in one and the same thread dedicated for that
purpose (i.e. have the other threads communicate to that thread using some
sort of inter-thread communication).

- Don't write to stdout but write to your own global (text) variable
instead while ensuring that only one thread is accessing the variable at a
time by your own means.

- Alternative of the above - override Write / WriteLn by your own routine
doing one of the former two.

Tomas

[Jonas Maebe]

Post by Tomas Hajny
I'm not sure if closing the handle using the platform specific API is
actually a good idea here, because the RTL is probably not opening the
file in this case

You're right, so you don't even need that special code. Just change the handle and you're fine.


Jonas

[Michael Schnell]

Post by Sven Barth
They are initialised when the thread is started (before your own
routine is called) to the same values that are used for StdIO on
application startup.

That means the StdIn, StdOut, and StdErr are opened multiple times by
the application ? While this might be possible for StdOut and StdErr
(mixing the outputs), I don't see what this means for StdIn.

-Michael

[Marco van de Voort]

Post by Anton Kavalenka

Post by Jonas Maebe

Post by Anton Kavalenka
Due to incompatibility of RTL between FPC and Delphi my capturing
tricks not work.
BTW what the real reason to make these files as threadvar?

To prevent multiple threads writing to stdin/stdout/sterr at the same
time from corrupting each other's data (all of these text files use an
internal buffer, even when the output is immediately flushed afterwards).

lock_output();
try
// do something nasty with the buffer
finally
flush(output);
unlock_output();
end;
There is NO other runtimes on this planet instead FPC RTL which take
care of per-thread buffer of output.
It is programmers duty to lock/flush buffer in multithreaded environments.

Such solutions are fine for slow apps, but in a fast pace, app, this means
that the logging with its locks has a significant impact on the behaviour of
the application.

And IMHO the RTL should be workable for all kinds of apps.

[Anton Kavalenka]

Post by Marco van de Voort

Post by Anton Kavalenka

Post by Jonas Maebe

Post by Anton Kavalenka
Due to incompatibility of RTL between FPC and Delphi my capturing
tricks not work.
BTW what the real reason to make these files as threadvar?

To prevent multiple threads writing to stdin/stdout/sterr at the same
time from corrupting each other's data (all of these text files use an
internal buffer, even when the output is immediately flushed afterwards).

lock_output();
try
// do something nasty with the buffer
finally
flush(output);
unlock_output();
end;
There is NO other runtimes on this planet instead FPC RTL which take
care of per-thread buffer of output.
It is programmers duty to lock/flush buffer in multithreaded environments.

Such solutions are fine for slow apps, but in a fast pace, app, this means
that the logging with its locks has a significant impact on the behaviour of
the application.
And IMHO the RTL should be workable for all kinds of apps.
_______________________________________________
http://lists.freepascal.org/mailman/listinfo/fpc-devel

OK, let's replace threadvar for those variables with var.

regards,
Anton