[fpc-devel] Linux thread priority mess (and possible implementation)

[Graeme Geldenhuys]

Hi,

Looking at the rtl/unix/cthread.pp unit, I noticed two $Warning lines sayin
that setting or getting thread priority is not implemented for any unix
system. WHAT???

I then thought I would go ahead and do the implementation. Seeing that
cthreads using libc and POSIX implementation, there was lots of docs about
it on the net, but the deeper I went the more confused I got.

In summary, this info is what I gathered so far:

* Linux thread priority is VERY confusing (compared to Windows).
* You have Static and Dynamic thread priority.
* You also have various Thread Scheduling priorities (different from
thread priorities).
* Thread priority values range from -15..15.
15 being real-time
0 being normal
-15 being idle
When this range is applied, I have no idea!
* Based on the scheduler used you also get other priority ranges.
SCHED_OTHER has a range of 0 (no range)
SCHED_FIFO (first in - first out) has a range of 1..99
SCHED_RR (round robin) has a range of 1..99
* Dynamic priority does not apply to processes with a non-zero static
priority. WTF, I'm lost again!
* Dynamic priorities have a range of -20..20
Just to f*ck with you even more, this range is now switched around.
-20 is real-time
0 is normal
20 is idle
* You can only set Dynamic priority with libc.setpriority() or libc.nice()
* A normal user can only use the Thread Scheduler SCHED_OTHER, all others
require super user rights. So no real-time apps for standard users.
* NICE levels range from 4..-5


All this information comes from these URLs:

* Linux Thread Priority
http://www.midnightbeach.com/Kylix.Sidebar.3.html

* Linux POSIX 1003.1c Thread Attribute Support
http://www2.roguewave.com/support/docs/leif/sourcepro/html/threadspl/4-3.html

* Java thread priority > Linux priority
http://www.javamex.com/tutorials/threads/priority_what.shtml

* Threads: Basic theory and libraries
http://www.cs.cf.ac.uk/Dave/C/node29.html#SECTION0029414000000000000000

NOTE: the last one shows an example implementation for thread priority.
I tried to translate that into Object Pascal or the cthreads
implementation shown below


I have created a Thread Manager demo that can use various Synchronization
Methods and Thread Priorities. But in all cases, the thread priority
doesn't seem to make any difference under Linux. I can see from the debug
lines (writeln statements below) that the thread priorities are being set
(seems to use the -20..20 range but wrong way round), but they all execute
at the same speed on my Linux quad core system.

Loading Image...


Can anybody make heads or tails about thread priority under Linux (or any
unix system for that matter). Is it possible to have various thread
priorities or not under Linux? I use multi-threading a lot in my code, and
assumed thread priority was implemented under FPC Linux, but looking at the
original cthreads.pp unit, it clearly is not (for any unix platform).

Is the following code correct?


---------------[ rtl/unix/cthreads.pp ]-------------------------------
function CThreadSetPriority (threadHandle : TThreadID; Prio: longint):
boolean;
var
param: sched_param;
ret: integer;
priority: integer;
begin
{.$Warning ThreadSetPriority needs to be implemented}
writeln('DEBUG: CThreadSetPriority priority=', Prio);
param.__sched_priority := Prio;
priority := 0; { SCHED_OTHER is always used }
ret := pthread_setschedparam(pthread_t(threadHandle), priority, @param);
Result := (ret = 0);
writeln('DEBUG: CThreadSetPriority - DONE');
end;


function CThreadGetPriority (threadHandle : TThreadID): Integer;
var
param: sched_param;
ret: integer;
priority: longint;
begin
{.$Warning ThreadGetPriority needs to be implemented}
writeln('DEBUG: CThreadGetPriority ');
priority := 0; { SCHED_OTHER is always used }
ret := pthread_getschedparam (pthread_t(threadHandle), @priority,
@param);
Result := param.__sched_priority;
writeln('DEBUG: CThreadGetPriority (priority=', Result, ') - DONE');
end;

-----------------------------[ end ]-----------------------


Regards,
- Graeme -

[Graeme Geldenhuys]

Post by Graeme Geldenhuys
* Dynamic priorities have a range of -20..20
Just to f*ck with you even more, this range is now switched around.
-20 is real-time
0 is normal
20 is idle

And here is why I say it is wrong... rtl/unix/tthread.inc

TThreadPriority = (tpIdle, tpLowest, tpLower, tpNormal, tpHigher, tpHighest,
tpTimeCritical);


const
// stupid, considering its not even implemented...
Priorities: array [TThreadPriority] of Integer =
(-20,-19,-10,0,9,18,19);


Here -20 is tpIdle, but in the quoted text from my previous email, dynamic
priority of -20 is real-time which should then be tpTimeCritical. The
comment in the code is also not very reassuring!



Regards,
- Graeme -

[Michael Schnell]

The term "priority" is used a bit misleadingly. Same for "real-time"


AFAIR:

- There are 100 basic priorities (for realtime-tasks) theses can only
be assigned to processes/thread if you have the appropriate rights.
- Theses priorities are strict. If a process/thread with a certain
priority wants to run, a thread with a lower priority never runs (unless
there is another CPU).
- If there are multiple processes/threads with the same priority, they
are scheduled according to the Scheduling definition (e.g. SHED-_FIFO)
- Additionally there is another "non-realtime" priority lower than all
others. Same is used for standard user processes/threads
- Same is scheduled using time slices, that are dynamically controlled
(e.g. by a "fair scheduler algorithm". there are several such algorithms
that can be configured)
- The time-slice allocation can be influenced by (sub-) "priories"
("nice"-definitions) -20 ... + 20 or something like that

-Michael

[Graeme Geldenhuys]

After Jonas told me in private (in his own kind words) that indeed
Thread Priority is not implemented for any unix-type system yet, and
that a bug report would be wise (no bug report was ever filed for
this).

So here it is:
http://bugs.freepascal.org/view.php?id=16785


Regards,
- Graeme -

[Paul van Helden]

Hi All,

"Threads are evil"?
http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf

Interesting view point. Perhaps that is why FPC doesn't bother too much
about thread priorities? :-)

Regards,

Paul.

[Michael Schnell]

I'm going to read it thoroughly.

But while I do see that threads (i.e. using static data in memory
concurrently with multiple execution units, which mostly is done with
threads and threads is the most common way to do that) impose a
sometimes underestimated level of complexity to the project, especially
the more and more common use of multiple CPUs (SMP) asks for running
procedures concurrently to speed up the task.

-Michael

[Michael Schnell]

Post by Paul van Helden
"Threads are evil"?
http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf

Looking forward to the "Coordination Pascal" extension of FPC. :)

-Michael

[Hans-Peter Diettrich]

Post by Paul van Helden
"Threads are evil"?
http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf
Interesting view point. Perhaps that is why FPC doesn't bother too much
about thread priorities? :-)

IMO the author neglects the existence of realtime systems, which have to
deal with concurrent processes and shared resources, as well as the
existence of multi-user multi-tasking systems. Such systems exist since
many decades, and the synchronization techniques have been researched
sufficiently - I learned all that in the early 70s.

Of course such "cooperative" systems do not work without proper
synchronziation, but IMO not threads are bad per se, instead the
assumptions of the author simply are impractical. With such assumptions
the use of shared files, filesystems or other resources are bad as well
- but what should that mean to software writers? Should we return to
"safe" single-tasking batch systems, or should we continue to share
resources with the *appropriate* synchronization mechanisms, at the
responsibility of the coder?

We all know that writing software is bad, from a
mathematical/philosophical VP, because nobody could proof the
correctness of a program till now, and that because almost nobody can
(or is willing to) provide the required information about the goal of a
non-trivial program. So I think that we can continue to do our very best
in writing useful code, even if it can not be verified by purists :-]

DoDi

[Graeme Geldenhuys]

Post by Hans-Peter Diettrich
synchronziation, but IMO not threads are bad per se, instead the
assumptions of the author simply are impractical. With such assumptions
the use of shared files, filesystems or other resources are bad as well

Plus the author is quick to point out the example of the Observer being bad
in a thread-safe implementation (as per the code he shows), yet doesn't
provide an "safe" example as per his ideas. He doesn't seem able to
practise what he preaches. :-/

An interesting read none the less.


Regards,
- Graeme -

[Michael Schnell]

Post by Hans-Peter Diettrich
or neglects the existence of realtime systems,

As I usually do embedded stuff, I had exactly this impression, too. If
realtime demands need to be considered, concurrent execution(i.e.
threads) not only serves the purpose of speeding things up, but
additionally are necessary to try to meed a timing restriction. So you
need to trade deterministic intermediate states vs deterministic timing.
Fine grained undeterminism, as preferred by the author above threads
does not help here.

Of course the problems he describes remain, in some projects doing
separate processes (i.e. not sharing memory) can help.

-Michael

[Hans-Peter Diettrich]

Post by Graeme Geldenhuys
Looking at the rtl/unix/cthread.pp unit, I noticed two $Warning lines sayin
that setting or getting thread priority is not implemented for any unix
system. WHAT???
I then thought I would go ahead and do the implementation. Seeing that
cthreads using libc and POSIX implementation, there was lots of docs about
it on the net, but the deeper I went the more confused I got.
* Linux thread priority is VERY confusing (compared to Windows).
* You have Static and Dynamic thread priority.
* You also have various Thread Scheduling priorities (different from
thread priorities).

[...]

There exists a broad range of scheduler algorithms and implementations,
that can vary even with customized kernels of the same version, or with
a user specific configuration. Some schedulers maintain a dynamic
priority for every process or thread, that varies over time, so that
long-running high priority jobs do not block the entire system. True
realtime systems impose further constraints on (high priority) threads,
e.g. that they must either finish in a given time period, or must reduce
their own priority after the really time critical parts have been done,
or otherwise the system will assign them a lower priority automatically.

Thus priorities should only be considered as *hints* to the scheduler,
with unpredictable consequences on arbitrary machines. When an
application is tuned for best performance on a specific machine, it may
behave miserably on a similar machine with only a different scheduler,
or on a very different platform.

DoDi

[Graeme Geldenhuys]

Thus priorities should only be considered as *hints* to the scheduler, with
unpredictable consequences on arbitrary machines. When an application is

So is there a kernel default which applies to most popular Linux
distros? If there is, does that default allow some form of thread
priority? For example, in tiOPF under Windows we set the Debug Logger
thread to low priority so as not to interfere much with the running
application. Logging is cached, so eventually the log output window
will catch up when the application is idle.

I would like to get the same behaviour under Linux. I'm using stock
standard Ubuntu 10.04 (most of our clients also use Ubuntu, but
versions may vary), so is there any hope in getting thread priority
working for our systems?

Another example of using threads in our application is checking for
global application statistics - again we require a thread of low
priority, because that feature is just a nice-to-have, and not
critical, so should not impair on the applications performance.

Also does multiple schedulers apply to other non-Linux platforms too?
eg: OSX, FreeBSD, NetBSD, Haiku, etc. Because currently any unix-type
platform doesn't have Thread Priority support in FPC.

[Jonas Maebe]

Post by Graeme Geldenhuys
Also does multiple schedulers apply to other non-Linux platforms too?
eg: OSX, FreeBSD, NetBSD, Haiku, etc. Because currently any unix-type
platform doesn't have Thread Priority support in FPC.

The POSIX pthread_setschedparam() api is not really appropriate in
this case. It's mainly geared at real-time threads, which is why POSIX
only define the behaviour of SCHED_RR and SCHED_FIFO parameters (both
of which are real-time schedulers in this context).

For SCHED_OTHER, the OS can do whatever it wants (including treating
it as another real-time scheduler class, or just as an alias for
SCHED_RR or SCHED_FIFO), and there is no guarantee that the
sched_priority field of the schedparam srtucture is used in that case
(the parameters are completely implementation-defined). Both http://opengroup.org/onlinepubs/007908775/xsh/pthread_setschedparam.html
and the Linux man page for pthread_setschedparam() mention that.

I'm not aware of any portable Unix API to set the priority of threads
scheduled by the default scheduler (unless that happens to be a
SCHED_FIFO or SCHED_RR real-time scheduler, which will only be the
case on real-time systems). I'm not even sure about what the low level
APIs are on particular Unix platforms such as Linux and Mac OS X.


Jonas

[Graeme Geldenhuys]

Post by Jonas Maebe
The POSIX pthread_setschedparam() api is not really appropriate in
this case. It's mainly geared at real-time threads, which is why POSIX

I also tried to use Google to see if there is any other way of setting
thread priority, and all Google returns is the pthread_setschedparam() API.
Even so, maybe TThread needs to be extended to include a Policy property as
well as a Priority property. eg:

TThread = class
...
property Policy: TSchedulePolicy read GetPolicy write SetPolicy;

This should be handy for embedded systems or other real-time apps. This
made we wonder how Kylix used to manage this... I'll try later tonight at
home where I have Kylix available.

Looking at the Kylix 3 Help, Kylix's TThread did indeed have a Policy
property. I guess one could implement it in FPC too, and will be ignored by
Windows, but used in any unix-type OS.

BTW:
Thanks to all that replied, I think I have a much better understanding now
between Thread Priority and Schedulers under Linux. It was quite confusing
at first, but the more articles I read on the web, the better I understood it.

----------[ Kylix 3 Help ]-------------------------------

Determines the schedule policy the thread uses (Linux only).


Delphi syntax:
property Policy: Integer;


Description

Policy is only available for applications that run under Linux. Set Policy
to assign a Schedule policy for this thread with respect to other threads
on the system. The value of Policy must be one of the three constants
listed below:

Policy Type Priority
--------------------------------------------------
SCHED_RR Real Time 1-99
SCHED_FIFO Real Time 1-99
SCHED_OTHER Regular 0

Note: The type of thread policy determines the Priority that can be
assigned to the thread, as indicated in the third column of the table above.

Note: The values SCHED_RR and SCHED_FIFO can only be set by root.
------------------------------------------

Post by Jonas Maebe
I'm not aware of any portable Unix API to set the priority of threads

Even if a API is not portable between *nix, at least we can implement it
per platform, and hopefully create some abstraction layer on top of that,
or translate TThreadPriority type to the individual platform
implementations. FPC has many API's implemented that are not portable
between platforms - so this should not be a stopping block.


Regards,
- Graeme -

[Jonas Maebe]

Post by Graeme Geldenhuys
----------[ Kylix 3 Help ]-------------------------------
Policy Type Priority
--------------------------------------------------
SCHED_RR Real Time 1-99
SCHED_FIFO Real Time 1-99
SCHED_OTHER Regular 0
Note: The type of thread policy determines the Priority that can be
assigned to the thread, as indicated in the third column of the table above.
------------------------------------------

The above means that Kylix also ignored setting the thread priority on
Linux for regular processes/threads (since the only allowed value for
SCHED_OTHER was "0").

Post by Graeme Geldenhuys
Even if a API is not portable between *nix, at least we can
implement it
per platform, and hopefully create some abstraction layer on top of that,
or translate TThreadPriority type to the individual platform
implementations. FPC has many API's implemented that are not portable
between platforms - so this should not be a stopping block.

That's why I added "I'm not even sure about what the low level APIs
are on particular Unix platforms such as Linux and Mac OS X".


Jonas