Branimir Maksimovic <branimir.maksimovic@gmail.com> writes:
> Problem is that signal handler is equivalent of interrupt service routine.
> It can be interupted by same interrupt/signal and called again.

sigaction lets you establish a mask in the signal handler; by default the
signal which caused the entry is blocked as well. This is much like
interrupt entry on pretty much every CPU I've used; you enter with interrupts
disabled, thus by default no reentry.

Andy Valencia
Home page: https://www.vsta.org/andy/
To contact me: https://www.vsta.org/contact/andy.html

On 2021-07-31, Andy Valencia <vandys@vsta.org> wrote:
> Branimir Maksimovic <branimir.maksimovic@gmail.com> writes:
>> Problem is that signal handler is equivalent of interrupt service routine.
>> It can be interupted by same interrupt/signal and called again.
>
> sigaction lets you establish a mask in the signal handler; by default the
> signal which caused the entry is blocked as well. This is much like
> interrupt entry on pretty much every CPU I've used; you enter with interrupts
> disabled, thus by default no reentry.
>
> Andy Valencia
> Home page: https://www.vsta.org/andy/
> To contact me: https://www.vsta.org/contact/andy.html
Honestly, I didn't know that :P
Thanks!

--
bmaxa now listens Ob-Neob Radio

Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
> On 2021-07-31, Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
> > On 2021-07-31, antispam@math.uni.wroc.pl <antispam@math.uni.wroc.pl> wrote:
> >> Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
> >>> On 2021-07-30, Ivan Godard <ivan@millcomputing.com> wrote:
> >>> > On 7/30/2021 2:47 PM, antispam@math.uni.wroc.pl wrote:
> >>> >> This is more question of implementating high-level languages,
> >>> >> but possible answers depend on computer architecture.
> >>> >>
> >>> >> Problem: We want to code in high-level language and we want
> >>> >> to respond to asynchronous events. Simplest example may
> >>> >> by timeout (response to timer interrupt). Or we may wish
> >>> >> to change what the program is doing in response to external
> >>> >> event. At hardware level there are interrupt, at operating
> >>> >> system level we may get signals, so no problem at low level.
> >>> >> However, high-level language may have data structures with
> >>> >> non-trivial invariants. If we abort modification in the
> >>> >> middle data structures may be left in inconsistent state.
> >>> >> So, what is needed is transfer of control to handler at
> >>> >> next safepoint, that is at next place where program may be
> >>> >> safely interrupted. I assume that high-level language
> >>> >> compiler knows where safepoints are, so the only problem
> >>> >> is how to perform control transfer from safepoint to
> >>> >> handler. This is essentially what (in)famous "came from"
> >>> >> instruction is doing, but I am not aware of any real
> >>> >> hardware implementing "came from"...
> >>> >>
> >>> >> Possible solutions:
> >>> >>
> >>> >> 1) Set a flag in signal handler and check for the flag at
> >>> >> each safepoint. Low-level variant od this would keep
> >>> >> interrupts disabled most of the time and enable them
> >>> >> only at safepoints. This is problematic, because
> >>> >> checking inside tight loop may lead to significant
> >>> >> performance drop. Not checking inside loop may
> >>> >> lead to long delay before reaction to external signal.
> >>> >> Inserting checking code by hand could work, but is
> >>> >> error-prone and labor-intensive, so I want automatic
> >>> >> solution, that is compiler generated checks.
> >>> >> 2) Using debug registers: debug register causes transfer
> >>> >> to handler after arriving at designated address. This
> >>> >> is quite close to what "came from" should do and would
> >>> >> be a satisfactory solution. Unfortunately, debug
> >>> >> registers are not standard part of architecture and
> >>> >> access to debug registers may be restricted by operating
> >>> >> system.
> >>> >> 3) Separate cleanup routines: each instruction would have
> >>> >> associated cleanup routine which will continue computations
> >>> >> up to safepoint, but then transfer control to handler.
> >>> >> This may be quite satisfactory from performance point
> >>> >> of view, but require quite a lot of code.
> >>> >> 4) Modyfing code to replace instruction at safepoint
> >>> >> by jump. This has usual disadvantages of self-modyfying
> >>> >> code, but otherwise looks reasonable.
> >>> >> 5) Dynamicaly generating cleanup code: we could copy code
> >>> >> from current instruction pointer up to setpoint and add
> >>> >> jump to handler at the end. Assuming that code is position
> >>> >> independent this should work and is resonably simple. We
> >>> >> avoid modification of "normal code", but it is not clear
> >>> >> for me if modern CPU-s like it better than 4.
> >>> >> 6) Interpret machine instructions up to safepoint. Shuld work,
> >>> >> but is lot of code and potentialy slow.
> >>> >> 7) Modification of 3 and 6: generate cleanups as bytecode and
> >>> >> interpret it. IIUC this is essentially what DWARF is doing
> >>> >> to handle C++ exceptions. My impression is that this is
> >>> >> complicated to implement, requires a lot of code space and
> >>> >> is slow a runtime. OTOH less complicated than 6, probably
> >>> >> runs faster than 6 and needs less code space than 3.
> >>> >>
> >>> >> I wonder if there is other possible implementation that I
> >>> >> overlooked? And advantages/disadvantages that I overlooked?
> >>> >>
> >>> >
> >>> > Simplest to implement would be to throw a C++ exception in the signal
> >>> > handler and catch at the safepoint (catches are always safepoints in
> >>> > C++). That lets C++ do the dwarfing instead of you.
> >>> In signal handler you can't call printf let alone throw C++ exception :P
> >>
> >> Well, from 'g++' help:
> >>
> >> -fasynchronous-unwind-tables Generate unwind tables that are exact at each
> >> instruction boundary.
> >>
> >> What is intended use of this option?
> >>
> > intended use is that throwing exception from ordinary code will not
> > break if async signal happens. But, if you throw from signal handler
> > it will certainly break.
> >
> >
> Besides, throwing from C callback is no no, as Linux is written in C.

There is good chance that 'raise' inside signal handler would
fail. However, AFAIK C++ runtime is already converting
synchronous signals (like SIGSEGV) into exceptions. With
asynchronous unwind tables the same could be done for all signals.
So the plumbing is all there, if you connect right things
you can have _effect_ of throwing exceptions from signal
handler. I do not go in this direction because for me C++
it has other disadvantages.

--
Waldek Hebisch

On 2021-07-31, antispam@math.uni.wroc.pl <antispam@math.uni.wroc.pl> wrote:
> Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
>> On 2021-07-31, Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
>> > On 2021-07-31, antispam@math.uni.wroc.pl <antispam@math.uni.wroc.pl> wrote:
>> >> Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
>> >>> On 2021-07-30, Ivan Godard <ivan@millcomputing.com> wrote:
>> >>> > On 7/30/2021 2:47 PM, antispam@math.uni.wroc.pl wrote:
>> >>> >> This is more question of implementating high-level languages,
>> >>> >> but possible answers depend on computer architecture.
>> >>> >>
>> >>> >> Problem: We want to code in high-level language and we want
>> >>> >> to respond to asynchronous events. Simplest example may
>> >>> >> by timeout (response to timer interrupt). Or we may wish
>> >>> >> to change what the program is doing in response to external
>> >>> >> event. At hardware level there are interrupt, at operating
>> >>> >> system level we may get signals, so no problem at low level.
>> >>> >> However, high-level language may have data structures with
>> >>> >> non-trivial invariants. If we abort modification in the
>> >>> >> middle data structures may be left in inconsistent state.
>> >>> >> So, what is needed is transfer of control to handler at
>> >>> >> next safepoint, that is at next place where program may be
>> >>> >> safely interrupted. I assume that high-level language
>> >>> >> compiler knows where safepoints are, so the only problem
>> >>> >> is how to perform control transfer from safepoint to
>> >>> >> handler. This is essentially what (in)famous "came from"
>> >>> >> instruction is doing, but I am not aware of any real
>> >>> >> hardware implementing "came from"...
>> >>> >>
>> >>> >> Possible solutions:
>> >>> >>
>> >>> >> 1) Set a flag in signal handler and check for the flag at
>> >>> >> each safepoint. Low-level variant od this would keep
>> >>> >> interrupts disabled most of the time and enable them
>> >>> >> only at safepoints. This is problematic, because
>> >>> >> checking inside tight loop may lead to significant
>> >>> >> performance drop. Not checking inside loop may
>> >>> >> lead to long delay before reaction to external signal.
>> >>> >> Inserting checking code by hand could work, but is
>> >>> >> error-prone and labor-intensive, so I want automatic
>> >>> >> solution, that is compiler generated checks.
>> >>> >> 2) Using debug registers: debug register causes transfer
>> >>> >> to handler after arriving at designated address. This
>> >>> >> is quite close to what "came from" should do and would
>> >>> >> be a satisfactory solution. Unfortunately, debug
>> >>> >> registers are not standard part of architecture and
>> >>> >> access to debug registers may be restricted by operating
>> >>> >> system.
>> >>> >> 3) Separate cleanup routines: each instruction would have
>> >>> >> associated cleanup routine which will continue computations
>> >>> >> up to safepoint, but then transfer control to handler.
>> >>> >> This may be quite satisfactory from performance point
>> >>> >> of view, but require quite a lot of code.
>> >>> >> 4) Modyfing code to replace instruction at safepoint
>> >>> >> by jump. This has usual disadvantages of self-modyfying
>> >>> >> code, but otherwise looks reasonable.
>> >>> >> 5) Dynamicaly generating cleanup code: we could copy code
>> >>> >> from current instruction pointer up to setpoint and add
>> >>> >> jump to handler at the end. Assuming that code is position
>> >>> >> independent this should work and is resonably simple. We
>> >>> >> avoid modification of "normal code", but it is not clear
>> >>> >> for me if modern CPU-s like it better than 4.
>> >>> >> 6) Interpret machine instructions up to safepoint. Shuld work,
>> >>> >> but is lot of code and potentialy slow.
>> >>> >> 7) Modification of 3 and 6: generate cleanups as bytecode and
>> >>> >> interpret it. IIUC this is essentially what DWARF is doing
>> >>> >> to handle C++ exceptions. My impression is that this is
>> >>> >> complicated to implement, requires a lot of code space and
>> >>> >> is slow a runtime. OTOH less complicated than 6, probably
>> >>> >> runs faster than 6 and needs less code space than 3.
>> >>> >>
>> >>> >> I wonder if there is other possible implementation that I
>> >>> >> overlooked? And advantages/disadvantages that I overlooked?
>> >>> >>
>> >>> >
>> >>> > Simplest to implement would be to throw a C++ exception in the signal
>> >>> > handler and catch at the safepoint (catches are always safepoints in
>> >>> > C++). That lets C++ do the dwarfing instead of you.
>> >>> In signal handler you can't call printf let alone throw C++ exception :P
>> >>
>> >> Well, from 'g++' help:
>> >>
>> >> -fasynchronous-unwind-tables Generate unwind tables that are exact at each
>> >> instruction boundary.
>> >>
>> >> What is intended use of this option?
>> >>
>> > intended use is that throwing exception from ordinary code will not
>> > break if async signal happens. But, if you throw from signal handler
>> > it will certainly break.
>> >
>> >
>> Besides, throwing from C callback is no no, as Linux is written in C.
>
> There is good chance that 'raise' inside signal handler would
> fail. However, AFAIK C++ runtime is already converting
> synchronous signals (like SIGSEGV) into exceptions. With
> asynchronous unwind tables the same could be done for all signals.
> So the plumbing is all there, if you connect right things
> you can have _effect_ of throwing exceptions from signal
> handler. I do not go in this direction because for me C++
> it has other disadvantages.
>
segfault cases program to be in inconsistent state. Kernel interrupts
program and dumps core usually. C++ exceptions are not implemented
same everywhere. There is space or runtime cost.
These with space(global tables of handlers) cost, can even work throwing from C callbacks :P
But catching signals and converting them to exceptions I never saw before :P
especially that C++ can't possibly know context where signal happened.

--
bmaxa now listens Ob-Neob Radio

On Friday, July 30, 2021 at 7:46:41 PM UTC-5, Branimir Maksimovic wrote:
> On 2021-07-30, Ivan Godard <iv...@millcomputing.com> wrote:
> > On 7/30/2021 2:47 PM, anti...@math.uni.wroc.pl wrote:
> >> This is more question of implementating high-level languages,
> >> but possible answers depend on computer architecture.

> > Simplest to implement would be to throw a C++ exception in the signal
> > handler and catch at the safepoint (catches are always safepoints in
> > C++). That lets C++ do the dwarfing instead of you.
<
> In signal handler you can't call printf let alone throw C++ exception :P
<
But you can put said signal in a list and have non-handler code throw it.
> --
> bmaxa now listens Yammat FM

On 7/31/2021 2:51 AM, David Brown wrote:

snip

> I've always thought a "disable interrupts for the next N instructions",
> with "N" being a small immediate constant, would be an extremely useful
> instruction on the kind of devices I use (single cpu microcontrollers).
> This instruction should work regardless of the current interrupt enable
> status, making it significantly more efficient than the usual store old
> status, disable interrupts, restore old status dance. It could also be
> allowable from user mode safely, unlike normal interrupt disable, since
> it is only temporary. And then you would have an easy and safe way to
> make short multi-instruction atomic sections, in a way that could be
> used by any language.
>
> That would not completely solve your complex invariant problem here, but
> it could be used to have atomic logs/trackers of the state of the
> object, allowing the program to identify and recover from inconsistent
> states.

I agree on its usefulness for your kind of application - those where you
control all the software. However on a general purpose system, I can
see lots of potential problems, mostly involving denial of service attacks.

--
- Stephen Fuld
(e-mail address disguised to prevent spam)

On Saturday, July 31, 2021 at 3:03:28 PM UTC-5, Stephen Fuld wrote:
> On 7/31/2021 2:51 AM, David Brown wrote:
>
> snip
> > I've always thought a "disable interrupts for the next N instructions",
> > with "N" being a small immediate constant, would be an extremely useful
> > instruction on the kind of devices I use (single cpu microcontrollers).
<
I can see utility here, but you need not only N as the number of instructions,
but also P the priority level below which you suppress interrupts and above
which you still allow interrupts.
<
In the realm of real CPUs (not microcontrollers) P is variable and the task/
thread might not know what privilege level is appropriate. In a hypervisor/
supervisor system, you might not be allowed to even know what level is
appropriate.
<
> > This instruction should work regardless of the current interrupt enable
> > status, making it significantly more efficient than the usual store old
> > status, disable interrupts, restore old status dance. It could also be
> > allowable from user mode safely, unlike normal interrupt disable, since
> > it is only temporary. And then you would have an easy and safe way to
> > make short multi-instruction atomic sections, in a way that could be
> > used by any language.
<
ASF (the AMD version of my ATOMIC stuff) had defined a watchdog
timeout that also suppressed interrupts for a short duration to enhance
the performance of ATOMICs without degrading interrupt responsiveness.
> >
> > That would not completely solve your complex invariant problem here, but
> > it could be used to have atomic logs/trackers of the state of the
> > object, allowing the program to identify and recover from inconsistent
> > states.
<
> I agree on its usefulness for your kind of application - those where you
> control all the software. However on a general purpose system, I can
> see lots of potential problems, mostly involving denial of service attacks.
>
The problem I see is that OP is desirous of something that smells ATOMIC
(his invariants) that are in no way surrounded by anything even remotely
smelling like an ATOMIC. Don't get me wrong, I think this brings up an
important point we may have missed in that:: in the presence of certain
invariants, one cannot prove a program to be in compliance at all points
between instructions, only at certain points where the invariants are known
to hold.
>
>
> --
> - Stephen Fuld
> (e-mail address disguised to prevent spam)

Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
> On 2021-07-31, aph@littlepinkcloud.invalid <aph@littlepinkcloud.invalid> wrote:
> > antispam@math.uni.wroc.pl wrote:
> >> This is more question of implementating high-level languages,
> >> but possible answers depend on computer architecture.
> >>
> >> Problem: We want to code in high-level language and we want
> >> to respond to asynchronous events. Simplest example may
> >> by timeout (response to timer interrupt). Or we may wish
> >> to change what the program is doing in response to external
> >> event. At hardware level there are interrupt, at operating
> >> system level we may get signals, so no problem at low level.
> >> However, high-level language may have data structures with
> >> non-trivial invariants. If we abort modification in the
> >> middle data structures may be left in inconsistent state.
> >> So, what is needed is transfer of control to handler at
> >> next safepoint, that is at next place where program may be
> >> safely interrupted. I assume that high-level language
> >> compiler knows where safepoints are, so the only problem
> >> is how to perform control transfer from safepoint to
> >> handler. This is essentially what (in)famous "came from"
> >> instruction is doing, but I am not aware of any real
> >> hardware implementing "came from"...
> >>
> > ...
> >> I wonder if there is other possible implementation that I
> >> overlooked? And advantages/disadvantages that I overlooked?
> >
> > Insert a load from a fixed address (thread specific, if you like) at
> > your safepoint. When you want to stop some thread, disable read
> > permissions for that address, and let the thread take a trap. In
> > practice the overhead in the common case is negligible because the
> > load is speculated and its value is never used.
> >
> > There is some overhead (on the order of a microsecond) though, from
> > the trip through the kernel when the trap actually happens, but it's
> > usually rare enough that this overhead doesn't matter. It would be
> > nice to have user-space fault handlers to make this overhead go away.
> >
> > Andrew.

I am not seeing orginal post by Andrew, so I am replying here.

This is interesting trick. On x86 it reduces overhead of common case
to single instruction. However, on ARM (at least version 5) it
seems that I need two instructions to pefrom load from constant
address: one instruction to load address to register (using PC
relative mode) and second for the load itself.

I admit that my hope is for zero runtime overhead in common case.

--
Waldek Hebisch

On 7/31/2021 1:58 PM, MitchAlsup wrote:
> On Saturday, July 31, 2021 at 3:03:28 PM UTC-5, Stephen Fuld wrote:
>> On 7/31/2021 2:51 AM, David Brown wrote:
>>
>> snip
>>> I've always thought a "disable interrupts for the next N instructions",
>>> with "N" being a small immediate constant, would be an extremely useful
>>> instruction on the kind of devices I use (single cpu microcontrollers).
> <
> I can see utility here, but you need not only N as the number of instructions,
> but also P the priority level below which you suppress interrupts and above
> which you still allow interrupts.
> <
> In the realm of real CPUs (not microcontrollers) P is variable and the task/
> thread might not know what privilege level is appropriate. In a hypervisor/
> supervisor system, you might not be allowed to even know what level is
> appropriate.

It isn't clear, at least to me, in a hypervisor system, whether
interrupts should be "locked out" for just the OS, or for other OSs
running under the hypervisor, or for the hypervisor itself.

>>> This instruction should work regardless of the current interrupt enable
>>> status, making it significantly more efficient than the usual store old
>>> status, disable interrupts, restore old status dance. It could also be
>>> allowable from user mode safely, unlike normal interrupt disable, since
>>> it is only temporary. And then you would have an easy and safe way to
>>> make short multi-instruction atomic sections, in a way that could be
>>> used by any language.

But simply locking out interrupts on the executing CPU doesn't force
atomic semantics on memory for multicore or multi processor systems.

> <
> ASF (the AMD version of my ATOMIC stuff) had defined a watchdog
> timeout that also suppressed interrupts for a short duration to enhance
> the performance of ATOMICs without degrading interrupt responsiveness.
>>>
>>> That would not completely solve your complex invariant problem here, but
>>> it could be used to have atomic logs/trackers of the state of the
>>> object, allowing the program to identify and recover from inconsistent
>>> states.
> <
>> I agree on its usefulness for your kind of application - those where you
>> control all the software. However on a general purpose system, I can
>> see lots of potential problems, mostly involving denial of service attacks.
>>
> The problem I see is that OP is desirous of something that smells ATOMIC
> (his invariants) that are in no way surrounded by anything even remotely
> smelling like an ATOMIC. Don't get me wrong, I think this brings up an
> important point we may have missed in that:: in the presence of certain
> invariants, one cannot prove a program to be in compliance at all points
> between instructions, only at certain points where the invariants are known
> to hold.
>>
>>
>> --
>> - Stephen Fuld
>> (e-mail address disguised to prevent spam)

--
- Stephen Fuld
(e-mail address disguised to prevent spam)

Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
> On 2021-07-31, aph@littlepinkcloud.invalid <aph@littlepinkcloud.invalid> wrote:
>>
>> Insert a load from a fixed address (thread specific, if you like) at
>> your safepoint. When you want to stop some thread, disable read
>> permissions for that address, and let the thread take a trap. In
>> practice the overhead in the common case is negligible because the
>> load is speculated and its value is never used.
>>
>> There is some overhead (on the order of a microsecond) though, from
>> the trip through the kernel when the trap actually happens, but it's
>> usually rare enough that this overhead doesn't matter. It would be
>> nice to have user-space fault handlers to make this overhead go away.
>>
> Isn't it simpler just to start another thread which will wait on events? :P

Um, what? The point is to interrupt a thread, in order to do stuff
like tracing its stack to do GC. You really have to stop the active
stack to do htat.

Andrew.

antispam@math.uni.wroc.pl wrote:
> Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
>> > ...
>> >> I wonder if there is other possible implementation that I
>> >> overlooked? And advantages/disadvantages that I overlooked?
>> >
>> > Insert a load from a fixed address (thread specific, if you like) at
>> > your safepoint. When you want to stop some thread, disable read
>> > permissions for that address, and let the thread take a trap. In
>> > practice the overhead in the common case is negligible because the
>> > load is speculated and its value is never used.
>> >
>> > There is some overhead (on the order of a microsecond) though, from
>> > the trip through the kernel when the trap actually happens, but it's
>> > usually rare enough that this overhead doesn't matter. It would be
>> > nice to have user-space fault handlers to make this overhead go away.
>
> I am not seeing orginal post by Andrew, so I am replying here.
>
> This is interesting trick. On x86 it reduces overhead of common case
> to single instruction. However, on ARM (at least version 5) it
> seems that I need two instructions to pefrom load from constant
> address: one instruction to load address to register (using PC
> relative mode) and second for the load itself.

It depends on whether you're looking to stop individual threads or all
of them. For individual threads, if you have a thread pointer in a
register you can put a page just before the thread and make the poll a
load with a negative offset. But the cost of a single immediate move
to set up the polling page address isn't very important.

> I admit that my hope is for zero runtime overhead in common case.

The greatest cost of the safepoint is setting everything up so that it
really is safe; a load doesn't add much.

Andrew.

On 2021-08-01, aph@littlepinkcloud.invalid <aph@littlepinkcloud.invalid> wrote:
> Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
>> On 2021-07-31, aph@littlepinkcloud.invalid <aph@littlepinkcloud.invalid> wrote:
>>>
>>> Insert a load from a fixed address (thread specific, if you like) at
>>> your safepoint. When you want to stop some thread, disable read
>>> permissions for that address, and let the thread take a trap. In
>>> practice the overhead in the common case is negligible because the
>>> load is speculated and its value is never used.
>>>
>>> There is some overhead (on the order of a microsecond) though, from
>>> the trip through the kernel when the trap actually happens, but it's
>>> usually rare enough that this overhead doesn't matter. It would be
>>> nice to have user-space fault handlers to make this overhead go away.
>>>
>> Isn't it simpler just to start another thread which will wait on events? :P
>
> Um, what? The point is to interrupt a thread, in order to do stuff
> like tracing its stack to do GC. You really have to stop the active
> stack to do htat.
>
> Andrew.
I think that he mentioned handling async events, while GC is already done,
which already have "stop the world implemented"....

--
bmaxa now listens rts_4

aph@littlepinkcloud.invalid wrote:
> antispam@math.uni.wroc.pl wrote:
> > Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
> >> > ...
> >> >> I wonder if there is other possible implementation that I
> >> >> overlooked? And advantages/disadvantages that I overlooked?
> >> >
> >> > Insert a load from a fixed address (thread specific, if you like) at
> >> > your safepoint. When you want to stop some thread, disable read
> >> > permissions for that address, and let the thread take a trap. In
> >> > practice the overhead in the common case is negligible because the
> >> > load is speculated and its value is never used.
> >> >
> >> > There is some overhead (on the order of a microsecond) though, from
> >> > the trip through the kernel when the trap actually happens, but it's
> >> > usually rare enough that this overhead doesn't matter. It would be
> >> > nice to have user-space fault handlers to make this overhead go away.
> >
> > I am not seeing orginal post by Andrew, so I am replying here.
> >
> > This is interesting trick. On x86 it reduces overhead of common case
> > to single instruction. However, on ARM (at least version 5) it
> > seems that I need two instructions to pefrom load from constant
> > address: one instruction to load address to register (using PC
> > relative mode) and second for the load itself.
>
> It depends on whether you're looking to stop individual threads or all
> of them. For individual threads, if you have a thread pointer in a
> register you can put a page just before the thread and make the poll a
> load with a negative offset. But the cost of a single immediate move
> to set up the polling page address isn't very important.
>
> > I admit that my hope is for zero runtime overhead in common case.
>
> The greatest cost of the safepoint is setting everything up so that it
> really is safe; a load doesn't add much.

Let me give an example. Today a loop generates the following
code (I skipped surrounding part, entry is via _L1):

_L2:
cmpq $0, i__031trap
jne _L5
cmpq %rbx, i__031userlim
jbe _L6
_L5:
call c__031checkall
_L6:
movq %r14, -8(%rbx)
movq 8(%rsp), %rsi
addq %rsi, %rsi
addq %r15, %rsi
movq -14(%rsi), %rax
subq $3, %rax
movq -8(%rbx), %r9
addq %rax, %r9
movq %r9, %r14
addq $4, %r15
_L1:
cmpq %r15, %r13
jge _L2

This is not great code, computational part can be improved
by standard compiler techniques. Comparison with
i__031userlim can be removed and equivalent check done
by setting a guard page. i__031trap is set to nonzero value
by signal handler. Already with this code checking for
for i__031trap adds notrivial overhead. And there is
also space overhead: compiler inserts checks at targets
of all backward branches (otherwise a loop could interate
for minutes without anybody looking for signals).

Several places in the loop are "unsafe", they use temporary
registers in way not understood by garbage collector.
Place at _L2 is safe. Indeally, after signal I would like
execution to continue to _L2, then low level signal handler
could simply transfer control to high-level handler
(currently high-level handler is called via c__031checkall).
To put is differently, I would like to get rid of code
between _L2 and _L6. Collapsing this to single instruction
would be an improvement. However, having _no_ extra
instructions would be better, and alternative methods
that I outlined in original post should allow this.

--
Waldek Hebisch

Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
> On 2021-08-01, aph@littlepinkcloud.invalid <aph@littlepinkcloud.invalid> wrote:
> > Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
> >> On 2021-07-31, aph@littlepinkcloud.invalid <aph@littlepinkcloud.invalid> wrote:
> >>>
> >>> Insert a load from a fixed address (thread specific, if you like) at
> >>> your safepoint. When you want to stop some thread, disable read
> >>> permissions for that address, and let the thread take a trap. In
> >>> practice the overhead in the common case is negligible because the
> >>> load is speculated and its value is never used.
> >>>
> >>> There is some overhead (on the order of a microsecond) though, from
> >>> the trip through the kernel when the trap actually happens, but it's
> >>> usually rare enough that this overhead doesn't matter. It would be
> >>> nice to have user-space fault handlers to make this overhead go away.
> >>>
> >> Isn't it simpler just to start another thread which will wait on events? :P
> >
> > Um, what? The point is to interrupt a thread, in order to do stuff
> > like tracing its stack to do GC. You really have to stop the active
> > stack to do htat.
> >
> > Andrew.
> I think that he mentioned handling async events, while GC is already done,
> which already have "stop the world implemented"....

Sorry for possible confusion. Currently there is single thread,
so to "stop" other things is easy: garbage collector is simply
not looking at flag set by low-level signal handler. To
have multiple threads I would have to implement stopping other
threads, which is possible using flags in shared memory and
checking for flags in other threads. But I would like to
save cost of flag checking, while shared memory would make
cheking much more expensive than it is now.

To put this differently: before going to multiple theads I
need efficient way to comunicate with asynchronous thread.
And implementing such efficient comunication is the whole
point of this (discussion) thread.
--
Waldek Hebisch

On 2021-08-01, antispam@math.uni.wroc.pl <antispam@math.uni.wroc.pl> wrote:
> Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
>> On 2021-08-01, aph@littlepinkcloud.invalid <aph@littlepinkcloud.invalid> wrote:
>> > Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
>> >> On 2021-07-31, aph@littlepinkcloud.invalid <aph@littlepinkcloud.invalid> wrote:
>> >>>
>> >>> Insert a load from a fixed address (thread specific, if you like) at
>> >>> your safepoint. When you want to stop some thread, disable read
>> >>> permissions for that address, and let the thread take a trap. In
>> >>> practice the overhead in the common case is negligible because the
>> >>> load is speculated and its value is never used.
>> >>>
>> >>> There is some overhead (on the order of a microsecond) though, from
>> >>> the trip through the kernel when the trap actually happens, but it's
>> >>> usually rare enough that this overhead doesn't matter. It would be
>> >>> nice to have user-space fault handlers to make this overhead go away.
>> >>>
>> >> Isn't it simpler just to start another thread which will wait on events? :P
>> >
>> > Um, what? The point is to interrupt a thread, in order to do stuff
>> > like tracing its stack to do GC. You really have to stop the active
>> > stack to do htat.
>> >
>> > Andrew.
>> I think that he mentioned handling async events, while GC is already done,
>> which already have "stop the world implemented"....
>
> Sorry for possible confusion. Currently there is single thread,
> so to "stop" other things is easy: garbage collector is simply
> not looking at flag set by low-level signal handler. To
> have multiple threads I would have to implement stopping other
> threads, which is possible using flags in shared memory and
> checking for flags in other threads. But I would like to
> save cost of flag checking, while shared memory would make
> cheking much more expensive than it is now.
>
> To put this differently: before going to multiple theads I
> need efficient way to comunicate with asynchronous thread.
> And implementing such efficient comunication is the whole
> point of this (discussion) thread.
This is from Hans's Boehm implementation:
to stop thread:
result = pthread_kill(p -> id, SIG_SUSPEND);
switch(result) {
case ESRCH:
/* Not really there anymore. Possible? */
n_live_threads--;
break;
case 0:
break;
default:
ABORT("pthread_kill failed");
}

to restart thread:
result = pthread_kill(p -> id, SIG_SUSPEND);
switch(result) {
case ESRCH:
/* Not really there anymore. Possible? */
n_live_threads--;
break;
case 0:
break;
default:
ABORT("pthread_kill failed");
}

--
bmaxa now listens rts_4

On 2021-08-01, Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
> On 2021-08-01, antispam@math.uni.wroc.pl <antispam@math.uni.wroc.pl> wrote:
>> Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
>>> On 2021-08-01, aph@littlepinkcloud.invalid <aph@littlepinkcloud.invalid> wrote:
>>> > Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
>>> >> On 2021-07-31, aph@littlepinkcloud.invalid <aph@littlepinkcloud.invalid> wrote:
>>> >>>
>>> >>> Insert a load from a fixed address (thread specific, if you like) at
>>> >>> your safepoint. When you want to stop some thread, disable read
>>> >>> permissions for that address, and let the thread take a trap. In
>>> >>> practice the overhead in the common case is negligible because the
>>> >>> load is speculated and its value is never used.
>>> >>>
>>> >>> There is some overhead (on the order of a microsecond) though, from
>>> >>> the trip through the kernel when the trap actually happens, but it's
>>> >>> usually rare enough that this overhead doesn't matter. It would be
>>> >>> nice to have user-space fault handlers to make this overhead go away.
>>> >>>
>>> >> Isn't it simpler just to start another thread which will wait on events? :P
>>> >
>>> > Um, what? The point is to interrupt a thread, in order to do stuff
>>> > like tracing its stack to do GC. You really have to stop the active
>>> > stack to do htat.
>>> >
>>> > Andrew.
>>> I think that he mentioned handling async events, while GC is already done,
>>> which already have "stop the world implemented"....
>>
>> Sorry for possible confusion. Currently there is single thread,
>> so to "stop" other things is easy: garbage collector is simply
>> not looking at flag set by low-level signal handler. To
>> have multiple threads I would have to implement stopping other
>> threads, which is possible using flags in shared memory and
>> checking for flags in other threads. But I would like to
>> save cost of flag checking, while shared memory would make
>> cheking much more expensive than it is now.
>>
>> To put this differently: before going to multiple theads I
>> need efficient way to comunicate with asynchronous thread.
>> And implementing such efficient comunication is the whole
>> point of this (discussion) thread.
> This is from Hans's Boehm implementation:
> to stop thread:
> result = pthread_kill(p -> id, SIG_SUSPEND);
> switch(result) {
> case ESRCH:
> /* Not really there anymore. Possible? */
> n_live_threads--;
> break;
> case 0:
> break;
> default:
> ABORT("pthread_kill failed");
> }
>
> to restart thread:
> result = pthread_kill(p -> id, SIG_SUSPEND);
> switch(result) {
> case ESRCH:
> /* Not really there anymore. Possible? */
> n_live_threads--;
> break;
> case 0:
> break;
> default:
> ABORT("pthread_kill failed");
> }
>
>
>

--
bmaxa nult = pthread_kill(p -> id, SIG_THR_RESTART);
switch(result) {
case ESRCH:
/* Not really there anymore. Possible? */
n_live_threads--;
break;
case 0:
break;
default:
ABORT("pthread_kill failed");
}
}
w listens rts_4

On 2021-08-01, Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
> On 2021-08-01, Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
>> On 2021-08-01, antispam@math.uni.wroc.pl <antispam@math.uni.wroc.pl> wrote:
>>> Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
>>>> On 2021-08-01, aph@littlepinkcloud.invalid <aph@littlepinkcloud.invalid> wrote:
>>>> > Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
>>>> >> On 2021-07-31, aph@littlepinkcloud.invalid <aph@littlepinkcloud.invalid> wrote:
>>>> >>>
>>>> >>> Insert a load from a fixed address (thread specific, if you like) at
>>>> >>> your safepoint. When you want to stop some thread, disable read
>>>> >>> permissions for that address, and let the thread take a trap. In
>>>> >>> practice the overhead in the common case is negligible because the
>>>> >>> load is speculated and its value is never used.
>>>> >>>
>>>> >>> There is some overhead (on the order of a microsecond) though, from
>>>> >>> the trip through the kernel when the trap actually happens, but it's
>>>> >>> usually rare enough that this overhead doesn't matter. It would be
>>>> >>> nice to have user-space fault handlers to make this overhead go away.
>>>> >>>
>>>> >> Isn't it simpler just to start another thread which will wait on events? :P
>>>> >
>>>> > Um, what? The point is to interrupt a thread, in order to do stuff
>>>> > like tracing its stack to do GC. You really have to stop the active
>>>> > stack to do htat.
>>>> >
>>>> > Andrew.
>>>> I think that he mentioned handling async events, while GC is already done,
>>>> which already have "stop the world implemented"....
>>>
>>> Sorry for possible confusion. Currently there is single thread,
>>> so to "stop" other things is easy: garbage collector is simply
>>> not looking at flag set by low-level signal handler. To
>>> have multiple threads I would have to implement stopping other
>>> threads, which is possible using flags in shared memory and
>>> checking for flags in other threads. But I would like to
>>> save cost of flag checking, while shared memory would make
>>> cheking much more expensive than it is now.
>>>
>>> To put this differently: before going to multiple theads I
>>> need efficient way to comunicate with asynchronous thread.
>>> And implementing such efficient comunication is the whole
>>> point of this (discussion) thread.
>> This is from Hans's Boehm implementation:
>> to stop thread:
>> result = pthread_kill(p -> id, SIG_SUSPEND);
>> switch(result) {
>> case ESRCH:
>> /* Not really there anymore. Possible? */
>> n_live_threads--;
>> break;
>> case 0:
>> break;
>> default:
>> ABORT("pthread_kill failed");
>> }
>>
>> to restart thread:
>> result = pthread_kill(p -> id, SIG_SUSPEND);
>> switch(result) {
>> case ESRCH:
>> /* Not really there anymore. Possible? */
>> n_live_threads--;
>> break;
>> case 0:
>> break;
>> default:
>> ABORT("pthread_kill failed");
>> }
>>
>>
>>
>
>
void GC_stop_init() {
struct sigaction act;

if (sem_init(&GC_suspend_ack_sem, 0, 0) != 0)
ABORT("sem_init failed");
# ifdef GC_NETBSD_THREADS_WORKAROUND
if (sem_init(&GC_restart_ack_sem, 0, 0) != 0)
ABORT("sem_init failed");
# endif

act.sa_flags = SA_RESTART | SA_SIGINFO;
if (sigfillset(&act.sa_mask) != 0) {
ABORT("sigfillset() failed");
}
GC_remove_allowed_signals(&act.sa_mask);
/* SIG_THR_RESTART is set in the resulting mask. */
/* It is unmasked by the handler when necessary. */
act.sa_sigaction = GC_suspend_handler;
if (sigaction(SIG_SUSPEND, &act, NULL) != 0) {
ABORT("Cannot set SIG_SUSPEND handler");
}

act.sa_flags &= ~ SA_SIGINFO;
act.sa_handler = GC_restart_handler;
if (sigaction(SIG_THR_RESTART, &act, NULL) != 0) {
ABORT("Cannot set SIG_THR_RESTART handler");
}

/* Inititialize suspend_handler_mask. It excludes SIG_THR_RESTART. */
if (sigfillset(&suspend_handler_mask) != 0) ABORT("sigfillset() failed");
GC_remove_allowed_signals(&suspend_handler_mask);
if (sigdelset(&suspend_handler_mask, SIG_THR_RESTART) != 0)
ABORT("sigdelset() failed");

/* Check for GC_RETRY_SIGNALS. */
if (0 != GETENV("GC_RETRY_SIGNALS")) {
GC_retry_signals = TRUE;
}
if (0 != GETENV("GC_NO_RETRY_SIGNALS")) {
GC_retry_signals = FALSE;
}
if (GC_print_stats && GC_retry_signals) {
GC_log_printf("Will retry suspend signal if necessary.\n");
}
}

--
bmaxa now listens rts_4

Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
> On 2021-08-01, antispam@math.uni.wroc.pl <antispam@math.uni.wroc.pl> wrote:
> > Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
> >> On 2021-08-01, aph@littlepinkcloud.invalid <aph@littlepinkcloud.invalid> wrote:
> >> > Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
> >> >> On 2021-07-31, aph@littlepinkcloud.invalid <aph@littlepinkcloud.invalid> wrote:
> >> >>>
> >> >>> Insert a load from a fixed address (thread specific, if you like) at
> >> >>> your safepoint. When you want to stop some thread, disable read
> >> >>> permissions for that address, and let the thread take a trap. In
> >> >>> practice the overhead in the common case is negligible because the
> >> >>> load is speculated and its value is never used.
> >> >>>
> >> >>> There is some overhead (on the order of a microsecond) though, from
> >> >>> the trip through the kernel when the trap actually happens, but it's
> >> >>> usually rare enough that this overhead doesn't matter. It would be
> >> >>> nice to have user-space fault handlers to make this overhead go away.
> >> >>>
> >> >> Isn't it simpler just to start another thread which will wait on events? :P
> >> >
> >> > Um, what? The point is to interrupt a thread, in order to do stuff
> >> > like tracing its stack to do GC. You really have to stop the active
> >> > stack to do htat.
> >> >
> >> > Andrew.
> >> I think that he mentioned handling async events, while GC is already done,
> >> which already have "stop the world implemented"....
> >
> > Sorry for possible confusion. Currently there is single thread,
> > so to "stop" other things is easy: garbage collector is simply
> > not looking at flag set by low-level signal handler. To
> > have multiple threads I would have to implement stopping other
> > threads, which is possible using flags in shared memory and
> > checking for flags in other threads. But I would like to
> > save cost of flag checking, while shared memory would make
> > cheking much more expensive than it is now.
> >
> > To put this differently: before going to multiple theads I
> > need efficient way to comunicate with asynchronous thread.
> > And implementing such efficient comunication is the whole
> > point of this (discussion) thread.
> This is from Hans's Boehm implementation:

Boehm garbage collector is conservative. What he is doing
does not work with precise collector (and much of complexity
that Boehm has is not needed in precise collector).

> to stop thread:
> result = pthread_kill(p -> id, SIG_SUSPEND);

This sends signal. Without handler SIG_SUSPEND it will
stop thread in some random place, must likely wrong one.
I need to continue execution up to next safepoint, which
is _not_ handled by this code.

--
Waldek Hebisch

On 2021-08-01, antispam@math.uni.wroc.pl <antispam@math.uni.wroc.pl> wrote:
> Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
>> On 2021-08-01, antispam@math.uni.wroc.pl <antispam@math.uni.wroc.pl> wrote:
>> > Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
>> >> On 2021-08-01, aph@littlepinkcloud.invalid <aph@littlepinkcloud.invalid> wrote:
>> >> > Branimir Maksimovic <branimir.maksimovic@gmail.com> wrote:
>> >> >> On 2021-07-31, aph@littlepinkcloud.invalid <aph@littlepinkcloud.invalid> wrote:
>> >> >>>
>> >> >>> Insert a load from a fixed address (thread specific, if you like) at
>> >> >>> your safepoint. When you want to stop some thread, disable read
>> >> >>> permissions for that address, and let the thread take a trap. In
>> >> >>> practice the overhead in the common case is negligible because the
>> >> >>> load is speculated and its value is never used.
>> >> >>>
>> >> >>> There is some overhead (on the order of a microsecond) though, from
>> >> >>> the trip through the kernel when the trap actually happens, but it's
>> >> >>> usually rare enough that this overhead doesn't matter. It would be
>> >> >>> nice to have user-space fault handlers to make this overhead go away.
>> >> >>>
>> >> >> Isn't it simpler just to start another thread which will wait on events? :P
>> >> >
>> >> > Um, what? The point is to interrupt a thread, in order to do stuff
>> >> > like tracing its stack to do GC. You really have to stop the active
>> >> > stack to do htat.
>> >> >
>> >> > Andrew.
>> >> I think that he mentioned handling async events, while GC is already done,
>> >> which already have "stop the world implemented"....
>> >
>> > Sorry for possible confusion. Currently there is single thread,
>> > so to "stop" other things is easy: garbage collector is simply
>> > not looking at flag set by low-level signal handler. To
>> > have multiple threads I would have to implement stopping other
>> > threads, which is possible using flags in shared memory and
>> > checking for flags in other threads. But I would like to
>> > save cost of flag checking, while shared memory would make
>> > cheking much more expensive than it is now.
>> >
>> > To put this differently: before going to multiple theads I
>> > need efficient way to comunicate with asynchronous thread.
>> > And implementing such efficient comunication is the whole
>> > point of this (discussion) thread.
>> This is from Hans's Boehm implementation:
>
> Boehm garbage collector is conservative. What he is doing
> does not work with precise collector (and much of complexity
> that Boehm has is not needed in precise collector).
>
>> to stop thread:
>> result = pthread_kill(p -> id, SIG_SUSPEND);
>
> This sends signal. Without handler SIG_SUSPEND it will
> stop thread in some random place, must likely wrong one.
> I need to continue execution up to next safepoint, which
> is _not_ handled by this code.
>
Look following two posts.

--
bmaxa now listens rts_4

antispam@math.uni.wroc.pl wrote:

> Several places in the loop are "unsafe", they use temporary
> registers in way not understood by garbage collector. Place at _L2
> is safe. Indeally, after signal I would like execution to continue
> to _L2, then low level signal handler could simply transfer control
> to high-level handler (currently high-level handler is called via
> c__031checkall). To put is differently, I would like to get rid of
> code between _L2 and _L6. Collapsing this to single instruction
> would be an improvement. However, having _no_ extra instructions
> would be better, and alternative methods that I outlined in original
> post should allow this.

I wonder which ones you have in mind? Here's my take:

1) Checking a flag at a safepoint is expensive.

2) Debug registers might work, but there are a limited number of
those, and potentially thousands of safepoints. I guess you could
interrupt the thread, look at its PC, then find its nearby safepoint
by disassembling, but it looks complex.

But that won't work if you want to stop a single thread.

3) The cleanup routines looks quite entertaining, but costly in other
ways.

4) Modifying code doesn't help, because how many loops are in a
program? You'd have to modify them all. I guess you could interrupt
the program, disassemble it to find the next nearby safepoint, patch
that and restart, but eww. But again, that won't work if you want to
stop a single thread.

Instead, you could stop a thread and single-step it by patching
instructions one at a time until it hits a safepoint. You can do that
by modifying code: you don't need debug registers.

5) Dynamically generating cleanup code from an arbitrary point looks
fun, if rather fiddly.

6) Is definitely do-able, but again rather fiddly.

7) is Hell's own complicated.

In contrast, a load at a safepoint costs very little to nothing (at
least on a Great-Big- out-of-order processor) and it's easy to
implement. But if you do manage to get one of these to work, I'd be
very interested to hear about it.

I think that trying to make the runtime code zero is something of a
mistake

Andrew.

On 2021-08-01, aph@littlepinkcloud.invalid <aph@littlepinkcloud.invalid> wrote:
> antispam@math.uni.wroc.pl wrote:
>
>> Several places in the loop are "unsafe", they use temporary
>> registers in way not understood by garbage collector. Place at _L2
>> is safe. Indeally, after signal I would like execution to continue
>> to _L2, then low level signal handler could simply transfer control
>> to high-level handler (currently high-level handler is called via
>> c__031checkall). To put is differently, I would like to get rid of
>> code between _L2 and _L6. Collapsing this to single instruction
>> would be an improvement. However, having _no_ extra instructions
>> would be better, and alternative methods that I outlined in original
>> post should allow this.
>
> I wonder which ones you have in mind? Here's my take:
>
> 1) Checking a flag at a safepoint is expensive.
>
> 2) Debug registers might work, but there are a limited number of
> those, and potentially thousands of safepoints. I guess you could
> interrupt the thread, look at its PC, then find its nearby safepoint
> by disassembling, but it looks complex.
>
> But that won't work if you want to stop a single thread.
>
> 3) The cleanup routines looks quite entertaining, but costly in other
> ways.
>
> 4) Modifying code doesn't help, because how many loops are in a
> program? You'd have to modify them all. I guess you could interrupt
> the program, disassemble it to find the next nearby safepoint, patch
> that and restart, but eww. But again, that won't work if you want to
> stop a single thread.
>
> Instead, you could stop a thread and single-step it by patching
> instructions one at a time until it hits a safepoint. You can do that
> by modifying code: you don't need debug registers.
>
> 5) Dynamically generating cleanup code from an arbitrary point looks
> fun, if rather fiddly.
>
> 6) Is definitely do-able, but again rather fiddly.
>
> 7) is Hell's own complicated.
>
> In contrast, a load at a safepoint costs very little to nothing (at
> least on a Great-Big- out-of-order processor) and it's easy to
> implement. But if you do manage to get one of these to work, I'd be
> very interested to hear about it.
>
> I think that trying to make the runtime code zero is something of a
> mistake
>
> Andrew.
_suspend_handler(int sig, siginfo_t *info, void *context)
{ int old_errno = errno;
GC_suspend_handler_inner((ptr_t)(word)sig, context);
errno = old_errno;
} #endif

void GC_suspend_handler_inner(ptr_t sig_arg, void *context)
{ int sig = (int)(word)sig_arg;
int dummy;
pthread_t my_thread = pthread_self();
GC_thread me;
# ifdef PARALLEL_MARK
word my_mark_no = GC_mark_no;
/* Marker can't proceed until we acknowledge. Thus this is */
/* guaranteed to be the mark_no correspending to our */
/* suspension, i.e. the marker can't have incremented it yet. */
# endif
AO_t my_stop_count = AO_load(&GC_stop_count);

if (sig != SIG_SUSPEND) ABORT("Bad signal in suspend_handler");

# if DEBUG_THREADS
GC_printf("Suspending 0x%x\n", (unsigned)my_thread);
# endif

me = GC_lookup_thread(my_thread);
/* The lookup here is safe, since I'm doing this on behalf */
/* of a thread which holds the allocation lock in order */
/* to stop the world. Thus concurrent modification of the */
/* data structure is impossible. */
if (me -> stop_info.last_stop_count == my_stop_count) {
/* Duplicate signal. OK if we are retrying. */
if (!GC_retry_signals) {
WARN("Duplicate suspend signal in thread %lx\n",
pthread_self());
}
return;
}
# ifdef SPARC
me -> stop_info.stack_ptr = GC_save_regs_in_stack();
# else
me -> stop_info.stack_ptr = (ptr_t)(&dummy);
# endif
# ifdef IA64
me -> backing_store_ptr = GC_save_regs_in_stack();
# endif

/* Tell the thread that wants to stop the world that this */
/* thread has been stopped. Note that sem_post() is */
/* the only async-signal-safe primitive in LinuxThreads. */
sem_post(&GC_suspend_ack_sem);
me -> stop_info.last_stop_count = my_stop_count;

/* Wait until that thread tells us to restart by sending */
/* this thread a SIG_THR_RESTART signal. */
/* SIG_THR_RESTART should be masked at this point. Thus there */
/* is no race. */
/* We do not continue until we receive a SIG_THR_RESTART, */
/* but we do not take that as authoritative. (We may be */
/* accidentally restarted by one of the user signals we */
/* don't block.) After we receive the signal, we use a */
/* primitive and expensive mechanism to wait until it's */
/* really safe to proceed. Under normal circumstances, */
/* this code should not be executed. */
do {
sigsuspend (&suspend_handler_mask);
} while (AO_load_acquire(&GC_world_is_stopped)
&& AO_load(&GC_stop_count) == my_stop_count);
/* If the RESTART signal gets lost, we can still lose. That should be */
/* less likely than losing the SUSPEND signal, since we don't do much */
/* between the sem_post and sigsuspend. */
/* We'd need more handshaking to work around that. */
/* Simply dropping the sigsuspend call should be safe, but is unlikely */
/* to be efficient. */

# if DEBUG_THREADS
GC_printf("Continuing 0x%x\n", (unsigned)my_thread);
# endif
}

void GC_restart_handler(int sig)
{ pthread_t my_thread = pthread_self();
GC_thread me;

if (sig != SIG_THR_RESTART) ABORT("Bad signal in suspend_handler");

# ifdef GC_NETBSD_THREADS_WORKAROUND
sem_post(&GC_restart_ack_sem);
# endif

/*
** Note: even if we don't do anything useful here,
** it would still be necessary to have a signal handler,
** rather than ignoring the signals, otherwise
** the signals will not be delivered at all, and
** will thus not interrupt the sigsuspend() above.
*/

# if DEBUG_THREADS
GC_printf("In GC_restart_handler for 0x%x\n", (unsigned)pthread_self());
# endif
}

aph@littlepinkcloud.invalid wrote:
> antispam@math.uni.wroc.pl wrote:
>
> > Several places in the loop are "unsafe", they use temporary
> > registers in way not understood by garbage collector. Place at _L2
> > is safe. Indeally, after signal I would like execution to continue
> > to _L2, then low level signal handler could simply transfer control
> > to high-level handler (currently high-level handler is called via
> > c__031checkall). To put is differently, I would like to get rid of
> > code between _L2 and _L6. Collapsing this to single instruction
> > would be an improvement. However, having _no_ extra instructions
> > would be better, and alternative methods that I outlined in original
> > post should allow this.
>
> I wonder which ones you have in mind? Here's my take:
>
> 1) Checking a flag at a safepoint is expensive.

Yes.
> 2) Debug registers might work, but there are a limited number of
> those, and potentially thousands of safepoints. I guess you could
> interrupt the thread, look at its PC, then find its nearby safepoint
> by disassembling, but it looks complex.

Well, in my case almost all control transfers are safepoints.
So basically I have linear piece of code and at the end there
is desired safepoint. Assuming that I keep table of addresses
of safepoints I can simply take next safepoint after current PC.
This is compiler generated code, and while compiler is not very
inteligent, it knows where safepoints are and can generate
appropriate table. Bottom line is that there is single
safepoint where code should arrive and its address is easy
to find. So single debug registe would do.

> But that won't work if you want to stop a single thread.

ATM I do not know what OS allows me to do with debug registers.
IIUC each core has its own debug registers. I hope that OS
ensures that each thread has its values of debug registers
(but I did not check). Assuming that OS works as I hope
there should be no problem with stopping single thread
without affecting other ones.

> 3) The cleanup routines looks quite entertaining, but costly in other
> ways.

Yes, there is significant space cost. Due to linearity of control
flow to the safepoints I could probably manage to do this in
double of current size (basically one copy doing normal work
and second copy transferring control to handler at safepoint.

> 4) Modifying code doesn't help, because how many loops are in a
> program? You'd have to modify them all.

As I wrote control should go to single safepoint which I can
easily find, so single place to modify.

> I guess you could interrupt
> the program, disassemble it to find the next nearby safepoint, patch
> that and restart, but eww. But again, that won't work if you want to
> stop a single thread.

Yes, self-modyfing code and multiple threads should not happen
together... One could probably stop all other threads (in this
case it does not matter if they are at safepoint or not), modify
code, run single thread, when it arrives at sefepoint restre code
and restart other threads.

> Instead, you could stop a thread and single-step it by patching
> instructions one at a time until it hits a safepoint. You can do that
> by modifying code: you don't need debug registers.

Repeated trips via signal handling machinery do not look very
attractive. In most cases there is only handful of instructions
to next safepoint, but 100 may happen.

> 5) Dynamically generating cleanup code from an arbitrary point looks
> fun, if rather fiddly.

As I wrote, I have linear block of code. And this code is
position-independent, so simple copy should do. Currently
it looks like best solution.
> 6) Is definitely do-able, but again rather fiddly.
>
> 7) is Hell's own complicated.

I would probably reverse order of descriptions, but neither
look very attractive due to amount of work needed for
implementation.

> In contrast, a load at a safepoint costs very little to nothing (at
> least on a Great-Big- out-of-order processor) and it's easy to
> implement.

Yes, much easier than my ideas.

> But if you do manage to get one of these to work, I'd be
> very interested to hear about it.
>
> I think that trying to make the runtime code zero is something of a
> mistake
>
> Andrew.

--
Waldek Hebisch

On 31/07/2021 11:59, Branimir Maksimovic wrote:
> On 2021-07-31, David Brown <david.brown@hesbynett.no> wrote:

>>
>> I've always thought a "disable interrupts for the next N instructions",
>> with "N" being a small immediate constant, would be an extremely useful
>> instruction on the kind of devices I use (single cpu microcontrollers).
>> This instruction should work regardless of the current interrupt enable
>> status, making it significantly more efficient than the usual store old
>> status, disable interrupts, restore old status dance. It could also be
>> allowable from user mode safely, unlike normal interrupt disable, since
>> it is only temporary. And then you would have an easy and safe way to
>> make short multi-instruction atomic sections, in a way that could be
>> used by any language.
>>
>> That would not completely solve your complex invariant problem here, but
>> it could be used to have atomic logs/trackers of the state of the
>> object, allowing the program to identify and recover from inconsistent
>> states.
>>
> Except that you cannot disable interupts in signal handler, neither
> touch (because of that) any global variable/flag, nor call any
> sideffect function....
>

I don't particularly care about signal handlers - they are not relevant
to the kind of programming I do.

But regardless - the point of such a limited and controlled instruction
is that it /would/ be safe even from signal handlers.

(There are still complications for multiple cores, of course.)

On 31/07/2021 22:03, Stephen Fuld wrote:
> On 7/31/2021 2:51 AM, David Brown wrote:
>
> snip
>
>> I've always thought a "disable interrupts for the next N instructions",
>> with "N" being a small immediate constant, would be an extremely useful
>> instruction on the kind of devices I use (single cpu microcontrollers).
>>   This instruction should work regardless of the current interrupt enable
>> status, making it significantly more efficient than the usual store old
>> status, disable interrupts, restore old status dance.  It could also be
>> allowable from user mode safely, unlike normal interrupt disable, since
>> it is only temporary.  And then you would have an easy and safe way to
>> make short multi-instruction atomic sections, in a way that could be
>> used by any language.
>>
>> That would not completely solve your complex invariant problem here, but
>> it could be used to have atomic logs/trackers of the state of the
>> object, allowing the program to identify and recover from inconsistent
>> states.
>
> I agree on its usefulness for your kind of application - those where you
> control all the software.  However on a general purpose system, I can
> see lots of potential problems, mostly involving denial of service attacks.
>

That is the point of having it limited - it would block interrupts for
up to N instructions (where N is a small constant), and then even if
followed by another "disable interrupts for the next N instructions",
interrupts could be handled between them.

I agree that on a bigger system, there are complications that would need
to be considered - multiple cores, long delays for memory, fault
handling, etc. Those would likely make it useless for such systems.
(On bigger systems, I would prefer to see dedicated hardware to handle
things like locks - it is insane that locks are slow, pageable
general-purpose memory instead of a small block of dedicated
core-coordinated locks.)

On 31/07/2021 22:58, MitchAlsup wrote:
> On Saturday, July 31, 2021 at 3:03:28 PM UTC-5, Stephen Fuld wrote:
>> On 7/31/2021 2:51 AM, David Brown wrote:
>>
>> snip
>>> I've always thought a "disable interrupts for the next N instructions",
>>> with "N" being a small immediate constant, would be an extremely useful
>>> instruction on the kind of devices I use (single cpu microcontrollers).
> <
> I can see utility here, but you need not only N as the number of instructions,
> but also P the priority level below which you suppress interrupts and above
> which you still allow interrupts.
> <

No - you block /all/ interrupts, of /all/ priorities. That's key to
such an instruction.

> In the realm of real CPUs (not microcontrollers) P is variable and the task/
> thread might not know what privilege level is appropriate. In a hypervisor/
> supervisor system, you might not be allowed to even know what level is
> appropriate.