On Wednesday, August 4, 2021 at 4:29:49 AM UTC-4, David Brown wrote:
> On 31/07/2021 22:58, MitchAlsup wrote:
> > On Saturday, July 31, 2021 at 3:03:28 PM UTC-5, Stephen Fuld wrote:
> >> On 7/31/2021 2:51 AM, David Brown wrote:
> >>
> >> snip
> >>> I've always thought a "disable interrupts for the next N instructions",
> >>> with "N" being a small immediate constant, would be an extremely useful
> >>> instruction on the kind of devices I use (single cpu microcontrollers).
> > <
> > I can see utility here, but you need not only N as the number of instructions,
> > but also P the priority level below which you suppress interrupts and above
> > which you still allow interrupts.
> > <
> No - you block /all/ interrupts, of /all/ priorities. That's key to
> such an instruction.
> > In the realm of real CPUs (not microcontrollers) P is variable and the task/
> > thread might not know what privilege level is appropriate. In a hypervisor/
> > supervisor system, you might not be allowed to even know what level is
> > appropriate.

Hey, I added this to ANY1, seemed like a good idea. It was fairly straight-forward to do. When the interrupt disable instruction (DI) hits the writeback stage, it simply sets a lockout flag in the following N queue entries which may or may not be populated yet with instructions. When the instructions hit writeback the lock out flag is cleared. In this case the max lockout is seven instructions, the size of the queue – 1. The DI instruction itself allows an interrupt to occur so it should not be possible to lock out interrupts using a string of DI instructions. Works in all modes of operation. The instruction was needed for several different operation that need to be atomic.

On 8/4/2021 1:27 AM, David Brown wrote:
> On 31/07/2021 22:03, Stephen Fuld wrote:
>> On 7/31/2021 2:51 AM, David Brown wrote:
>>
>> snip
>>
>>> I've always thought a "disable interrupts for the next N instructions",
>>> with "N" being a small immediate constant, would be an extremely useful
>>> instruction on the kind of devices I use (single cpu microcontrollers).
>>>   This instruction should work regardless of the current interrupt enable
>>> status, making it significantly more efficient than the usual store old
>>> status, disable interrupts, restore old status dance.  It could also be
>>> allowable from user mode safely, unlike normal interrupt disable, since
>>> it is only temporary.  And then you would have an easy and safe way to
>>> make short multi-instruction atomic sections, in a way that could be
>>> used by any language.
>>>
>>> That would not completely solve your complex invariant problem here, but
>>> it could be used to have atomic logs/trackers of the state of the
>>> object, allowing the program to identify and recover from inconsistent
>>> states.
>>
>> I agree on its usefulness for your kind of application - those where you
>> control all the software.  However on a general purpose system, I can
>> see lots of potential problems, mostly involving denial of service attacks.
>>
>
> That is the point of having it limited - it would block interrupts for
> up to N instructions (where N is a small constant), and then even if
> followed by another "disable interrupts for the next N instructions",
> interrupts could be handled between them.

I understand that. But you can reduce the "responsivness" to
interrupts, and reduce the number of interrupts per unit time that you
can handle. I am not sure that is a problem, but it nags at the back of
my mind.

--
- Stephen Fuld
(e-mail address disguised to prevent spam)

Stephen Fuld wrote:
> On 8/4/2021 1:27 AM, David Brown wrote:
>> On 31/07/2021 22:03, Stephen Fuld wrote:
>>> On 7/31/2021 2:51 AM, David Brown wrote:
>>>
>>> snip
>>>
>>>> I've always thought a "disable interrupts for the next N instructions",
>>>> with "N" being a small immediate constant, would be an extremely useful
>>>> instruction on the kind of devices I use (single cpu microcontrollers).
>>>> This instruction should work regardless of the current interrupt
>>>> enable
>>>> status, making it significantly more efficient than the usual store old
>>>> status, disable interrupts, restore old status dance. It could also be
>>>> allowable from user mode safely, unlike normal interrupt disable, since
>>>> it is only temporary. And then you would have an easy and safe way to
>>>> make short multi-instruction atomic sections, in a way that could be
>>>> used by any language.
>>>>
>>>> That would not completely solve your complex invariant problem here,
>>>> but
>>>> it could be used to have atomic logs/trackers of the state of the
>>>> object, allowing the program to identify and recover from inconsistent
>>>> states.
>>>
>>> I agree on its usefulness for your kind of application - those where you
>>> control all the software. However on a general purpose system, I can
>>> see lots of potential problems, mostly involving denial of service
>>> attacks.
>>>
>>
>> That is the point of having it limited - it would block interrupts for
>> up to N instructions (where N is a small constant), and then even if
>> followed by another "disable interrupts for the next N instructions",
>> interrupts could be handled between them.
>
> I understand that. But you can reduce the "responsivness" to
> interrupts, and reduce the number of interrupts per unit time that you
> can handle. I am not sure that is a problem, but it nags at the back of
> my mind.

Operating systems go to great lengths to layer their code
exactly so that interrupts do NOT need to be completely disabled.

Interrupts of the same or lower priority are blocked by HW while
an Interrupt Service Routine runs, but the ISR actions should be
only a relatively few instructions, maybe dozens,
before it queues further work to a lower priority and executes
a Return From Interrupt to re enable that interrupt priority.
The deferred work is processed when all nested ISR's return
and interrupts are again fully enabled.

There is of course more to it than that, a lot due to x86/x64 clunkiness.

User mode code should use Exchange or CompareExchange, FetchAdd, etc.
to perform non-interruptible operations. They don't need to be atomic if
the variables are not shared between threads (are thread local store).
Load-Lock/Store-Conditional clear the lock if an interrupt occurs
so can be used to perform non-interruptible sequences without
memory barriers if the variables are thread local.

According to David Brown <david.brown@hesbynett.no>:
>I've always thought a "disable interrupts for the next N instructions",
>with "N" being a small immediate constant, would be an extremely useful
>instruction on the kind of devices I use (single cpu microcontrollers).

Back in the 1960s the GE 635 had an interrupt blocking bit in each instruction
that you could use to do that. A timer forced an interrupt if too
many consecutive instructions set the bit. It worked in both system and user modes
(unforunately known at the time as master and slave.)

It was fine for what it was used for, atomic data structure updates on
a single threaded in-order machine. The 635 was sort of a better 7094
so I wouldn't be surprised if other scientific machines of the era had
something similar. Compare the IBM 360's test-and-set, followed by
compare-and-swap which avoided spin loops and worked better with
multiple CPUs.
--
Regards,
John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

On 8/4/2021 9:59 AM, EricP wrote:
> Stephen Fuld wrote:
>> On 8/4/2021 1:27 AM, David Brown wrote:
>>> On 31/07/2021 22:03, Stephen Fuld wrote:
>>>> On 7/31/2021 2:51 AM, David Brown wrote:
>>>>
>>>> snip
>>>>
>>>>> I've always thought a "disable interrupts for the next N
>>>>> instructions",
>>>>> with "N" being a small immediate constant, would be an extremely
>>>>> useful
>>>>> instruction on the kind of devices I use (single cpu
>>>>> microcontrollers).
>>>>>    This instruction should work regardless of the current interrupt
>>>>> enable
>>>>> status, making it significantly more efficient than the usual store
>>>>> old
>>>>> status, disable interrupts, restore old status dance.  It could
>>>>> also be
>>>>> allowable from user mode safely, unlike normal interrupt disable,
>>>>> since
>>>>> it is only temporary.  And then you would have an easy and safe way to
>>>>> make short multi-instruction atomic sections, in a way that could be
>>>>> used by any language.
>>>>>
>>>>> That would not completely solve your complex invariant problem
>>>>> here, but
>>>>> it could be used to have atomic logs/trackers of the state of the
>>>>> object, allowing the program to identify and recover from inconsistent
>>>>> states.
>>>>
>>>> I agree on its usefulness for your kind of application - those where
>>>> you
>>>> control all the software.  However on a general purpose system, I can
>>>> see lots of potential problems, mostly involving denial of service
>>>> attacks.
>>>>
>>>
>>> That is the point of having it limited - it would block interrupts for
>>> up to N instructions (where N is a small constant), and then even if
>>> followed by another "disable interrupts for the next N instructions",
>>> interrupts could be handled between them.
>>
>> I understand that.  But you can reduce the "responsivness" to
>> interrupts, and reduce the number of interrupts per unit time that you
>> can handle.  I am not sure that is a problem, but it nags at the back
>> of my mind.
>
> Operating systems go to great lengths to layer their code
> exactly so that interrupts do NOT need to be completely disabled.
>
> Interrupts of the same or lower priority are blocked by HW while
> an Interrupt Service Routine runs, but the ISR actions should be
> only a relatively few instructions, maybe dozens,
> before it queues further work to a lower priority and executes
> a Return From Interrupt to re enable that interrupt priority.
> The deferred work is processed when all nested ISR's return
> and interrupts are again fully enabled.
>
> There is of course more to it than that, a lot due to x86/x64 clunkiness.
>
> User mode code should use Exchange or CompareExchange, FetchAdd, etc.
> to perform non-interruptible operations. They don't need to be atomic if
> the variables are not shared between threads (are thread local store).
> Load-Lock/Store-Conditional clear the lock if an interrupt occurs
> so can be used to perform non-interruptible sequences without
> memory barriers if the variables are thread local.

You have made a bunch of assumptions that may not be true. If you look
back to see the kind of environment David was talking about, (single CPU
microcontrollers), he may not have a full OS, probably doesn't have
instructions like fetch and add or compare and swap, etc.

If David would give a few more specifics about his world, it might show
you a whole different world of programming.

--
- Stephen Fuld
(e-mail address disguised to prevent spam)

On 2021-08-04, David Brown <david.brown@hesbynett.no> wrote:
> On 31/07/2021 22:58, MitchAlsup wrote:
>> On Saturday, July 31, 2021 at 3:03:28 PM UTC-5, Stephen Fuld wrote:
>>> On 7/31/2021 2:51 AM, David Brown wrote:
>>>
>>> snip
>>>> I've always thought a "disable interrupts for the next N instructions",
>>>> with "N" being a small immediate constant, would be an extremely useful
>>>> instruction on the kind of devices I use (single cpu microcontrollers).
>> <
>> I can see utility here, but you need not only N as the number of instructions,
>> but also P the priority level below which you suppress interrupts and above
>> which you still allow interrupts.
>> <
>
> No - you block /all/ interrupts, of /all/ priorities. That's key to
> such an instruction.

Except non maskable interrupts.

>

--
bmaxa now listens rock.mp3

On 04/08/2021 19:51, Branimir Maksimovic wrote:
> On 2021-08-04, David Brown <david.brown@hesbynett.no> wrote:
>> On 31/07/2021 22:58, MitchAlsup wrote:
>>> On Saturday, July 31, 2021 at 3:03:28 PM UTC-5, Stephen Fuld wrote:
>>>> On 7/31/2021 2:51 AM, David Brown wrote:
>>>>
>>>> snip
>>>>> I've always thought a "disable interrupts for the next N instructions",
>>>>> with "N" being a small immediate constant, would be an extremely useful
>>>>> instruction on the kind of devices I use (single cpu microcontrollers).
>>> <
>>> I can see utility here, but you need not only N as the number of instructions,
>>> but also P the priority level below which you suppress interrupts and above
>>> which you still allow interrupts.
>>> <
>>
>> No - you block /all/ interrupts, of /all/ priorities. That's key to
>> such an instruction.
>
> Except non maskable interrupts.
>

Yes, but those only occur in a disaster (such as critical hardware fault).

On Wednesday, August 4, 2021 at 3:27:59 AM UTC-5, David Brown wrote:
> On 31/07/2021 22:03, Stephen Fuld wrote:
> > On 7/31/2021 2:51 AM, David Brown wrote:
> >
> > snip
> >
> >> I've always thought a "disable interrupts for the next N instructions",
> >> with "N" being a small immediate constant, would be an extremely useful
> >> instruction on the kind of devices I use (single cpu microcontrollers).
> >> This instruction should work regardless of the current interrupt enable
> >> status, making it significantly more efficient than the usual store old
> >> status, disable interrupts, restore old status dance. It could also be
> >> allowable from user mode safely, unlike normal interrupt disable, since
> >> it is only temporary. And then you would have an easy and safe way to
> >> make short multi-instruction atomic sections, in a way that could be
> >> used by any language.
> >>
> >> That would not completely solve your complex invariant problem here, but
> >> it could be used to have atomic logs/trackers of the state of the
> >> object, allowing the program to identify and recover from inconsistent
> >> states.
> >
> > I agree on its usefulness for your kind of application - those where you
> > control all the software. However on a general purpose system, I can
> > see lots of potential problems, mostly involving denial of service attacks.
> >
> That is the point of having it limited - it would block interrupts for
> up to N instructions (where N is a small constant), and then even if
> followed by another "disable interrupts for the next N instructions",
> interrupts could be handled between them.
<
Are any of these instructions where the interrupts are blocked capable
of taking hundreds of clock cycles to be performed ? gamma()
or take several dozen clocks to be performed ? atan2()
Are the capable of throwing exceptions? PAGEFAULT
And what happens after control returns from PAGEFALT?
Are the interrupts still blocked for n-k instructions ?
And to what end ?????

On 04/08/2021 19:46, Stephen Fuld wrote:
> On 8/4/2021 9:59 AM, EricP wrote:
>> Stephen Fuld wrote:
>>> On 8/4/2021 1:27 AM, David Brown wrote:
>>>> On 31/07/2021 22:03, Stephen Fuld wrote:
>>>>> On 7/31/2021 2:51 AM, David Brown wrote:
>>>>>
>>>>> snip
>>>>>
>>>>>> I've always thought a "disable interrupts for the next N
>>>>>> instructions",
>>>>>> with "N" being a small immediate constant, would be an extremely
>>>>>> useful
>>>>>> instruction on the kind of devices I use (single cpu
>>>>>> microcontrollers).
>>>>>>    This instruction should work regardless of the current
>>>>>> interrupt enable
>>>>>> status, making it significantly more efficient than the usual
>>>>>> store old
>>>>>> status, disable interrupts, restore old status dance.  It could
>>>>>> also be
>>>>>> allowable from user mode safely, unlike normal interrupt disable,
>>>>>> since
>>>>>> it is only temporary.  And then you would have an easy and safe
>>>>>> way to
>>>>>> make short multi-instruction atomic sections, in a way that could be
>>>>>> used by any language.
>>>>>>
>>>>>> That would not completely solve your complex invariant problem
>>>>>> here, but
>>>>>> it could be used to have atomic logs/trackers of the state of the
>>>>>> object, allowing the program to identify and recover from
>>>>>> inconsistent
>>>>>> states.
>>>>>
>>>>> I agree on its usefulness for your kind of application - those
>>>>> where you
>>>>> control all the software.  However on a general purpose system, I can
>>>>> see lots of potential problems, mostly involving denial of service
>>>>> attacks.
>>>>>
>>>>
>>>> That is the point of having it limited - it would block interrupts for
>>>> up to N instructions (where N is a small constant), and then even if
>>>> followed by another "disable interrupts for the next N instructions",
>>>> interrupts could be handled between them.
>>>
>>> I understand that.  But you can reduce the "responsivness" to
>>> interrupts, and reduce the number of interrupts per unit time that
>>> you can handle.  I am not sure that is a problem, but it nags at the
>>> back of my mind.
>>
>> Operating systems go to great lengths to layer their code
>> exactly so that interrupts do NOT need to be completely disabled.
>>
>> Interrupts of the same or lower priority are blocked by HW while
>> an Interrupt Service Routine runs, but the ISR actions should be
>> only a relatively few instructions, maybe dozens,
>> before it queues further work to a lower priority and executes
>> a Return From Interrupt to re enable that interrupt priority.
>> The deferred work is processed when all nested ISR's return
>> and interrupts are again fully enabled.
>>
>> There is of course more to it than that, a lot due to x86/x64 clunkiness.
>>
>> User mode code should use Exchange or CompareExchange, FetchAdd, etc.
>> to perform non-interruptible operations. They don't need to be atomic if
>> the variables are not shared between threads (are thread local store).
>> Load-Lock/Store-Conditional clear the lock if an interrupt occurs
>> so can be used to perform non-interruptible sequences without
>> memory barriers if the variables are thread local.
>
> You have made a bunch of assumptions that may not be true.  If you look
> back to see the kind of environment David was talking about, (single CPU
> microcontrollers), he may not have a full OS, probably doesn't have
> instructions like fetch and add or compare and swap, etc.
>
> If David would give a few more specifics about his world, it might show
> you a whole different world of programming.
>

A typical device for me these days would be based on an ARM Cortex M
single-core microcontroller. The OS's I use are real-time OS's (such as
FreeRTOS) that are linked directly to a single executable along with the
program - the systems typically have only a single binary. (There might
be an independent bootloader, but that stops once the main program starts.)

Processors in this class generally do not have atomic swap, CAS,
FetchAdd, etc., instructions. Most microcontrollers have RISC cpus
(except some of the small, old-fashioned 8-bit devices), and typically
the only tool you have other than disabling interrupts is a
load-link/store-conditional pair (ldrex/strex on ARM).

Use of ldrex/strex can avoid disabling interrupts. But they are big and
slow. Disabling interrupts around a critical section (such as an atomic
increment, or accessing a 64-bit variable) is simpler, more efficient
and more predictable.

And of critical importance in this world, it works in conjunction with
interrupts.

Any locking mechanism that does not disable interrupts works by using a
loop to get a lock (a memory location), doing the critical work, then
releasing the lock. This will not work if thread-level code and an
interrupt both want access to the same object, as the thread-level code
can never pre-empty the interrupt code. Thus if the thread has taken
the lock, and the interrupt hits, it cannot get the lock.

If you have a larger section of work that needs to be in the critical
section, you use an RTOS mutex and keep everything in threads -
interrupt routines should not be doing "large sections of work" anyway,
and the RTOS will handle the priority inversions needed to avoid deadlock.

Stephen Fuld wrote:
> On 8/4/2021 9:59 AM, EricP wrote:
>> Stephen Fuld wrote:
>>> On 8/4/2021 1:27 AM, David Brown wrote:
>>>> On 31/07/2021 22:03, Stephen Fuld wrote:
>>>>> On 7/31/2021 2:51 AM, David Brown wrote:
>>>>>
>>>>> snip
>>>>>
>>>>>> I've always thought a "disable interrupts for the next N
>>>>>> instructions",
>>>>>> with "N" being a small immediate constant, would be an extremely
>>>>>> useful
>>>>>> instruction on the kind of devices I use (single cpu
>>>>>> microcontrollers).
>>>>>> This instruction should work regardless of the current
>>>>>> interrupt enable
>>>>>> status, making it significantly more efficient than the usual
>>>>>> store old
>>>>>> status, disable interrupts, restore old status dance. It could
>>>>>> also be
>>>>>> allowable from user mode safely, unlike normal interrupt disable,
>>>>>> since
>>>>>> it is only temporary. And then you would have an easy and safe
>>>>>> way to
>>>>>> make short multi-instruction atomic sections, in a way that could be
>>>>>> used by any language.
>>>>>>
>>>>>> That would not completely solve your complex invariant problem
>>>>>> here, but
>>>>>> it could be used to have atomic logs/trackers of the state of the
>>>>>> object, allowing the program to identify and recover from
>>>>>> inconsistent
>>>>>> states.
>>>>>
>>>>> I agree on its usefulness for your kind of application - those
>>>>> where you
>>>>> control all the software. However on a general purpose system, I can
>>>>> see lots of potential problems, mostly involving denial of service
>>>>> attacks.
>>>>>
>>>>
>>>> That is the point of having it limited - it would block interrupts for
>>>> up to N instructions (where N is a small constant), and then even if
>>>> followed by another "disable interrupts for the next N instructions",
>>>> interrupts could be handled between them.
>>>
>>> I understand that. But you can reduce the "responsivness" to
>>> interrupts, and reduce the number of interrupts per unit time that
>>> you can handle. I am not sure that is a problem, but it nags at the
>>> back of my mind.
>>
>> Operating systems go to great lengths to layer their code
>> exactly so that interrupts do NOT need to be completely disabled.
>>
>> Interrupts of the same or lower priority are blocked by HW while
>> an Interrupt Service Routine runs, but the ISR actions should be
>> only a relatively few instructions, maybe dozens,
>> before it queues further work to a lower priority and executes
>> a Return From Interrupt to re enable that interrupt priority.
>> The deferred work is processed when all nested ISR's return
>> and interrupts are again fully enabled.
>>
>> There is of course more to it than that, a lot due to x86/x64 clunkiness.
>>
>> User mode code should use Exchange or CompareExchange, FetchAdd, etc.
>> to perform non-interruptible operations. They don't need to be atomic if
>> the variables are not shared between threads (are thread local store).
>> Load-Lock/Store-Conditional clear the lock if an interrupt occurs
>> so can be used to perform non-interruptible sequences without
>> memory barriers if the variables are thread local.
>
> You have made a bunch of assumptions that may not be true. If you look
> back to see the kind of environment David was talking about, (single CPU
> microcontrollers), he may not have a full OS, probably doesn't have
> instructions like fetch and add or compare and swap, etc.
>
> If David would give a few more specifics about his world, it might show
> you a whole different world of programming.

Not really. Any microcontroller with interrupts has a disable capability.
That means a non-interruptible sequence is always possible
and the discussion is about the efficiency on a particular uC.

One designs the software using HalNonInterruptibleExchange and
HalNonInterruptibleCompareExchange subroutines or macros.
If those map to single instructions on a uC, great.
If not, then they do the equivalent PUSHF, INTD, LD, ST, POPF sequence.
If an INTD_N instruction is available then it saves a PUSHF and POPF.
But LL/SC saves any disable and is more generally useful even on a uC.

Many uC I have seen had an XCHG [mem],reg instruction but not
compare-exchange as most are accumulator architectures and many
instructions are a non-interruptible R-M-W sequence anyway.

David Brown wrote:
> On 04/08/2021 19:46, Stephen Fuld wrote:
>> On 8/4/2021 9:59 AM, EricP wrote:
>>> Stephen Fuld wrote:
>>>> On 8/4/2021 1:27 AM, David Brown wrote:
>>>>> On 31/07/2021 22:03, Stephen Fuld wrote:
>>>>>> On 7/31/2021 2:51 AM, David Brown wrote:
>>>>>>
>>>>>> snip
>>>>>>
>>>>>>> I've always thought a "disable interrupts for the next N
>>>>>>> instructions",
>>>>>>> with "N" being a small immediate constant, would be an extremely
>>>>>>> useful
>>>>>>> instruction on the kind of devices I use (single cpu
>>>>>>> microcontrollers).
>>>>>>> This instruction should work regardless of the current
>>>>>>> interrupt enable
>>>>>>> status, making it significantly more efficient than the usual
>>>>>>> store old
>>>>>>> status, disable interrupts, restore old status dance. It could
>>>>>>> also be
>>>>>>> allowable from user mode safely, unlike normal interrupt disable,
>>>>>>> since
>>>>>>> it is only temporary. And then you would have an easy and safe
>>>>>>> way to
>>>>>>> make short multi-instruction atomic sections, in a way that could be
>>>>>>> used by any language.
>>>>>>>
>>>>>>> That would not completely solve your complex invariant problem
>>>>>>> here, but
>>>>>>> it could be used to have atomic logs/trackers of the state of the
>>>>>>> object, allowing the program to identify and recover from
>>>>>>> inconsistent
>>>>>>> states.
>>>>>> I agree on its usefulness for your kind of application - those
>>>>>> where you
>>>>>> control all the software. However on a general purpose system, I can
>>>>>> see lots of potential problems, mostly involving denial of service
>>>>>> attacks.
>>>>>>
>>>>> That is the point of having it limited - it would block interrupts for
>>>>> up to N instructions (where N is a small constant), and then even if
>>>>> followed by another "disable interrupts for the next N instructions",
>>>>> interrupts could be handled between them.
>>>> I understand that. But you can reduce the "responsivness" to
>>>> interrupts, and reduce the number of interrupts per unit time that
>>>> you can handle. I am not sure that is a problem, but it nags at the
>>>> back of my mind.
>>> Operating systems go to great lengths to layer their code
>>> exactly so that interrupts do NOT need to be completely disabled.
>>>
>>> Interrupts of the same or lower priority are blocked by HW while
>>> an Interrupt Service Routine runs, but the ISR actions should be
>>> only a relatively few instructions, maybe dozens,
>>> before it queues further work to a lower priority and executes
>>> a Return From Interrupt to re enable that interrupt priority.
>>> The deferred work is processed when all nested ISR's return
>>> and interrupts are again fully enabled.
>>>
>>> There is of course more to it than that, a lot due to x86/x64 clunkiness.
>>>
>>> User mode code should use Exchange or CompareExchange, FetchAdd, etc.
>>> to perform non-interruptible operations. They don't need to be atomic if
>>> the variables are not shared between threads (are thread local store).
>>> Load-Lock/Store-Conditional clear the lock if an interrupt occurs
>>> so can be used to perform non-interruptible sequences without
>>> memory barriers if the variables are thread local.
>> You have made a bunch of assumptions that may not be true. If you look
>> back to see the kind of environment David was talking about, (single CPU
>> microcontrollers), he may not have a full OS, probably doesn't have
>> instructions like fetch and add or compare and swap, etc.
>>
>> If David would give a few more specifics about his world, it might show
>> you a whole different world of programming.
>>
>
> A typical device for me these days would be based on an ARM Cortex M
> single-core microcontroller. The OS's I use are real-time OS's (such as
> FreeRTOS) that are linked directly to a single executable along with the
> program - the systems typically have only a single binary. (There might
> be an independent bootloader, but that stops once the main program starts.)
>
> Processors in this class generally do not have atomic swap, CAS,
> FetchAdd, etc., instructions. Most microcontrollers have RISC cpus
> (except some of the small, old-fashioned 8-bit devices), and typically
> the only tool you have other than disabling interrupts is a
> load-link/store-conditional pair (ldrex/strex on ARM).

ARMv5 had SWP Swap Word and SWPB Swap Byte exchange instructions.
I suppose before v5 super mode had to disable interrupts.
User mode would have to make a system call to do an an exchange.

> Use of ldrex/strex can avoid disabling interrupts. But they are big and
> slow. Disabling interrupts around a critical section (such as an atomic
> increment, or accessing a 64-bit variable) is simpler, more efficient
> and more predictable.

ldrex and strex came later with ARMv6.
I am surprised you say that on ARM they are big and slow.
The whole point of a LL/SC mechanism is that it is extremely lean.

LL loads a word as usual sets a FF in the cache controller to
indicate the line is "linked" and sets a clock counter to some value.
If any other processor reads the linked cache line, the FF is reset.
If an interrupt occurs the FF is reset.
SC stores the word as usual but checks if the FF is still set
and the clock counter is > 0. If both are true the store occurs.

The memory coherence detector should be present even on a uniprocessor as
a cache line may be invalidated by DMA. But that is just an address XOR.

> And of critical importance in this world, it works in conjunction with
> interrupts.
>
> Any locking mechanism that does not disable interrupts works by using a
> loop to get a lock (a memory location), doing the critical work, then
> releasing the lock. This will not work if thread-level code and an
> interrupt both want access to the same object, as the thread-level code
> can never pre-empty the interrupt code. Thus if the thread has taken
> the lock, and the interrupt hits, it cannot get the lock.
>
> If you have a larger section of work that needs to be in the critical
> section, you use an RTOS mutex and keep everything in threads -
> interrupt routines should not be doing "large sections of work" anyway,
> and the RTOS will handle the priority inversions needed to avoid deadlock.

On 04/08/2021 22:45, EricP wrote:
> David Brown wrote:
>> On 04/08/2021 19:46, Stephen Fuld wrote:
>>> On 8/4/2021 9:59 AM, EricP wrote:
>>>> Stephen Fuld wrote:
>>>>> On 8/4/2021 1:27 AM, David Brown wrote:
>>>>>> On 31/07/2021 22:03, Stephen Fuld wrote:
>>>>>>> On 7/31/2021 2:51 AM, David Brown wrote:
>>>>>>>
>>>>>>> snip
>>>>>>>
>>>>>>>> I've always thought a "disable interrupts for the next N
>>>>>>>> instructions",
>>>>>>>> with "N" being a small immediate constant, would be an extremely
>>>>>>>> useful
>>>>>>>> instruction on the kind of devices I use (single cpu
>>>>>>>> microcontrollers).
>>>>>>>>    This instruction should work regardless of the current
>>>>>>>> interrupt enable
>>>>>>>> status, making it significantly more efficient than the usual
>>>>>>>> store old
>>>>>>>> status, disable interrupts, restore old status dance.  It could
>>>>>>>> also be
>>>>>>>> allowable from user mode safely, unlike normal interrupt disable,
>>>>>>>> since
>>>>>>>> it is only temporary.  And then you would have an easy and safe
>>>>>>>> way to
>>>>>>>> make short multi-instruction atomic sections, in a way that
>>>>>>>> could be
>>>>>>>> used by any language.
>>>>>>>>
>>>>>>>> That would not completely solve your complex invariant problem
>>>>>>>> here, but
>>>>>>>> it could be used to have atomic logs/trackers of the state of the
>>>>>>>> object, allowing the program to identify and recover from
>>>>>>>> inconsistent
>>>>>>>> states.
>>>>>>> I agree on its usefulness for your kind of application - those
>>>>>>> where you
>>>>>>> control all the software.  However on a general purpose system, I
>>>>>>> can
>>>>>>> see lots of potential problems, mostly involving denial of service
>>>>>>> attacks.
>>>>>>>
>>>>>> That is the point of having it limited - it would block interrupts
>>>>>> for
>>>>>> up to N instructions (where N is a small constant), and then even if
>>>>>> followed by another "disable interrupts for the next N instructions",
>>>>>> interrupts could be handled between them.
>>>>> I understand that.  But you can reduce the "responsivness" to
>>>>> interrupts, and reduce the number of interrupts per unit time that
>>>>> you can handle.  I am not sure that is a problem, but it nags at the
>>>>> back of my mind.
>>>> Operating systems go to great lengths to layer their code
>>>> exactly so that interrupts do NOT need to be completely disabled.
>>>>
>>>> Interrupts of the same or lower priority are blocked by HW while
>>>> an Interrupt Service Routine runs, but the ISR actions should be
>>>> only a relatively few instructions, maybe dozens,
>>>> before it queues further work to a lower priority and executes
>>>> a Return From Interrupt to re enable that interrupt priority.
>>>> The deferred work is processed when all nested ISR's return
>>>> and interrupts are again fully enabled.
>>>>
>>>> There is of course more to it than that, a lot due to x86/x64
>>>> clunkiness.
>>>>
>>>> User mode code should use Exchange or CompareExchange, FetchAdd, etc.
>>>> to perform non-interruptible operations. They don't need to be
>>>> atomic if
>>>> the variables are not shared between threads (are thread local store).
>>>> Load-Lock/Store-Conditional clear the lock if an interrupt occurs
>>>> so can be used to perform non-interruptible sequences without
>>>> memory barriers if the variables are thread local.
>>> You have made a bunch of assumptions that may not be true.  If you look
>>> back to see the kind of environment David was talking about, (single CPU
>>> microcontrollers), he may not have a full OS, probably doesn't have
>>> instructions like fetch and add or compare and swap, etc.
>>>
>>> If David would give a few more specifics about his world, it might show
>>> you a whole different world of programming.
>>>
>>
>> A typical device for me these days would be based on an ARM Cortex M
>> single-core microcontroller.  The OS's I use are real-time OS's (such as
>> FreeRTOS) that are linked directly to a single executable along with the
>> program - the systems typically have only a single binary.  (There might
>> be an independent bootloader, but that stops once the main program
>> starts.)
>>
>> Processors in this class generally do not have atomic swap, CAS,
>> FetchAdd, etc., instructions.  Most microcontrollers have RISC cpus
>> (except some of the small, old-fashioned 8-bit devices), and typically
>> the only tool you have other than disabling interrupts is a
>> load-link/store-conditional pair (ldrex/strex on ARM).
>
> ARMv5 had SWP Swap Word and SWPB Swap Byte exchange instructions.
> I suppose before v5 super mode had to disable interrupts.
> User mode would have to make a system call to do an an exchange.

SWP and SWPB were deprecated for ARMv6, and dropped in the Thumb-2
instruction set used by the Cortex-M devices.

>
>> Use of ldrex/strex can avoid disabling interrupts.  But they are big and
>> slow.  Disabling interrupts around a critical section (such as an atomic
>> increment, or accessing a 64-bit variable) is simpler, more efficient
>> and more predictable.
>
> ldrex and strex came later with ARMv6.
> I am surprised you say that on ARM they are big and slow.
> The whole point of a LL/SC mechanism is that it is extremely lean.

Compared to disabling interrupts, LL/SC sequences are big (and therefore
slow). They take something like 7 or 8 instructions, with a loop (and
in RTOS systems, worst-case timings are important even if the loop is
usually only run once). Disabling interrupts are two or three instructions.

There are two important aspects of LL/SC, compared to older "lock
everything" mechanisms such disabling interrupts (for single core
devices) or x86 locked instructions. It splits the locking in two so
that you don't have the delays seen in the x86 world, and it is vastly
easier to implement in the hardware of a load/store processor than fully
locked instructions.

But on a single core system, it is bigger and slower than disabling
interrupts. And it does not work inside interrupt functions (which is a
big issue for microcontroller usage).

>
> LL loads a word as usual sets a FF in the cache controller to
> indicate the line is "linked" and sets a clock counter to some value.
> If any other processor reads the linked cache line, the FF is reset.
> If an interrupt occurs the FF is reset.
> SC stores the word as usual but checks if the FF is still set
> and the clock counter is > 0. If both are true the store occurs.
>
> The memory coherence detector should be present even on a uniprocessor as
> a cache line may be invalidated by DMA. But that is just an address XOR.
>

Yes. And all of that is slower than a couple of instructions to disable
interrupts.

>> And of critical importance in this world, it works in conjunction with
>> interrupts.
>>
>> Any locking mechanism that does not disable interrupts works by using a
>> loop to get a lock (a memory location), doing the critical work, then
>> releasing the lock.  This will not work if thread-level code and an
>> interrupt both want access to the same object, as the thread-level code
>> can never pre-empty the interrupt code.  Thus if the thread has taken
>> the lock, and the interrupt hits, it cannot get the lock.
>>
>> If you have a larger section of work that needs to be in the critical
>> section, you use an RTOS mutex and keep everything in threads -
>> interrupt routines should not be doing "large sections of work" anyway,
>> and the RTOS will handle the priority inversions needed to avoid
>> deadlock.
>
>

On 04/08/2021 21:55, EricP wrote:
> Stephen Fuld wrote:
>> On 8/4/2021 9:59 AM, EricP wrote:
>>> Stephen Fuld wrote:
>>>> On 8/4/2021 1:27 AM, David Brown wrote:
>>>>> On 31/07/2021 22:03, Stephen Fuld wrote:
>>>>>> On 7/31/2021 2:51 AM, David Brown wrote:
>>>>>>
>>>>>> snip
>>>>>>
>>>>>>> I've always thought a "disable interrupts for the next N
>>>>>>> instructions",
>>>>>>> with "N" being a small immediate constant, would be an extremely
>>>>>>> useful
>>>>>>> instruction on the kind of devices I use (single cpu
>>>>>>> microcontrollers).
>>>>>>>    This instruction should work regardless of the current
>>>>>>> interrupt enable
>>>>>>> status, making it significantly more efficient than the usual
>>>>>>> store old
>>>>>>> status, disable interrupts, restore old status dance.  It could
>>>>>>> also be
>>>>>>> allowable from user mode safely, unlike normal interrupt disable,
>>>>>>> since
>>>>>>> it is only temporary.  And then you would have an easy and safe
>>>>>>> way to
>>>>>>> make short multi-instruction atomic sections, in a way that could be
>>>>>>> used by any language.
>>>>>>>
>>>>>>> That would not completely solve your complex invariant problem
>>>>>>> here, but
>>>>>>> it could be used to have atomic logs/trackers of the state of the
>>>>>>> object, allowing the program to identify and recover from
>>>>>>> inconsistent
>>>>>>> states.
>>>>>>
>>>>>> I agree on its usefulness for your kind of application - those
>>>>>> where you
>>>>>> control all the software.  However on a general purpose system, I can
>>>>>> see lots of potential problems, mostly involving denial of service
>>>>>> attacks.
>>>>>>
>>>>>
>>>>> That is the point of having it limited - it would block interrupts for
>>>>> up to N instructions (where N is a small constant), and then even if
>>>>> followed by another "disable interrupts for the next N instructions",
>>>>> interrupts could be handled between them.
>>>>
>>>> I understand that.  But you can reduce the "responsivness" to
>>>> interrupts, and reduce the number of interrupts per unit time that
>>>> you can handle.  I am not sure that is a problem, but it nags at the
>>>> back of my mind.
>>>
>>> Operating systems go to great lengths to layer their code
>>> exactly so that interrupts do NOT need to be completely disabled.
>>>
>>> Interrupts of the same or lower priority are blocked by HW while
>>> an Interrupt Service Routine runs, but the ISR actions should be
>>> only a relatively few instructions, maybe dozens,
>>> before it queues further work to a lower priority and executes
>>> a Return From Interrupt to re enable that interrupt priority.
>>> The deferred work is processed when all nested ISR's return
>>> and interrupts are again fully enabled.
>>>
>>> There is of course more to it than that, a lot due to x86/x64
>>> clunkiness.
>>>
>>> User mode code should use Exchange or CompareExchange, FetchAdd, etc.
>>> to perform non-interruptible operations. They don't need to be atomic if
>>> the variables are not shared between threads (are thread local store).
>>> Load-Lock/Store-Conditional clear the lock if an interrupt occurs
>>> so can be used to perform non-interruptible sequences without
>>> memory barriers if the variables are thread local.
>>
>> You have made a bunch of assumptions that may not be true.  If you
>> look back to see the kind of environment David was talking about,
>> (single CPU microcontrollers), he may not have a full OS, probably
>> doesn't have instructions like fetch and add or compare and swap, etc.
>>
>> If David would give a few more specifics about his world, it might
>> show you a whole different world of programming.
>
> Not really. Any microcontroller with interrupts has a disable capability.
> That means a non-interruptible sequence is always possible
> and the discussion is about the efficiency on a particular uC.
>
> One designs the software using HalNonInterruptibleExchange and
> HalNonInterruptibleCompareExchange subroutines or macros.

No, one does not.

You view the interrupt disabling / restoring code (inline functions,
macros, or - my preference - RAII in C++ or gcc-extended C) as brackets
around your critical code section. You don't use it to duplicate other
mechanisms such as compare-exchange or CAS.

> If those map to single instructions on a uC, great.
> If not, then they do the equivalent PUSHF, INTD, LD, ST, POPF sequence.
> If an INTD_N instruction is available then it saves a PUSHF and POPF.
> But LL/SC saves any disable and is more generally useful even on a uC.
>

No, it is not more useful - because it does not work in conjunction with
interrupts.

Typical uses for short critical sections might be to make an atomic
variable, or to keep some related variables consistent. These will
often be used alongside interrupts - for more "normal" variables, you'd
likely be using RTOS synchronisation systems like messaging or mutexes,
and know when you can access shared data without any more locking. And
if you try to use LL/SC mechanisms to lock the data your thread shares
with an interrupt function, it will work perfectly during all your lab
testing and fail at the customer site.

(There are also other occasions when you might want such
interrupt-disabled critical sections, such as during a watchdog update,
that cannot be handled by any kind of software locking mechanism.)

> Many uC I have seen had an XCHG [mem],reg instruction but not
> compare-exchange as most are accumulator architectures and many
> instructions are a non-interruptible R-M-W sequence anyway.
>

That is the case for a number of old 8-bit CISC designs - not for more
modern microcontrollers which are almost always RISC based. And on
these old 8-bit designs, you regularly need to have interrupt disable
sequences in order to make atomic access to variables of more than 8 bits.

David Brown wrote:
> On 04/08/2021 22:45, EricP wrote:
>> David Brown wrote:
>>> A typical device for me these days would be based on an ARM Cortex M
>>> single-core microcontroller. The OS's I use are real-time OS's (such as
>>> FreeRTOS) that are linked directly to a single executable along with the
>>> program - the systems typically have only a single binary. (There might
>>> be an independent bootloader, but that stops once the main program
>>> starts.)
>>>
>>> Processors in this class generally do not have atomic swap, CAS,
>>> FetchAdd, etc., instructions. Most microcontrollers have RISC cpus
>>> (except some of the small, old-fashioned 8-bit devices), and typically
>>> the only tool you have other than disabling interrupts is a
>>> load-link/store-conditional pair (ldrex/strex on ARM).
>> ARMv5 had SWP Swap Word and SWPB Swap Byte exchange instructions.
>> I suppose before v5 super mode had to disable interrupts.
>> User mode would have to make a system call to do an an exchange.
>
> SWP and SWPB were deprecated for ARMv6, and dropped in the Thumb-2
> instruction set used by the Cortex-M devices.

I checked the ARMv7 manual before posting that and it is still there.
It was removed for ARMv7-M which is a subset of ARMv7-R.

>>> Use of ldrex/strex can avoid disabling interrupts. But they are big and
>>> slow. Disabling interrupts around a critical section (such as an atomic
>>> increment, or accessing a 64-bit variable) is simpler, more efficient
>>> and more predictable.
>> ldrex and strex came later with ARMv6.
>> I am surprised you say that on ARM they are big and slow.
>> The whole point of a LL/SC mechanism is that it is extremely lean.
>
> Compared to disabling interrupts, LL/SC sequences are big (and therefore
> slow). They take something like 7 or 8 instructions, with a loop (and
> in RTOS systems, worst-case timings are important even if the loop is
> usually only run once). Disabling interrupts are two or three instructions.

A non-interruptable exchange on Alpha would be

loop:
LDL_L rx,[mem]
STL_C rx,[mem]
BEQ rx,loop

though strictly speaking that should be a forward branch to
a backward branch so it is predicted as not taken.

> There are two important aspects of LL/SC, compared to older "lock
> everything" mechanisms such disabling interrupts (for single core
> devices) or x86 locked instructions. It splits the locking in two so
> that you don't have the delays seen in the x86 world, and it is vastly
> easier to implement in the hardware of a load/store processor than fully
> locked instructions.
>
> But on a single core system, it is bigger and slower than disabling
> interrupts. And it does not work inside interrupt functions (which is a
> big issue for microcontroller usage).

Then Arm must have done something seriously wrong with its implementation.
There is nothing in any Arm manual that I have seen to indicate a
problem with ldrex/strex and interrupts, or that they are inherently
slower than normal LD and ST.

>> LL loads a word as usual sets a FF in the cache controller to
>> indicate the line is "linked" and sets a clock counter to some value.
>> If any other processor reads the linked cache line, the FF is reset.
>> If an interrupt occurs the FF is reset.
>> SC stores the word as usual but checks if the FF is still set
>> and the clock counter is > 0. If both are true the store occurs.
>>
>> The memory coherence detector should be present even on a uniprocessor as
>> a cache line may be invalidated by DMA. But that is just an address XOR.
>>
>
> Yes. And all of that is slower than a couple of instructions to disable
> interrupts.

I keep wondering if you left the DMB memory barrier instructions in?
If a cpu is just talking between its own interrupts and its own
non-interrupt levels then it doesn't need memory barriers.
A cpu loads and stores are always consistent with itself.

David Brown wrote:
> On 04/08/2021 21:55, EricP wrote:
>> Stephen Fuld wrote:
>>> You have made a bunch of assumptions that may not be true. If you
>>> look back to see the kind of environment David was talking about,
>>> (single CPU microcontrollers), he may not have a full OS, probably
>>> doesn't have instructions like fetch and add or compare and swap, etc.
>>>
>>> If David would give a few more specifics about his world, it might
>>> show you a whole different world of programming.
>> Not really. Any microcontroller with interrupts has a disable capability.
>> That means a non-interruptible sequence is always possible
>> and the discussion is about the efficiency on a particular uC.
>>
>> One designs the software using HalNonInterruptibleExchange and
>> HalNonInterruptibleCompareExchange subroutines or macros.
>
> No, one does not.
>
> You view the interrupt disabling / restoring code (inline functions,
> macros, or - my preference - RAII in C++ or gcc-extended C) as brackets
> around your critical code section. You don't use it to duplicate other
> mechanisms such as compare-exchange or CAS.

If, as Stephen Fuld's scenario said, the microcontroller doesn't have
a swap instruction then your philosophy is moot. You have no choice.

>> If those map to single instructions on a uC, great.
>> If not, then they do the equivalent PUSHF, INTD, LD, ST, POPF sequence.
>> If an INTD_N instruction is available then it saves a PUSHF and POPF.
>> But LL/SC saves any disable and is more generally useful even on a uC.
>>
>
> No, it is not more useful - because it does not work in conjunction with
> interrupts.

Its not clear what you think does not work with interrupts but if it is
LL/SC the interrupt should clear the lock flag and prevent the SC.

The ARMv7-M manual says exceptions do clear the flag:
section A3.4.4 Context switch support
"the local monitor is changed to Open Access automatically
as part of an exception entry or exit"

In ARM parlance Open Access means not exclusive - the load-lock is canceled.

> Typical uses for short critical sections might be to make an atomic
> variable, or to keep some related variables consistent. These will
> often be used alongside interrupts - for more "normal" variables, you'd
> likely be using RTOS synchronisation systems like messaging or mutexes,
> and know when you can access shared data without any more locking. And
> if you try to use LL/SC mechanisms to lock the data your thread shares
> with an interrupt function, it will work perfectly during all your lab
> testing and fail at the customer site.

I'm not referring to atomically shared variables,
just interrupts within a single cpu.

LL/SC is used to build a non interruptible exchange,
which is used to queue deferred work packets from interrupt level
to non interrupt kernel level, and for kernel to pop deferred work
packets from the queue, all without disabling interrupts.

Its how RSX, VMS, and WNT deferred work from interrupts
without having to constantly disable and enable interrupts,
and maybe BSD and Linux too.

>
> (There are also other occasions when you might want such
> interrupt-disabled critical sections, such as during a watchdog update,
> that cannot be handled by any kind of software locking mechanism.)
>
>> Many uC I have seen had an XCHG [mem],reg instruction but not
>> compare-exchange as most are accumulator architectures and many
>> instructions are a non-interruptible R-M-W sequence anyway.
>>
>
> That is the case for a number of old 8-bit CISC designs - not for more
> modern microcontrollers which are almost always RISC based. And on
> these old 8-bit designs, you regularly need to have interrupt disable
> sequences in order to make atomic access to variables of more than 8 bits.

I don't count 32 bits as a microcontroller - those are microprocessors.
Last I read the 8-bit 8051 is still the most popular microcontroller.

David Brown wrote:
> On 04/08/2021 19:51, Branimir Maksimovic wrote:
>> On 2021-08-04, David Brown <david.brown@hesbynett.no> wrote:
>>> On 31/07/2021 22:58, MitchAlsup wrote:
>>>> On Saturday, July 31, 2021 at 3:03:28 PM UTC-5, Stephen Fuld wrote:
>>>>> On 7/31/2021 2:51 AM, David Brown wrote:
>>>>>
>>>>> snip
>>>>>> I've always thought a "disable interrupts for the next N instructions",
>>>>>> with "N" being a small immediate constant, would be an extremely useful
>>>>>> instruction on the kind of devices I use (single cpu microcontrollers).
>>>> <
>>>> I can see utility here, but you need not only N as the number of instructions,
>>>> but also P the priority level below which you suppress interrupts and above
>>>> which you still allow interrupts.
>>>> <
>>> No - you block /all/ interrupts, of /all/ priorities. That's key to
>>> such an instruction.
>> Except non maskable interrupts.
>>
>
> Yes, but those only occur in a disaster (such as critical hardware fault).

Yeah, well, only if people read the fucking HW manual first.

However it seems none of those who designed the PC did as they
wired the 8087 FPU coprocessor operation completion signal into
the NMI and afterwards everything had to be backwards compatible.
And of couse all PC OS's have to deal with the possibility that
disabling interrupts won't block FPU NMI's.

Later, from what I've read, someone wired an NMI enable/disable register.
But they connected it to the IO port bus, which for compatibility
was stuck at 5 MHz no matter what the system bus speed was.
So NMI disables and enables were really slow.

The PC Convertible had the NMI attached to the diskette,
real-time clock, keyboard, and the system suspend interrupts.

The PCjr had NMI attached to the keyboard interrupt.

On 05/08/2021 04:56, EricP wrote:
> David Brown wrote:
>> On 04/08/2021 19:51, Branimir Maksimovic wrote:
>>> On 2021-08-04, David Brown <david.brown@hesbynett.no> wrote:
>>>> On 31/07/2021 22:58, MitchAlsup wrote:
>>>>> On Saturday, July 31, 2021 at 3:03:28 PM UTC-5, Stephen Fuld wrote:
>>>>>> On 7/31/2021 2:51 AM, David Brown wrote:
>>>>>> snip
>>>>>>> I've always thought a "disable interrupts for the next N
>>>>>>> instructions", with "N" being a small immediate constant, would
>>>>>>> be an extremely useful instruction on the kind of devices I use
>>>>>>> (single cpu microcontrollers).
>>>>> <
>>>>> I can see utility here, but you need not only N as the number of
>>>>> instructions,
>>>>> but also P the priority level below which you suppress interrupts
>>>>> and above
>>>>> which you still allow interrupts.
>>>>> <
>>>> No - you block /all/ interrupts, of /all/ priorities.  That's key to
>>>> such an instruction.
>>> Except non maskable interrupts.
>>>
>>
>> Yes, but those only occur in a disaster (such as critical hardware
>> fault).
>
> Yeah, well, only if people read the fucking HW manual first.
>
> However it seems none of those who designed the PC did as they
> wired the 8087 FPU coprocessor operation completion signal into
> the NMI and afterwards everything had to be backwards compatible.
> And of couse all PC OS's have to deal with the possibility that
> disabling interrupts won't block FPU NMI's.
>
> Later, from what I've read, someone wired an NMI enable/disable register.
> But they connected it to the IO port bus, which for compatibility
> was stuck at 5 MHz no matter what the system bus speed was.
> So NMI disables and enables were really slow.
>
> The PC Convertible had the NMI attached to the diskette,
> real-time clock, keyboard, and the system suspend interrupts.
>
> The PCjr had NMI attached to the keyboard interrupt.
>

I am in the lucky position of being heavily involved in the design of
most of the systems I program - and I am very careful about how an NMI
can be used. (I don't design the microcontrollers themselves, of course
- and sometimes these have a few astoundingly stupid features.) My work
on PC's is all at a higher level - usually in a higher level language
too, such as Python. So PC's NMI problems are an SEP from my viewpoint!

On 05/08/2021 03:36, EricP wrote:
> David Brown wrote:
>> On 04/08/2021 22:45, EricP wrote:
>>> David Brown wrote:
>>>> A typical device for me these days would be based on an ARM Cortex M
>>>> single-core microcontroller.  The OS's I use are real-time OS's
>>>> (such as
>>>> FreeRTOS) that are linked directly to a single executable along with
>>>> the
>>>> program - the systems typically have only a single binary.  (There
>>>> might
>>>> be an independent bootloader, but that stops once the main program
>>>> starts.)
>>>>
>>>> Processors in this class generally do not have atomic swap, CAS,
>>>> FetchAdd, etc., instructions.  Most microcontrollers have RISC cpus
>>>> (except some of the small, old-fashioned 8-bit devices), and typically
>>>> the only tool you have other than disabling interrupts is a
>>>> load-link/store-conditional pair (ldrex/strex on ARM).
>>> ARMv5 had SWP Swap Word and SWPB Swap Byte exchange instructions.
>>> I suppose before v5 super mode had to disable interrupts.
>>> User mode would have to make a system call to do an an exchange.
>>
>> SWP and SWPB were deprecated for ARMv6, and dropped in the Thumb-2
>> instruction set used by the Cortex-M devices.
>
> I checked the ARMv7 manual before posting that and it is still there.
> It was removed for ARMv7-M which is a subset of ARMv7-R.

Yes, it is still in the ARMv7 - but deprecated. And it does not exist
at all in the Thumb-2 instruction set, which is the only available set
in the Cortex-M family. So you are right that it is in the architecture
(though I believe it is completely gone in ARMv8), but it is missing
from the devices I have been talking about.

>
>>>> Use of ldrex/strex can avoid disabling interrupts.  But they are big
>>>> and
>>>> slow.  Disabling interrupts around a critical section (such as an
>>>> atomic
>>>> increment, or accessing a 64-bit variable) is simpler, more efficient
>>>> and more predictable.
>>> ldrex and strex came later with ARMv6.
>>> I am surprised you say that on ARM they are big and slow.
>>> The whole point of a LL/SC mechanism is that it is extremely lean.
>>
>> Compared to disabling interrupts, LL/SC sequences are big (and therefore
>> slow).  They take something like 7 or 8 instructions, with a loop (and
>> in RTOS systems, worst-case timings are important even if the loop is
>> usually only run once).  Disabling interrupts are two or three
>> instructions.
>
> A non-interruptable exchange on Alpha would be
>
>   loop:
>     LDL_L  rx,[mem]
>     STL_C  rx,[mem]
>     BEQ    rx,loop
>
> though strictly speaking that should be a forward branch to
> a backward branch so it is predicted as not taken.
>
>> There are two important aspects of LL/SC, compared to older "lock
>> everything" mechanisms such disabling interrupts (for single core
>> devices) or x86 locked instructions.  It splits the locking in two so
>> that you don't have the delays seen in the x86 world, and it is vastly
>> easier to implement in the hardware of a load/store processor than fully
>> locked instructions.
>>
>> But on a single core system, it is bigger and slower than disabling
>> interrupts.  And it does not work inside interrupt functions (which is a
>> big issue for microcontroller usage).
>
> Then Arm must have done something seriously wrong with its implementation.
> There is nothing in any Arm manual that I have seen to indicate a
> problem with ldrex/strex and interrupts, or that they are inherently
> slower than normal LD and ST.
>

I am not sure I a explaining things well here.

The assembly for a 32-bit increment (of a variable in memory) in Thumb-2
will be something along the lines of :

ldr r3, [r2]
add r3, r3, #1
str r3, [r2]

Three instructions, with one load and one store. With tightly-coupled
memory (or on an M0 to M4 microcontroller, on-board ram), loads and
stores are two cycles. So that is 3 instructions, 5 cycles.

Making this atomic on an M0 microcontroller is :

cpsid i

ldr r3, [r2]
add r3, r3, #1
str r3, [r2]

cpsie

Two extra instructions, at two cycles. (These will generally not be
needed inside an interrupt function, unless the same data will be
accessed by different interrupt routines with different priorities.)

If you want a more flexible sequence that saves and restores interrupt
status, and also need it safe for the dual-issue M7, you can use:

mrs r1, primask
cpsid i
dsb

ldr r3, [r2]
add r3, r3, #1
str r3, [r2]

msr primask, r1

Four instructions, four cycles overhead for making the sequence atomic.

The equivalent for this using ldrex/strex is indeed short and fast:

loop:
ldrex r3, [r2]
add r3, r3, 1
strex r1, r3, [r2]
cmp r1, #0
bne loop
dsb

The ldrex and strex instructions don't take any longer than their
non-locking equivalents. And this is safe to use in interrupts. In
real-time work, it is vital to track worst-case execution time, not
best-case or common-case. So though the best case might be 3 cycles
overhead, an extra round of the loop might be another 9 cycles. (Single
extra rounds will happen occasionally - multiple extra rounds should not
be realistically possible.)

So it seems to be a viable alternative to disabling interrupts, with
approximately the same overhead. However, it has two major
disadvantages to go with its obvious advantage of not blocking interrupts.

It will only work for sequences ending in a single write of 32 bits or
less, and it will only work for restartable sequences.

Suppose, instead, that the atomic operation you want is not a simple
increment of a 32-bit value, but is storing a 64-bit value. With the
interrupt disable strategy, you still have exactly the same 4
instruction, 4 cycle overhead (or two instructions in the simplest
version), for both read and write routines. How do you do this with
ldrex/strex ?

You can't use the same setup as earlier. Suppose task A takes the
exclusive monitor lock, writes the first half of the 64-bit item, then
there is an interrupt. Task B wants to read the value - it takes the
lock, reads the 64-bit value, and releases it, thinking all is well.
When task A resumes, its write to the second half using strex fails, and
it restarts the write. In the meantime, task B is left with a corrupted
read that it thinks is valid. This is typically a very low probability
event - you never see it in testing, but it /will/ happen when you have
deployed thousands of systems at customers.

So you now use ldrex/strex to control access to an independent lock flag
(a simple semaphore). That works for tasks, but the overhead is now
much bigger, and there is the possibility of failure - if another task
has the semaphore, the current one must cede control to it. And that
means blocking the task, changing dynamic priority levels so that the
other task can run, etc. - a full-blown RTOS mutex solution. These are
very powerful and useful locking mechanisms, but a /vast/ overhead
compared to the four instructions for interrupt disabling, and the
single instruction and 3 cycles needed for the 64-bit load or store.

And what happens in interrupts? An interrupt function must not block
(though it can trigger a scheduler run when it exits). If a task has
the lock when the interrupt is triggered, it will not be able to do its job.

In summary, ldrex/strex /can/ be used to implement high-level,
high-power, high-overhead locking mechanisms such as an RTOS mutex or
queue (though interrupt disabling works there too). It can be used as
an alternative to interrupt locking for a limited (albeit common) subset
of atomic operations, with little more cost in run-time but
significantly greater source-code complexity (and therefore scope for
programmer error).

In general, you want to avoid disabling interrupts for any significant
length of time. But a system that can't cope with them being disabled
for the time taken to do a small atomic access, is broken anyway.

This is a different world from multi-core processors, or systems where
reading from memory might take 200 clock cycles due to cache misses, or
where it might trigger page faults. Then ldrex/strex becomes essential.

>>> LL loads a word as usual sets a FF in the cache controller to
>>> indicate the line is "linked" and sets a clock counter to some value.
>>> If any other processor reads the linked cache line, the FF is reset.
>>> If an interrupt occurs the FF is reset.
>>> SC stores the word as usual but checks if the FF is still set
>>> and the clock counter is > 0. If both are true the store occurs.
>>>
>>> The memory coherence detector should be present even on a
>>> uniprocessor as
>>> a cache line may be invalidated by DMA. But that is just an address XOR.
>>>
>>
>> Yes.  And all of that is slower than a couple of instructions to disable
>> interrupts.
>
> I keep wondering if you left the DMB memory barrier instructions in?
> If a cpu is just talking between its own interrupts and its own
> non-interrupt levels then it doesn't need memory barriers.
> A cpu loads and stores are always consistent with itself.
>

On 05/08/2021 04:40, EricP wrote:
> David Brown wrote:
>> On 04/08/2021 21:55, EricP wrote:
>>> Stephen Fuld wrote:
>>>> You have made a bunch of assumptions that may not be true.  If you
>>>> look back to see the kind of environment David was talking about,
>>>> (single CPU microcontrollers), he may not have a full OS, probably
>>>> doesn't have instructions like fetch and add or compare and swap, etc.
>>>>
>>>> If David would give a few more specifics about his world, it might
>>>> show you a whole different world of programming.
>>> Not really. Any microcontroller with interrupts has a disable
>>> capability.
>>> That means a non-interruptible sequence is always possible
>>> and the discussion is about the efficiency on a particular uC.
>>>
>>> One designs the software using HalNonInterruptibleExchange and
>>> HalNonInterruptibleCompareExchange subroutines or macros.
>>
>> No, one does not.
>>
>> You view the interrupt disabling / restoring code (inline functions,
>> macros, or - my preference - RAII in C++ or gcc-extended C) as brackets
>> around your critical code section.  You don't use it to duplicate other
>> mechanisms such as compare-exchange or CAS.
>
> If, as Stephen Fuld's scenario said, the microcontroller doesn't have
> a swap instruction then your philosophy is moot. You have no choice.
>
>>> If those map to single instructions on a uC, great.
>>> If not, then they do the equivalent PUSHF, INTD, LD, ST, POPF sequence.
>>> If an INTD_N instruction is available then it saves a PUSHF and POPF.
>>> But LL/SC saves any disable and is more generally useful even on a uC.
>>>
>>
>> No, it is not more useful - because it does not work in conjunction with
>> interrupts.
>
> Its not clear what you think does not work with interrupts but if it is
> LL/SC the interrupt should clear the lock flag and prevent the SC.
>
> The ARMv7-M manual says exceptions do clear the flag:
> section A3.4.4 Context switch support
> "the local monitor is changed to Open Access automatically
> as part of an exception entry or exit"
>
> In ARM parlance Open Access means not exclusive - the load-lock is
> canceled.

Yes, I realise that. This is not primarily designed to make ldrex/strex
work in an interrupt function, but to ensure that you don't have
problems when a task gets interrupted in the middle of a ldrex/strex
sequence, then another task starts a new sequence (claiming the lock for
itself), then the first task resumes again and thinks it has the lock.

>
>> Typical uses for short critical sections might be to make an atomic
>> variable, or to keep some related variables consistent.  These will
>> often be used alongside interrupts - for more "normal" variables, you'd
>> likely be using RTOS synchronisation systems like messaging or mutexes,
>> and know when you can access shared data without any more locking.  And
>> if you try to use LL/SC mechanisms to lock the data your thread shares
>> with an interrupt function, it will work perfectly during all your lab
>> testing and fail at the customer site.
>
> I'm not referring to atomically shared variables,
> just interrupts within a single cpu.
>
> LL/SC is used to build a non interruptible exchange,
> which is used to queue deferred work packets from interrupt level
> to non interrupt kernel level, and for kernel to pop deferred work
> packets from the queue, all without disabling interrupts.
>
> Its how RSX, VMS, and WNT deferred work from interrupts
> without having to constantly disable and enable interrupts,
> and maybe BSD and Linux too.
>

Yes. However, I am talking about locking for much smaller atomic
sequences - avoiding the need for OS-level locking with its much higher
overheads (but more features, of course).

>>
>> (There are also other occasions when you might want such
>> interrupt-disabled critical sections, such as during a watchdog update,
>> that cannot be handled by any kind of software locking mechanism.)
>>
>>> Many uC I have seen had an XCHG [mem],reg instruction but not
>>> compare-exchange as most are accumulator architectures and many
>>> instructions are a non-interruptible R-M-W sequence anyway.
>>>
>>
>> That is the case for a number of old 8-bit CISC designs - not for more
>> modern microcontrollers which are almost always RISC based.  And on
>> these old 8-bit designs, you regularly need to have interrupt disable
>> sequences in order to make atomic access to variables of more than 8
>> bits.
>
> I don't count 32 bits as a microcontroller - those are microprocessors.
> Last I read the 8-bit 8051 is still the most popular microcontroller.
>

The 8051 has been greatly used - to call it "popular" suggests that lots
of people like it, which is a different matter. Like Windows, the main
reason people used the 8051 is because a lot of people used the 8051.
(Okay, I suppose /some/ people like - I don't.)

ARM Cortex-M cores now dominate in most new microcontroller designs.
And yes, they /are/ microcontrollers - as are dual-core 64-bit PPC
microcontrollers, or 600 MHz Cortex-M7 devices. "Microcontroller" means
you have an integrated device with a cpu core, a range of peripherals,
and you are working primarily with a single binary image that handles
the entire system - OS (if you have one at all) and application code
combined.

There are no hard boundaries or definitions, however. There are devices
that some people use as microcontrollers, others as microprocessor SoC's
combined with general-purpose OS's (Linux being the most common). Some
microcontrollers have more than one program running at the same time
(this is common for Bluetooth and Wifi stacks).

I have been using 32-bit microcontrollers for 25 years - though it is
perhaps only the last 10 years since 32-bit has been more common than
8-bit or 16-bit in the boards we make. And we still make some boards
with 8-bit and 16-bit microcontrollers for specific requirements.
However, given that 32-bit microcontrollers can be had for $0.50 and are
smaller and lower power than most 8-bit devices, you'd need a really
good reason to actually /choose/ to use an 8051.

EricP wrote:
> David Brown wrote:
>> On 04/08/2021 19:51, Branimir Maksimovic wrote:
>>> On 2021-08-04, David Brown <david.brown@hesbynett.no> wrote:
>>>> No - you block /all/ interrupts, of /all/ priorities. That's key to
>>>> such an instruction.
>>> Except non maskable interrupts.
>>
>> Yes, but those only occur in a disaster (such as critical hardware
>> fault).
>
> Yeah, well, only if people read the fucking HW manual first.

For the curious and any masochists out there,
after 40 years of NMI abuse and misdesigns
here is what Linux has do to deal with NMI on x64.

https://0xax.gitbooks.io/linux-insides/content/Interrupts/linux-interrupts-6.html

Intel and AMD are proposing to clean up at least some of the mess.
Linus's thoughts on the proposals below.
https://www.realworldtech.com/forum/?threadid=200812&curpostid=200822

Intel Flexible Return and Event Delivery (FRED) Jun-2021
https://software.intel.com/content/dam/develop/external/us/en/documents-tps/346446-flexible-return-and-event-delivery.pdf

*Warning*: My reader warns that this PDF contains JavaScript *
AMD Supervisor Entry Extensions Feb-2021
https://www.amd.com/system/files/TechDocs/57115.pdf

On 8/5/2021 6:55 AM, David Brown wrote:

snip

> I am not sure I a explaining things well here.

I snipped all the details because I just wanted to thank you for
responding to my request to post more details about your work
environment. I think it has led to, and continues to provide, an
interesting sub-thread.

And BTW, I think you are explaining the differences between yours and a
"typical" big, non-embedded environment quite well.

--
- Stephen Fuld
(e-mail address disguised to prevent spam)

On 05/08/2021 17:48, Stephen Fuld wrote:
> On 8/5/2021 6:55 AM, David Brown wrote:
>
> snip
>
>> I am not sure I a explaining things well here.
>
> I snipped all the details because I just wanted to thank you for
> responding to my request to post more details about your work
> environment.  I think it has led to, and continues to provide, an
> interesting sub-thread.
>
> And BTW, I think you are explaining the differences between yours and a
> "typical" big, non-embedded environment quite well.
>

I'm on holiday, so I have plenty of time :-)

This thread has inspired me to look into more details of ldrex/strex
myself, and learn a number of new things. It's also shown me some of
the other architectures with ways of disabling interrupts. All in all,
a good thread.

David Brown wrote:
>
> I am not sure I a explaining things well here.

Your explanations are fine but we have been talking about different things.

You have been mostly referring to atomic memory operations between
threads or smp cpus, and I explicitly excluded atomic operations.

I was expanding on Mitch's earlier point about turning asynchronous
events delivered using interrupt semantics into synchronous ones,
which is what the original poster was asking about.

By atomic I mean memory operations between concurrent threads or cpus.
Non interruptible sequence is orthogonal, within a single thread or cpu.
E.g. on x86 ADD [mem],reg is a non interruptible R-M-W sequence but
is not atomic. LCK ADD [mem],reg is both atomic and non interruptible.

> The assembly for a 32-bit increment (of a variable in memory) in Thumb-2
> will be something along the lines of :
>
> ldr r3, [r2]
> add r3, r3, #1
> str r3, [r2]
>
> Three instructions, with one load and one store. With tightly-coupled
> memory (or on an M0 to M4 microcontroller, on-board ram), loads and
> stores are two cycles. So that is 3 instructions, 5 cycles.
>
> Making this atomic on an M0 microcontroller is :
>
> cpsid i
>
> ldr r3, [r2]
> add r3, r3, #1
> str r3, [r2]
>
> cpsie
>
> Two extra instructions, at two cycles. (These will generally not be
> needed inside an interrupt function, unless the same data will be
> accessed by different interrupt routines with different priorities.)
>
> If you want a more flexible sequence that saves and restores interrupt
> status, and also need it safe for the dual-issue M7, you can use:
>
> mrs r1, primask
> cpsid i
> dsb
>
> ldr r3, [r2]
> add r3, r3, #1
> str r3, [r2]
>
> msr primask, r1
>
> Four instructions, four cycles overhead for making the sequence atomic.

Yes, most OS programming standards require interrupt state
be saved and restored so the code is reentrant.

> The equivalent for this using ldrex/strex is indeed short and fast:
>
> loop:
> ldrex r3, [r2]
> add r3, r3, 1
> strex r1, r3, [r2]
> cmp r1, #0
> bne loop
> dsb

I looked at the ARMv7 manual again and I still think,
_for code that is within a single thread or cpu as per
the original question about asynchronous signals_
that the DSB is unnecessary in this particular case.
A cpu will always see its own instructions in program order.
As DSB stalls the cpu until all prior memory and branches have
completed it would impose an extra overhead on the sequence.

> The ldrex and strex instructions don't take any longer than their
> non-locking equivalents. And this is safe to use in interrupts. In
> real-time work, it is vital to track worst-case execution time, not
> best-case or common-case. So though the best case might be 3 cycles
> overhead, an extra round of the loop might be another 9 cycles. (Single
> extra rounds will happen occasionally - multiple extra rounds should not
> be realistically possible.)
>
> So it seems to be a viable alternative to disabling interrupts, with
> approximately the same overhead. However, it has two major
> disadvantages to go with its obvious advantage of not blocking interrupts.
>
> It will only work for sequences ending in a single write of 32 bits or
> less, and it will only work for restartable sequences.

Yes, this is a limitation of LL/SC - it cannot do double wide operations.

A few years ago I proposed an enhancement here which
allows LL/SC to multiple locations within a single cache line.

LL - retains the lock if load is to the same cache line
SCH - store conditional and hold stores and retains lock
SCR - store conditional and release stores and releases lock
CLL - Clear lock

The extra cost in the cache controller is holding the first line updates
separate until the release occurs in case it needs to roll back.
For the modified line it can either use a separate cache line buffer,
or if L2 is inclusive it can update L1 and either keep or toss that copy.

> Suppose, instead, that the atomic operation you want is not a simple
> increment of a 32-bit value, but is storing a 64-bit value. With the
> interrupt disable strategy, you still have exactly the same 4
> instruction, 4 cycle overhead (or two instructions in the simplest
> version), for both read and write routines. How do you do this with
> ldrex/strex ?

Yes unfortunately in general you don't.

However there are specific situations where it can be done.
E.g. reading a 64-bit clock on a 32-bit cpu since the clock is always
increasing one can read high1 read low, read high2, compare high1,high2

Another example is updating a 64-bit PTE on a 32-bit system without using
spinlocks and never leave the intermediate PTE in an illegal state.
One does the reads and writes setting and clearing bits
in a particular order.

> You can't use the same setup as earlier. Suppose task A takes the
> exclusive monitor lock, writes the first half of the 64-bit item, then
> there is an interrupt. Task B wants to read the value - it takes the
> lock, reads the 64-bit value, and releases it, thinking all is well.
> When task A resumes, its write to the second half using strex fails, and
> it restarts the write. In the meantime, task B is left with a corrupted
> read that it thinks is valid. This is typically a very low probability
> event - you never see it in testing, but it /will/ happen when you have
> deployed thousands of systems at customers.
>
> So you now use ldrex/strex to control access to an independent lock flag
> (a simple semaphore). That works for tasks, but the overhead is now
> much bigger, and there is the possibility of failure - if another task
> has the semaphore, the current one must cede control to it. And that
> means blocking the task, changing dynamic priority levels so that the
> other task can run, etc. - a full-blown RTOS mutex solution. These are
> very powerful and useful locking mechanisms, but a /vast/ overhead
> compared to the four instructions for interrupt disabling, and the
> single instruction and 3 cycles needed for the 64-bit load or store.

Right, this is all the atomic stuff which I am avoiding by narrowly
constraining the problem to e.g. signals within a single thread.

> And what happens in interrupts? An interrupt function must not block
> (though it can trigger a scheduler run when it exits). If a task has
> the lock when the interrupt is triggered, it will not be able to do its job.

By design this is illegal in any OS I know of
and will panic (crash/halt/bugcheck) the system.

Mutexes coordinate threads, spinlocks coordinate cpu kernels.
Spinlocks coordinate interrupts at the same interrupt priority and must
never be shared across interrupt priority levels as they _WILL_ deadlock.
A spinlock initialized for, say, IPL 3 should panic the OS on
attempts to acquire or release it at any other IPL.

> In summary, ldrex/strex /can/ be used to implement high-level,
> high-power, high-overhead locking mechanisms such as an RTOS mutex or
> queue (though interrupt disabling works there too). It can be used as
> an alternative to interrupt locking for a limited (albeit common) subset
> of atomic operations, with little more cost in run-time but
> significantly greater source-code complexity (and therefore scope for
> programmer error).

Yes.

> In general, you want to avoid disabling interrupts for any significant
> length of time. But a system that can't cope with them being disabled
> for the time taken to do a small atomic access, is broken anyway.
>
>
> This is a different world from multi-core processors, or systems where
> reading from memory might take 200 clock cycles due to cache misses, or
> where it might trigger page faults. Then ldrex/strex becomes essential.

I'm thinking that exchange, compare-exchange may be essential too.
Too many algorithms need non interruptable pointer swap.

Plus 32-bit cpus with 64-bit PTE's need 64-bit LD, ST, Xchg, CmpXchg
even if restricted to 8 byte alignment.

I'm warming to your interrupt-disable-for-N instruction.
One can avoid any denial of service problems by simply ignoring
further disable_N requests while current N is still counting down
and limiting N to, say, less than 8.

>>>> LL loads a word as usual sets a FF in the cache controller to
>>>> indicate the line is "linked" and sets a clock counter to some value.
>>>> If any other processor reads the linked cache line, the FF is reset.
>>>> If an interrupt occurs the FF is reset.
>>>> SC stores the word as usual but checks if the FF is still set
>>>> and the clock counter is > 0. If both are true the store occurs.
>>>>
>>>> The memory coherence detector should be present even on a
>>>> uniprocessor as
>>>> a cache line may be invalidated by DMA. But that is just an address XOR.
>>>>
>>> Yes. And all of that is slower than a couple of instructions to disable
>>> interrupts.
>> I keep wondering if you left the DMB memory barrier instructions in?
>> If a cpu is just talking between its own interrupts and its own
>> non-interrupt levels then it doesn't need memory barriers.
>> A cpu loads and stores are always consistent with itself.
>>
>
> The Cortex-M7 has a dual-issue core, and there are pipelines involved.
> That means instructions following the interrupt disable could be started
> or slightly re-organised, and you need to avoid that. A DSB is cheap in
> these devices, and recommended by ARM in connection with interrupt
> disables or ldrex/strex sequences. (The M0 core is simpler, and does
> not need it.)

David Brown wrote:
> On 05/08/2021 04:40, EricP wrote:
>> David Brown wrote:
>>> On 04/08/2021 21:55, EricP wrote:
>>
>>>> If those map to single instructions on a uC, great.
>>>> If not, then they do the equivalent PUSHF, INTD, LD, ST, POPF sequence.
>>>> If an INTD_N instruction is available then it saves a PUSHF and POPF.
>>>> But LL/SC saves any disable and is more generally useful even on a uC.
>>>>
>>> No, it is not more useful - because it does not work in conjunction with
>>> interrupts.
>> Its not clear what you think does not work with interrupts but if it is
>> LL/SC the interrupt should clear the lock flag and prevent the SC.
>>
>> The ARMv7-M manual says exceptions do clear the flag:
>> section A3.4.4 Context switch support
>> "the local monitor is changed to Open Access automatically
>> as part of an exception entry or exit"
>>
>> In ARM parlance Open Access means not exclusive - the load-lock is
>> canceled.
>
> Yes, I realise that. This is not primarily designed to make ldrex/strex
> work in an interrupt function, but to ensure that you don't have
> problems when a task gets interrupted in the middle of a ldrex/strex
> sequence, then another task starts a new sequence (claiming the lock for
> itself), then the first task resumes again and thinks it has the lock.

I don't think LL/SC can function correctly if it does not cancel the
lock on interrupt. Alpha documents this with the LL and SC instructions.

It is unfortunate that ARM documents the lock-cancel-on-interrupt separately
(and I might add, making it almost impossible to find unless you
pretty much know what to look for - that section took 1/2 hour to find)
as this might give the impression that it is a secondary side effect
rather than critical to the correct functioning.

So lock-cancel-on-interrupt should not be viewed as a side effect
but as a basic feature that you may use to your advantage.

On Friday, August 6, 2021 at 10:58:19 AM UTC-5, EricP wrote:
> David Brown wrote:
> >
> > I am not sure I a explaining things well here.
> Your explanations are fine but we have been talking about different things.
>
> You have been mostly referring to atomic memory operations between
> threads or smp cpus, and I explicitly excluded atomic operations.
>
> I was expanding on Mitch's earlier point about turning asynchronous
> events delivered using interrupt semantics into synchronous ones,
> which is what the original poster was asking about.
>
> By atomic I mean memory operations between concurrent threads or cpus.
> Non interruptible sequence is orthogonal, within a single thread or cpu.
> E.g. on x86 ADD [mem],reg is a non interruptible R-M-W sequence but
> is not atomic. LCK ADD [mem],reg is both atomic and non interruptible.
> > The assembly for a 32-bit increment (of a variable in memory) in Thumb-2
> > will be something along the lines of :
> >
> > ldr r3, [r2]
> > add r3, r3, #1
> > str r3, [r2]
> >
> > Three instructions, with one load and one store. With tightly-coupled
> > memory (or on an M0 to M4 microcontroller, on-board ram), loads and
> > stores are two cycles. So that is 3 instructions, 5 cycles.
> >
> > Making this atomic on an M0 microcontroller is :
> >
> > cpsid i
> >
> > ldr r3, [r2]
> > add r3, r3, #1
> > str r3, [r2]
> >
> > cpsie
> >
> > Two extra instructions, at two cycles. (These will generally not be
> > needed inside an interrupt function, unless the same data will be
> > accessed by different interrupt routines with different priorities.)
> >
> > If you want a more flexible sequence that saves and restores interrupt
> > status, and also need it safe for the dual-issue M7, you can use:
> >
> > mrs r1, primask
> > cpsid i
> > dsb
> >
> > ldr r3, [r2]
> > add r3, r3, #1
> > str r3, [r2]
> >
> > msr primask, r1
> >
> > Four instructions, four cycles overhead for making the sequence atomic.
> Yes, most OS programming standards require interrupt state
> be saved and restored so the code is reentrant.
> > The equivalent for this using ldrex/strex is indeed short and fast:
> >
> > loop:
> > ldrex r3, [r2]
> > add r3, r3, 1
> > strex r1, r3, [r2]
> > cmp r1, #0
> > bne loop
> > dsb
> I looked at the ARMv7 manual again and I still think,
> _for code that is within a single thread or cpu as per
> the original question about asynchronous signals_
> that the DSB is unnecessary in this particular case.
> A cpu will always see its own instructions in program order.
> As DSB stalls the cpu until all prior memory and branches have
> completed it would impose an extra overhead on the sequence.
> > The ldrex and strex instructions don't take any longer than their
> > non-locking equivalents. And this is safe to use in interrupts. In
> > real-time work, it is vital to track worst-case execution time, not
> > best-case or common-case. So though the best case might be 3 cycles
> > overhead, an extra round of the loop might be another 9 cycles. (Single
> > extra rounds will happen occasionally - multiple extra rounds should not
> > be realistically possible.)
> >
> > So it seems to be a viable alternative to disabling interrupts, with
> > approximately the same overhead. However, it has two major
> > disadvantages to go with its obvious advantage of not blocking interrupts.
> >
> > It will only work for sequences ending in a single write of 32 bits or
> > less, and it will only work for restartable sequences.
> Yes, this is a limitation of LL/SC - it cannot do double wide operations.
>
> A few years ago I proposed an enhancement here which
> allows LL/SC to multiple locations within a single cache line.
>
> LL - retains the lock if load is to the same cache line
> SCH - store conditional and hold stores and retains lock
> SCR - store conditional and release stores and releases lock
> CLL - Clear lock
>
> The extra cost in the cache controller is holding the first line updates
> separate until the release occurs in case it needs to roll back.
> For the modified line it can either use a separate cache line buffer,
> or if L2 is inclusive it can update L1 and either keep or toss that copy.
<
And then there is ESM (Exotic Synchronization Method) where one
can LL up to 8 cache lines and perform a number of LDs and STs
between these cache lines (and other places) to effect what smells
like an ATOMIC event. Here one can move an Item from one place in
a concurrent data structure to another in a single ATOMIC event
(It requires 6 cache lines to participate in the event.)
<
CAS, CDAS, CDASD, are all writeable subsets of ESM
<
One can apply timestamps and thread numbers to the pointers to
make figuring out when and where things went wrong straightforward.
<
> > Suppose, instead, that the atomic operation you want is not a simple
> > increment of a 32-bit value, but is storing a 64-bit value. With the
> > interrupt disable strategy, you still have exactly the same 4
> > instruction, 4 cycle overhead (or two instructions in the simplest
> > version), for both read and write routines. How do you do this with
> > ldrex/strex ?
<
Suppose you want to alter 3 forward pointing pointers and 3 backward
pointing pointers in a single ATOMIC event ?
<
> Yes unfortunately in general you don't.
>
> However there are specific situations where it can be done.
> E.g. reading a 64-bit clock on a 32-bit cpu since the clock is always
> increasing one can read high1 read low, read high2, compare high1,high2
>
> Another example is updating a 64-bit PTE on a 32-bit system without using
> spinlocks and never leave the intermediate PTE in an illegal state.
> One does the reads and writes setting and clearing bits
> in a particular order.
> > You can't use the same setup as earlier. Suppose task A takes the
> > exclusive monitor lock, writes the first half of the 64-bit item, then
> > there is an interrupt. Task B wants to read the value - it takes the
> > lock, reads the 64-bit value, and releases it, thinking all is well.
> > When task A resumes, its write to the second half using strex fails, and
> > it restarts the write. In the meantime, task B is left with a corrupted
> > read that it thinks is valid. This is typically a very low probability
> > event - you never see it in testing, but it /will/ happen when you have
> > deployed thousands of systems at customers.
> >
> > So you now use ldrex/strex to control access to an independent lock flag
> > (a simple semaphore). That works for tasks, but the overhead is now
> > much bigger, and there is the possibility of failure - if another task
> > has the semaphore, the current one must cede control to it. And that
> > means blocking the task, changing dynamic priority levels so that the
> > other task can run, etc. - a full-blown RTOS mutex solution. These are
> > very powerful and useful locking mechanisms, but a /vast/ overhead
> > compared to the four instructions for interrupt disabling, and the
> > single instruction and 3 cycles needed for the 64-bit load or store.
> Right, this is all the atomic stuff which I am avoiding by narrowly
> constraining the problem to e.g. signals within a single thread.
> > And what happens in interrupts? An interrupt function must not block
> > (though it can trigger a scheduler run when it exits). If a task has
> > the lock when the interrupt is triggered, it will not be able to do its job.
> By design this is illegal in any OS I know of
> and will panic (crash/halt/bugcheck) the system.
>
> Mutexes coordinate threads, spinlocks coordinate cpu kernels.
> Spinlocks coordinate interrupts at the same interrupt priority and must
> never be shared across interrupt priority levels as they _WILL_ deadlock.
> A spinlock initialized for, say, IPL 3 should panic the OS on
> attempts to acquire or release it at any other IPL.
> > In summary, ldrex/strex /can/ be used to implement high-level,
> > high-power, high-overhead locking mechanisms such as an RTOS mutex or
> > queue (though interrupt disabling works there too). It can be used as
> > an alternative to interrupt locking for a limited (albeit common) subset
> > of atomic operations, with little more cost in run-time but
> > significantly greater source-code complexity (and therefore scope for
> > programmer error).
> Yes.
> > In general, you want to avoid disabling interrupts for any significant
> > length of time. But a system that can't cope with them being disabled
> > for the time taken to do a small atomic access, is broken anyway.
> >
> >
> > This is a different world from multi-core processors, or systems where
> > reading from memory might take 200 clock cycles due to cache misses, or
> > where it might trigger page faults. Then ldrex/strex becomes essential.
> I'm thinking that exchange, compare-exchange may be essential too.
> Too many algorithms need non interruptable pointer swap.
>
> Plus 32-bit cpus with 64-bit PTE's need 64-bit LD, ST, Xchg, CmpXchg
> even if restricted to 8 byte alignment.
>
> I'm warming to your interrupt-disable-for-N instruction.
> One can avoid any denial of service problems by simply ignoring
> further disable_N requests while current N is still counting down
> and limiting N to, say, less than 8.
> >>>> LL loads a word as usual sets a FF in the cache controller to
> >>>> indicate the line is "linked" and sets a clock counter to some value.
> >>>> If any other processor reads the linked cache line, the FF is reset.
> >>>> If an interrupt occurs the FF is reset.
> >>>> SC stores the word as usual but checks if the FF is still set
> >>>> and the clock counter is > 0. If both are true the store occurs.
> >>>>
> >>>> The memory coherence detector should be present even on a
> >>>> uniprocessor as
> >>>> a cache line may be invalidated by DMA. But that is just an address XOR.
> >>>>
> >>> Yes. And all of that is slower than a couple of instructions to disable
> >>> interrupts.
> >> I keep wondering if you left the DMB memory barrier instructions in?
> >> If a cpu is just talking between its own interrupts and its own
> >> non-interrupt levels then it doesn't need memory barriers.
> >> A cpu loads and stores are always consistent with itself.
> >>
> >
> > The Cortex-M7 has a dual-issue core, and there are pipelines involved.
> > That means instructions following the interrupt disable could be started
> > or slightly re-organised, and you need to avoid that. A DSB is cheap in
> > these devices, and recommended by ARM in connection with interrupt
> > disables or ldrex/strex sequences. (The M0 core is simpler, and does
> > not need it.)
> For non-atomic, ie within a single thread between its normal and
> its own signal level, or between a single cpu and its interrupts,
> the data dependencies themselves look to me to be sufficient
> ordering which is why I thought the DSB superfluous.