On 5/12/23 3:25 PM, Clair Grant wrote:
> Virtual Box 7.0.6, Lenovo ThinkBook, Windows 11
>
> I put SHOW TIME in a 10 minute loop and ran it 7 hours. Every output
> showed exactly 10 minutes and 0 seconds from the previous. I did the
> same thing on
>
> ESXi 8.0, DL580
>
> and got the same result. If we have a problem, it is certainly not
> universal.
>
> Clair

Thanks for the reply and for running your own test. The problem is
indeed not universal. I'm beginning to think in my case there is some
disagreement between VirtualBox and macOS, and/or something specific to
the particular laptop hardware of the host.

On 5/15/2023 6:17 AM, Johnny Billquist wrote:
> On 2023-05-13 02:16, Arne Vajhøj wrote:
>> On 5/12/2023 1:30 PM, Simon Clubley wrote:
>>> On 2023-05-12, Dave Froble <davef@tsoft-inc.com> wrote:
>>>> On 5/12/2023 8:14 AM, Simon Clubley wrote:
>>>>> That's going to make for some "interesting" real-time program
>>>>> behaviour... :-)
>>>>
>>>> Do you think any serious real time programmer will run a real time
>>>> task inside a
>>>> VM?  I'm not a real time programmer, and I'd still not do that.
>>>
>>> As well as traditional real-time stuff (which I agree with you about
>>> BTW),
>>
>> I would not want to do it on a type 2 hypervisor - there must be
>> cases where what is happening on the host OS impact the performance
>> of the guest OS.
>>
>> But with a type 1 hypervisor and no over allocation of resources -
>> would it be worse than running on bare metal?
>
> For sure. You have no guarantee that you will get the CPU cycles when
> you need them. No matter what kind of hypervisor we're talking about,
> there is overhead in the host that can affect things way more than what
> might happen on bare metal.

What host? A type 1 hypervisor does not run a host OS.

And why would the CPU not be available, when there is no
over allocation of resources? It seems pretty silly of
the hypervisor to not want to give the CPU if no other
VM can get it.

Arne

On 5/15/2023 7:37 AM, Craig A. Berry wrote:
> On 5/12/23 3:25 PM, Clair Grant wrote:
>> Virtual Box 7.0.6, Lenovo ThinkBook, Windows 11
>>
>> I put SHOW TIME in a 10 minute loop and ran it 7 hours. Every output
>> showed exactly 10 minutes and 0 seconds from the previous. I did the
>> same thing on
>>
>> ESXi 8.0, DL580
>>
>> and got the same result. If we have a problem, it is certainly not
>> universal.
>
> Thanks for the reply and for running your own test.  The problem is
> indeed not universal.  I'm beginning to think in my case there is some
> disagreement between VirtualBox and macOS, and/or something specific to
> the particular laptop hardware of the host.

From the high level perspective then I think we need to realize that:

number of virtualization platforms (software and version) x number of OS
platforms (software and version - discard if type 1 hypervisor) x number
of HW platforms (CPU + MB + more combos) > number VMS x86-64 users

Impossible to test everything.

And in some cases there will be glitches.

Bug in VMS x86-64 or bug in virtualization software or bug in host OS
(if present) or bug in HW.

So far I believe the indication is that VMS x86-64 quality is
pretty good.

Let us call it VMS quality! :-) :-) :-)

Arne

In article <8b6c6dbd-5dc4-41aa-982e-7c25c5ebb29an@googlegroups.com>,
gah4 <gah4@u.washington.edu> wrote:
>[snip]
>In the case of virtual machines, each one will have its own CPU timer,
>but only one TOD clock. STCK is not a privileged instruction, so programs
>(or OS) see the host clock.

This would seem to violate Popek and Goldberg's virtualization
criteria; how is that squared?

- Dan C.

On 2023-05-13, Arne Vajhøj <arne@vajhoej.dk> wrote:
>
> VAX got real time priorities 16-31.
>
> Alpha got real time priorities 16-63.
>
> I assume Itanium and x86-64 has 16-63 as well.
>
> But I believe there is more to real time than
> having those (no quantum end and no dynamic
> adjustment).
>

Real-time at its core is about meeting timing deadlines.

If you miss the odd timing deadline and can still function (maybe in
a degraded way), it's called soft real-time.

If you miss a timing deadline and hence go BOOM!!! (or even fully fail
in a less dramatic way :-)), it's called hard real-time.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

>
> On 5/12/23 3:25 PM, Clair Grant wrote:
>> Virtual Box 7.0.6, Lenovo ThinkBook, Windows 11
>>
>> I put SHOW TIME in a 10 minute loop and ran it 7 hours. Every
>> output showed exactly 10 minutes and 0 seconds from the previous.

I'm not sure I fully understand how that test was done.

If you ask VMS to wait for 10 min and then ask VMS for the time
passed, I would expect VMS to say that "10 min has passed", no
matter how long those 10 min was in reallity.

Of course VMS in it self belives that 10 min has passed,
it doesn't know better, so to speak.

Or, fully possible of course, I'm just missing something here... :-)

Jan-Erik.

On 2023-05-12, Arne Vajhøj <arne@vajhoej.dk> wrote:
> On 5/12/2023 1:30 PM, Simon Clubley wrote:
>> On 2023-05-12, Dave Froble <davef@tsoft-inc.com> wrote:
>>> On 5/12/2023 8:14 AM, Simon Clubley wrote:
>>>> That's going to make for some "interesting" real-time program behaviour... :-)
>>>
>>> Do you think any serious real time programmer will run a real time task inside a
>>> VM? I'm not a real time programmer, and I'd still not do that.
>>
>> As well as traditional real-time stuff (which I agree with you about BTW),
>
> I would not want to do it on a type 2 hypervisor - there must be
> cases where what is happening on the host OS impact the performance
> of the guest OS.
>
> But with a type 1 hypervisor and no over allocation of resources -
> would it be worse than running on bare metal?
>

A type 1 hypervisor is still a software layer between the hardware and
the RTOS. It would have to _guarantee_ that it would _never_ get in the
way of the timing guarantees that a program running under a RTOS needs.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 2023-05-15, Jan-Erik Söderholm <jan-erik.soderholm@telia.com> wrote:
>>
>> On 5/12/23 3:25 PM, Clair Grant wrote:
>>> Virtual Box 7.0.6, Lenovo ThinkBook, Windows 11
>>>
>>> I put SHOW TIME in a 10 minute loop and ran it 7 hours. Every
>>> output showed exactly 10 minutes and 0 seconds from the previous.
>
> I'm not sure I fully understand how that test was done.
>
> If you ask VMS to wait for 10 min and then ask VMS for the time
> passed, I would expect VMS to say that "10 min has passed", no
> matter how long those 10 min was in reallity.
>

I agree. I don't think it's a real test unless both VMS and the host OS
are both doing other things while also running this timing loop and there
is a way to compare the reported VMS time with the underlying host OS time.

> Of course VMS in it self belives that 10 min has passed,
> it doesn't know better, so to speak.
>
> Or, fully possible of course, I'm just missing something here... :-)
>

I don't think you are. I think for the test to be valid, it needs to be
done on a loaded system and a way of comparing the VMS reported time to
the actual host time needs to be devised.

Given the reported problems, I also think the test needs to have an
test interval that's way under 10 minutes.

It needs some kind of script, running on the host OS, that is connected to
VMS and runs a $ SHOW TIME every couple of minutes for several hours.

It needs to write both the output from $ SHOW TIME and the current host
time to a log file, along with the delta difference value between the two.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 5/15/2023 8:20 AM, Simon Clubley wrote:
> On 2023-05-12, Arne Vajhøj <arne@vajhoej.dk> wrote:
>> On 5/12/2023 1:30 PM, Simon Clubley wrote:
>>> On 2023-05-12, Dave Froble <davef@tsoft-inc.com> wrote:
>>>> On 5/12/2023 8:14 AM, Simon Clubley wrote:
>>>>> That's going to make for some "interesting" real-time program behaviour... :-)
>>>>
>>>> Do you think any serious real time programmer will run a real time task inside a
>>>> VM? I'm not a real time programmer, and I'd still not do that.
>>>
>>> As well as traditional real-time stuff (which I agree with you about BTW),
>>
>> I would not want to do it on a type 2 hypervisor - there must be
>> cases where what is happening on the host OS impact the performance
>> of the guest OS.
>>
>> But with a type 1 hypervisor and no over allocation of resources -
>> would it be worse than running on bare metal?
>
> A type 1 hypervisor is still a software layer between the hardware and
> the RTOS. It would have to _guarantee_ that it would _never_ get in the
> way of the timing guarantees that a program running under a RTOS needs.

It is software that is active between the VMS and the HW.

But it is not obvious to me where any delay would come from.

If we talk type 2 then it is easy to understand. You have a host
OS running 200 processes - 1 VM and 199 other stuff, and 4 CPU
so those 200 processes get scheduled on those 4 CPU's, and certain
interrupts may happen in the host OS. That VM will have
a problem providing real time capabilities.

But in the type 1 with no over allocation of resources then
what will cause any delays? The VM got its own CPU or CPU's
that no other VM want, so I would expect the hypervisor to
keep them permanently allocated to the VM. And I don't
see the need for any interrupts in the hypervisor either.

Arne

On 2023-05-15 12:59, gah4 wrote:
> On Monday, May 15, 2023 at 3:06:16 AM UTC-7, Johnny Billquist wrote:
>
> (snip, I wrote)
>
>>> IBM S/360 uses the same system for both, as you describe.
>
>> So does VMS, all Unixes that I've ever encountered, all PDP-11 OSes I've
>> ever encountered, all RTOSes I've ever encountered, and everything else
>> I've ever encountered...
>
> The S/360 interval timer is a 32 bit word in memory that decrement such that
> bit 23 (counting from the left) seems to decrement at 300Hz. That allows for
> an actual 50Hz or 60Hz timer, subtracting 6 or 5. An interrupt is generated
> when it goes below zero.

PDP-11 clock interrupts are usually also at 50Hz or 60Hz depending on
what frequence is used on main lines power. Not an unusual frequency for
computers to use to count time.

It used to be a *very* reliable clock source. Power grids used to
compensate for drift because of load, so that over 24 hours, you have a
very exact number of peaks on the power. All type of clocks were
designed with this clock source. Sadly, I think power grids no longer
are caring as much about the accuracy of the frequency.

VAX didn't, but instead have a 100 Hz clock source for interrupts, which
was less accurate than the power based one.

> One way to look at it is a microcode interrupt. It only decrements between
> instructions. It does not need to generate a real interrupt at that rate.

I don't know the S/360 at all, but I would expect that it also uses the
50/60Hz as a clock source with interrupts generated at that frequency...

Johnny

On 5/15/23 7:17 AM, Jan-Erik Söderholm wrote:
>>
>> On 5/12/23 3:25 PM, Clair Grant wrote:
>>> Virtual Box 7.0.6, Lenovo ThinkBook, Windows 11
>>>
>>> I put SHOW TIME in a 10 minute loop and ran it 7 hours. Every
>>> output showed exactly 10 minutes and 0 seconds from the previous.
>
> I'm not sure I fully understand how that test was done.
>
> If you ask VMS to wait for 10 min and then ask VMS for the time
> passed, I would expect VMS to say that "10 min has passed", no
> matter how long those 10 min was in reallity.
>
> Of course VMS in it self belives that 10 min has passed,
> it doesn't know better, so to speak.
>
> Or, fully possible of course, I'm just missing something here... :-)

I was assuming that the total time of 7 hours was independently
corroborated, but you're right, he didn't explicitly say that.

On 2023-05-15 13:37, Craig A. Berry wrote:
>
> On 5/12/23 3:25 PM, Clair Grant wrote:
>> Virtual Box 7.0.6, Lenovo ThinkBook, Windows 11
>>
>> I put SHOW TIME in a 10 minute loop and ran it 7 hours. Every output
>> showed exactly 10 minutes and 0 seconds from the previous. I did the
>> same thing on
>>
>> ESXi 8.0, DL580
>>
>> and got the same result. If we have a problem, it is certainly not
>> universal.
>>
>> Clair
>
>
> Thanks for the reply and for running your own test.  The problem is
> indeed not universal.  I'm beginning to think in my case there is some
> disagreement between VirtualBox and macOS, and/or something specific to
> the particular laptop hardware of the host.

In your case, I would agree. The amount of errors you mentioned is big.
Definitely sounds like some disagreement between the components about
the clock handling.

Johnny

On 2023-05-15 14:17, Jan-Erik Söderholm wrote:
>>
>> On 5/12/23 3:25 PM, Clair Grant wrote:
>>> Virtual Box 7.0.6, Lenovo ThinkBook, Windows 11
>>>
>>> I put SHOW TIME in a 10 minute loop and ran it 7 hours. Every
>>> output showed exactly 10 minutes and 0 seconds from the previous.
>
> I'm not sure I fully understand how that test was done.
>
> If you ask VMS to wait for 10 min and then ask VMS for the time
> passed, I would expect VMS to say that "10 min has passed", no
> matter how long those 10 min was in reallity.
>
> Of course VMS in it self belives that 10 min has passed,
> it doesn't know better, so to speak.
>
> Or, fully possible of course, I'm just missing something here... :-)

Ha! A very good point. The test don't actually prove anything. You are
right. :-)

But if you were to compare to an external clock, I would suspect you
would not see big deviations either, but I would expect there to be
small glitches in the short timescale and if you look with enough
resolution.

Johnny

On 2023-05-15 14:03, Arne Vajhøj wrote:
> On 5/15/2023 6:17 AM, Johnny Billquist wrote:
>> On 2023-05-13 02:16, Arne Vajhøj wrote:
>>> On 5/12/2023 1:30 PM, Simon Clubley wrote:
>>>> On 2023-05-12, Dave Froble <davef@tsoft-inc.com> wrote:
>>>>> On 5/12/2023 8:14 AM, Simon Clubley wrote:
>>>>>> That's going to make for some "interesting" real-time program
>>>>>> behaviour... :-)
>>>>>
>>>>> Do you think any serious real time programmer will run a real time
>>>>> task inside a
>>>>> VM?  I'm not a real time programmer, and I'd still not do that.
>>>>
>>>> As well as traditional real-time stuff (which I agree with you about
>>>> BTW),
>>>
>>> I would not want to do it on a type 2 hypervisor - there must be
>>> cases where what is happening on the host OS impact the performance
>>> of the guest OS.
>>>
>>> But with a type 1 hypervisor and no over allocation of resources -
>>> would it be worse than running on bare metal?
>>
>> For sure. You have no guarantee that you will get the CPU cycles when
>> you need them. No matter what kind of hypervisor we're talking about,
>> there is overhead in the host that can affect things way more than
>> what might happen on bare metal.
>
> What host? A type 1 hypervisor does not run a host OS.

Um? What is the hypervisor then?

> And why would the CPU not be available, when there is no
> over allocation of resources? It seems pretty silly of
> the hypervisor to not want to give the CPU if no other
> VM can get it.

Any kind of hypervisor means we're talking about virtualization. That
means you have real hardware which exists outside of this
virtualization. And that hardware can generate interrupts and demand
service, which then forces the hypervisor to wait before it can actually
get any cycles. Outside the control of the hypervisor...

Johnny

On 2023-05-15 14:36, Arne Vajhøj wrote:
> On 5/15/2023 8:20 AM, Simon Clubley wrote:
>> On 2023-05-12, Arne Vajhøj <arne@vajhoej.dk> wrote:
>>> On 5/12/2023 1:30 PM, Simon Clubley wrote:
>>>> On 2023-05-12, Dave Froble <davef@tsoft-inc.com> wrote:
>>>>> On 5/12/2023 8:14 AM, Simon Clubley wrote:
>>>>>> That's going to make for some "interesting" real-time program
>>>>>> behaviour... :-)
>>>>>
>>>>> Do you think any serious real time programmer will run a real time
>>>>> task inside a
>>>>> VM?  I'm not a real time programmer, and I'd still not do that.
>>>>
>>>> As well as traditional real-time stuff (which I agree with you about
>>>> BTW),
>>>
>>> I would not want to do it on a type 2 hypervisor - there must be
>>> cases where what is happening on the host OS impact the performance
>>> of the guest OS.
>>>
>>> But with a type 1 hypervisor and no over allocation of resources -
>>> would it be worse than running on bare metal?
>>
>> A type 1 hypervisor is still a software layer between the hardware and
>> the RTOS. It would have to _guarantee_ that it would _never_ get in the
>> way of the timing guarantees that a program running under a RTOS needs.
>
> It is software that is active between the VMS and the HW.
>
> But it is not obvious to me where any delay would come from.
>
> If we talk type 2 then it is easy to understand. You have a host
> OS running 200 processes - 1 VM and 199 other stuff, and 4 CPU
> so those 200 processes get scheduled on those 4 CPU's, and certain
> interrupts may happen in the host OS. That VM will have
> a problem providing real time capabilities.
>
> But in the type 1 with no over allocation of resources then
> what will cause any delays? The VM got its own CPU or CPU's
> that no other VM want, so I would expect the hypervisor to
> keep them permanently allocated to the VM. And I don't
> see the need for any interrupts in the hypervisor either.

Can you guarantee that no interrupts happens on the CPUs the hypervisor
is using? I would suspect "no", which means you do not have full control.

It's kindof weird to me that people can even believe that it would be
possible to add a software layer in between that have no actual impact.

Johnny

On 5/15/2023 8:51 AM, Johnny Billquist wrote:
> On 2023-05-15 14:03, Arne Vajhøj wrote:
>> On 5/15/2023 6:17 AM, Johnny Billquist wrote:
>>> On 2023-05-13 02:16, Arne Vajhøj wrote:
>>>> On 5/12/2023 1:30 PM, Simon Clubley wrote:
>>>>> On 2023-05-12, Dave Froble <davef@tsoft-inc.com> wrote:
>>>>>> On 5/12/2023 8:14 AM, Simon Clubley wrote:
>>>>>>> That's going to make for some "interesting" real-time program
>>>>>>> behaviour... :-)
>>>>>>
>>>>>> Do you think any serious real time programmer will run a real time
>>>>>> task inside a
>>>>>> VM?  I'm not a real time programmer, and I'd still not do that.
>>>>>
>>>>> As well as traditional real-time stuff (which I agree with you
>>>>> about BTW),
>>>>
>>>> I would not want to do it on a type 2 hypervisor - there must be
>>>> cases where what is happening on the host OS impact the performance
>>>> of the guest OS.
>>>>
>>>> But with a type 1 hypervisor and no over allocation of resources -
>>>> would it be worse than running on bare metal?
>>>
>>> For sure. You have no guarantee that you will get the CPU cycles when
>>> you need them. No matter what kind of hypervisor we're talking about,
>>> there is overhead in the host that can affect things way more than
>>> what might happen on bare metal.
>>
>> What host? A type 1 hypervisor does not run a host OS.
>
> Um? What is the hypervisor then?

The hypervisor software like ESXi.

It is not running on top of a host OS like a type 2 (VirtualBox etc.) does.

>> And why would the CPU not be available, when there is no
>> over allocation of resources? It seems pretty silly of
>> the hypervisor to not want to give the CPU if no other
>> VM can get it.
>
> Any kind of hypervisor means we're talking about virtualization. That
> means you have real hardware which exists outside of this
> virtualization. And that hardware can generate interrupts and demand
> service, which then forces the hypervisor to wait before it can actually
> get any cycles. Outside the control of the hypervisor...

Maybe. But what would that be?

Arne

On 5/15/2023 8:53 AM, Johnny Billquist wrote:
> On 2023-05-15 14:36, Arne Vajhøj wrote:
>> On 5/15/2023 8:20 AM, Simon Clubley wrote:
>>> On 2023-05-12, Arne Vajhøj <arne@vajhoej.dk> wrote:
>>>> On 5/12/2023 1:30 PM, Simon Clubley wrote:
>>>>> On 2023-05-12, Dave Froble <davef@tsoft-inc.com> wrote:
>>>>>> On 5/12/2023 8:14 AM, Simon Clubley wrote:
>>>>>>> That's going to make for some "interesting" real-time program
>>>>>>> behaviour... :-)
>>>>>>
>>>>>> Do you think any serious real time programmer will run a real time
>>>>>> task inside a
>>>>>> VM?  I'm not a real time programmer, and I'd still not do that.
>>>>>
>>>>> As well as traditional real-time stuff (which I agree with you
>>>>> about BTW),
>>>>
>>>> I would not want to do it on a type 2 hypervisor - there must be
>>>> cases where what is happening on the host OS impact the performance
>>>> of the guest OS.
>>>>
>>>> But with a type 1 hypervisor and no over allocation of resources -
>>>> would it be worse than running on bare metal?
>>>
>>> A type 1 hypervisor is still a software layer between the hardware and
>>> the RTOS. It would have to _guarantee_ that it would _never_ get in the
>>> way of the timing guarantees that a program running under a RTOS needs.
>>
>> It is software that is active between the VMS and the HW.
>>
>> But it is not obvious to me where any delay would come from.
>>
>> If we talk type 2 then it is easy to understand. You have a host
>> OS running 200 processes - 1 VM and 199 other stuff, and 4 CPU
>> so those 200 processes get scheduled on those 4 CPU's, and certain
>> interrupts may happen in the host OS. That VM will have
>> a problem providing real time capabilities.
>>
>> But in the type 1 with no over allocation of resources then
>> what will cause any delays? The VM got its own CPU or CPU's
>> that no other VM want, so I would expect the hypervisor to
>> keep them permanently allocated to the VM. And I don't
>> see the need for any interrupts in the hypervisor either.
>
> Can you guarantee that no interrupts happens on the CPUs the hypervisor
> is using? I would suspect "no", which means you do not have full control.

What would those interrupts do?

A type 1 hypervisor does not do that much.

It has not very much in common with a type 2 hypervisor. I see it
as more of a software equivalent of system partitioning (DEC Galaxy,
IBM LPAR).

Arne

On 5/15/2023 9:45 AM, Arne Vajhøj wrote:
> On 5/15/2023 8:53 AM, Johnny Billquist wrote:
>> On 2023-05-15 14:36, Arne Vajhøj wrote:
>>> On 5/15/2023 8:20 AM, Simon Clubley wrote:
>>>> On 2023-05-12, Arne Vajhøj <arne@vajhoej.dk> wrote:
>>>>> On 5/12/2023 1:30 PM, Simon Clubley wrote:
>>>>>> On 2023-05-12, Dave Froble <davef@tsoft-inc.com> wrote:
>>>>>>> On 5/12/2023 8:14 AM, Simon Clubley wrote:
>>>>>>>> That's going to make for some "interesting" real-time program
>>>>>>>> behaviour... :-)
>>>>>>>
>>>>>>> Do you think any serious real time programmer will run a real
>>>>>>> time task inside a
>>>>>>> VM?  I'm not a real time programmer, and I'd still not do that.
>>>>>>
>>>>>> As well as traditional real-time stuff (which I agree with you
>>>>>> about BTW),
>>>>>
>>>>> I would not want to do it on a type 2 hypervisor - there must be
>>>>> cases where what is happening on the host OS impact the performance
>>>>> of the guest OS.
>>>>>
>>>>> But with a type 1 hypervisor and no over allocation of resources -
>>>>> would it be worse than running on bare metal?
>>>>
>>>> A type 1 hypervisor is still a software layer between the hardware and
>>>> the RTOS. It would have to _guarantee_ that it would _never_ get in the
>>>> way of the timing guarantees that a program running under a RTOS needs.
>>>
>>> It is software that is active between the VMS and the HW.
>>>
>>> But it is not obvious to me where any delay would come from.
>>>
>>> If we talk type 2 then it is easy to understand. You have a host
>>> OS running 200 processes - 1 VM and 199 other stuff, and 4 CPU
>>> so those 200 processes get scheduled on those 4 CPU's, and certain
>>> interrupts may happen in the host OS. That VM will have
>>> a problem providing real time capabilities.
>>>
>>> But in the type 1 with no over allocation of resources then
>>> what will cause any delays? The VM got its own CPU or CPU's
>>> that no other VM want, so I would expect the hypervisor to
>>> keep them permanently allocated to the VM. And I don't
>>> see the need for any interrupts in the hypervisor either.
>>
>> Can you guarantee that no interrupts happens on the CPUs the
>> hypervisor is using? I would suspect "no", which means you do not have
>> full control.
>
> What would those interrupts do?
>
> A type 1 hypervisor does not do that much.
>
> It has not very much in common with a type 2 hypervisor. I see it
> as more of a software equivalent of system partitioning (DEC Galaxy,
> IBM LPAR).

As I don't know much about hypervisors and real time, then
no need to take my opinion too serious.

But one can look at the market.

VMWare officially talks about real-time:

https://docs.vmware.com/en/VMware-Edge-Compute-Stack/1.0/ecs-enterprise-edge-ref-arch/GUID-99972F62-3FAE-42B2-A850-66402274B616.html

And a whole bunch of special type 1 hypervisors for real time usage exist:

Wind River Helix -
https://www.windriver.com/solutions/learning/hypervisor and
https://www.windriver.com/products/helix

RTS - https://www.real-time-systems.com/

Acontis - https://www.acontis.com/en/acontis-hypervisor.html

Some people besides me seems to think that VM and real time is possible.

Arne

In article <u3t0se$5jt$6@news.misty.com>,
Johnny Billquist <bqt@softjar.se> wrote:
>On 2023-05-13 02:16, Arne Vajhøj wrote:
>> On 5/12/2023 1:30 PM, Simon Clubley wrote:
>>> On 2023-05-12, Dave Froble <davef@tsoft-inc.com> wrote:
>>>> On 5/12/2023 8:14 AM, Simon Clubley wrote:
>>>>> That's going to make for some "interesting" real-time program
>>>>> behaviour... :-)
>>>>
>>>> Do you think any serious real time programmer will run a real time
>>>> task inside a
>>>> VM?  I'm not a real time programmer, and I'd still not do that.
>>>
>>> As well as traditional real-time stuff (which I agree with you about
>>> BTW),
>>
>> I would not want to do it on a type 2 hypervisor - there must be
>> cases where what is happening on the host OS impact the performance
>> of the guest OS.
>>
>> But with a type 1 hypervisor and no over allocation of resources -
>> would it be worse than running on bare metal?
>
>For sure. You have no guarantee that you will get the CPU cycles when
>you need them. No matter what kind of hypervisor we're talking about,
>there is overhead in the host that can affect things way more than what
>might happen on bare metal.

There are many issues at play when we talk about virtualization
and soft real-time systems. What you're describing has to do
with scheduling of guest tasks, and best-in-class techniques can
mostly deal with that (e.g., schedulers like Tableau or applying
an EDF scheduler to VCPU management).

But even without over-commit, there are other aspects of the
overall system configuration that can affect performance. For
example, there is overhead in managing the guest's physical
address space: while current x86 architectures can support a set
of second-level page tables for this, there is still the issue
of needing to walk those tables, the pressure that puts on the
equivalent of the TLB, etc. Not to mention that the real TLB
for virtual addresses (both guest and host) will be put under
pressure due to both the host and other guest tasks (even a
non-overcommitted type-1 hypervisor may move tasks around
between physical host CPUs). And if you're using nested virt
or shadow-paging...well, good luck.

And separately there are issues of IO: in a type-1 scenario, IO
is offloaded to another VM (or even a separate system) on behalf
of a guest; that introduces non-determinate latency and
noisy-neighbor problems without careful construction. Even with
SR-IOV and passthru techniques for hypervisor-bypass, there are
classes of IO that must go through the host, and these can take
arbitrarily long. What do you do when the `outb` to your
virtualized console UART blocks? There are some techniques to
amortize the overhead of these events (posted IOs, for example)
but they are not perfect.

With careful construction, one can side-step most of the cost
and get latencies down to within a couple of percentage points
of running on bare-metal, but there _is_ overhead, and in many
scenarios, it can be unbounded, even if in practice it is
relatively small.

- Dan C.

Den 2023-05-15 kl. 16:09, skrev Arne Vajhøj:
> On 5/15/2023 9:45 AM, Arne Vajhøj wrote:
>> On 5/15/2023 8:53 AM, Johnny Billquist wrote:
>>> On 2023-05-15 14:36, Arne Vajhøj wrote:
>>>> On 5/15/2023 8:20 AM, Simon Clubley wrote:
>>>>> On 2023-05-12, Arne Vajhøj <arne@vajhoej.dk> wrote:
>>>>>> On 5/12/2023 1:30 PM, Simon Clubley wrote:
>>>>>>> On 2023-05-12, Dave Froble <davef@tsoft-inc.com> wrote:
>>>>>>>> On 5/12/2023 8:14 AM, Simon Clubley wrote:
>>>>>>>>> That's going to make for some "interesting" real-time program
>>>>>>>>> behaviour... :-)
>>>>>>>>
>>>>>>>> Do you think any serious real time programmer will run a real time
>>>>>>>> task inside a
>>>>>>>> VM?  I'm not a real time programmer, and I'd still not do that.
>>>>>>>
>>>>>>> As well as traditional real-time stuff (which I agree with you about
>>>>>>> BTW),
>>>>>>
>>>>>> I would not want to do it on a type 2 hypervisor - there must be
>>>>>> cases where what is happening on the host OS impact the performance
>>>>>> of the guest OS.
>>>>>>
>>>>>> But with a type 1 hypervisor and no over allocation of resources -
>>>>>> would it be worse than running on bare metal?
>>>>>
>>>>> A type 1 hypervisor is still a software layer between the hardware and
>>>>> the RTOS. It would have to _guarantee_ that it would _never_ get in the
>>>>> way of the timing guarantees that a program running under a RTOS needs.
>>>>
>>>> It is software that is active between the VMS and the HW.
>>>>
>>>> But it is not obvious to me where any delay would come from.
>>>>
>>>> If we talk type 2 then it is easy to understand. You have a host
>>>> OS running 200 processes - 1 VM and 199 other stuff, and 4 CPU
>>>> so those 200 processes get scheduled on those 4 CPU's, and certain
>>>> interrupts may happen in the host OS. That VM will have
>>>> a problem providing real time capabilities.
>>>>
>>>> But in the type 1 with no over allocation of resources then
>>>> what will cause any delays? The VM got its own CPU or CPU's
>>>> that no other VM want, so I would expect the hypervisor to
>>>> keep them permanently allocated to the VM. And I don't
>>>> see the need for any interrupts in the hypervisor either.
>>>
>>> Can you guarantee that no interrupts happens on the CPUs the hypervisor
>>> is using? I would suspect "no", which means you do not have full control.
>>
>> What would those interrupts do?
>>
>> A type 1 hypervisor does not do that much.
>>
>> It has not very much in common with a type 2 hypervisor. I see it
>> as more of a software equivalent of system partitioning (DEC Galaxy,
>> IBM LPAR).
>
> As I don't know much about hypervisors and real time, then
> no need to take my opinion too serious.
>
> But one can look at the market.
>
> VMWare officially talks about real-time:
>
> https://docs.vmware.com/en/VMware-Edge-Compute-Stack/1.0/ecs-enterprise-edge-ref-arch/GUID-99972F62-3FAE-42B2-A850-66402274B616.html
>
> And a whole bunch of special type 1 hypervisors for real time usage exist:
>
> Wind River Helix - https://www.windriver.com/solutions/learning/hypervisor
> and https://www.windriver.com/products/helix
>
> RTS - https://www.real-time-systems.com/
>
> Acontis - https://www.acontis.com/en/acontis-hypervisor.html
>
> Some people besides me seems to think that VM and real time is possible.
>
> Arne
>
>

And I use to say that "real time" is worthless if you do not
specify what you mean with it. What are your expections?

For a user standing with some I/O devices like a hand-scanner
and a label printer, "real time" can be "within 1-2 seconds".

For a system talkning to some machinery, "real time" can be a
few 10's of milliseconds or even less.

On Monday, May 15, 2023 at 10:16:42 AM UTC-4, Jan-Erik Söderholm wrote:
> Den 2023-05-15 kl. 16:09, skrev Arne Vajhøj:
> > On 5/15/2023 9:45 AM, Arne Vajhøj wrote:
> >> On 5/15/2023 8:53 AM, Johnny Billquist wrote:
> >>> On 2023-05-15 14:36, Arne Vajhøj wrote:
> >>>> On 5/15/2023 8:20 AM, Simon Clubley wrote:
> >>>>> On 2023-05-12, Arne Vajhøj <ar...@vajhoej.dk> wrote:
> >>>>>> On 5/12/2023 1:30 PM, Simon Clubley wrote:
> >>>>>>> On 2023-05-12, Dave Froble <da...@tsoft-inc.com> wrote:
> >>>>>>>> On 5/12/2023 8:14 AM, Simon Clubley wrote:
> >>>>>>>>> That's going to make for some "interesting" real-time program
> >>>>>>>>> behaviour... :-)
> >>>>>>>>
> >>>>>>>> Do you think any serious real time programmer will run a real time
> >>>>>>>> task inside a
> >>>>>>>> VM? I'm not a real time programmer, and I'd still not do that.
> >>>>>>>
> >>>>>>> As well as traditional real-time stuff (which I agree with you about
> >>>>>>> BTW),
> >>>>>>
> >>>>>> I would not want to do it on a type 2 hypervisor - there must be
> >>>>>> cases where what is happening on the host OS impact the performance
> >>>>>> of the guest OS.
> >>>>>>
> >>>>>> But with a type 1 hypervisor and no over allocation of resources -
> >>>>>> would it be worse than running on bare metal?
> >>>>>
> >>>>> A type 1 hypervisor is still a software layer between the hardware and
> >>>>> the RTOS. It would have to _guarantee_ that it would _never_ get in the
> >>>>> way of the timing guarantees that a program running under a RTOS needs.
> >>>>
> >>>> It is software that is active between the VMS and the HW.
> >>>>
> >>>> But it is not obvious to me where any delay would come from.
> >>>>
> >>>> If we talk type 2 then it is easy to understand. You have a host
> >>>> OS running 200 processes - 1 VM and 199 other stuff, and 4 CPU
> >>>> so those 200 processes get scheduled on those 4 CPU's, and certain
> >>>> interrupts may happen in the host OS. That VM will have
> >>>> a problem providing real time capabilities.
> >>>>
> >>>> But in the type 1 with no over allocation of resources then
> >>>> what will cause any delays? The VM got its own CPU or CPU's
> >>>> that no other VM want, so I would expect the hypervisor to
> >>>> keep them permanently allocated to the VM. And I don't
> >>>> see the need for any interrupts in the hypervisor either.
> >>>
> >>> Can you guarantee that no interrupts happens on the CPUs the hypervisor
> >>> is using? I would suspect "no", which means you do not have full control.
> >>
> >> What would those interrupts do?
> >>
> >> A type 1 hypervisor does not do that much.
> >>
> >> It has not very much in common with a type 2 hypervisor. I see it
> >> as more of a software equivalent of system partitioning (DEC Galaxy,
> >> IBM LPAR).
> >
> > As I don't know much about hypervisors and real time, then
> > no need to take my opinion too serious.
> >
> > But one can look at the market.
> >
> > VMWare officially talks about real-time:
> >
> > https://docs.vmware.com/en/VMware-Edge-Compute-Stack/1.0/ecs-enterprise-edge-ref-arch/GUID-99972F62-3FAE-42B2-A850-66402274B616.html
> >
> > And a whole bunch of special type 1 hypervisors for real time usage exist:
> >
> > Wind River Helix - https://www.windriver.com/solutions/learning/hypervisor
> > and https://www.windriver.com/products/helix
> >
> > RTS - https://www.real-time-systems.com/
> >
> > Acontis - https://www.acontis.com/en/acontis-hypervisor.html
> >
> > Some people besides me seems to think that VM and real time is possible..
> >
> > Arne
> >
> >
> And I use to say that "real time" is worthless if you do not
> specify what you mean with it. What are your expections?
>
> For a user standing with some I/O devices like a hand-scanner
> and a label printer, "real time" can be "within 1-2 seconds".
>
> For a system talkning to some machinery, "real time" can be a
> few 10's of milliseconds or even less.
Of course you are right. My test did not prove anything. That was really stupid.

Den 2023-05-15 kl. 16:33, skrev Clair Grant:
> On Monday, May 15, 2023 at 10:16:42 AM UTC-4, Jan-Erik Söderholm wrote:
>> Den 2023-05-15 kl. 16:09, skrev Arne Vajhøj:
>>> On 5/15/2023 9:45 AM, Arne Vajhøj wrote:
>>>> On 5/15/2023 8:53 AM, Johnny Billquist wrote:
>>>>> On 2023-05-15 14:36, Arne Vajhøj wrote:
>>>>>> On 5/15/2023 8:20 AM, Simon Clubley wrote:
>>>>>>> On 2023-05-12, Arne Vajhøj <ar...@vajhoej.dk> wrote:
>>>>>>>> On 5/12/2023 1:30 PM, Simon Clubley wrote:
>>>>>>>>> On 2023-05-12, Dave Froble <da...@tsoft-inc.com> wrote:
>>>>>>>>>> On 5/12/2023 8:14 AM, Simon Clubley wrote:
>>>>>>>>>>> That's going to make for some "interesting" real-time program
>>>>>>>>>>> behaviour... :-)
>>>>>>>>>>
>>>>>>>>>> Do you think any serious real time programmer will run a real time
>>>>>>>>>> task inside a
>>>>>>>>>> VM? I'm not a real time programmer, and I'd still not do that.
>>>>>>>>>
>>>>>>>>> As well as traditional real-time stuff (which I agree with you about
>>>>>>>>> BTW),
>>>>>>>>
>>>>>>>> I would not want to do it on a type 2 hypervisor - there must be
>>>>>>>> cases where what is happening on the host OS impact the performance
>>>>>>>> of the guest OS.
>>>>>>>>
>>>>>>>> But with a type 1 hypervisor and no over allocation of resources -
>>>>>>>> would it be worse than running on bare metal?
>>>>>>>
>>>>>>> A type 1 hypervisor is still a software layer between the hardware and
>>>>>>> the RTOS. It would have to _guarantee_ that it would _never_ get in the
>>>>>>> way of the timing guarantees that a program running under a RTOS needs.
>>>>>>
>>>>>> It is software that is active between the VMS and the HW.
>>>>>>
>>>>>> But it is not obvious to me where any delay would come from.
>>>>>>
>>>>>> If we talk type 2 then it is easy to understand. You have a host
>>>>>> OS running 200 processes - 1 VM and 199 other stuff, and 4 CPU
>>>>>> so those 200 processes get scheduled on those 4 CPU's, and certain
>>>>>> interrupts may happen in the host OS. That VM will have
>>>>>> a problem providing real time capabilities.
>>>>>>
>>>>>> But in the type 1 with no over allocation of resources then
>>>>>> what will cause any delays? The VM got its own CPU or CPU's
>>>>>> that no other VM want, so I would expect the hypervisor to
>>>>>> keep them permanently allocated to the VM. And I don't
>>>>>> see the need for any interrupts in the hypervisor either.
>>>>>
>>>>> Can you guarantee that no interrupts happens on the CPUs the hypervisor
>>>>> is using? I would suspect "no", which means you do not have full control.
>>>>
>>>> What would those interrupts do?
>>>>
>>>> A type 1 hypervisor does not do that much.
>>>>
>>>> It has not very much in common with a type 2 hypervisor. I see it
>>>> as more of a software equivalent of system partitioning (DEC Galaxy,
>>>> IBM LPAR).
>>>
>>> As I don't know much about hypervisors and real time, then
>>> no need to take my opinion too serious.
>>>
>>> But one can look at the market.
>>>
>>> VMWare officially talks about real-time:
>>>
>>> https://docs.vmware.com/en/VMware-Edge-Compute-Stack/1.0/ecs-enterprise-edge-ref-arch/GUID-99972F62-3FAE-42B2-A850-66402274B616.html
>>>
>>> And a whole bunch of special type 1 hypervisors for real time usage exist:
>>>
>>> Wind River Helix - https://www.windriver.com/solutions/learning/hypervisor
>>> and https://www.windriver.com/products/helix
>>>
>>> RTS - https://www.real-time-systems.com/
>>>
>>> Acontis - https://www.acontis.com/en/acontis-hypervisor.html
>>>
>>> Some people besides me seems to think that VM and real time is possible.
>>>
>>> Arne
>>>
>>>
>> And I use to say that "real time" is worthless if you do not
>> specify what you mean with it. What are your expections?
>>
>> For a user standing with some I/O devices like a hand-scanner
>> and a label printer, "real time" can be "within 1-2 seconds".
>>
>> For a system talkning to some machinery, "real time" can be a
>> few 10's of milliseconds or even less.
> Of course you are right. My test did not prove anything. That was really stupid.

Maybe not stupid, but I'd say that it missed an important point. :-)

And I guess you are replying to my note about using the same system
to measure the systems own clock. It will always be "spot-on".
Not really my post about real-time...

On Monday, May 15, 2023 at 8:45:00 AM UTC-4, Johnny Billquist wrote:
> It used to be a *very* reliable clock source. Power grids used to
> compensate for drift because of load, so that over 24 hours, you have a
> very exact number of peaks on the power. All type of clocks were
> designed with this clock source. Sadly, I think power grids no longer
> are caring as much about the accuracy of the frequency.

You can see the (near) real-time frequency deviation as well as the
cumulative clock drift that it causes here:
https://www.swissgrid.ch/en/home/operation/grid-data/current-data.html#frequency

5 or years ago, a Kosovo / Serbia dispute caused much larger drift -
on the order of 6 minutes. Do a web search for 'kosovo makes clocks
run slow'

On 2023-05-15 15:35, Arne Vajhøj wrote:
> On 5/15/2023 8:51 AM, Johnny Billquist wrote:
>> On 2023-05-15 14:03, Arne Vajhøj wrote:
>>> On 5/15/2023 6:17 AM, Johnny Billquist wrote:
>>>> On 2023-05-13 02:16, Arne Vajhøj wrote:
>>>>> On 5/12/2023 1:30 PM, Simon Clubley wrote:
>>>>>> On 2023-05-12, Dave Froble <davef@tsoft-inc.com> wrote:
>>>>>>> On 5/12/2023 8:14 AM, Simon Clubley wrote:
>>>>>>>> That's going to make for some "interesting" real-time program
>>>>>>>> behaviour... :-)
>>>>>>>
>>>>>>> Do you think any serious real time programmer will run a real
>>>>>>> time task inside a
>>>>>>> VM?  I'm not a real time programmer, and I'd still not do that.
>>>>>>
>>>>>> As well as traditional real-time stuff (which I agree with you
>>>>>> about BTW),
>>>>>
>>>>> I would not want to do it on a type 2 hypervisor - there must be
>>>>> cases where what is happening on the host OS impact the performance
>>>>> of the guest OS.
>>>>>
>>>>> But with a type 1 hypervisor and no over allocation of resources -
>>>>> would it be worse than running on bare metal?
>>>>
>>>> For sure. You have no guarantee that you will get the CPU cycles
>>>> when you need them. No matter what kind of hypervisor we're talking
>>>> about, there is overhead in the host that can affect things way more
>>>> than what might happen on bare metal.
>>>
>>> What host? A type 1 hypervisor does not run a host OS.
>>
>> Um? What is the hypervisor then?
>
> The hypervisor software like ESXi.
>
> It is not running on top of a host OS like a type 2 (VirtualBox etc.) does.

I never assumed you had a full OS. However, they Hypervisor is
intercepting access to hardware, and is responsible for the actual
interaction with the hardware, adding a layer between the guest OS and
the hardware. Which means that the hypvervisor is in the end doing
interrupt handling, for example. Which means that the CPU might be doing
things that suspends the execution for the hypervisor outside the
control of the hypervisor itself, and further more makes things like
interrupts to the guest OS happen at some random later time, including
clock interrupts, which are what affects things like real-time reponse
times.

>>> And why would the CPU not be available, when there is no
>>> over allocation of resources? It seems pretty silly of
>>> the hypervisor to not want to give the CPU if no other
>>> VM can get it.
>>
>> Any kind of hypervisor means we're talking about virtualization. That
>> means you have real hardware which exists outside of this
>> virtualization. And that hardware can generate interrupts and demand
>> service, which then forces the hypervisor to wait before it can
>> actually get any cycles. Outside the control of the hypervisor...
>
> Maybe. But what would that be?

Primarily I would be worried about interrupts, which introduce unknown
amout of delay.

But yes, for things like memory, if it is fixed allocated to VM
instances, and always available, then another potential source of delays
are at least not there.

Depending on what the hypervisor does, you might still also have issues
like page table caches, and normal memory caches, which will have rather
different behaviors compared to plain hardware. Caches are a significant
factor is performance these days.

Johnny

On 2023-05-15 16:12, Dan Cross wrote:
> In article <u3t0se$5jt$6@news.misty.com>,
> Johnny Billquist <bqt@softjar.se> wrote:
>> On 2023-05-13 02:16, Arne Vajhøj wrote:
>>> On 5/12/2023 1:30 PM, Simon Clubley wrote:
>>>> On 2023-05-12, Dave Froble <davef@tsoft-inc.com> wrote:
>>>>> On 5/12/2023 8:14 AM, Simon Clubley wrote:
>>>>>> That's going to make for some "interesting" real-time program
>>>>>> behaviour... :-)
>>>>>
>>>>> Do you think any serious real time programmer will run a real time
>>>>> task inside a
>>>>> VM?  I'm not a real time programmer, and I'd still not do that.
>>>>
>>>> As well as traditional real-time stuff (which I agree with you about
>>>> BTW),
>>>
>>> I would not want to do it on a type 2 hypervisor - there must be
>>> cases where what is happening on the host OS impact the performance
>>> of the guest OS.
>>>
>>> But with a type 1 hypervisor and no over allocation of resources -
>>> would it be worse than running on bare metal?
>>
>> For sure. You have no guarantee that you will get the CPU cycles when
>> you need them. No matter what kind of hypervisor we're talking about,
>> there is overhead in the host that can affect things way more than what
>> might happen on bare metal.
>
> There are many issues at play when we talk about virtualization
> and soft real-time systems. What you're describing has to do
> with scheduling of guest tasks, and best-in-class techniques can
> mostly deal with that (e.g., schedulers like Tableau or applying
> an EDF scheduler to VCPU management).
>
> But even without over-commit, there are other aspects of the
> overall system configuration that can affect performance. For
> example, there is overhead in managing the guest's physical
> address space: while current x86 architectures can support a set
> of second-level page tables for this, there is still the issue
> of needing to walk those tables, the pressure that puts on the
> equivalent of the TLB, etc. Not to mention that the real TLB
> for virtual addresses (both guest and host) will be put under
> pressure due to both the host and other guest tasks (even a
> non-overcommitted type-1 hypervisor may move tasks around
> between physical host CPUs). And if you're using nested virt
> or shadow-paging...well, good luck.
>
> And separately there are issues of IO: in a type-1 scenario, IO
> is offloaded to another VM (or even a separate system) on behalf
> of a guest; that introduces non-determinate latency and
> noisy-neighbor problems without careful construction. Even with
> SR-IOV and passthru techniques for hypervisor-bypass, there are
> classes of IO that must go through the host, and these can take
> arbitrarily long. What do you do when the `outb` to your
> virtualized console UART blocks? There are some techniques to
> amortize the overhead of these events (posted IOs, for example)
> but they are not perfect.
>
> With careful construction, one can side-step most of the cost
> and get latencies down to within a couple of percentage points
> of running on bare-metal, but there _is_ overhead, and in many
> scenarios, it can be unbounded, even if in practice it is
> relatively small.

Thanks for the exhaustive answer. I don't think I need to add much more
to this.

Johnny