In article <u3tr77$6e9$1@news.misty.com>,
Johnny Billquist <bqt@softjar.se> wrote:
>On 2023-05-15 15:35, Arne Vajhøj wrote:
>[snip]
>>>> What host? A type 1 hypervisor does not run a host OS.
>>>
>>> Um? What is the hypervisor then?
>>
>> The hypervisor software like ESXi.
>>
>> It is not running on top of a host OS like a type 2 (VirtualBox etc.) does.
>
>I never assumed you had a full OS. However, they Hypervisor is
>intercepting access to hardware, and is responsible for the actual
>interaction with the hardware, adding a layer between the guest OS and
>the hardware. Which means that the hypvervisor is in the end doing
>interrupt handling, for example.

Yes and no. For many things, the guest necessarily traps back
into the host environment for handling (indeed, on x86, the
`CPUID` instruction forces a mandatory VM exit). However, there
are techniques whereby one can bypass the hypervisor entirely,
including for interrupt injection/delivery. For example, again
on x86, the "posted interrupt" functionality exposed by both VMX
on Intel and SVM on AMD can be used in conjunction with SR-IOV
and the IOMMU on both platforms to allow virtual functions on
high-speed devices to deliver interrupts directly into a guest,
provided one is using PCI pass-through. Similarly, IPI delivery
on x86 can use this same mechanism (though IPI generation, which
involves a write to ICR on the LAPIC, often requires hypervisor
intervention, even when using LAPIC virtualization: AMD has a
technique whereby this can _mostly_ be bypassed, but it is
buggy and not recommended for use).

And of course, the hypervisor must necessarily own the PCI
configuration space, and virtualize access to it, even if it
provides pass-thru to some functions.

Sadly, there's little support for virtual timers right now,
relating to another part of this discussion, so the local APIC
timer (for example) is usually multiplexed and requires a VM
exit to acknowledge the interrupt in the host and inject into
the guest.

>Which means that the CPU might be doing
>things that suspends the execution for the hypervisor outside the
>control of the hypervisor itself, and further more makes things like
>interrupts to the guest OS happen at some random later time, including
>clock interrupts, which are what affects things like real-time reponse
>times.

The list of things that a type-1 hypervisor must own is large,
but in general I agree with you: it is de facto the kernel
running on the bare hardware, even if it does not provide many
of the services usually associated with a traditional operating
system (e.g., a filesystem or a user-visible process model).

>>>> And why would the CPU not be available, when there is no
>>>> over allocation of resources? It seems pretty silly of
>>>> the hypervisor to not want to give the CPU if no other
>>>> VM can get it.
>>>
>>> Any kind of hypervisor means we're talking about virtualization. That
>>> means you have real hardware which exists outside of this
>>> virtualization. And that hardware can generate interrupts and demand
>>> service, which then forces the hypervisor to wait before it can
>>> actually get any cycles. Outside the control of the hypervisor...
>>
>> Maybe. But what would that be?
>
>Primarily I would be worried about interrupts, which introduce unknown
>amout of delay.

I would generalize that to virtualized IO, not just interrupts.
Again, in a type-1 hypervisor, handling of IO requests is often
delegated to a root VM (Dom0 in Xen terms), which will introduce
its own latency issues. "Disks" as seen by a VM may even be
synthesized by software and moved off-node.

>But yes, for things like memory, if it is fixed allocated to VM
>instances, and always available, then another potential source of delays
>are at least not there.

Again, see my earlier post: you still have issues with
management of the nested page tables, including cache and
TLB pressure, and a TLB miss is substantially more expensive:
conceptually, one must walk the second-level page table for
_every_ physical access. To do a bog-standard 4-level page
walk, that can 16 memory accesses.

>Depending on what the hypervisor does, you might still also have issues
>like page table caches, and normal memory caches, which will have rather
>different behaviors compared to plain hardware. Caches are a significant
>factor is performance these days.

Yes. See my earlier post in this thread.

- Dan C.

On 5/15/2023 1:47 PM, Johnny Billquist wrote:
> On 2023-05-15 15:35, Arne Vajhøj wrote:
>> On 5/15/2023 8:51 AM, Johnny Billquist wrote:
>>> On 2023-05-15 14:03, Arne Vajhøj wrote:
>>>> On 5/15/2023 6:17 AM, Johnny Billquist wrote:
>>>>> On 2023-05-13 02:16, Arne Vajhøj wrote:
>>>>>> On 5/12/2023 1:30 PM, Simon Clubley wrote:
>>>>>>> On 2023-05-12, Dave Froble <davef@tsoft-inc.com> wrote:
>>>>>>>> On 5/12/2023 8:14 AM, Simon Clubley wrote:
>>>>>>>>> That's going to make for some "interesting" real-time program
>>>>>>>>> behaviour... :-)
>>>>>>>>
>>>>>>>> Do you think any serious real time programmer will run a real
>>>>>>>> time task inside a
>>>>>>>> VM?  I'm not a real time programmer, and I'd still not do that.
>>>>>>>
>>>>>>> As well as traditional real-time stuff (which I agree with you
>>>>>>> about BTW),
>>>>>>
>>>>>> I would not want to do it on a type 2 hypervisor - there must be
>>>>>> cases where what is happening on the host OS impact the performance
>>>>>> of the guest OS.
>>>>>>
>>>>>> But with a type 1 hypervisor and no over allocation of resources -
>>>>>> would it be worse than running on bare metal?
>>>>>
>>>>> For sure. You have no guarantee that you will get the CPU cycles
>>>>> when you need them. No matter what kind of hypervisor we're talking
>>>>> about, there is overhead in the host that can affect things way
>>>>> more than what might happen on bare metal.
>>>>
>>>> What host? A type 1 hypervisor does not run a host OS.
>>>
>>> Um? What is the hypervisor then?
>>
>> The hypervisor software like ESXi.
>>
>> It is not running on top of a host OS like a type 2 (VirtualBox etc.)
>> does.
>
> I never assumed you had a full OS. However, they Hypervisor is
> intercepting access to hardware, and is responsible for the actual
> interaction with the hardware, adding a layer between the guest OS and
> the hardware.

The hypervisor is there.

But no host OS.

> Which means that the hypvervisor is in the end doing
> interrupt handling, for example. Which means that the CPU might be doing
> things that suspends the execution for the hypervisor outside the
> control of the hypervisor itself,

If the HW interrupts the CPU assigned to the VM and that interrupt
is not intended for the VM and it actually require some processing,
then it will steal CPU time from the VM.

But because the hypervisor does so little it is not obvious to
me what that would be.

> and further more makes things like
> interrupts to the guest OS happen at some random later time, including
> clock interrupts, which are what affects things like real-time reponse
> times.

Why would that happen at a random later time?

If the hypervisor gets the interrupt wouldn't it pass
it on to the guest OS ASAP instead of queuing it up?

>>>> And why would the CPU not be available, when there is no
>>>> over allocation of resources? It seems pretty silly of
>>>> the hypervisor to not want to give the CPU if no other
>>>> VM can get it.
>>>
>>> Any kind of hypervisor means we're talking about virtualization. That
>>> means you have real hardware which exists outside of this
>>> virtualization. And that hardware can generate interrupts and demand
>>> service, which then forces the hypervisor to wait before it can
>>> actually get any cycles. Outside the control of the hypervisor...
>>
>> Maybe. But what would that be?
>
> Primarily I would be worried about interrupts, which introduce unknown
> amout of delay.
>
> But yes, for things like memory, if it is fixed allocated to VM
> instances, and always available, then another potential source of delays
> are at least not there.
>
> Depending on what the hypervisor does, you might still also have issues
> like page table caches, and normal memory caches, which will have rather
> different behaviors compared to plain hardware. Caches are a significant
> factor is performance these days.

Yes.

But if the cache is per core and the VM get assigned not only
a specific number of cores but a specific set of cores, then
it is still predictable.

Only shared L3 cache can give some uncertainty.

And we are in ns space not us space here.

Arne

On 5/15/2023 10:09 AM, Arne Vajhøj wrote:
> As I don't know much about hypervisors and real time, then
> no need to take my opinion too serious.
>
> But one can look at the market.
>
> VMWare officially talks about real-time:
>
> https://docs.vmware.com/en/VMware-Edge-Compute-Stack/1.0/ecs-enterprise-edge-ref-arch/GUID-99972F62-3FAE-42B2-A850-66402274B616.html
>
> And a whole bunch of special type 1 hypervisors for real time usage exist:
>
> Wind River Helix -
> https://www.windriver.com/solutions/learning/hypervisor and
> https://www.windriver.com/products/helix
>
> RTS - https://www.real-time-systems.com/
>
> Acontis - https://www.acontis.com/en/acontis-hypervisor.html

QNX -
https://blackberry.qnx.com/content/dam/qnx/products/hypervisor/qnx-hypervisor-gem-product-brief-v5.pdf

Arne

On Monday, May 15, 2023 at 10:33:11 AM UTC-4, Clair Grant wrote:
> Of course you are right. My test did not prove anything. That was really stupid.

I resurrected a 30 year old Fortran program that did an NTP query to a server
and reported the time difference. I stopped the local NTP server (and no DTSS
running either) and used a command procedure to query every 10 minutes for
18 hours. There was no drift seen, so VirtualBox was apparently keeping the
clock interrupt stable (or my laptop has a very good clock).

In article <8cba8b92-ddbb-4f30-8e29-9563a9c7224dn@googlegroups.com>,
David Jones <osuvman50@gmail.com> wrote:
>On Monday, May 15, 2023 at 10:33:11 AM UTC-4, Clair Grant wrote:
>> Of course you are right. My test did not prove anything. That was really stupid.
>
>I resurrected a 30 year old Fortran program that did an NTP query to a server
>and reported the time difference. I stopped the local NTP server (and no DTSS
>running either) and used a command procedure to query every 10 minutes for
>18 hours. There was no drift seen, so VirtualBox was apparently keeping the
>clock interrupt stable (or my laptop has a very good clock).

In fairness to your laptop, it probably has a very good clock.
Drift of the kind discussed in this thread would take place over
days or weeks, not just hours.

- Dan C.

In article <u3tv4e$34ma3$1@dont-email.me>,
Arne Vajhøj <arne@vajhoej.dk> wrote:
>On 5/15/2023 1:47 PM, Johnny Billquist wrote:
>> On 2023-05-15 15:35, Arne Vajhøj wrote:
>>> On 5/15/2023 8:51 AM, Johnny Billquist wrote:
>>>> Um? What is the hypervisor then?
>>>
>>> The hypervisor software like ESXi.
>>>
>>> It is not running on top of a host OS like a type 2 (VirtualBox etc.)
>>> does.
>>
>> I never assumed you had a full OS. However, they Hypervisor is
>> intercepting access to hardware, and is responsible for the actual
>> interaction with the hardware, adding a layer between the guest OS and
>> the hardware.
>
>The hypervisor is there.
>
>But no host OS.

For all intents and purposes, the hypervisor _is_ the host OS.
Some software component must drive the hardware, run VCPUs, etc:
that's the hypervisor.

>> Which means that the hypvervisor is in the end doing
>> interrupt handling, for example. Which means that the CPU might be doing
>> things that suspends the execution for the hypervisor outside the
>> control of the hypervisor itself,
>
>If the HW interrupts the CPU assigned to the VM and that interrupt
>is not intended for the VM and it actually require some processing,
>then it will steal CPU time from the VM.
>
>But because the hypervisor does so little it is not obvious to
>me what that would be.

Where do you think the device models that are exposed to the
guest are implemented?

>> and further more makes things like
>> interrupts to the guest OS happen at some random later time, including
>> clock interrupts, which are what affects things like real-time reponse
>> times.
>
>Why would that happen at a random later time?
>
>If the hypervisor gets the interrupt wouldn't it pass
>it on to the guest OS ASAP instead of queuing it up?

See above. Where do you think that the device models exposed to
a guest are implemented? What do you think controls IO, or e.g.
PCIe configuration space? What do you think emulates legacy IO
devices like (on the PC) the PS/2 keyboard controller, PIC, PIT,
HPET, RTC, UART, etc?

IO in hypervisors does not, in general, go directly to a device.

The arbitrary distinction between "Type-1" and "Type-2" style
hypervisors is also a bit illusory among those who work on them;
like all approximations, it is just that.

>> Primarily I would be worried about interrupts, which introduce unknown
>> amout of delay.
>>
>> But yes, for things like memory, if it is fixed allocated to VM
>> instances, and always available, then another potential source of delays
>> are at least not there.
>>
>> Depending on what the hypervisor does, you might still also have issues
>> like page table caches, and normal memory caches, which will have rather
>> different behaviors compared to plain hardware. Caches are a significant
>> factor is performance these days.
>
>Yes.
>
>But if the cache is per core and the VM get assigned not only
>a specific number of cores but a specific set of cores, then
>it is still predictable.

Any VM exit, such as mandatory exits for handling e.g. the
`CPUID` instruction, will necessarily conflict with the guest
cache (since the L1 and L2 I and D caches are shared with the
guest).

- Dan C.

On Monday, May 15, 2023 at 4:37:47 PM UTC-4, Dan Cross wrote:
> In fairness to your laptop, it probably has a very good clock.
> Drift of the kind discussed in this thread would take place over
> days or weeks, not just hours.
>
> - Dan C.

For connecting to my Raspberry Pi's, I bought a cheap PCF8523 RTC
($7) and a better DS3231 RTC ($25), the better one being accurate to
3 seconds per month. A drift of 100ms per day can be detected over
12-18 hours or less. What incentive do they have to put a super
accurate clock in a computer they expect to be on the network and
talking to time servers?

Why did I buy an RTC for my Pi? I'm doing data collection with it and
Pi's don't have an RTC on the card. If it boots after a power outage,
there is a window where timestamps will be wrong because it starts
the system time at the last checkpoint and runs with that until it can
get a correct time from either an NTP source or an external clock. I
have a cron job that syncs the RTC with an NTP source every 10
days.

In article <024ad569-cc96-4908-be2d-55f6f7b1ac6an@googlegroups.com>,
David Jones <osuvman50@gmail.com> wrote:
>On Monday, May 15, 2023 at 4:37:47 PM UTC-4, Dan Cross wrote:
>> In fairness to your laptop, it probably has a very good clock.
>> Drift of the kind discussed in this thread would take place over
>> days or weeks, not just hours.
>
>For connecting to my Raspberry Pi's, I bought a cheap PCF8523 RTC
>($7) and a better DS3231 RTC ($25), the better one being accurate to
>3 seconds per month. A drift of 100ms per day can be detected over
>12-18 hours or less. What incentive do they have to put a super
>accurate clock in a computer they expect to be on the network and
>talking to time servers?

It would be interesting to stop any NNTP service you may be
running (e.g., on a host) and see if you observe that.

>Why did I buy an RTC for my Pi? I'm doing data collection with it and
>Pi's don't have an RTC on the card. If it boots after a power outage,
>there is a window where timestamps will be wrong because it starts
>the system time at the last checkpoint and runs with that until it can
>get a correct time from either an NTP source or an external clock. I
>have a cron job that syncs the RTC with an NTP source every 10
>days.

Nifty.

- Dan C.

> It would be interesting to stop any NNTP service you may be
> running (e.g., on a host) and see if you observe that.

Pretty sure any usenet server is making timekeeping a lot WORSE. :)

De

As another datapoint, I'm running Virtual Box on my W10 system. I have a VM
running for over 4 days doing lots of compiler testing, etc. The SHOW TIME
output is still correct (it matches my W10 clock AND my phone).

In article <3Y2dnVQzJd-UBf75nZ2dnZfqnPGdnZ2d@giganews.com>,
Dennis Boone <drb@ihatespam.msu.edu> wrote:
> > It would be interesting to stop any NNTP service you may be
> > running (e.g., on a host) and see if you observe that.
>
>Pretty sure any usenet server is making timekeeping a lot WORSE. :)

Ha! Freudian slip. :-)

- Dan C.

On 2023-05-16 17:07, John Reagan wrote:
> As another datapoint, I'm running Virtual Box on my W10 system. I have a VM
> running for over 4 days doing lots of compiler testing, etc. The SHOW TIME
> output is still correct (it matches my W10 clock AND my phone).

I would suspect/expect that your W10 system is having a synchronized
clock (they do use NTP by default), so it will not drift. And I expect
Virtual Box will be picking up time from W10 to check that it's
generating the correct amount of clock interrupts, so that would also be
accurate over time.

So I'm not at all surprised. You would have jitter on the small scale,
but in the long run, it should be just fine. That is, unless your host
OS do not have any time synchronization, in which case I would expect
the guest OS to show the same drift you would observe on the host OS.

Johnny

On 2023-05-15 19:43, terry-...@glaver.org wrote:
> On Monday, May 15, 2023 at 8:45:00 AM UTC-4, Johnny Billquist wrote:
>> It used to be a *very* reliable clock source. Power grids used to
>> compensate for drift because of load, so that over 24 hours, you have a
>> very exact number of peaks on the power. All type of clocks were
>> designed with this clock source. Sadly, I think power grids no longer
>> are caring as much about the accuracy of the frequency.
>
> You can see the (near) real-time frequency deviation as well as the
> cumulative clock drift that it causes here:
> https://www.swissgrid.ch/en/home/operation/grid-data/current-data.html#frequency

This is way cool. I hadn't seen that one. Did you know I was in
Switzerland by the way, or was that just random luck?

Johnny

Den 2023-05-16 kl. 21:23, skrev Johnny Billquist:
> On 2023-05-15 19:43, terry-...@glaver.org wrote:
>> On Monday, May 15, 2023 at 8:45:00 AM UTC-4, Johnny Billquist wrote:
>>> It used to be a *very* reliable clock source. Power grids used to
>>> compensate for drift because of load, so that over 24 hours, you have a
>>> very exact number of peaks on the power. All type of clocks were
>>> designed with this clock source. Sadly, I think power grids no longer
>>> are caring as much about the accuracy of the frequency.
>>
>> You can see the (near) real-time frequency deviation as well as the
>> cumulative clock drift that it causes here:
>> https://www.swissgrid.ch/en/home/operation/grid-data/current-data.html#frequency
>
> This is way cool. I hadn't seen that one. Did you know I was in Switzerland
> by the way, or was that just random luck?
>
>   Johnny
>

You have written that a couple of times over the years... :-)
And it was easy to find these:
http://www.softjar.se/
https://ch.linkedin.com/in/sillbit

:-)

Jan-Erik

On Monday, May 15, 2023 at 7:14:08 AM UTC-7, Dan Cross wrote:

(snip)

> And separately there are issues of IO: in a type-1 scenario, IO
> is offloaded to another VM (or even a separate system) on behalf
> of a guest; that introduces non-determinate latency and
> noisy-neighbor problems without careful construction. Even with
> SR-IOV and passthru techniques for hypervisor-bypass, there are
> classes of IO that must go through the host, and these can take
> arbitrarily long. What do you do when the `outb` to your
> virtualized console UART blocks? There are some techniques to
> amortize the overhead of these events (posted IOs, for example)
> but they are not perfect.
(snip)

and then there is NFS, where I/O can take arbitrarily long, even
without a hypervisor.

I once sold a machine that clients had NFS mounts from.
That is, hard mounts.

On Monday, May 15, 2023 at 5:15:41 AM UTC-7, Dan Cross wrote:
> In article <8b6c6dbd-5dc4-41aa...@googlegroups.com>,
> gah4 <ga...@u.washington.edu> wrote:
> >[snip]
> >In the case of virtual machines, each one will have its own CPU timer,
> >but only one TOD clock. STCK is not a privileged instruction, so programs
> >(or OS) see the host clock.

> This would seem to violate Popek and Goldberg's virtualization
> criteria; how is that squared?
IBM was either lucky or smart, in the design of S/370, except for that one.

As well as I know, it mostly wasn't a problem until Y2K testing. Who would
want a machine with the clock wrong on purpose?

It might have changed later, as not much pre ESA/390 hardware was still
running by Y2K. The favorite IBM method when a feature changes, is to put
in a control register bit that allows the new or old operation.

Wondering about some of those, I found this one:

https://www.vmware.com/files/pdf/techpaper/Timekeeping-In-VirtualMachines.pdf

which is the VMware description of how they virtualize clocks.

On 2023-05-17 11:42, gah4 wrote:
> On Monday, May 15, 2023 at 5:15:41 AM UTC-7, Dan Cross wrote:
>> In article <8b6c6dbd-5dc4-41aa...@googlegroups.com>,
>> gah4 <ga...@u.washington.edu> wrote:
>>> [snip]
>>> In the case of virtual machines, each one will have its own CPU timer,
>>> but only one TOD clock. STCK is not a privileged instruction, so programs
>>> (or OS) see the host clock.
>
>> This would seem to violate Popek and Goldberg's virtualization
>> criteria; how is that squared?
>
> IBM was either lucky or smart, in the design of S/370, except for that one.
>
> As well as I know, it mostly wasn't a problem until Y2K testing. Who would
> want a machine with the clock wrong on purpose?
>
> It might have changed later, as not much pre ESA/390 hardware was still
> running by Y2K. The favorite IBM method when a feature changes, is to put
> in a control register bit that allows the new or old operation.
>
> Wondering about some of those, I found this one:
>
> https://www.vmware.com/files/pdf/techpaper/Timekeeping-In-VirtualMachines.pdf
>
> which is the VMware description of how they virtualize clocks.

Thanks. That pretty much repeat what I've been writing in a number of
posts now. I hope this can now be laid to rest.

Johnny

In article <5d2fe9bd-f475-44df-89c7-93f61f27ee09n@googlegroups.com>,
gah4 <gah4@u.washington.edu> wrote:
>On Monday, May 15, 2023 at 5:15:41 AM UTC-7, Dan Cross wrote:
>> In article <8b6c6dbd-5dc4-41aa...@googlegroups.com>,
>> gah4 <ga...@u.washington.edu> wrote:
>> >[snip]
>> >In the case of virtual machines, each one will have its own CPU timer,
>> >but only one TOD clock. STCK is not a privileged instruction, so programs
>> >(or OS) see the host clock.
>
>> This would seem to violate Popek and Goldberg's virtualization
>> criteria; how is that squared?
>
>IBM was either lucky or smart, in the design of S/370, except for that one.
>
>As well as I know, it mostly wasn't a problem until Y2K testing. Who would
>want a machine with the clock wrong on purpose?

Who can say? Timezones have been a thing since railroads began
popping up. But I agree: it may have been considered totally
innocuous. Still, I find it odd that it wasn't considered for
virtualization particularly given that VM started life on S/360.
Then again, S/370 was 1970, and Popek and Goldberg didn't
publish until 1974.

This _can_ be pretty easily papered over by using a hardware
technique similar to VMX (which gives the VMM a fair amount of
capability to virtualize the TSC, for example), but I don't know
whether they do that or not. It stands to reason that they may.

It strikes me that setting the clock must be a privileged
operation, and it wouldn't take much for the hypervisor to save
and restore the clock on switches between VCPUs (and of course
it would have its own offsets for maintaining time in each VM
relative to the host). In that case, allowing unfettered reads
wouldn't be too much of an issue. Perhaps that is what they do.

>It might have changed later, as not much pre ESA/390 hardware was still
>running by Y2K. The favorite IBM method when a feature changes, is to put
>in a control register bit that allows the new or old operation.
>
>Wondering about some of those, I found this one:
>
>https://www.vmware.com/files/pdf/techpaper/Timekeeping-In-VirtualMachines.pdf
>
>which is the VMware description of how they virtualize clocks.

This describes most of the PC time sources, and how they are
virtualized in VMWare ESXi, but omits the important technique of
supporting enlightenments. To be clear, a hypervisor can
provide sufficiently robust implementations of most PC time
sources that an unmodified guest won't really be able to tell
the difference between the virtualized and host environments,
but at the cost of non-trivial complexity and cost (particularly
when one starts to consider, e.g., migration of VMs between
physical hosts, as is common in cloud environments). Thus, this
is one area where it is advantageous to paravirtualize; if the
OS can detect that it is running in a virtualized enviornment,
it can coordinate with the host to provide e.g., more robust and
accurate timing services. This is supported by Hyper-V, KVM,
Bhyve, and a few other x86 hypervisors; it wouldn't surprise me
if similar techniques were employed on the mainframe as well.

Incidentally, I believe that Microsoft coined the term
"enlightenment" with Hyper-V.

- Dan C.

In article <25f6569c-fa35-46de-8a5d-0a886a11e899n@googlegroups.com>,
gah4 <gah4@u.washington.edu> wrote:
>On Monday, May 15, 2023 at 7:14:08 AM UTC-7, Dan Cross wrote:
>
>(snip)
>
>> And separately there are issues of IO: in a type-1 scenario, IO
>> is offloaded to another VM (or even a separate system) on behalf
>> of a guest; that introduces non-determinate latency and
>> noisy-neighbor problems without careful construction. Even with
>> SR-IOV and passthru techniques for hypervisor-bypass, there are
>> classes of IO that must go through the host, and these can take
>> arbitrarily long. What do you do when the `outb` to your
>> virtualized console UART blocks? There are some techniques to
>> amortize the overhead of these events (posted IOs, for example)
>> but they are not perfect.
>
>(snip)
>
>and then there is NFS, where I/O can take arbitrarily long, even
>without a hypervisor.
>
>I once sold a machine that clients had NFS mounts from.
>That is, hard mounts.

The one that'll really flip your noggin is if your virtualized
storage device is backed by a file on an NFS server somewhere...

- Dan C.

On Wednesday, May 17, 2023 at 4:59:24 AM UTC-7, Johnny Billquist wrote:

(snip, I wrote)

> > Wondering about some of those, I found this one:
> > https://www.vmware.com/files/pdf/techpaper/Timekeeping-In-VirtualMachines.pdf
> > which is the VMware description of how they virtualize clocks.

> Thanks. That pretty much repeat what I've been writing in a number of
> posts now. I hope this can now be laid to rest.
Well, that one is 32 pages, which I don't think you have written so far.

A lot of detail that one might need tracking down bugs.

On Wednesday, May 17, 2023 at 6:37:32 AM UTC-7, Dan Cross wrote:
> In article <5d2fe9bd-f475-44df...@googlegroups.com>,

(snip, I wrote)

> >IBM was either lucky or smart, in the design of S/370, except for that one.

> >As well as I know, it mostly wasn't a problem until Y2K testing. Who would
> >want a machine with the clock wrong on purpose?

> Who can say? Timezones have been a thing since railroads began
> popping up. But I agree: it may have been considered totally
> innocuous. Still, I find it odd that it wasn't considered for
> virtualization particularly given that VM started life on S/360.
> Then again, S/370 was 1970, and Popek and Goldberg didn't
> publish until 1974.
OS/360, which in later versions has support for S/370 hardware,
uses a different epoch. But other than that, there is a standard
epoch, and you are supposed to add/subtract for time zones.

The usual epoch is midnight, Jan 1, 1900, I believe GMT.
(UTC wasn't invented in 1900.) Using that one, the high order bit
is always 1, as no were built before 1971. That is supposed to be
the test for the usual epoch.

Note that it overflows in 2042.

(snip)

> It strikes me that setting the clock must be a privileged
> operation, and it wouldn't take much for the hypervisor to save
> and restore the clock on switches between VCPUs (and of course
> it would have its own offsets for maintaining time in each VM
> relative to the host). In that case, allowing unfettered reads
> wouldn't be too much of an issue. Perhaps that is what they do.
The design is that there is a button on the operators panel that you
press to allow changes. But worse, there is no way to change it,
and keep the previous value. (Funny, as IBM is pretty good at doing
that everywhere else.) If you disable interrupts, you can minimize
the counts lost. But you have to ask the operator to press the button.

The OS/360 interval timer, described above, is designed so one can
read the value at the same time (one instruction) as setting a new value.
No ticks are lost. (Careful use of the MVC instruction.)

On Wednesday, May 17, 2023 at 6:39:09 AM UTC-7, Dan Cross wrote:

(snip, I wrote)

> >I once sold a machine that clients had NFS mounts from.
> >That is, hard mounts.

> The one that'll really flip your noggin is if your virtualized
> storage device is backed by a file on an NFS server somewhere...
I once thought about doing it for the P/370.
One reason for not doing it, is that 100Mbit Microchannel Ethernet
cards are hard to find, and apparently not that fast when you do
find them. I also thought about FDDI, but didn't do that, either.

The P/370 I/O system is based on OS/2, which does have
support for NFS disks.

https://en.wikipedia.org/wiki/P/370

On 5/15/2023 4:47 PM, Dan Cross wrote:
> In article <u3tv4e$34ma3$1@dont-email.me>,
> Arne Vajhøj <arne@vajhoej.dk> wrote:
>> On 5/15/2023 1:47 PM, Johnny Billquist wrote:
>>> On 2023-05-15 15:35, Arne Vajhøj wrote:
>>>> On 5/15/2023 8:51 AM, Johnny Billquist wrote:
>>>>> Um? What is the hypervisor then?
>>>>
>>>> The hypervisor software like ESXi.
>>>>
>>>> It is not running on top of a host OS like a type 2 (VirtualBox etc.)
>>>> does.
>>>
>>> I never assumed you had a full OS. However, they Hypervisor is
>>> intercepting access to hardware, and is responsible for the actual
>>> interaction with the hardware, adding a layer between the guest OS and
>>> the hardware.
>>
>> The hypervisor is there.
>>
>> But no host OS.
>
> For all intents and purposes, the hypervisor _is_ the host OS.
> Some software component must drive the hardware, run VCPUs, etc:
> that's the hypervisor.

A hypervisor is not a host OS.

Not in terminology and not in functionality.

The stacks are:

guest OS
type 1 hypervisor
hardware

and:

guest OS
type 2 hypervisor
host OS
hardware

Host OS and hypervisors are terms used for different things.

And used for things that provide different functionality:

a host OS is a normal OS like Linux or Windows able to run
applications in general including but certainly not limited
to type 2 hypervisors

a type 1 hypervisor like VMWare ESXi is a special piece of
software able to run VM's and partition the hardware resources
between the VM's - usually they are extremely thin and with only
this very specialized functionality - any heavy stuff
are usually put elsewhere in a VM (often known as a Dom0
VM) or a remote management application

a type 2 hypervisor like VMWare Player is an application running
on a host OS that runs a VM

> The arbitrary distinction between "Type-1" and "Type-2" style
> hypervisors is also a bit illusory among those who work on them;
> like all approximations, it is just that.

Anyone that works with them will know the difference between type
1 and type 2 (and the difference between a hypervisor and a host OS).

It is not an approximation. It is distinct as the difference
between 3 and 4.

Arne

On 5/17/2023 8:25 PM, Arne Vajhøj wrote:
> On 5/15/2023 4:47 PM, Dan Cross wrote:
>> For all intents and purposes, the hypervisor _is_ the host OS.
>> Some software component must drive the hardware, run VCPUs, etc:
>> that's the hypervisor.
>
> A hypervisor is not a host OS.
>
> Not in terminology and not in functionality.
>
> The stacks are:
>
> guest OS
> type 1 hypervisor
> hardware
>
> and:
>
> guest OS
> type 2 hypervisor
> host OS
> hardware
>
> Host OS and hypervisors are terms used for different things.
>
> And used for things that provide different functionality:
>
> a host OS is a normal OS like Linux or Windows able to run
> applications in general including but certainly not limited
> to type 2 hypervisors
>
> a type 1 hypervisor like VMWare ESXi is a special piece of
> software able to run VM's and partition the hardware resources
> between the VM's - usually they are extremely thin and with only
> this very specialized functionality - any heavy stuff
> are usually put elsewhere in a VM (often known as a Dom0
> VM) or a remote management application
>
> a type 2 hypervisor like VMWare Player is an application running
> on a host OS that runs a VM
>
>> The arbitrary distinction between "Type-1" and "Type-2" style
>> hypervisors is also a bit illusory among those who work on them;
>> like all approximations, it is just that.
>
> Anyone that works with them will know the difference between type
> 1 and type 2 (and the difference between a hypervisor and a host OS).
>
> It is not an approximation. It is distinct as the difference
> between 3 and 4.

There is one hypervisor that tend to muddy discussions: KVM.

KVM comes with Linux but is a type 1 hypervisor.

If people think of it as:

guest OS
KVM
Linux
hardware

then KVM looks like a type 2 hypervisor.

But KVM is not a Linux application - KVM is part of the Linux kernel.

So the correct picture is:

guest OS
Linux with KVM
hardware

so it is a type 1 hypervisor combined with an OS.

But it is not a thin hypervisor not doing anything else than
running VM's and managing hardware.

This sub-thread started about running real time stuff in a VM. And
I would not put KVM on the top of my hypervisor list for real time
VM's, because of the fact that it comes with an entire Linux
as part of the hypervisor. But the KVM people consider it doable.
They have guidelines on how to setup KVM for VM's running real time
Linux with real time applications.

Arne

On Wednesday, May 17, 2023 at 6:49:24 PM UTC-4, gah4 wrote:
> The design is that there is a button on the operators panel that you
> press to allow changes.

It is a momentary toggle labeled "TOD CLOCK". The up position is labeled "SECURE" and the down position is labeled "ENABLE SET". The OS would prompt you to toggle the switch during boot (IPL).

At least on the 370/138, it is conveniently located next to the power off button...