On 2022-12-19, Mark Daniel <mark.daniel@wasd.vsm.com.au> wrote:
> On 20/12/2022 5:40 am, Simon Clubley wrote:
>> That means operating systems running under it should be running at close
>> to native hardware performance speeds and it means you should be testing
>> the x86-64 performance against physical Alpha machines, not emulated ones.
>
> As a performance comparison tool, why not?
>

Because I was expecting an emulated Alpha to be slower than a real Alpha
so the results Dan gets might not be like-for-like.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On Tuesday, December 20, 2022 at 9:00:35 AM UTC-5, Simon Clubley wrote:
> On 2022-12-19, Mark Daniel <mark....@wasd.vsm.com.au> wrote:
> > On 20/12/2022 5:40 am, Simon Clubley wrote:
> >> That means operating systems running under it should be running at close
> >> to native hardware performance speeds and it means you should be testing
> >> the x86-64 performance against physical Alpha machines, not emulated ones.
> >
> > As a performance comparison tool, why not?
> >
> Because I was expecting an emulated Alpha to be slower than a real Alpha
> so the results Dan gets might not be like-for-like.
> Simon.
>
> --
> Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
> Walking destinations on a map are further away than they appear.
Using this test for V9.x is not the same as using it to test emulated environments. As stated prior, the current state of compiler code generation is such that performance tests are not relevant yet.

I would expect similar performance levels to real Alpha hardware for emulated Alphas on 3.5Ghz or faster machines (except for some of the larger GS class machines). I am aware of ES40 and ES45 class emulated machines achieving same or better performance than the real Alpha equivalents. CPU speed on par with I/O and network performance better than real Alphas.

Dan

On 12/20/22 13:56, Andreas Gruhl wrote:
> Mark Daniel schrieb am Montag, 19. Dezember 2022 um 06:01:45 UTC+1:
>> On 17/12/2022 10:13 am, Mark Daniel wrote:
>>> On 16/12/2022 10:27 pm, Mark Daniel wrote:
>>>> Now, before everyone piles on, I understand the procedure provides an
>>>> indicative/comparative/finger-in-the-air measurement of the relative
>>>> performance of a VMS CPU relative to "the original VAX processor".
>>> 8< snip 8<
>>>
>>> Thanks to all those who contributed to this thread.
>>>
>>> The followup that caught my eye was from Simon Clubley who provided an
>>> entirely convincing explanation for the elevated X86 Kernel mode.
>>>
>>> Also his pointer to BogoMips. Most interesting. I read the FAQ and
>>> accessed the github code. Quite straighforward. Might be a good
>>> replacement as a general performance metric.
>>>
>>> https://github.com/vitalyvch/Bogo/blob/BogoMIPS_v1.3/bogomips.c
>> 8< snip 8<
>>
>> For general VMS comparative usage something more VMS-measuring is
>> needed. I looked about the 'net and nothing sprang out. I wonder what
>> VSI are using for metrics on X86 development? Anything lurking in the
>> DECUS/Freeware repositories I missed?
>>
>> Anyway, in the absence of anything else, I was thinking about what may
>> consume "non-productive" VMS cycles (i.e. non-USER mode crunching :-)
>> and all I could think of were the transitions between USER, EXEC and
>> KERNEL modes. As required by RMS, $QIO, drivers, etc., etc. No SUPER
>> modes measured here.
>>
>> With this in mind I knocked-up a small program to repeatedly call a
>> function using $CMEXEC which calls a function using $CMKRNL and that is
>> that. It measures how much effort is required compared to the simple
>> USER mode loop and reports it as b[ogo]VUPs.
>>
>> https://wasd.vsm.com.au/wasd_tmp/bogovups.c
>>
>> The real disappointment is my X86 VM. The rest of the results seem in
>> line with expectations.
>>
>> PS. Looking for ideas, suggestions, criticism(s), etc. here...
>> --
> I ran your program in two of our own environments with the following rsults:
>
> |AlphaServer ES45 Model 1B 4 CPUs 16384MB V8.4-2L2
> |Calculating IPS...
> |321.000000 cpu-ticks, 3.279087 seconds, 609925873 / second
> |Calculating bVUPs...
> |1645.000000 cpu-ticks, 16.614445 seconds, 370.8 bVUPs
> |9157.000000 cpu-ticks, 91.599084 seconds, 10.8 bVUPs
>
> |HP DL380 2.6 GHz VMware, Inc. 4 CPUs 15868MB V9.2
> |Calculating IPS...
> |459.000000 cpu-ticks, 4.589954 seconds, 435734205 / second
> |Calculating bVUPs...
> |7470.000000 cpu-ticks, 74.799252 seconds, 58.3 bVUPs
>
> Sorry to say, but your bVUPs computation is not very useful.
> This is because you divide dins [instructions/sec] by dticks [0.01 sec].
> Insead of being dimensionless (like a good VUP ought to be)
> your result has the physical unit of [ 1/sec²].
> My own interpretation of the figures given above is:
> X86 integer performance reaches 71% of ES45 (321/459 Ticks)
> X86 change mode performance reaches 22% of ES45 (1645/7470 Ticks)
>
> This not bad given the fact, that the C crosscompiler currently attempts no
> optimization at all (as I know from an extraordinarily well informed source).
> Andreas

None of this makes much sense. spec.org have been devising cpu tests
for decades and have specialist tests for different workloads. That
includes all the info on compilers and code used. Probably the most
accurate data around and is supported by system and cpu vendors as
well. Too many variables involved, so some sort of level playing
field approach is the only way to get accuracy.

Can be fun devising simple tests, but would never used that as a
basis for purchasing decisions...

Chris

> None of this makes much sense. spec.org have been devising cpu tests
> for decades and have specialist tests for different workloads. That
> includes all the info on compilers and code used. Probably the most
> accurate data around and is supported by system and cpu vendors as
> well. Too many variables involved, so some sort of level playing
> field approach is the only way to get accuracy.
>
> Can be fun devising simple tests, but would never used that as a
> basis for purchasing decisions...
>
> Chris

The big problem with these standard benchmarks is that some compilers will look for these and insert some "special" optimizations specifically for those benchmarks. You are better served using a homegrown benchmark of some type that more closely reflects your application environment.

Dan

On 20/12/2022 10:43 pm, Volker Halle wrote:
> Mark,
>
> on my Intel i5-9600K @3.7 GHz with 6 Cores, Windows 10 Pro 22H2 with VMware Player 16, running my VUPS.COM reports about 730 VUPS with VSI OpenVMS x86-64 V9.2 with 2 vCPUs.

Nice. Perhaps a little more than AU$300.00 :-} Mine a paltry 280 VUPs.

I'm looking forward to retiring my PWS for the Dell (or something
equivalent); 145W x 24hr x 365 days, cf. 15W idle 35W processing.

> $ MONI MODE/INT=1/AVERAGE reports about 10% Kernel, 21% Exec and 51% Supervisor mode while running VUPS.COM
>
> Running a simple LOOP.COM
>
> $ i=0
> $loop:
> $ i=i+1
> $ GOTO loop
>
> consumes 22% Kernel, 33% Exec and 45% Supervisor mode
>
> Running the same LOOP.COM procedure on an OpenVMS V8.2 rx2600 1.3 GHz (1 CPU) consumes: 8% Kernel, 31% Exec and 61% Supervisor mode.
> Running VUPS.COM on that Itanium reports about 1762 VUPS with 5% Kernel, 17% Exec and 77% Supervisor mode.
>
> The 'CPU work' is done in Supervisor mode in a small loop in VUPS.COM, Exec and Kernel could be considered (necessary) 'overhead' and this 'overhead' is more significant on x86-64.
>
> AFAIK VSI has not yet publicized any performance data and may still be concentrating on function vs. performance.
>
> Volker.

Again, thanks to all who contributed. Unfortunately, I thoughtlessly
began this thread at the wrong end of the year. I'll be occupied with
other things for a couple of weeks.

> The 'CPU work' is done in Supervisor mode in a small loop in VUPS.COM, Exec and Kernel could be considered (necessary) 'overhead' and this 'overhead' is more significant on x86-64.
>
> AFAIK VSI has not yet publicized any performance data and may still be concentrating on function vs. performance.

Understand these comments, made by several posters. Not trying to say
anything about the X86 port, per se. Just trying to get a *feel* for
the relative performance/behaviour of the respective platforms
(including X86).

My summary...

The loop described above

$ i=0
$loop:
$ i=i+1
$ GOTO loop

is the essence of VUPS.COM and seems as good a finger-in-the-air as any
other. Just forget the "VUPS" as a unit and compare with other
platforms as the measure.

Using the above DCL and MON MODES/INT=1/AVE, my X86 shows

|Combined for 2 CPUs
| Kernel Mode 26 |▒▒▒▒▒
| Executive Mode 26 |▒▒▒▒▒
| Supervisor Mode 43 |▒▒▒▒▒▒▒▒
| Idle Time 103 |▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒

after a couple of minutes. The PWS 500

| Executive Mode 26 |▒▒▒▒▒▒▒▒▒▒
| Supervisor Mode 70 |▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒
| Idle Time 3 |▒

And the RX2600

|Combined for 4 CPUs
| Executive Mode 28 |▒▒
| Supervisor Mode 61 |▒▒▒▒▒▒
| User Mode 4 |
| Idle Time 299 |▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒

And the reported @VUPS.COM "VUPs" correspond to expected platform
performance.

Approximate System VUPs Rating : 286.6 ( min: 285.4 max: 287.8 )
Approximate System VUPs Rating : 135.8 ( min: 135.8 max: 135.8 )
Approximate System VUPs Rating : 486.3 ( min: 483.8 max: 488.8 )

This also shows the (granted, unoptimised) X86 performance to be
none-too-shoddy, especially for AU$300.00

I withdraw the bogoVUPs.c suggestion.

--
Anyone, who using social-media, forms an opinion regarding anything
other than the relative cuteness of this or that puppy-dog, needs
seriously to examine their critical thinking.

On 12/20/22 15:50, abrsvc wrote:
>> None of this makes much sense. spec.org have been devising cpu tests
>> for decades and have specialist tests for different workloads. That
>> includes all the info on compilers and code used. Probably the most
>> accurate data around and is supported by system and cpu vendors as
>> well. Too many variables involved, so some sort of level playing
>> field approach is the only way to get accuracy.
>>
>> Can be fun devising simple tests, but would never used that as a
>> basis for purchasing decisions...
>>
>> Chris
>
> The big problem with these standard benchmarks is that some compilers will look for these and insert some "special" optimizations specifically for those benchmarks. You are better served using a homegrown benchmark of some type that more closely reflects your application environment.
>
> Dan

All the conditions are published, including compiler flags,
which compiler and more. Must be more accurate than a home
grown ad hoc test which ignores so many variables that could
influence the results.

If you want to measure something, use the best and most
accurate tools available...

Chris

On Tuesday, December 20, 2022 at 11:33:13 AM UTC-5, chris wrote:
> On 12/20/22 15:50, abrsvc wrote:
> >> None of this makes much sense. spec.org have been devising cpu tests
> >> for decades and have specialist tests for different workloads. That
> >> includes all the info on compilers and code used. Probably the most
> >> accurate data around and is supported by system and cpu vendors as
> >> well. Too many variables involved, so some sort of level playing
> >> field approach is the only way to get accuracy.
> >>
> >> Can be fun devising simple tests, but would never used that as a
> >> basis for purchasing decisions...
> >>
> >> Chris
> >
> > The big problem with these standard benchmarks is that some compilers will look for these and insert some "special" optimizations specifically for those benchmarks. You are better served using a homegrown benchmark of some type that more closely reflects your application environment.
> >
> > Dan
> All the conditions are published, including compiler flags,
> which compiler and more. Must be more accurate than a home
> grown ad hoc test which ignores so many variables that could
> influence the results.
>
> If you want to measure something, use the best and most
> accurate tools available...
>
> Chris
I will disagree. How many standard benchmarks bear any relevance to an actual application? I suppose you can use them for relative machine performance information, but without knowing how your own application performs relative to those, they are useless. SPEC benchmarks mean little to I/O bound applications. Great, my new machine can perform calculations 10 times as fast. But... the application is bound by disk performance limits, so I see little to nothing for the speed improvement. just one extreme example.

On 12/20/22 16:43, abrsvc wrote:
> On Tuesday, December 20, 2022 at 11:33:13 AM UTC-5, chris wrote:
>> On 12/20/22 15:50, abrsvc wrote:
>>>> None of this makes much sense. spec.org have been devising cpu tests
>>>> for decades and have specialist tests for different workloads. That
>>>> includes all the info on compilers and code used. Probably the most
>>>> accurate data around and is supported by system and cpu vendors as
>>>> well. Too many variables involved, so some sort of level playing
>>>> field approach is the only way to get accuracy.
>>>>
>>>> Can be fun devising simple tests, but would never used that as a
>>>> basis for purchasing decisions...
>>>>
>>>> Chris
>>>
>>> The big problem with these standard benchmarks is that some compilers will look for these and insert some "special" optimizations specifically for those benchmarks. You are better served using a homegrown benchmark of some type that more closely reflects your application environment.
>>>
>>> Dan
>> All the conditions are published, including compiler flags,
>> which compiler and more. Must be more accurate than a home
>> grown ad hoc test which ignores so many variables that could
>> influence the results.
>>
>> If you want to measure something, use the best and most
>> accurate tools available...
>>
>> Chris
> I will disagree. How many standard benchmarks bear any relevance to an actual application? I suppose you can use them for relative machine performance information, but without knowing how your own application performs relative to those, they are useless. SPEC benchmarks mean little to I/O bound applications. Great, my new machine can perform calculations 10 times as fast. But... the application is bound by disk performance limits, so I see little to nothing for the speed improvement. just one extreme example.

The spec tests do target various workloads, database, web,
scientific and more, so why not use them ?. I;m sure there
must be other sites that do similar work, though haven't looked
recently.

If you want to find out where the bottlenecks are, you need to
start with single core throughput, to establish a baseline. That
without io, which would otherwise dominate most measurements,
orders of magnitude difference. How can you determine anything
by measuring at vm level only, which means you can have no idea
if it's the cpu, os or vm layer having the most influence ?.

Suspect there is little difference between most server vendors,
since they are all using the same cpu ranges and designs and it will
be some variant of the cpu vendors reference design anyway. Same
for disk and network io, as they are all using common controller chips
and vendors as well. Ever increasing complexity and R&D cost means
that only those like IBM can afford to go their own way. Not worth
the investment otherwise.

What will probably make the most difference is the OS and the
intimate knowhow that allows a vendor to make best use of the
processor, cache size and more. Same for application software
as well, some better than others. Too many variables
really to allow any meaningful results based on ad hoc tests.

So you dream up an ad hoc test and get results, so what does that
tell you comparison to anything else ?...

Chris

On 12/20/2022 11:43 AM, abrsvc wrote:
> On Tuesday, December 20, 2022 at 11:33:13 AM UTC-5, chris wrote:
>> On 12/20/22 15:50, abrsvc wrote:
>>>> None of this makes much sense. spec.org have been devising cpu tests
>>>> for decades and have specialist tests for different workloads. That
>>>> includes all the info on compilers and code used. Probably the most
>>>> accurate data around and is supported by system and cpu vendors as
>>>> well. Too many variables involved, so some sort of level playing
>>>> field approach is the only way to get accuracy.
>>>>
>>>> Can be fun devising simple tests, but would never used that as a
>>>> basis for purchasing decisions...
>>>
>>> The big problem with these standard benchmarks is that some
>>> compilers will look for these and insert some "special"
>>> optimizations specifically for those benchmarks. You are better
>>> served using a homegrown benchmark of some type that more closely
>>> reflects your application environment. >>>
>> All the conditions are published, including compiler flags,
>> which compiler and more. Must be more accurate than a home
>> grown ad hoc test which ignores so many variables that could
>> influence the results.
>>
>> If you want to measure something, use the best and most
>> accurate tools available...
>>
> I will disagree. How many standard benchmarks bear any relevance to
> an actual application? I suppose you can use them for relative
> machine performance information, but without knowing how your own
> application performs relative to those, they are useless. SPEC
> benchmarks mean little to I/O bound applications. Great, my new
> machine can perform calculations 10 times as fast. But... the
> application is bound by disk performance limits, so I see little to
> nothing for the speed improvement. just one extreme example.

Testing with the actual application instead of an
artificial benchmark is obviously better.

But given how much effort has gone into developing
the modern benchmarks, then they should be better
than a simple homegrown benchmark unless one has a rather
unique context.

Obviously one need to pick the right benchmark. Like:

CPU integer => SPEC CPU SPECint
CPU floating point => SPEC CPU SPECfp
CPU floating point linear algebra => LINPACK
Database OLTP => TPC-C
Database DWH => TPC-H
Java app servers => SPECjEnterprise

If we talk old 1980's benchmarks like Dhrystone/Whetstone, then
it is probably not too much work to come up a homegrown benchmark
as good or better.

Arne

Arne

On Tuesday, December 20, 2022 at 7:07:50 PM UTC-5, Arne Vajhøj wrote:
> On 12/20/2022 11:43 AM, abrsvc wrote:
> > On Tuesday, December 20, 2022 at 11:33:13 AM UTC-5, chris wrote:
> >> On 12/20/22 15:50, abrsvc wrote:
> >>>> None of this makes much sense. spec.org have been devising cpu tests
> >>>> for decades and have specialist tests for different workloads. That
> >>>> includes all the info on compilers and code used. Probably the most
> >>>> accurate data around and is supported by system and cpu vendors as
> >>>> well. Too many variables involved, so some sort of level playing
> >>>> field approach is the only way to get accuracy.
> >>>>
> >>>> Can be fun devising simple tests, but would never used that as a
> >>>> basis for purchasing decisions...
> >>>
> >>> The big problem with these standard benchmarks is that some
> >>> compilers will look for these and insert some "special"
> >>> optimizations specifically for those benchmarks. You are better
> >>> served using a homegrown benchmark of some type that more closely
> >>> reflects your application environment. >>>
> >> All the conditions are published, including compiler flags,
> >> which compiler and more. Must be more accurate than a home
> >> grown ad hoc test which ignores so many variables that could
> >> influence the results.
> >>
> >> If you want to measure something, use the best and most
> >> accurate tools available...
> >>
> > I will disagree. How many standard benchmarks bear any relevance to
> > an actual application? I suppose you can use them for relative
> > machine performance information, but without knowing how your own
> > application performs relative to those, they are useless. SPEC
> > benchmarks mean little to I/O bound applications. Great, my new
> > machine can perform calculations 10 times as fast. But... the
> > application is bound by disk performance limits, so I see little to
> > nothing for the speed improvement. just one extreme example.
> Testing with the actual application instead of an
> artificial benchmark is obviously better.
>
> But given how much effort has gone into developing
> the modern benchmarks, then they should be better
> than a simple homegrown benchmark unless one has a rather
> unique context.
>
> Obviously one need to pick the right benchmark. Like:
>
> CPU integer => SPEC CPU SPECint
> CPU floating point => SPEC CPU SPECfp
> CPU floating point linear algebra => LINPACK
> Database OLTP => TPC-C
> Database DWH => TPC-H
> Java app servers => SPECjEnterprise
>
> If we talk old 1980's benchmarks like Dhrystone/Whetstone, then
> it is probably not too much work to come up a homegrown benchmark
> as good or better.
>
> Arne
>
>
>
>
> Arne
Perhaps the point I wsa trying to make has not been clear.

Standard benchmarks can provide raw throughput numbers for certain classes of functions (CPU, raw I/O , database functions, etc.).
But... How these relate to a real application environment is required in order to use these to predict performance of a system. A home grown benchmark is less of a raw performance indicator than a more accurate predictor of the specific application environment for any new hardware. If you know the relationship, then I would guess that industry standard benchmarks are useful. In many cases where I have been involved, no simple correlation could be made. You mileage will vary...

On Friday, December 16, 2022 at 6:57:36 AM UTC-5, Mark Daniel wrote:
> Now, before everyone piles on, I understand the procedure provides an
> indicative/comparative/finger-in-the-air measurement of the relative
> performance of a VMS CPU relative to "the original VAX processor".
>
> It states as much in the prologue:
>
> |$! Provides an estimate of system CPU performance on OpenVMS systems.
> |$! Use at your own risk.
>
> To clarify the platform of a particular run I added:
>
> |$ write sys$output f$fao("!AS with !UL CPU and !ULMB running VMS !AS",-
> |f$edit(f$getsyi("hw_name"),"compress,trim"),-
> |f$getsyi("availcpu_cnt"),-
> |(f$getsyi("memsize")*(f$getsyi("page_size")/512)/2048),-
> |f$edit(f$getsyi("version"),"collapse"))
>
> which provides the likes of:
>
> |HP rx2660 (1.40GHz/6.0MB) with 4 CPU and 14335MB running VMS V8.4-2L1
> |Digital Personal WorkStation with 1 CPU and 1536MB running VMS V8.4-2L1
> |innotek GmbH VirtualBox with 2 CPU and 7574MB running VMS V9.2
>
> It seems to be implemented as a tight DCL loop that executes almost
> entirely in inner modes (I'm sure Brian can explain why).
>
> $ start_cputime = f$getjpi(0,"CPUTIM")
> $ loop_index = 0
> $ 10$:
> $ loop_index = loop_index + 1
> $ if loop_index .ne. init_loop_maximum then goto 10$
> $ end_cputime = f$getjpi(0,"CPUTIM")
>
> |Combined for 2 CPUs 0 50 100 150 200
> | Interrupt State | |
> | MP Synchronization | |
> | Kernel Mode 21 |▒▒▒▒ |
> | Executive Mode 21 |▒▒▒▒ |
> | Supervisor Mode 58 |▒▒▒▒▒▒▒▒▒▒▒ |
> | User Mode | |
> | Compatibility Mode | |
> | Idle Time 99 |▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ |
>
> 100% (or all-but) of this execution appears to be in inner modes.
> Although X86 (above) seems to to have much more Kernel than other
> architectures, e.g. IA64 below). There is no USER mode displayed in either.
>
> |Combined for 4 CPUs 0 100 200 300 400
> | Interrupt State 1 | |
> | MP Synchronization | |
> | Kernel Mode 5 | |
> | Executive Mode 18 |▒ |
> | Supervisor Mode 78 |▒▒▒▒▒▒▒ |
> | User Mode | |
> | Not Available | |
> | Idle Time 299 |▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ |
>
> There appear to be (at least) two versions of these procedures. The
> later contains:
>
> |$! Modified: MAY-2010: Code updated by Volker Halle to address the
> |$! following issues:
>
> and tweaks a few of the calculations.
>
> There also appear to be earlier tweaks allowing for Alpha processors
>
> |$ cpu_multiplier = 10 ! VAX = 10 - Alpha/AXP = 40
> |$ cpu_round_add = 1 ! VAX = 1 - Alpha/AXP = 9
> th
> but none for Itanium.
>
> Are the Alpha tweaks sufficient to allow relevance for all 64bit CPUs?
>
> Are further tweaks required to make measurements on Itania relevant?
>
> And of course the same question for the successor to all three
> architectures?
>
> --
> Anyone, who using social-media, forms an opinion regarding anything
> other than the relative cuteness of this or that puppy-dog, needs
> seriously to examine their critical thinking.

Been a bit busy the past few weeks with various things.

The best quote on benchmarks has been the US Environmental Protection Agency's disclaimer on it's automobile dynamometer-based fuel economy ratings "Your mileage may vary", generally rendered as "YMMV".

When CPU, and for that matter, mass storage devices were simple devices, without pipelines, caches, and the like, one could do simple benchmarks and obtain a useful result.

As far back as the late 1970s, pipelines and caches created a benchmark terrain full of cliffs, sinkholes, and plateaus. Back then, my research team saw benchmarks of the CDC 6600 vs the IBM System/370 Model 168 Submodel 3. Depewaysnding on the benchmark the comparison was a factor of 300%; both ways. In other words, essentially an order of magnitude range.

Toss in three levels of CPU/memory caches, some of which are shared; virtualized mass storage at various levels; and other factors. The sum is that one is that getting a raw benchmark is only the beginning of the journey. Tuning can, and often does, have a very large field to explore.

- Bob Gezelter, http://www.rlgsc.com

I use the old Bytemark benchmark to do crude CPU comparison (old as in
'normalized to a 90 Mhz Pentium'). The results are very sensitive to the compiler
and optimization levels, more so for gcc on X86 than the DEC OVMS compilers
on Alpha. Gcc defaults to no optimization and can improve 3-4 times while VSI
C defaults to a fairly high level but doesn't improve as much over /nooptimize.

The current C cross compiler does no optimization and the result I've seen for
a 1.8 GHz gen 8 Xeon (released 2012) is about what I got for a 617 Mhz DS10
in 2001. Bytemark on my Mac mini (3 ghz i5) shows integer performance 20
times better and floating point 10 times better than the DS10.

On 12/21/2022 7:42 AM, abrsvc wrote:
> On Tuesday, December 20, 2022 at 7:07:50 PM UTC-5, Arne Vajhøj wrote:
>> On 12/20/2022 11:43 AM, abrsvc wrote:
>>> I will disagree. How many standard benchmarks bear any relevance to
>>> an actual application? I suppose you can use them for relative
>>> machine performance information, but without knowing how your own
>>> application performs relative to those, they are useless. SPEC
>>> benchmarks mean little to I/O bound applications. Great, my new
>>> machine can perform calculations 10 times as fast. But... the
>>> application is bound by disk performance limits, so I see little to
>>> nothing for the speed improvement. just one extreme example.

>> Testing with the actual application instead of an
>> artificial benchmark is obviously better.
>>
>> But given how much effort has gone into developing
>> the modern benchmarks, then they should be better
>> than a simple homegrown benchmark unless one has a rather
>> unique context.
>>
>> Obviously one need to pick the right benchmark. Like:
>>
>> CPU integer => SPEC CPU SPECint
>> CPU floating point => SPEC CPU SPECfp
>> CPU floating point linear algebra => LINPACK
>> Database OLTP => TPC-C
>> Database DWH => TPC-H
>> Java app servers => SPECjEnterprise
>>
>> If we talk old 1980's benchmarks like Dhrystone/Whetstone, then
>> it is probably not too much work to come up a homegrown benchmark
>> as good or better.

> Standard benchmarks can provide raw throughput numbers for certain
> classes of functions (CPU, raw I/O , database functions, etc.).
> But... How these relate to a real application environment is
> required in order to use these to predict performance of a system. A
> home grown benchmark is less of a raw performance indicator than a
> more accurate predictor of the specific application environment for
> any new hardware. If you know the relationship, then I would guess
> that industry standard benchmarks are useful. In many cases where I
> have been involved, no simple correlation could be made. You mileage
> will vary...

If ones application does not match one of the standard benchmarks then
one has to create a custom benchmark.

But creating good benchmarks is not easy. Most quickly put
together custom benchmarks are pretty bad. There is a long
history of bad custom benchmarks producing misleading results.

Arne

On Friday, 16 December 2022 at 11:57:36 UTC, Mark Daniel wrote:
> Now, before everyone piles on, I understand the procedure provides an
> indicative/comparative/finger-in-the-air measurement of the relative
> performance of a VMS CPU relative to "the original VAX processor".
>
> It states as much in the prologue:
>
> |$! Provides an estimate of system CPU performance on OpenVMS systems.
> |$! Use at your own risk.
>
> To clarify the platform of a particular run I added:
>
> |$ write sys$output f$fao("!AS with !UL CPU and !ULMB running VMS !AS",-
> |f$edit(f$getsyi("hw_name"),"compress,trim"),-
> |f$getsyi("availcpu_cnt"),-
> |(f$getsyi("memsize")*(f$getsyi("page_size")/512)/2048),-
> |f$edit(f$getsyi("version"),"collapse"))
>
> which provides the likes of:
>
> |HP rx2660 (1.40GHz/6.0MB) with 4 CPU and 14335MB running VMS V8.4-2L1
> |Digital Personal WorkStation with 1 CPU and 1536MB running VMS V8.4-2L1
> |innotek GmbH VirtualBox with 2 CPU and 7574MB running VMS V9.2
>
> It seems to be implemented as a tight DCL loop that executes almost
> entirely in inner modes (I'm sure Brian can explain why).
>
> $ start_cputime = f$getjpi(0,"CPUTIM")
> $ loop_index = 0
> $ 10$:
> $ loop_index = loop_index + 1
> $ if loop_index .ne. init_loop_maximum then goto 10$
> $ end_cputime = f$getjpi(0,"CPUTIM")
>
> |Combined for 2 CPUs 0 50 100 150 200
> | Interrupt State | |
> | MP Synchronization | |
> | Kernel Mode 21 |▒▒▒▒ |
> | Executive Mode 21 |▒▒▒▒ |
> | Supervisor Mode 58 |▒▒▒▒▒▒▒▒▒▒▒ |
> | User Mode | |
> | Compatibility Mode | |
> | Idle Time 99 |▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ |
>
> 100% (or all-but) of this execution appears to be in inner modes.
> Although X86 (above) seems to to have much more Kernel than other
> architectures, e.g. IA64 below). There is no USER mode displayed in either.
>
> |Combined for 4 CPUs 0 100 200 300 400
> | Interrupt State 1 | |
> | MP Synchronization | |
> | Kernel Mode 5 | |
> | Executive Mode 18 |▒ |
> | Supervisor Mode 78 |▒▒▒▒▒▒▒ |
> | User Mode | |
> | Not Available | |
> | Idle Time 299 |▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ |
>
> There appear to be (at least) two versions of these procedures. The
> later contains:
>
> |$! Modified: MAY-2010: Code updated by Volker Halle to address the
> |$! following issues:
>
> and tweaks a few of the calculations.
>
> There also appear to be earlier tweaks allowing for Alpha processors
>
> |$ cpu_multiplier = 10 ! VAX = 10 - Alpha/AXP = 40
> |$ cpu_round_add = 1 ! VAX = 1 - Alpha/AXP = 9
>
> but none for Itanium.
>
> Are the Alpha tweaks sufficient to allow relevance for all 64bit CPUs?
>
> Are further tweaks required to make measurements on Itania relevant?
>
> And of course the same question for the successor to all three
> architectures?
>
> --
> Anyone, who using social-media, forms an opinion regarding anything
> other than the relative cuteness of this or that puppy-dog, needs
> seriously to examine their critical thinking.

Not sure if people remember these, but the DEC/Compaq Enterprise Capacity Planner had *.DBA files that listed SPEC values for pretty much every vendor at the with normalised data. The Enterprise Capacity Planner was bought out by its engineers in 2001 (as that was the only part of Polycenter (PSPA/PSDC/PSCP) not given to CA in 1997) and today lists most vendors, and CPUs to for SPEC 95 to SPEC 2017. Both spec int and spec rate are included. www..perfcap.com. I still use the ECP today, which was ported to Linux. The data is interesting, performance is getting wider *more cores, not faster (faster cores). The files are user editiable to add future predicted CPU speeds for modelling purposes.
Merry Christmas one and all.
Paul.

On 2022-12-22 09:44:30 +0000, pos said:

> ... The Enterprise Capacity Planner was bought out by its engineers in
> 2001 (as that was the only part of Polycenter (PSPA/PSDC/PSCP) not
> given to CA in 1997) ...

Pedantic: POLYCENTER Software Installation Utility (PCSI) was (also) retained.

--
Pure Personal Opinion | HoffmanLabs LLC

On 12/26/2022 3:05 PM, Stephen Hoffman wrote:
> On 2022-12-22 09:44:30 +0000, pos said:
>
>> ... The Enterprise Capacity Planner was bought out by its engineers in 2001
>> (as that was the only part of Polycenter (PSPA/PSDC/PSCP) not given to CA in
>> 1997) ...
>
> Pedantic: POLYCENTER Software Installation Utility (PCSI) was (also) retained.

POLYCENTER File Optimizer was retained as well.

--

--- Rob