It's been a while since I last posted here, but now that the OpenVMS x86 V9..2-1 release is out with a community license, I've finally set aside some time to create and configure a new VM in VirtualBox 7.0.10.

Since I knew what I was doing, it didn't take me too many hours to install OpenVMS, C, CivetWeb, DECset, COBOL, C++, Fortran, Lua, OpenJDK 8.0, OpenSSH, SSL 1.1.1, SSL 3, vGit, VMSI18N, WebUI, x86 macro (llvm-mc), and ZeroMQ. I did think to myself how daunting and confusing the whole post-installation process must be to someone new to VMS.

I'm a big fan of the Phoronix Test Suite, so I thought it would be educational to select the benchmarks that I know will have a good chance of running on VMS, then comparing to Linux. So I used PTS to run Java and other benchmarks first in an Ubuntu Server 22.04 VM (I'm running VirtualBox itself on Ubuntu 22.04 LTS and connecting remotely via PuTTY from Windows) with an identical VM config, then I ran them manually in OpenVMS.

The hardware is a Lenovo ThinkStation P340 Tower with an Intel Xeon W-1290P CPU @ 3.7 GHz, with performance CPU governor set with "cpupower", all systemd timer services disabled, and using the legacy COM ports to log in to OpenVMS instead of sshd. The host has 64 GB RAM, and the VMs have 6 CPUs (the max for the community VMS license) and 16 GB RAM.

My OpenVMS VM has two LsiLogic SCSI disks, and I should've used SCSI for the Ubuntu Server system disk, but I used SATA, and later made a second LsiLogic SCSI disk for disk benchmarks, which I'll cover in a later post. I used "VBoxManage modifyvm [name] --x2apic=on" and "VBoxManage modifyvm [name] --hpet=on" for both VMS and Linux (the VMS bootloader notices the presence/absence of X2APIC and I remember reading in a VSI doc that it benefits from HPET as well).

I remember seeing a post here complaining about the VSI port of the JDK being "only" Java 8, but I've found that a lot of Java programs, including some of the benchmarks I ran, either failed on Java 17 or occasionally ran much slower than Java 8 (Apache Spark ALS specifically). I uploaded my Linux results for both Java 8 and Java 17 to OpenBenchmarking.org, so you can compare the results here:

https://openbenchmarking.org/result/2310050-JHAM-UBUNTUV47,2310052-JHAM-UBUNTUV60

I can't use the Phoronix harness on OpenVMS, or at least not yet, as it's written in PHP but also dependent on UNIX-specific and even Linux-specific shell scripts and utilities. As a side note, I have a plan to port GNU bash to C++, with support to compile under C++03 or newer (specifically for VMS Alpha/Itanium), but optimized for newer C++ versions (which will help for x86).

In fact, my bash C++ / VMS port is in progress at my GitHub page, but it currently doesn't compile. I'll have to review my latest commits to jog my memory as to where I was in the project. My plan to not be dependent on fork() is dependent on getting rid of all the global variables, which is the major part of the work remaining, and then when fork() would normally be called, I'll exec() myself with a special command-line flag and then push the bash shell state to the child via a mailbox. That's my plan, anyway.

I'll write more about bash, coreutils, and the rest of GNV in a future post.. Back to Java on VMS. I had to run each benchmark by hand, so I'll post the results in tables below, along with the ratio of VMS to Linux speed. Here are the quotas that I set for my user account (and SYSTEM), based on merging recommendations from layered product release notes, including Samba. I had no problems with running out of quota.

Maxjobs: 0 Fillm: 1024 Bytlm: 3000000
Maxacctjobs: 0 Shrfillm: 0 Pbytlm: 0
Maxdetach: 0 BIOlm: 32767 JTquota: 4096
Prclm: 50 DIOlm: 32767 WSdef: 4096
Prio: 4 ASTlm: 2048 WSquo: 8192
Queprio: 4 TQElm: 100 WSextent: 16384
CPU: (none) Enqlm: 4000 Pgflquo: 10000000

First up, Java SciMark 2.0, which is a numerical benchmark, so it should have relatively good results on VMS, and it does. VMS scores are the average of 3 runs, followed by the speed percentage vs. Linux:

Composite Score: 3099.525 (99.7%)
FFT (1024): 1907.459 (99.4%)
SOR (100x100): 1832.324 (99.5%)
Monte Carlo : 1238.935 (99.5%)
Sparse matmult (N=1000, nz=5000): 2853.442 (97.3%)
LU (100x100): 7665.467 (100.76%)

So we can safely conclude that the performance of OpenJDK 1.8.0_372 for purely CPU-bound tasks in OpenVMS x86 is nearly identical to Linux on the same hardware and VM params. The differences between the two are within the margin of error considering the variance between individual runs.

Next up, DaCapo benchmarks. I'll give the description for the three that worked on VMS. Results are times in ms (lower is better), with the average of 2 runs, or 3 for Eclipse, plus % ratio of Linux / VMS time.

"H2 executes a TPC-C like benchmark written by Apache as part of the Apache Derby database. The application models: customers, districts, warehouses, purchases and deliveries.",

VMS score: 27135.0 (10.13% of Linux)

"jython executes (interprets) the pybench benchmark"

VMS score: 22000.0 (14.92% of Linux)

"Run eclipse's jdt (non-gui) performance tests"

VMS score: 169355.33 (9.09% of Linux)

These results are a bit more dismal. Let's see what the Renaissance Java benchmarks look like. In addition to having to "define/job JAVA$DELETE_ALL_VERSIONS TRUE" for Scala Dotty to work (otherwise it fails with a directory not empty error trying to delete a subdir), for the Apache Spark benchmarks to work, I also had to add "-Dspark.file.transferTo=false" to the Java line or else I get this exception:

23/10/07 19:16:10 ERROR Executor: Exception in task 5.0 in stage 1.0 (TID 6)
java.lang.AssertionError: assertion failed:
Current position 0 do not equal to expected position 27650
after transferTo, please check your kernel version to see if it is 2.6.32,
this is a kernel bug which will lead to unexpected behavior when using transferTo.
You can set spark.file.transferTo = false to disable this NIO feature.

at scala.Predef$.assert(Predef.scala:279)
at org.apache.spark.util.Utils$.$anonfun$copyFileStreamNIO$2(Utils.scala
:459)
at org.apache.spark.util.Utils$.$anonfun$copyFileStreamNIO$2$adapted(Uti
ls.scala:450)
at scala.Option.foreach(Option.scala:437)
at org.apache.spark.util.Utils$.copyFileStreamNIO(Utils.scala:450)
at org.apache.spark.util.Utils.copyFileStreamNIO(Utils.scala)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.writeParti
tionedDataWithChannel(BypassMergeSortShuffleWriter.java:246)

It's wonderful that the exception told me how to work around the bug, but I'm a little scared to think about who may still be using Linux 2.6.32!

The in-memory database test failed with an "org.mapdb.DBException$VolumeIOError" exception which I didn't investigate, and the ALS Movie Lens benchmark failed due to missing "chmod" command. This time, the benchmark results were in CSV format, and I averaged them without the first run's score (which tended to be noticeably slower with the JIT having to warm up) and converted to ms.

Scala Dotty: 8042.981 (10.39%)
Random Forest: 4008.678 (14.53%)
Apache Spark ALS: 34253.451 (5.52%)
Apache Spark Bayes: 31865.199 (5.73%)
Savina Reactors.IO: 24028.649 (24.79%)
Apache Spark PageRank: 10485.158 (22.20%)
Finagle HTTP Requests: 74394.711 (3.91%)
Akka Unbalanced Cobwebbed Tree: 34890.521 (25.9%)
Genetic Algorithm (Jenetics + Futures): 2520.791 (62.56%)

Finally, the Sunflow ray-tracing benchmark.

VMS score: 4470 ms (36.22%)

I hope this info has been of interest, and I look forward to trying out a future OpenJDK 11 release. As I noted up top, it's not in VSI's interest to release a Java 14 or Java 17 port without first releasing Java 8 and Java 11 ports, since the newer versions of the JDK and the command-line options for the JVM simply aren't backwards compatible, and I've found that many popular Java apps have a max supported Java version.

Regards,
Jake Hamby

On Monday, October 9, 2023 at 5:34:22 AM UTC+1, Jake Hamby (Solid State Jake) wrote:
> It's been a while since I last posted here, but now that the OpenVMS x86 V9.2-1 release is out with a community license, I've finally set aside some time to create and configure a new VM in VirtualBox 7.0.10.
>
> Since I knew what I was doing, it didn't take me too many hours to install OpenVMS, C, CivetWeb, DECset, COBOL, C++, Fortran, Lua, OpenJDK 8.0, OpenSSH, SSL 1.1.1, SSL 3, vGit, VMSI18N, WebUI, x86 macro (llvm-mc), and ZeroMQ. I did think to myself how daunting and confusing the whole post-installation process must be to someone new to VMS.
>
> I'm a big fan of the Phoronix Test Suite, so I thought it would be educational to select the benchmarks that I know will have a good chance of running on VMS, then comparing to Linux. So I used PTS to run Java and other benchmarks first in an Ubuntu Server 22.04 VM (I'm running VirtualBox itself on Ubuntu 22.04 LTS and connecting remotely via PuTTY from Windows) with an identical VM config, then I ran them manually in OpenVMS.
>
> The hardware is a Lenovo ThinkStation P340 Tower with an Intel Xeon W-1290P CPU @ 3.7 GHz, with performance CPU governor set with "cpupower", all systemd timer services disabled, and using the legacy COM ports to log in to OpenVMS instead of sshd. The host has 64 GB RAM, and the VMs have 6 CPUs (the max for the community VMS license) and 16 GB RAM.
>
> My OpenVMS VM has two LsiLogic SCSI disks, and I should've used SCSI for the Ubuntu Server system disk, but I used SATA, and later made a second LsiLogic SCSI disk for disk benchmarks, which I'll cover in a later post. I used "VBoxManage modifyvm [name] --x2apic=on" and "VBoxManage modifyvm [name] --hpet=on" for both VMS and Linux (the VMS bootloader notices the presence/absence of X2APIC and I remember reading in a VSI doc that it benefits from HPET as well).
>
> I remember seeing a post here complaining about the VSI port of the JDK being "only" Java 8, but I've found that a lot of Java programs, including some of the benchmarks I ran, either failed on Java 17 or occasionally ran much slower than Java 8 (Apache Spark ALS specifically). I uploaded my Linux results for both Java 8 and Java 17 to OpenBenchmarking.org, so you can compare the results here:
>
> https://openbenchmarking.org/result/2310050-JHAM-UBUNTUV47,2310052-JHAM-UBUNTUV60
>
> I can't use the Phoronix harness on OpenVMS, or at least not yet, as it's written in PHP but also dependent on UNIX-specific and even Linux-specific shell scripts and utilities. As a side note, I have a plan to port GNU bash to C++, with support to compile under C++03 or newer (specifically for VMS Alpha/Itanium), but optimized for newer C++ versions (which will help for x86).
>
> In fact, my bash C++ / VMS port is in progress at my GitHub page, but it currently doesn't compile. I'll have to review my latest commits to jog my memory as to where I was in the project. My plan to not be dependent on fork() is dependent on getting rid of all the global variables, which is the major part of the work remaining, and then when fork() would normally be called, I'll exec() myself with a special command-line flag and then push the bash shell state to the child via a mailbox. That's my plan, anyway.
>
> I'll write more about bash, coreutils, and the rest of GNV in a future post. Back to Java on VMS. I had to run each benchmark by hand, so I'll post the results in tables below, along with the ratio of VMS to Linux speed. Here are the quotas that I set for my user account (and SYSTEM), based on merging recommendations from layered product release notes, including Samba. I had no problems with running out of quota.
>
> Maxjobs: 0 Fillm: 1024 Bytlm: 3000000
> Maxacctjobs: 0 Shrfillm: 0 Pbytlm: 0
> Maxdetach: 0 BIOlm: 32767 JTquota: 4096
> Prclm: 50 DIOlm: 32767 WSdef: 4096
> Prio: 4 ASTlm: 2048 WSquo: 8192
> Queprio: 4 TQElm: 100 WSextent: 16384
> CPU: (none) Enqlm: 4000 Pgflquo: 10000000
>
> First up, Java SciMark 2.0, which is a numerical benchmark, so it should have relatively good results on VMS, and it does. VMS scores are the average of 3 runs, followed by the speed percentage vs. Linux:
>
> Composite Score: 3099.525 (99.7%)
> FFT (1024): 1907.459 (99.4%)
> SOR (100x100): 1832.324 (99.5%)
> Monte Carlo : 1238.935 (99.5%)
> Sparse matmult (N=1000, nz=5000): 2853.442 (97.3%)
> LU (100x100): 7665.467 (100.76%)
>
> So we can safely conclude that the performance of OpenJDK 1.8.0_372 for purely CPU-bound tasks in OpenVMS x86 is nearly identical to Linux on the same hardware and VM params. The differences between the two are within the margin of error considering the variance between individual runs.
>
> Next up, DaCapo benchmarks. I'll give the description for the three that worked on VMS. Results are times in ms (lower is better), with the average of 2 runs, or 3 for Eclipse, plus % ratio of Linux / VMS time.
>
> "H2 executes a TPC-C like benchmark written by Apache as part of the Apache Derby database. The application models: customers, districts, warehouses, purchases and deliveries.",
>
> VMS score: 27135.0 (10.13% of Linux)
>
> "jython executes (interprets) the pybench benchmark"
>
> VMS score: 22000.0 (14.92% of Linux)
>
> "Run eclipse's jdt (non-gui) performance tests"
>
> VMS score: 169355.33 (9.09% of Linux)
>
> These results are a bit more dismal. Let's see what the Renaissance Java benchmarks look like. In addition to having to "define/job JAVA$DELETE_ALL_VERSIONS TRUE" for Scala Dotty to work (otherwise it fails with a directory not empty error trying to delete a subdir), for the Apache Spark benchmarks to work, I also had to add "-Dspark.file.transferTo=false" to the Java line or else I get this exception:
>
> 23/10/07 19:16:10 ERROR Executor: Exception in task 5.0 in stage 1.0 (TID 6)
> java.lang.AssertionError: assertion failed:
> Current position 0 do not equal to expected position 27650
> after transferTo, please check your kernel version to see if it is 2.6.32,
> this is a kernel bug which will lead to unexpected behavior when using transferTo.
> You can set spark.file.transferTo = false to disable this NIO feature.
>
> at scala.Predef$.assert(Predef.scala:279)
> at org.apache.spark.util.Utils$.$anonfun$copyFileStreamNIO$2(Utils.scala
> :459)
> at org.apache.spark.util.Utils$.$anonfun$copyFileStreamNIO$2$adapted(Uti
> ls.scala:450)
> at scala.Option.foreach(Option.scala:437)
> at org.apache.spark.util.Utils$.copyFileStreamNIO(Utils.scala:450)
> at org.apache.spark.util.Utils.copyFileStreamNIO(Utils.scala)
> at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.writeParti
> tionedDataWithChannel(BypassMergeSortShuffleWriter.java:246)
>
> It's wonderful that the exception told me how to work around the bug, but I'm a little scared to think about who may still be using Linux 2.6.32!
>
> The in-memory database test failed with an "org.mapdb.DBException$VolumeIOError" exception which I didn't investigate, and the ALS Movie Lens benchmark failed due to missing "chmod" command. This time, the benchmark results were in CSV format, and I averaged them without the first run's score (which tended to be noticeably slower with the JIT having to warm up) and converted to ms.
>
> Scala Dotty: 8042.981 (10.39%)
> Random Forest: 4008.678 (14.53%)
> Apache Spark ALS: 34253.451 (5.52%)
> Apache Spark Bayes: 31865.199 (5.73%)
> Savina Reactors.IO: 24028.649 (24.79%)
> Apache Spark PageRank: 10485.158 (22.20%)
> Finagle HTTP Requests: 74394.711 (3.91%)
> Akka Unbalanced Cobwebbed Tree: 34890.521 (25.9%)
> Genetic Algorithm (Jenetics + Futures): 2520.791 (62.56%)
>
> Finally, the Sunflow ray-tracing benchmark.
>
> VMS score: 4470 ms (36.22%)
>
> I hope this info has been of interest, and I look forward to trying out a future OpenJDK 11 release. As I noted up top, it's not in VSI's interest to release a Java 14 or Java 17 port without first releasing Java 8 and Java 11 ports, since the newer versions of the JDK and the command-line options for the JVM simply aren't backwards compatible, and I've found that many popular Java apps have a max supported Java version.
>
> Regards,
> Jake Hamby

Hopefully VSI will notice these results and someone with some time reproduce or extend your work.

Jake, thank you so much for the testing. I need to analyze these tests and try to fix our JDK a little.

On 10/9/23 12:54 PM, Vitaly Pustovetov wrote:
> Jake, thank you so much for the testing. I need to analyze these tests and try to fix our JDK a little.

I was wondering if JIT was implicitly or explicitly turned off to see
things on VMS that much slower than on Linux.

On 10/9/2023 12:34 AM, Jake Hamby (Solid State Jake) wrote:
> I'm a big fan of the Phoronix Test Suite, so I thought it would be
> educational to select the benchmarks that I know will have a good
> chance of running on VMS, then comparing to Linux. So I used PTS to
> run Java and other benchmarks first in an Ubuntu Server 22.04 VM (I'm
> running VirtualBox itself on Ubuntu 22.04 LTS and connecting remotely
> via PuTTY from Windows) with an identical VM config, then I ran them
> manually in OpenVMS.
>
> The hardware is a Lenovo ThinkStation P340 Tower with an Intel Xeon
> W-1290P CPU @ 3.7 GHz, with performance CPU governor set with
> "cpupower", all systemd timer services disabled, and using the legacy
> COM ports to log in to OpenVMS instead of sshd. The host has 64 GB
> RAM, and the VMs have 6 CPUs (the max for the community VMS license)
> and 16 GB RAM.
>
> My OpenVMS VM has two LsiLogic SCSI disks, and I should've used SCSI
> for the Ubuntu Server system disk, but I used SATA, and later made a
> second LsiLogic SCSI disk for disk benchmarks, which I'll cover in a
> later post. I used "VBoxManage modifyvm [name] --x2apic=on" and
> "VBoxManage modifyvm [name] --hpet=on" for both VMS and Linux (the
> VMS bootloader notices the presence/absence of X2APIC and I remember
> reading in a VSI doc that it benefits from HPET as well).
>
> I remember seeing a post here complaining about the VSI port of the
> JDK being "only" Java 8, but I've found that a lot of Java programs,
> including some of the benchmarks I ran, either failed on Java 17 or
> occasionally ran much slower than Java 8 (Apache Spark ALS
> specifically). I uploaded my Linux results for both Java 8 and Java
> 17 to OpenBenchmarking.org, so you can compare the results here:
>
> https://openbenchmarking.org/result/2310050-JHAM-UBUNTUV47,2310052-JHAM-UBUNTUV60

Different benchmarks can produce different results.

My favorite stupid DIY benchmarks gives:

21 - 120
17 - 116
11 - 109
8 - 96

> I'll write more about bash, coreutils, and the rest of GNV in a
> future post. Back to Java on VMS. I had to run each benchmark by
> hand, so I'll post the results in tables below, along with the ratio
> of VMS to Linux speed. Here are the quotas that I set for my user
> account (and SYSTEM), based on merging recommendations from layered
> product release notes, including Samba. I had no problems with
> running out of quota.
>
> Maxjobs: 0 Fillm: 1024 Bytlm: 3000000
> Maxacctjobs: 0 Shrfillm: 0 Pbytlm: 0 Maxdetach:
> 0 BIOlm: 32767 JTquota: 4096 Prclm: 50 DIOlm:
> 32767 WSdef: 4096 Prio: 4 ASTlm: 2048
> WSquo: 8192 Queprio: 4 TQElm: 100 WSextent:
> 16384 CPU: (none) Enqlm: 4000 Pgflquo: 10000000
>
> First up, Java SciMark 2.0, which is a numerical benchmark, so it
> should have relatively good results on VMS, and it does. VMS scores
> are the average of 3 runs, followed by the speed percentage vs.
> Linux:
>
> Composite Score: 3099.525 (99.7%)
> FFT (1024): 1907.459 (99.4%)
> SOR (100x100): 1832.324 (99.5%)
> Monte Carlo : 1238.935 (99.5%)
> Sparse matmult (N=1000, nz=5000): 2853.442 (97.3%)
> LU (100x100): 7665.467 (100.76%)
>
> So we can safely conclude that the performance of OpenJDK 1.8.0_372
> for purely CPU-bound tasks in OpenVMS x86 is nearly identical to
> Linux on the same hardware and VM params. The differences between the
> two are within the margin of error considering the variance between
> individual runs.

I would have been very surprised if the results had been different.

Linux does:

Java source--(OpenJDK javac)-->Java byte code--(OpenJDK JVM JIT for
x86-64)-->x86-64 instructions

VMS does:

Java source--(OpenJDK javac)-->Java byte code--(OpenJDK JVM JIT for
x86-64)-->x86-64 instructions

There is no need for VSI to do anything VMS specific in either javac
or JVM JIT.

So basically the CPU execute the same instructions.

Only difference will be small differences caused be
paging and scheduling.

> Next up, DaCapo benchmarks. I'll give the description for the three
> that worked on VMS. Results are times in ms (lower is better), with
> the average of 2 runs, or 3 for Eclipse, plus % ratio of Linux / VMS
> time.
>
> "H2 executes a TPC-C like benchmark written by Apache as part of the
> Apache Derby database. The application models: customers, districts,
> warehouses, purchases and deliveries.",
>
> VMS score: 27135.0 (10.13% of Linux)
>
> "jython executes (interprets) the pybench benchmark"
>
> VMS score: 22000.0 (14.92% of Linux)
>
> "Run eclipse's jdt (non-gui) performance tests"
>
> VMS score: 169355.33 (9.09% of Linux)
>
> These results are a bit more dismal. Let's see what the Renaissance
> Java benchmarks look like. In addition to having to "define/job
> JAVA$DELETE_ALL_VERSIONS TRUE" for Scala Dotty to work (otherwise it
> fails with a directory not empty error trying to delete a subdir),
> for the Apache Spark benchmarks to work,

> The in-memory database test failed with an
> "org.mapdb.DBException$VolumeIOError" exception which I didn't
> investigate, and the ALS Movie Lens benchmark failed due to missing
> "chmod" command. This time, the benchmark results were in CSV format,
> and I averaged them without the first run's score (which tended to be
> noticeably slower with the JIT having to warm up) and converted to
> ms.
>
> Scala Dotty: 8042.981 (10.39%)
> Random Forest: 4008.678 (14.53%)
> Apache Spark ALS: 34253.451 (5.52%)
> Apache Spark Bayes: 31865.199 (5.73%)
> Savina Reactors.IO: 24028.649 (24.79%)
> Apache Spark PageRank: 10485.158 (22.20%)
> Finagle HTTP Requests: 74394.711 (3.91%)
> Akka Unbalanced Cobwebbed Tree: 34890.521 (25.9%)
> Genetic Algorithm (Jenetics + Futures): 2520.791 (62.56%)
>
> Finally, the Sunflow ray-tracing benchmark.
>
> VMS score: 4470 ms (36.22%)

I assume these must include some disk file IO and network IO.

It is well known that disk file IO are slower on VMS
than on *nix. Benchmarks has consistently shown that
for 25 years.

I would also not be surprised if network IO on VMS is slower than
on *nix.

> I hope this info has been of interest, and I look forward to trying
> out a future OpenJDK 11 release. As I noted up top, it's not in VSI's
> interest to release a Java 14 or Java 17 port without first releasing
> Java 8 and Java 11 ports, since the newer versions of the JDK and the
> command-line options for the JVM simply aren't backwards compatible,
> and I've found that many popular Java apps have a max supported Java
> version.

There is no reason to go Java 14.

The LTS versions are:

8 -> 11 -> 17 -> 21

Arne

On 10/9/2023 1:54 PM, Vitaly Pustovetov wrote:
> Jake, thank you so much for the testing. I need to analyze these
> tests and try to fix our JDK a little.

You can always take a look, but the problem is not necessarily
in JDK.

For IO it is:

Java application-->Java RT (Java)-->Java RT (native)-->C
RTL-->RMS-->VMS-->disk

I have never looked at the source code, but I would assume
that the call is mostly passthrough until it hits C RTL.

If that assumption is correct, then all you can do is look
at RMS tuning options.

(well unless you want to skip C RTL and RMS and go directly
for SYS$IO_PERFORM(W), but that would be "unusual")

Arne

On 10/9/2023 3:51 PM, Craig A. Berry wrote:
> On 10/9/23 12:54 PM, Vitaly Pustovetov wrote:
>> Jake, thank you so much for the testing. I need to analyze these tests
>> and try to fix our JDK a little.
>
> I was wondering if JIT was implicitly or explicitly turned off to see
> things on VMS that much slower than on Linux.

The CPU bound test results were fine.

Arne

On 10/9/2023 6:49 PM, Arne Vajhøj wrote:
> On 10/9/2023 12:34 AM, Jake Hamby (Solid State Jake) wrote:
>> First up, Java SciMark 2.0, which is a numerical benchmark, so it
>> should have relatively good results on VMS, and it does. VMS scores
>> are the average of 3 runs, followed by the speed percentage vs.
>> Linux:
>>
>> Composite Score: 3099.525 (99.7%)
>> FFT (1024): 1907.459 (99.4%)
>> SOR (100x100):  1832.324 (99.5%)
>> Monte Carlo : 1238.935 (99.5%)
>> Sparse matmult (N=1000, nz=5000): 2853.442 (97.3%)
>> LU (100x100): 7665.467 (100.76%)
>>
>> So we can safely conclude that the performance of OpenJDK 1.8.0_372
>> for purely CPU-bound tasks in OpenVMS x86 is nearly identical to
>> Linux on the same hardware and VM params. The differences between the
>> two are within the margin of error considering the variance between
>> individual runs.
>
> I would have been very surprised if the results had been different.
>
> Linux does:
>
> Java source--(OpenJDK javac)-->Java byte code--(OpenJDK JVM JIT for
> x86-64)-->x86-64 instructions
>
> VMS does:
>
> Java source--(OpenJDK javac)-->Java byte code--(OpenJDK JVM JIT for
> x86-64)-->x86-64 instructions
>
> There is no need for VSI to do anything VMS specific in either javac
> or JVM JIT.
>
> So basically the CPU execute the same instructions.
>
> Only difference will be small differences caused be
> paging and scheduling.

My stupid DIY benchmark produces similar result:

VMS 9 Windows 10 Ubuntu 22 Windows 10
Java 8 Java 8 Java 17 Java 17
int MOPS 655 660 782 774
fp MOPS 239 242 245 245
str MOPS 12.2 11.8 18.7 19.5

Same Java version produces same result no matter the platform.

It is really a micro-benchmark - some may call it a
nano-benchmark - and the result is manual picked and rounded
from 10 runs, so not a particular good test.
But the code is attached below, so everybody can run
it themselves.

Arne

****

import java.text.NumberFormat;
import java.text.DecimalFormat;

public class JvmTest {
private static NumberFormat nf = new DecimalFormat("0.0000");
private static void printres(long t1, long t2, int n1, int n2,
String ops) {
double xperf = (double)n1 * (double)n2 / ((t2 - t1) / 1000.0) ;
String sperf = nf.format(xperf / 1000000);
System.out.println(sperf + " million " + ops + " per second");
}
private final static int NINT = 10000;
private final static int NFP = 1000;
private final static int NSTR = 1000;
private final static int N = 1000000;
public static void testint(int scale) {
int nintscale = NINT / scale;
long t1 = System.currentTimeMillis();
for(int i = 0; i < nintscale; i++) {
int sum = i;
for(int j = 0; j < N; j++) {
sum = ((sum + 1) * 2 + 1) / 2;
}
if(sum != (i + N)) {
System.out.println("Integer test error");
System.exit(1);
}
}
long t2 = System.currentTimeMillis();
printres(t1, t2, nintscale, N, "integer operations");
}
public static void testfp(int scale) {
int nfpscale = NFP / scale;
long t1 = System.currentTimeMillis();
for(int i = 0; i < nfpscale; i++) {
double sum = i;
for(int j = 0; j < N; j++) {
sum = ((sum + 1) * 2 + 1) / 2;
}
if(Math.abs(sum - (i + 1.5 * N)) > 1) {
System.out.println("Floating point test error");
System.exit(1);
}
}
long t2 = System.currentTimeMillis();
printres(t1, t2, nfpscale, N, "floating point operations");
}
private final static String ALFA = "ABCDEFGHIJKLMNOPQRSTUVWXYZ";
private final static int VMS_FACTOR = 100;
public static void teststr(int scale) {
int nstrscale = NSTR / scale;
long t1 = System.currentTimeMillis();
for(int i = 0; i < nstrscale * VMS_FACTOR; i++) {
StringBuffer sb = new StringBuffer("");
for(int j = 0; j < N / VMS_FACTOR; j = j + 10) {
String s = ALFA + ALFA;
int ix = (i + j) % ALFA.length();
sb.append(s.substring(ix, ix + 1) + s.substring(ix + 1,
ix + 3) + s.substring(ix + 3, ix + 6) + s.substring(ix + 6, ix + 10));
}
int ix1 = (N / VMS_FACTOR) / 3;
int ix2 = 2 * (N / VMS_FACTOR) / 3;
if(sb.length() != (N / VMS_FACTOR) || sb.charAt(ix1) !=
ALFA.charAt((i + ix1) % ALFA.length()) || sb.charAt(ix2) !=
ALFA.charAt((i + ix2) % ALFA.length())) {
System.out.println("String test error");
System.exit(1);
}
}
long t2 = System.currentTimeMillis();
printres(t1, t2, nstrscale, N / 10, "string operations");
}
private final static int REP = 10;
public static void main(String[] args) {
System.out.println("Java:");
int scale = 1;
if(args.length > 0) {
scale = Integer.parseInt(args[0]);
}
for(int i = 0; i < REP; i++) {
testint(scale);
}
for(int i = 0; i < REP; i++) {
testfp(scale);
}
for(int i = 0; i < REP; i++) {
teststr(scale);
}
}
}

On 10/9/2023 6:49 PM, Arne Vajhøj wrote:
> On 10/9/2023 12:34 AM, Jake Hamby (Solid State Jake) wrote:
>> Next up, DaCapo benchmarks. I'll give the description for the three
>> that worked on VMS. Results are times in ms (lower is better), with
>> the average of 2 runs, or 3 for Eclipse, plus % ratio of Linux / VMS
>> time.
>>
>> "H2 executes a TPC-C like benchmark written by Apache as part of the
>> Apache Derby database. The application models: customers, districts,
>> warehouses, purchases and deliveries.",
>>
>> VMS score: 27135.0 (10.13% of Linux)
>>
>> "jython executes (interprets) the pybench benchmark"
>>
>> VMS score: 22000.0 (14.92% of Linux)
>>
>> "Run eclipse's jdt (non-gui) performance tests"
>>
>> VMS score: 169355.33 (9.09% of Linux)
>>
>> These results are a bit more dismal. Let's see what the Renaissance
>> Java benchmarks look like. In addition to having to "define/job
>> JAVA$DELETE_ALL_VERSIONS TRUE" for Scala Dotty to work (otherwise it
>> fails with a directory not empty error trying to delete a subdir),
>> for the Apache Spark benchmarks to work,
>
>> The in-memory database test failed with an
>> "org.mapdb.DBException$VolumeIOError" exception which I didn't
>> investigate, and the ALS Movie Lens benchmark failed due to missing
>> "chmod" command. This time, the benchmark results were in CSV format,
>> and I averaged them without the first run's score (which tended to be
>> noticeably slower with the JIT having to warm up) and converted to
>> ms.
>>
>> Scala Dotty: 8042.981 (10.39%)
>> Random Forest: 4008.678 (14.53%)
>> Apache Spark ALS: 34253.451 (5.52%)
>> Apache Spark Bayes: 31865.199 (5.73%)
>> Savina Reactors.IO: 24028.649 (24.79%)
>> Apache Spark PageRank: 10485.158 (22.20%)
>> Finagle HTTP Requests: 74394.711 (3.91%)
>> Akka Unbalanced Cobwebbed Tree: 34890.521 (25.9%)
>> Genetic Algorithm (Jenetics + Futures): 2520.791 (62.56%)
>>
>> Finally, the Sunflow ray-tracing benchmark.
>>
>> VMS score: 4470 ms (36.22%)
>
> I assume these must include some disk file IO and network IO.
>
> It is well known that disk file IO are slower on VMS
> than on *nix. Benchmarks has consistently shown that
> for 25 years.
>
> I would also not be surprised if network IO on VMS is slower than
> on *nix.

I also had some code testing disk file IO. A silly line
counting program. Code again attached below.

VMS 9 Windows 10 Ubuntu 22 Windows 10
Java 8 Java 8 Java 17 Java 17
read lines 4.6s 3.3s 3.2s 3.3s
map file 1.0s 0.5s 0.3s 0.5s

VMS is significant slower.

But actually less than what I expected.

Arne

****

import java.io.FileWriter;
import java.io.IOException;
import java.io.PrintWriter;

public class Gen {
private static final int SIZE = 1000000000;
public static void main(String[] args) throws IOException {
System.out.printf("%s %s / %s %s\n",
System.getProperty("java.vendor"), System.getProperty("java.version"),
System.getProperty("os.name"), System.getProperty("os.version"));
long t1 = System.currentTimeMillis();
try(PrintWriter pw = new PrintWriter(new FileWriter(args[0]))) {
for(int i = 0; i < SIZE / 10; i++) {
for(int j = 0; j < 10 -
System.getProperty("line.separator").length(); j++) {
pw.print(Integer.toString(i % 10));
}
pw.println();
}
}
long t2 = System.currentTimeMillis();
System.out.printf("Gen : %d ms\n", t2 - t1);
}
}

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;

public class Read2 {
private static void test(String fnm) throws IOException {
long t1 = System.currentTimeMillis();
try(BufferedReader br = new BufferedReader(new FileReader(fnm))) {
int n = 0;
@SuppressWarnings("unused")
String line;
while((line = br.readLine()) != null) {
n++;
}
System.out.printf("%d lines\n", n);
}
long t2 = System.currentTimeMillis();
System.out.printf("Read one line at a time : %d ms\n", t2 - t1);
}
public static void main(String[] args) throws IOException {
System.out.printf("%s %s / %s %s\n",
System.getProperty("java.vendor"), System.getProperty("java.version"),
System.getProperty("os.name"), System.getProperty("os.version"));
for(int i = 0; i < 10; i++) {
test(args[0]);
}
}
}

import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.channels.FileChannel.MapMode;
import java.nio.file.Paths;

public class Read3 {
private static final int OFFSET = 0;
private static final int SIZE = 1000000000;
private static void test(String fnm) throws IOException {
long t1 = System.currentTimeMillis();
try(FileChannel fc = FileChannel.open(Paths.get(fnm))) {
ByteBuffer bb = fc.map(MapMode.READ_ONLY, OFFSET, SIZE);
bb.rewind();
int n = 0;
for(int ix = 0; ix < SIZE; ix++) {
byte b = bb.get(ix);
if(b == '\n') {
n++;
}
}
System.out.printf("%d lines\n", n);
}
long t2 = System.currentTimeMillis();
System.out.printf("Map file to memory : %d ms\n", t2 - t1);
}
public static void main(String[] args) throws IOException {
System.out.printf("%s %s / %s %s\n",
System.getProperty("java.vendor"), System.getProperty("java.version"),
System.getProperty("os.name"), System.getProperty("os.version"));
for(int i = 0; i < 10; i++) {
test(args[0]);
}
}
}

On 10/9/2023 8:28 PM, Arne Vajhøj wrote:
> On 10/9/2023 6:49 PM, Arne Vajhøj wrote:
>> It is well known that disk file IO are slower on VMS
>> than on *nix. Benchmarks has consistently shown that
>> for 25 years.
>>
>> I would also not be surprised if network IO on VMS is slower than
>> on *nix.
>
> I also had some code testing disk file IO. A silly line
> counting program. Code again attached below.
>
>
>              VMS 9     Windows 10   Ubuntu 22   Windows 10
>              Java 8    Java 8       Java 17     Java 17
> read lines    4.6s       3.3s         3.2s        3.3s
> map file      1.0s       0.5s         0.3s        0.5s
>
> VMS is significant slower.
>
> But actually less than what I expected.

Oh and Read3 crashes pretty hard after 3 rounds.

Which someone should probably look at.

Arne

вторник, 10 октября 2023 г. в 02:47:48 UTC+2, Arne Vajhøj:
> Oh and Read3 crashes pretty hard after 3 rounds.
Yes :( I've created a ticket to fix the issue.

On Monday, October 9, 2023 at 10:55:38 PM UTC-7, Vitaly Pustovetov wrote:
> вторник, 10 октября 2023 г. в 02:47:48 UTC+2, Arne Vajhøj:
> > Oh and Read3 crashes pretty hard after 3 rounds.
> Yes :( I've created a ticket to fix the issue.

I'm going to start a new thread with my results for OpenSSH, C++ microbenchmarks, Perl, PHP, and PostMark.

Jake

понедельник, 9 октября 2023 г. в 06:34:22 UTC+2, Jake Hamby (Solid State Jake):
.....
> I also had to add "-Dspark.file.transferTo=false" to the Java line or else I get this exception:
>
> 23/10/07 19:16:10 ERROR Executor: Exception in task 5.0 in stage 1.0 (TID 6)
> java.lang.AssertionError: assertion failed:
> Current position 0 do not equal to expected position 27650
> after transferTo, please check your kernel version to see if it is 2.6.32,
> this is a kernel bug which will lead to unexpected behavior when using transferTo.
> You can set spark.file.transferTo = false to disable this NIO feature.
>
> at scala.Predef$.assert(Predef.scala:279)
> at org.apache.spark.util.Utils$.$anonfun$copyFileStreamNIO$2(Utils.scala
> :459)
> at org.apache.spark.util.Utils$.$anonfun$copyFileStreamNIO$2$adapted(Uti
> ls.scala:450)
> at scala.Option.foreach(Option.scala:437)
> at org.apache.spark.util.Utils$.copyFileStreamNIO(Utils.scala:450)
> at org.apache.spark.util.Utils.copyFileStreamNIO(Utils.scala)
> at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.writeParti
> tionedDataWithChannel(BypassMergeSortShuffleWriter.java:246)

I hope I've fixed the bug. After fix
$ java -jar DISK$WORK:[000000.xxxxxxx]renaissance-gpl-0^.14^.2.jar dec-tree
.....
====== dec-tree (apache-spark) [default], iteration 0 started =====GC before operation: completed in 739.993 ms, heap usage 129.732 MB -> 28.748 MB.
====== dec-tree (apache-spark) [default], iteration 0 completed (91859.081 ms) =========== dec-tree (apache-spark) [default], iteration 1 started =====GC before operation: completed in 1639.984 ms, heap usage 74.755 MB -> 53.232 MB.
====== dec-tree (apache-spark) [default], iteration 1 completed (14959.850 ms) =========== dec-tree (apache-spark) [default], iteration 2 started =====GC before operation: completed in 1409.986 ms, heap usage 209.433 MB -> 53.864 MB.
====== dec-tree (apache-spark) [default], iteration 2 completed (12819.872 ms) =====