Quick heads-up: I just finished porting Charles Cazabon's open source "memtester" utility to run with 64-bit pointers on OpenVMS:

https://github.com/jhamby/vms-memtester/

Apart from the expected cursing that size_t is only 32 bits, and after setting my WSMAX, WSQUOTA, and WSEXTENT to sufficiently high values, I discovered that the version of memset() in OpenVMS V8.4-2L3 I64 (I don't have enough RAM on my Alpha to test this) can only fill (memtester fills with 0xff) up to 4GB minus 64KB. If I increase the size beyond that, to (4GB - 8KB) or (4GB - 32KB), the memory region doesn't get fully filled.

Hopefully someone can file as a bug. Definitely a bug to keep in mind when porting apps. I had to add a simple loop to fill the 64-bit region being tested, 4GB - 64KB at a time. I really, really hope that as part of the x86-64 port, VMS will acquire a new ABI mode where size_t (and ideally long as well) are promoted to 64 bits, so that we don't get silent failures by forgetting that memset() can't just fill 6GB in one call.

Jake

On 5/12/22 3:42 PM, Jake Hamby wrote:
> Quick heads-up: I just finished porting Charles Cazabon's open source
> "memtester" utility to run with 64-bit pointers on OpenVMS:
>
> https://github.com/jhamby/vms-memtester/
>
> Apart from the expected cursing that size_t is only 32 bits, and
> after setting my WSMAX, WSQUOTA, and WSEXTENT to sufficiently high
> values, I discovered that the version of memset() in OpenVMS V8.4-2L3
> I64 (I don't have enough RAM on my Alpha to test this) can only fill
> (memtester fills with 0xff) up to 4GB minus 64KB. If I increase the
> size beyond that, to (4GB - 8KB) or (4GB - 32KB), the memory region
> doesn't get fully filled.
>
> Hopefully someone can file as a bug. Definitely a bug to keep in mind
> when porting apps. I had to add a simple loop to fill the 64-bit
> region being tested, 4GB - 64KB at a time. I really, really hope that
> as part of the x86-64 port, VMS will acquire a new ABI mode where
> size_t (and ideally long as well) are promoted to 64 bits, so that we
> don't get silent failures by forgetting that memset() can't just fill
> 6GB in one call.

It may eventually be possible to have a compiler mode that changes the
model from LLP64, as it is now, to LP64 or ILP64, at least for the
clang-based compilers.[1] I'm not actually sure that LLP64 even
considers the possibility that size_t and ptrdiff_t won't match the size
of a long pointer as is the case now on VMS. When Alpha was new, the
top priority for VMS was making all the existing VAX code work without
modification. That priority hasn't really changed in the intervening
decades and will likely be evident in initial compiler releases on
x86_64. Tru64 did not have that limitation since it was brand new with
Alpha and so was (I think) LP64 from the start.

[1] https://unix.org/version2/whatsnew/lp64_wp.html

On Thursday, May 12, 2022 at 4:42:17 PM UTC-4, jake....@gmail.com wrote:
> Quick heads-up: I just finished porting Charles Cazabon's open source "memtester" utility to run with 64-bit pointers on OpenVMS:
>
> https://github.com/jhamby/vms-memtester/
>
> Apart from the expected cursing that size_t is only 32 bits, and after setting my WSMAX, WSQUOTA, and WSEXTENT to sufficiently high values, I discovered that the version of memset() in OpenVMS V8.4-2L3 I64 (I don't have enough RAM on my Alpha to test this) can only fill (memtester fills with 0xff) up to 4GB minus 64KB. If I increase the size beyond that, to (4GB - 8KB) or (4GB - 32KB), the memory region doesn't get fully filled.
>
> Hopefully someone can file as a bug. Definitely a bug to keep in mind when porting apps. I had to add a simple loop to fill the 64-bit region being tested, 4GB - 64KB at a time. I really, really hope that as part of the x86-64 port, VMS will acquire a new ABI mode where size_t (and ideally long as well) are promoted to 64 bits, so that we don't get silent failures by forgetting that memset() can't just fill 6GB in one call.
>
> Jake
Yeah, _memset64 zero-extends the size argument before calling OTS$FILL (which handles 64-bit lengths just fine). As you note, it is all related to size_t being 32-bits and assuming that no single variable can be larger than that. So memset would never have to fill more than that.

However, I don't see this 64kb issue you talk about.

Introducing a new ABI mode is certainly one of the things we are looking at.. Testing C++/clang right now with LLP64 is a challenge.

On Thursday, May 12, 2022 at 6:31:05 PM UTC-4, Craig A. Berry wrote:
> On 5/12/22 3:42 PM, Jake Hamby wrote:
> > Quick heads-up: I just finished porting Charles Cazabon's open source
> > "memtester" utility to run with 64-bit pointers on OpenVMS:
> >
> > https://github.com/jhamby/vms-memtester/
> >
> > Apart from the expected cursing that size_t is only 32 bits, and
> > after setting my WSMAX, WSQUOTA, and WSEXTENT to sufficiently high
> > values, I discovered that the version of memset() in OpenVMS V8.4-2L3
> > I64 (I don't have enough RAM on my Alpha to test this) can only fill
> > (memtester fills with 0xff) up to 4GB minus 64KB. If I increase the
> > size beyond that, to (4GB - 8KB) or (4GB - 32KB), the memory region
> > doesn't get fully filled.
> >
> > Hopefully someone can file as a bug. Definitely a bug to keep in mind
> > when porting apps. I had to add a simple loop to fill the 64-bit
> > region being tested, 4GB - 64KB at a time. I really, really hope that
> > as part of the x86-64 port, VMS will acquire a new ABI mode where
> > size_t (and ideally long as well) are promoted to 64 bits, so that we
> > don't get silent failures by forgetting that memset() can't just fill
> > 6GB in one call.
> It may eventually be possible to have a compiler mode that changes the
> model from LLP64, as it is now, to LP64 or ILP64, at least for the
> clang-based compilers.[1] I'm not actually sure that LLP64 even
> considers the possibility that size_t and ptrdiff_t won't match the size
> of a long pointer as is the case now on VMS. When Alpha was new, the
> top priority for VMS was making all the existing VAX code work without
> modification. That priority hasn't really changed in the intervening
> decades and will likely be evident in initial compiler releases on
> x86_64. Tru64 did not have that limitation since it was brand new with
> Alpha and so was (I think) LP64 from the start.
>
> [1] https://unix.org/version2/whatsnew/lp64_wp.html
There is so much bad code out there that uses size_t or ptrdiff_t instead of intptr_t
that I'm getting tired of even trying to justify it anymore.

In article <af8912e6-0a4c-48e5-8674-d8c75fa29c98n@googlegroups.com>, John Reagan <xyzzy1959@gmail.com> writes:
>{...snip..}
>There is so much bad code out there that uses size_t or ptrdiff_t instead of intptr_t
>that I'm getting tired of even trying to justify it anymore.

But, but, but, ... C!

--
VAXman- A Bored Certified VMS Kernel Mode Hacker VAXman(at)TMESIS(dot)ORG

I speak to machines with the voice of humanity.

On 2022-05-12 22:42, Jake Hamby wrote:
> Quick heads-up: I just finished porting Charles Cazabon's open source "memtester" utility to run with 64-bit pointers on OpenVMS:
>
> https://github.com/jhamby/vms-memtester/
>
> Apart from the expected cursing that size_t is only 32 bits, and after setting my WSMAX, WSQUOTA, and WSEXTENT to sufficiently high values, I discovered that the version of memset() in OpenVMS V8.4-2L3 I64 (I don't have enough RAM on my Alpha to test this) can only fill (memtester fills with 0xff) up to 4GB minus 64KB. If I increase the size beyond that, to (4GB - 8KB) or (4GB - 32KB), the memory region doesn't get fully filled.

I would have expected that you could run this even if you have way less
ram. That's what virtual memory and demand paging is there for.

> Hopefully someone can file as a bug. Definitely a bug to keep in mind when porting apps. I had to add a simple loop to fill the 64-bit region being tested, 4GB - 64KB at a time. I really, really hope that as part of the x86-64 port, VMS will acquire a new ABI mode where size_t (and ideally long as well) are promoted to 64 bits, so that we don't get silent failures by forgetting that memset() can't just fill 6GB in one call.

Sound like it might behaving exactly as it should.
The size argument to memset is a size_t. If size_t is only 32 bits, how
would memset ever be able to set more than 4G?

I would have expected the compiler to also warn you if you threw a
64-bit value as the argument to memset, pointing out the type
incompatibility and risk for truncation.

Johnny

On Friday, May 13, 2022 at 11:48:19 AM UTC-7, Johnny Billquist wrote:
> Sound like it might behaving exactly as it should.
> The size argument to memset is a size_t. If size_t is only 32 bits, how
> would memset ever be able to set more than 4G?
>
> I would have expected the compiler to also warn you if you threw a
> 64-bit value as the argument to memset, pointing out the type
> incompatibility and risk for truncation.

I think I may have been unclear in my original post. I was able to memset up to 0xffff0000 bytes successfully, but not any higher. I originally tried to loop and fill 0xffffe000 (4GB minus one 8K page), and then discovered that memtester failed because the entire range wasn't filled properly. I then verified that memset(buf, 255, 0xffff8000) fails but memset(buf, 255, 0xffff0000) succeeds, at least on OpenVMS V8.4-2L3 on a dual-CPU rx2620.

You're right, I would've expected the compiler to warn me about my original attempt to memset() with a 64-bit value. Maybe I should've added /STANDARD=LATEST. The C compiler is usually so strict about type mismatches by default, but not here.

As for virtual memory and demand paging, this program tests RAM by repeatedly writing to two halves of a giant memory buffer, then comparing them for equality. If I had a week to wait for it, I could possibly run memtester on >4G of virtual memory on an Alpha with 2GB of physical memory, except that my PAGEFILE.SYS isn't big enough and I'm not going to wait a week for it to get to the part where it would pass or fail. :)

Jake

On Thursday, May 12, 2022 at 8:02:49 PM UTC-7, xyzz...@gmail.com wrote:
> Yeah, _memset64 zero-extends the size argument before calling OTS$FILL (which handles 64-bit lengths just fine). As you note, it is all related to size_t being 32-bits and assuming that no single variable can be larger than that. So memset would never have to fill more than that.
>
> However, I don't see this 64kb issue you talk about.

Strange. It's really easy for me to reproduce, by fiddling with the value of max_clear_bytes on line 525 of memtester.c in my fork of the tree: https://github.com/jhamby/vms-memtester

If I call memset() with a value larger than 0xffff0000, then on the third test, after filling the region with random values and then calling memset(buf, 255, length) at the end of the second test, the following tests perform XOR, AND, OR, etc. on the data that's in the buffer, and expect it to be identical in both halves afterwards, because it should've been completely filled with 0xff at the end of the previous test.

> Introducing a new ABI mode is certainly one of the things we are looking at. Testing C++/clang right now with LLP64 is a challenge.

That's good to hear!

Jake

On Saturday, May 14, 2022 at 7:34:53 AM UTC-7, Jake Hamby wrote:
> > Introducing a new ABI mode is certainly one of the things we are looking at. Testing C++/clang right now with LLP64 is a challenge.
> That's good to hear!

I'm hoping that a 64-bit time_t would be part of the new ABI mode as well. The current unsigned 32-bit type is yet another portability issue. At least you no longer have to support VAX floating-point in the new modes, so it should hopefully be just one new mode and set of overrides for all the C/C++ standard library functions that use the modified types.

BTW, I'm assuming that OTS$FILL is an internal routine since it isn't documented or listed in OTS$ROUTINES.H? I did look there, and the closest thing that I saw was OTS$MOVE5, for VAX compatibility, which does more than I needed.

Jake

On Saturday, May 14, 2022 at 10:42:29 AM UTC-4, jake....@gmail.com wrote:
> On Saturday, May 14, 2022 at 7:34:53 AM UTC-7, Jake Hamby wrote:
> > > Introducing a new ABI mode is certainly one of the things we are looking at. Testing C++/clang right now with LLP64 is a challenge.
> > That's good to hear!
> I'm hoping that a 64-bit time_t would be part of the new ABI mode as well.. The current unsigned 32-bit type is yet another portability issue. At least you no longer have to support VAX floating-point in the new modes, so it should hopefully be just one new mode and set of overrides for all the C/C++ standard library functions that use the modified types.
>
> BTW, I'm assuming that OTS$FILL is an internal routine since it isn't documented or listed in OTS$ROUTINES.H? I did look there, and the closest thing that I saw was OTS$MOVE5, for VAX compatibility, which does more than I needed.
>
> Jake
VAX floating isn't related to any pointer/size_t issues. We have VAX floating on the non-C++ compilers. For C++, I'm thinking about writing a C++ class to provide VAX floating support without hacking it into clang.

OTS$FILL is in LIBOTS.EXE, not LIBRTL.EXE. While the LIBOTS routines use the OTS$ prefix, they are not related to the documented OTS$ROUTINES.H routines. They are compiler-private calls (much like how compiler_rt is to LLVM).

On Saturday, May 14, 2022 at 10:34:53 AM UTC-4, jake....@gmail.com wrote:
> On Thursday, May 12, 2022 at 8:02:49 PM UTC-7, xyzz...@gmail.com wrote:
> > Yeah, _memset64 zero-extends the size argument before calling OTS$FILL (which handles 64-bit lengths just fine). As you note, it is all related to size_t being 32-bits and assuming that no single variable can be larger than that. So memset would never have to fill more than that.
> >
> > However, I don't see this 64kb issue you talk about.
> Strange. It's really easy for me to reproduce, by fiddling with the value of max_clear_bytes on line 525 of memtester.c in my fork of the tree: https://github.com/jhamby/vms-memtester
>
> If I call memset() with a value larger than 0xffff0000, then on the third test, after filling the region with random values and then calling memset(buf, 255, length) at the end of the second test, the following tests perform XOR, AND, OR, etc. on the data that's in the buffer, and expect it to be identical in both halves afterwards, because it should've been completely filled with 0xff at the end of the previous test.
> > Introducing a new ABI mode is certainly one of the things we are looking at. Testing C++/clang right now with LLP64 is a challenge.
> That's good to hear!
>
> Jake
I didn't take much time (<1 minute) to try to reproduce. I don't have lots of time to track down github sources. If you can send one to me, I'd appreciate it.

On 2022-05-14 16:26, Jake Hamby wrote:
> On Friday, May 13, 2022 at 11:48:19 AM UTC-7, Johnny Billquist wrote:
>> Sound like it might behaving exactly as it should.
>> The size argument to memset is a size_t. If size_t is only 32 bits, how
>> would memset ever be able to set more than 4G?
>>
>> I would have expected the compiler to also warn you if you threw a
>> 64-bit value as the argument to memset, pointing out the type
>> incompatibility and risk for truncation.
>
> I think I may have been unclear in my original post. I was able to memset up to 0xffff0000 bytes successfully, but not any higher. I originally tried to loop and fill 0xffffe000 (4GB minus one 8K page), and then discovered that memtester failed because the entire range wasn't filled properly. I then verified that memset(buf, 255, 0xffff8000) fails but memset(buf, 255, 0xffff0000) succeeds, at least on OpenVMS V8.4-2L3 on a dual-CPU rx2620.

I think you missed that in your original post you said you tried to
clear 6G. :-)
That was more what my response targeted.

The part about missing the last few bytes do sound like a bug, but I
can't really tell if it actually is there or not, since I don't have a
machine to test on (I only have VAXen here...).

> You're right, I would've expected the compiler to warn me about my original attempt to memset() with a 64-bit value. Maybe I should've added /STANDARD=LATEST. The C compiler is usually so strict about type mismatches by default, but not here.

That do sounds broken if it didn't complain.

> As for virtual memory and demand paging, this program tests RAM by repeatedly writing to two halves of a giant memory buffer, then comparing them for equality. If I had a week to wait for it, I could possibly run memtester on >4G of virtual memory on an Alpha with 2GB of physical memory, except that my PAGEFILE.SYS isn't big enough and I'm not going to wait a week for it to get to the part where it would pass or fail. :)

Fair enough. Execution time can certainly be horrible. But it sounded
like you said it was not possible, which is what I was disagreeing with.
I would most likely not want to try and run this either.

Johnny

On Saturday, May 14, 2022 at 8:11:27 AM UTC-7, xyzz...@gmail.com wrote:
> > Jake
> I didn't take much time (<1 minute) to try to reproduce. I don't have lots of time to track down github sources. If you can send one to me, I'd appreciate it.

I'll put together a standalone test case. BTW, I forgot to mention this would be the __memset() builtin provided by the C compiler and not the one in the CRTL. I haven't tested the non-builtin memset yet.