I don't want to resurrect the long off-topic thread that started out asking about writing Icelandic words in a DECterm, but I do have something to add to the topic of ISO-8859 and UTF-8 on OpenVMS.

I've been working on updating the GNV ports, starting from the latest Mercurial versions, and then patching to use __NEW_STARLET and the latest C99 features from V8.4-2L1 (+ the CRTL C99 ECO), since both are now available to hobbyists. Finally, a C99 version of <stdint.h> and other functions I've been wanting for 20 years.

When it came time to test how well bash 4.4.23 worked after my patches and updates, I discovered that one self-test was extremely slow and CPU-bound. Profiling it, I found that 90% of the entire CPU time was spent in setlocale(), because bash apparently likes to call setlocale() to save and restore the LC_* settings, a lot.

I realized that I'd previously defined SYS$LC_ALL to "EN_US_ISO8859-1-EURO" in my sylogicals .com, so I figured I could undefine that and that the default built-in "C" locale would be much faster, and that's exactly what I found. Returning to the default locale made the calls to setlocale() vanish into the noise.

I did discover that the DECC$LOCALE_CACHE_SIZE feature does absolutely nothing worthwhile, whether I set it to 1MB, 10MB, or 100MB. I also tried setting "UTF8-50" as a locale, but I had problems with earlier test cases when I did that. I can't imagine it'd be faster.

This is on Alpha, so I'll test later if this issue was fixed in 8.4-2L3 on I64. As it is, I'd be wary of setting my user locale to anything but "C", much less the system locale as I'd been doing.

GNV bash has many hacks to avoid spawning subprocesses, and one of them is to save/restore its own environment when calling other shell scripts, rather than fork()'ing a separate shell. As a consequence of that, it's making many, many calls to setlocale(), both to get and to set the locale, so it's not a matter of caching the values that it returns.

Relatedly, when I installed the I18N kit on Alpha, I couldn't start the DECwindows server any more: it would hang at the crosshatch screen because dtlogin wasn't starting. Fortunately, I found a workaround by commenting out the lines that deassigned the system logical DECW$USEXLIBXPG4, which is defined by an earlier startup script (sys$startup:VMS$LPBEGIN-050_STARTUP). So apparently DECW$USEXLIBXPG4 is still required, for some reason, when the locales are installed.

Maybe that bug has been fixed already, because I haven't seen that issue with the I18N package on V8.4-2L3. I've definitely seen it in V8.4, so it's something to be aware of, along with setlocale() being very slow and the locale cache being seemingly useless at speeding repeated calls to it.

On Wednesday, June 1, 2022 at 3:48:58 PM UTC-4, jake....@gmail.com wrote:
> I don't want to resurrect the long off-topic thread that started out asking about writing Icelandic words in a DECterm, but I do have something to add to the topic of ISO-8859 and UTF-8 on OpenVMS.
>
> I've been working on updating the GNV ports, starting from the latest Mercurial versions, and then patching to use __NEW_STARLET and the latest C99 features from V8.4-2L1 (+ the CRTL C99 ECO), since both are now available to hobbyists. Finally, a C99 version of <stdint.h> and other functions I've been wanting for 20 years.
>
> When it came time to test how well bash 4.4.23 worked after my patches and updates, I discovered that one self-test was extremely slow and CPU-bound.. Profiling it, I found that 90% of the entire CPU time was spent in setlocale(), because bash apparently likes to call setlocale() to save and restore the LC_* settings, a lot.
>
> I realized that I'd previously defined SYS$LC_ALL to "EN_US_ISO8859-1-EURO" in my sylogicals .com, so I figured I could undefine that and that the default built-in "C" locale would be much faster, and that's exactly what I found. Returning to the default locale made the calls to setlocale() vanish into the noise.
>
> I did discover that the DECC$LOCALE_CACHE_SIZE feature does absolutely nothing worthwhile, whether I set it to 1MB, 10MB, or 100MB. I also tried setting "UTF8-50" as a locale, but I had problems with earlier test cases when I did that. I can't imagine it'd be faster.
>
> This is on Alpha, so I'll test later if this issue was fixed in 8.4-2L3 on I64. As it is, I'd be wary of setting my user locale to anything but "C", much less the system locale as I'd been doing.
>
> GNV bash has many hacks to avoid spawning subprocesses, and one of them is to save/restore its own environment when calling other shell scripts, rather than fork()'ing a separate shell. As a consequence of that, it's making many, many calls to setlocale(), both to get and to set the locale, so it's not a matter of caching the values that it returns.
>
> Relatedly, when I installed the I18N kit on Alpha, I couldn't start the DECwindows server any more: it would hang at the crosshatch screen because dtlogin wasn't starting. Fortunately, I found a workaround by commenting out the lines that deassigned the system logical DECW$USEXLIBXPG4, which is defined by an earlier startup script (sys$startup:VMS$LPBEGIN-050_STARTUP). So apparently DECW$USEXLIBXPG4 is still required, for some reason, when the locales are installed.
>
> Maybe that bug has been fixed already, because I haven't seen that issue with the I18N package on V8.4-2L3. I've definitely seen it in V8.4, so it's something to be aware of, along with setlocale() being very slow and the locale cache being seemingly useless at speeding repeated calls to it.
I'll have somebody look at it.

On 6/1/22 8:37 PM, John Reagan wrote:
> On Wednesday, June 1, 2022 at 3:48:58 PM UTC-4, jake....@gmail.com wrote:
>> I don't want to resurrect the long off-topic thread that started out asking about writing Icelandic words in a DECterm, but I do have something to add to the topic of ISO-8859 and UTF-8 on OpenVMS.
>>
>> I've been working on updating the GNV ports, starting from the latest Mercurial versions, and then patching to use __NEW_STARLET and the latest C99 features from V8.4-2L1 (+ the CRTL C99 ECO), since both are now available to hobbyists. Finally, a C99 version of <stdint.h> and other functions I've been wanting for 20 years.
>>
>> When it came time to test how well bash 4.4.23 worked after my patches and updates, I discovered that one self-test was extremely slow and CPU-bound. Profiling it, I found that 90% of the entire CPU time was spent in setlocale(), because bash apparently likes to call setlocale() to save and restore the LC_* settings, a lot.
>>
>> I realized that I'd previously defined SYS$LC_ALL to "EN_US_ISO8859-1-EURO" in my sylogicals .com, so I figured I could undefine that and that the default built-in "C" locale would be much faster, and that's exactly what I found. Returning to the default locale made the calls to setlocale() vanish into the noise.
>>
>> I did discover that the DECC$LOCALE_CACHE_SIZE feature does absolutely nothing worthwhile, whether I set it to 1MB, 10MB, or 100MB. I also tried setting "UTF8-50" as a locale, but I had problems with earlier test cases when I did that. I can't imagine it'd be faster.
>>
>> This is on Alpha, so I'll test later if this issue was fixed in 8.4-2L3 on I64. As it is, I'd be wary of setting my user locale to anything but "C", much less the system locale as I'd been doing.
>>
>> GNV bash has many hacks to avoid spawning subprocesses, and one of them is to save/restore its own environment when calling other shell scripts, rather than fork()'ing a separate shell. As a consequence of that, it's making many, many calls to setlocale(), both to get and to set the locale, so it's not a matter of caching the values that it returns.
>>
>> Relatedly, when I installed the I18N kit on Alpha, I couldn't start the DECwindows server any more: it would hang at the crosshatch screen because dtlogin wasn't starting. Fortunately, I found a workaround by commenting out the lines that deassigned the system logical DECW$USEXLIBXPG4, which is defined by an earlier startup script (sys$startup:VMS$LPBEGIN-050_STARTUP). So apparently DECW$USEXLIBXPG4 is still required, for some reason, when the locales are installed.
>>
>> Maybe that bug has been fixed already, because I haven't seen that issue with the I18N package on V8.4-2L3. I've definitely seen it in V8.4, so it's something to be aware of, along with setlocale() being very slow and the locale cache being seemingly useless at speeding repeated calls to it.
> I'll have somebody look at it.

David Jones discovered some years ago that strerror() has similar
problems. Every invocation hunts through a search list to find, open,
read from, and close the locale-specific message catalog file.

OK. I've sended a patch for the issue to QA. So if everything goes well, then the patch will be in the next CRTL ECO kit.

Suite: testlocale_long
Test: testlocale() ...passed
Test: testlocale_cache() ...passed

Run Summary - Run Failed Inactive Skipped
Suites : 1 0 0 0
Asserts : 200006 0 n/a n/a
Tests : 2 0 0 0

Elapsed Time: 38.430(s)

and after the patch
$ define DECC$SHR WORK:[XXX.ACRTL.BIN]DECC$SHR.EXE

Suite: testlocale_long
Test: testlocale() ...passed
Test: testlocale_cache() ...passed

Run Summary - Run Failed Inactive Skipped
Suites : 1 0 0 0
Asserts : 200006 0 n/a n/a
Tests : 2 0 0 0

Elapsed Time: 0.470(s)