I updated/upgraded my main VMS system (rx2600) to the VSI Community
stuff (VMS V8.4-2L3, et al.) last week, without too much pain.

Since then, on multiple occasions, I've caught one or both CPUs being
occupied 100% by one (or two) of those APACHE$SWS00xx subprocesses.
(When the second one appears, the system seems to slow noticeably.
Which did not amaze me.) I never saw this behavior before installing
the new software. (Before I shut it down for the update, up-time was
about 888 days, which seemed to me to be a fair sample size.)

In some cases, SYS$STARTUP:APACHE$SHUTDOWN.COM does not kill them all
(within a minute or so, at least). STOP /IDENT works.

The only significant changes to the http.conf file should be those
required by the new version. (Remove "AcceptMutex vmsdlm" and
"DefaultTypetext", fiddle with the LoadModule directives, ...) But, in
any case, I wouldn't expect anything legal to cause such runaway
processes.

Am I the first/only one to observe this?

Is there any useful information which an ignoramus might collect to
help someone who knew something to analyze the problem?

P.S. Just happened again, APACHE$SWS000E

On 12/27/2021 7:36 PM, Steven Schweda wrote:
> I updated/upgraded my main VMS system (rx2600) to the VSI Community
> stuff (VMS V8.4-2L3, et al.) last week, without too much pain.

I know nearly nothing about any of the open-source software on VMS.

Did your upgrade involve a new version of Apache?

--
-- Rob

> Did your upgrade involve a new version of Apache?

Yes. It's not integrated with the VMS kit, but the Community stuff
included a fresh Apache. I didn't carefully note what I was running
before, but I do have a HP-I64VMS-CSWS-V0202-1-1.PCSI$COMPRESSED* kit
lying around, so I'd guess that that's it. In principle, I could
restore the backup and review the old environment, if necessary.

Steven,

you can at least try to collect some more information to determine, whether the problem is inside Apache or OpenVMS.

$ SHOW PROC/CONT/ID=<pid-of-looping-process>

Any resource consumption except CPU ?
Collect some PC values. In process or system space ?

$ ANALYSE/SYS
SDA> SET PROC/ID=<pid-of-looping-process>
SDA> SHOW PROC/CHAN
.... any Busy channels ?
SDA> SHOW CALL/SUMMARY
.... collect some stack traces. Always the same routine names on the stack ?
SDA> EXA/INS <pc-values-collected-above> - which routine names ?

Volker.

On Tuesday, December 28, 2021 at 8:01:15 AM UTC-5, Volker Halle wrote:
> Steven,
>
> you can at least try to collect some more information to determine, whether the problem is inside Apache or OpenVMS.
>
> $ SHOW PROC/CONT/ID=<pid-of-looping-process>
>
> Any resource consumption except CPU ?
> Collect some PC values. In process or system space ?
>
> $ ANALYSE/SYS
> SDA> SET PROC/ID=<pid-of-looping-process>
> SDA> SHOW PROC/CHAN
> ... any Busy channels ?
> SDA> SHOW CALL/SUMMARY
> ... collect some stack traces. Always the same routine names on the stack ?
> SDA> EXA/INS <pc-values-collected-above> - which routine names ?
>
> Volker.

Set the process priority down to zero too. This will help. I have used the technique mentioned by Volker many times to resolve "hanging" processes. Setting the priority down, will allow for better capture of PCs in a tight loop as little CPU will be used if the system is relatively busy given the process priority.

Apparently, the big TCPIP patch (VSI-I64VMS-TCPIP_PAT-V0507-ECO5O-4,
which was supplied on the I640842L3OE DVD) resolves this problem (along
with the exploding BIND server processes), mercifully obviating any
data collection or actual analysis on my part. Thanks again.

On Wednesday, December 29, 2021 at 10:26:21 PM UTC-5, Steven Schweda wrote:
> Apparently, the big TCPIP patch (VSI-I64VMS-TCPIP_PAT-V0507-ECO5O-4,
> which was supplied on the I640842L3OE DVD) resolves this problem (along
> with the exploding BIND server processes), mercifully obviating any
> data collection or actual analysis on my part. Thanks again.

Wow, a first pertinent and useful writing in many months!

A refreshing change from the unfortunately usual back and forth over largely irrelevant subjest.
The same folks over and over again with their Editor wars, favorite programming languages, how horrible OpenVMS is (Hoff!), how to promote OpenVMS.
That's all fun over a beer, in person, but I find it excruciatingly painful to skip over here in c.o.v.
178 replies plus offspring on 'where is Eisner' ?!
Thank you Steven!

On 12/30/2021 11:43 AM, Hein RMS van den Heuvel wrote:
> On Wednesday, December 29, 2021 at 10:26:21 PM UTC-5, Steven Schweda wrote:

>> Apparently, the big TCPIP patch (VSI-I64VMS-TCPIP_PAT-V0507-ECO5O-4,
>> which was supplied on the I640842L3OE DVD) resolves this problem (along
>> with the exploding BIND server processes), mercifully obviating any
>> data collection or actual analysis on my part. Thanks again.
>
> Wow, a first pertinent and useful writing in many months!

[...]

> Thank you Steven!

Steven's signal-to-noise ratio is about the highest among c.o.v posters.

--
-- Rob

In article <a5fede44-5751-4a14-8428-e91a386cfdfcn@googlegroups.com>, Hein RMS van den Heuvel <heinvandenheuvel@gmail.com> writes:
>On Wednesday, December 29, 2021 at 10:26:21 PM UTC-5, Steven Schweda wrote:
>> Apparently, the big TCPIP patch (VSI-I64VMS-TCPIP_PAT-V0507-ECO5O-4,
>> which was supplied on the I640842L3OE DVD) resolves this problem (along
>> with the exploding BIND server processes), mercifully obviating any
>> data collection or actual analysis on my part. Thanks again.
>
>Wow, a first pertinent and useful writing in many months!
>
>A refreshing change from the unfortunately usual back and forth over largely irrelevant subjest.
>The same folks over and over again with their Editor wars, favorite programming languages, how horrible OpenVMS is (Hoff!), how to promote OpenVMS.
>That's all fun over a beer, in person, but I find it excruciatingly painful to skip over here in c.o.v.
>178 replies plus offspring on 'where is Eisner' ?!
>Thank you Steven!

It's a great workout, Hein. I sit here in the perpetual WuFlu lockdown getting
my exercise by shaking my head as I read the faff and folderol herein.

--
VAXman- A Bored Certified VMS Kernel Mode Hacker VAXman(at)TMESIS(dot)ORG

I speak to machines with the voice of humanity.

On 2021-12-30, Hein RMS van den Heuvel <heinvandenheuvel@gmail.com> wrote:
> On Wednesday, December 29, 2021 at 10:26:21 PM UTC-5, Steven Schweda wrote:
>> Apparently, the big TCPIP patch (VSI-I64VMS-TCPIP_PAT-V0507-ECO5O-4,
>> which was supplied on the I640842L3OE DVD) resolves this problem (along
>> with the exploding BIND server processes), mercifully obviating any
>> data collection or actual analysis on my part. Thanks again.
>
> Wow, a first pertinent and useful writing in many months!
>
> A refreshing change from the unfortunately usual back and forth over largely irrelevant subjest.
> The same folks over and over again with their Editor wars, favorite programming languages, how horrible OpenVMS is (Hoff!), how to promote OpenVMS.
> That's all fun over a beer, in person, but I find it excruciatingly painful to skip over here in c.o.v.
> 178 replies plus offspring on 'where is Eisner' ?!
> Thank you Steven!

What's wrong with coming up with ideas to promote VMS ?

This is exactly the place I would have thought such discussions were
welcome (and needed).

Also, you may not like it, but Stephen is right to say the things he does.
Just because VMS was a leader decades ago, that doesn't mean it doesn't
need changes to stay relevant in today's world. The world is a changing
place and that applies to VMS just as much as it does to everything else.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

> Applying that patch kit seems to have stopped the runaway Apache
> subprocesses, too.

Wrong again.

> $ SHOW PROC/CONT/ID=<pid-of-looping-process>

Process APACHE$SWS000F 14:41:22

State CUR Working set 1648
Cur/base priority 4/4 Virtual pages 13431
Current PC FFFFFFFF.8038C640 CPU time 0 01:20:58.66
Current PSL 00000000 Direct I/O 1504
Current user SP 7ACC90E0 Buffered I/O 10917
PID 20200AEC Page faults 1276
UIC [AP_HTTPD,APACHE$WWW] Event flags C0000001
C0000000

ITS$DKA0:[SYS0.SYSCOMMON.APACHE.][000000]APACHE$HTTPD.EXE;1

Current PC seems to bounce around FFFFFFFF.80388EA0, .80382880,
..803935D0, .800FA270, .801B5270, ..., so not an extremely tight loop?

> Any resource consumption except CPU ?

Not that I've noticed. MONITOR SYSTEM is mostly empty, except for
the runaway(s). Most of the 24GB of physical memory is available
(Total: 3145680, Free: 2485002).

> SDA> SHOW PROC/CHAN

Process index: 02EC Name: APACHE$SWS000F Extended PID: 20200AEC
--------------------------------------------------------------------

Process active channels
-----------------------

Channel CCB Window Status Device/file accessed
------- --- ------ ------ --------------------
0010 7FF06000 00000000 ITS$DKA0:
0020 7FF06020 8A218440 ITS$DKA0:[VMS$COMMON.APACHE]APACHE$HTTPD
..EXE;1
0030 7FF06040 8943A280 ITS$DKA0:[VMS$COMMON.SYSLIB]DECC$SHR.EXE
;1 (section file)
0040 7FF06060 89439B00 ITS$DKA0:[VMS$COMMON.SYSLIB]DPML$SHR.EXE
;1 (section file)
0050 7FF06080 894451C0 ITS$DKA0:[VMS$COMMON.SYSEXE]DCL.EXE;1 (s
ection file)
0060 7FF060A0 89435380 ITS$DKA0:[VMS$COMMON.SYSLIB]DCLTABLES.EX
E;213 (section file)
0070 7FF060C0 89A28600 ITS$DKA0:[VMS$COMMON.APACHE.SPECIFIC.ITS
]APACHE$SWS000F.LOG;1135
0080 7FF060E0 89DB4E40 ITS$DKA0:[VMS$COMMON.APACHE.SPECIFIC.ITS
]APACHE$SWS000F.COM;2
0090 7FF06100 89438380 ITS$DKA0:[VMS$COMMON.SYSLIB]CMA$TIS_SHR.
EXE;1 (section file)
00A0 7FF06120 89435D00 ITS$DKA0:[VMS$COMMON.SYSLIB]LIBOTS.EXE;1
(section file)
00B0 7FF06140 89435500 ITS$DKA0:[VMS$COMMON.SYSLIB]LIBRTL.EXE;1
(section file)
00C0 7FF06160 8A2182C0 ITS$DKA0:[VMS$COMMON.APACHE]APACHE$HTTPD
_SHR.EXE;1
00D0 7FF06180 89437E00 ITS$DKA0:[VMS$COMMON.SYSLIB]SECURESHR.EX
E;1 (section file)
00E0 7FF061A0 89434380 ITS$DKA0:[VMS$COMMON.SYSLIB]SECURESHRP.E
XE;1 (section file)
00F0 7FF061C0 89919140 ITS$DKA0:[VMS$COMMON.APACHE]APACHE$APU_S
HR.EXE;1 (section file)
0100 7FF061E0 89439E00 ITS$DKA0:[VMS$COMMON.SYSLIB]PTHREAD$RTL.
EXE;1 (section file)
0110 7FF06200 89434A80 ITS$DKA0:[VMS$COMMON.SYSLIB]RMISHR.EXE;1
(section file)
0120 7FF06220 89918300 ITS$DKA0:[VMS$COMMON.APACHE]APACHE$APR_S
HR.EXE;1 (section file)
0130 7FF06240 89917B80 ITS$DKA0:[VMS$COMMON.APACHE]APACHE$APR_S
HRP.EXE;1 (section file)
0140 7FF06260 895761C0 ITS$DKA0:[VMS$COMMON.SYSLIB]TCPIP$IPC_SH
R.EXE;1 (section file)
0150 7FF06280 89437300 ITS$DKA0:[VMS$COMMON.SYSLIB]MAILSHR.EXE;
1 (section file)
0160 7FF062A0 89433900 ITS$DKA0:[VMS$COMMON.SYSLIB]MAILSHRP.EXE
;1 (section file)
0170 7FF062C0 89575DC0 ITS$DKA0:[VMS$COMMON.SYSLIB]TCPIP$ACCESS
_SHR.EXE;1 (section file)
0180 7FF062E0 89A02B40 ITS$DKA0:[VMS$COMMON.SYSEXE]RIGHTSLIST.D
AT;1
0190 7FF06300 00000000 MBA2959:
01A0 7FF06320 8A2179C0 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
ACCESS_COMPAT.EXE;1
01B0 7FF06340 89448F40 ITS$DKA0:[VMS$COMMON.SYSMSG]DECC$MSG.EXE
;1 (section file)
01C0 7FF06360 8944A3C0 ITS$DKA0:[VMS$COMMON.SYSMSG]SHRIMGMSG.EX
E;1 (section file)
01D0 7FF06380 89449CC0 ITS$DKA0:[VMS$COMMON.SYSMSG]CLIUTLMSG.EX
E;1 (section file)
01E0 7FF063A0 89575BC0 ITS$DKA0:[VMS$COMMON.SYSMSG]TCPIP$MSG.EX
E;1 (section file)
01F0 7FF063C0 8A217E40 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
ACTIONS.EXE;1
0200 7FF063E0 8A216640 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
ALIAS.EXE;1
0210 7FF06400 8A217B40 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
ASIS.EXE;1
0220 7FF06420 8A215740 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
AUTHN_CORE.EXE;1
0230 7FF06440 8A2164C0 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
AUTHZ_CORE.EXE;1
0240 7FF06460 8A215D40 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
AUTH_BASIC.EXE;1
0250 7FF06480 8A217840 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
AUTHNZ_OPENVMS.EXE;1
0260 7FF064A0 8A2152C0 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
AUTOINDEX.EXE;1
0270 7FF064C0 8A215A40 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
CGI.EXE;1
0280 7FF064E0 8A214CC0 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
DIR.EXE;1
0290 7FF06500 8A2155C0 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
INCLUDE.EXE;1
02A0 7FF06520 8A214240 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
LOG_CONFIG.EXE;1
02B0 7FF06540 8A215140 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
MIME.EXE;1
02C0 7FF06560 8A2137C0 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
NEGOTIATION.EXE;1
02D0 7FF06580 8A2140C0 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
REWRITE.EXE;1
02E0 7FF065A0 8A213DC0 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
SETENVIF.EXE;1
02F0 7FF065C0 8A214840 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
SOCACHE_SHMCB.EXE;1
0300 7FF065E0 8A2128C0 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
UNIQUE_ID.EXE;1
0310 7FF06600 8A213AC0 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
UNIXD.EXE;1
0320 7FF06620 8A2125C0 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
USERDIR.EXE;1
0330 7FF06640 00000000 BG11169:
0340 7FF06660 8A219940 ITS$DKA0:[VMS$COMMON.APACHE.MODULES]MOD_
SSL.EXE;1
0350 7FF06680 00000000 BG11170:
0360 7FF066A0 00000000 MBA2959:
0370 7FF066C0 00000000 MBA2958:
0380 7FF066E0 00000000 MBA2958:
0390 7FF06700 00000000 MBA2959:
03A0 7FF06720 00000000 MBA2959:
03B0 7FF06740 00000000 MBA2960:
03C0 7FF06760 00000000 MBA2960:
03D0 7FF06780 00000000 MBA2978:
03E0 7FF067A0 8A212140 ITS$DKA0:[VMS$COMMON.SYSLIB]SYS$PUBLIC_V
ECTORS.EXE;1
03F0 7FF067C0 00000000 BG11334:
0400 7FF067E0 00000000 BG33460:

Total number of open channels : 64.
SDA>

That "64" looked suspicious, but I didn't see any system or account
quotas which look like that.

SDA> SHOW CALL/SUMMARY

Frame Type (mode) Handle Current PC
------------------------ ----------------- -----------------
Memory Stack Frame (K) 00000000.7FF2E1C8 FFFFFFFF.803A15A0 EXCEPTION+833
A0
Register Stack Frame (K) 00000000.7FF2E0E0 FFFFFFFF.800E9010 SYSTEM_PRIMIT
IVES_MIN+E9010
Cannot display further call frames (invalid CFM)

------------------------ ----------------- -----------------
Register Stack Frame (U) 000007FD.BFFD5458 FFFFFFFF.80388EA0 EXCEPTION+6AC
A0
Memory Stack Frame (U) 000007FD.BFFD53E8 FFFFFFFF.80378600 EXCEPTION+5A4
00
Cannot display further call frames (invalid CFM)

------------------------ ----------------- -----------------
Register Stack Frame (U) 000007FD.BFFD53B0 FFFFFFFF.803884F1 EXCEPTION+6A2
F1
Memory Stack Frame (U) 00000000.0000000F FFFFFFFF.803809D0 EXCEPTION+627
D0
Cannot display further call frames (no unwind descriptor)

------------------------ ----------------- -----------------
Memory Stack Frame (U) 000007FD.BFFD53C8 FFFFFFFF.803879D0 EXCEPTION+697
D0
Memory Stack Frame (U) 00000000.7ACC8920 FFFFFFFF.80385BC0 EXCEPTION+679
C0
Memory Stack Frame (U) 00000000.7ACC8920 FFFFFFFF.80385BC0 EXCEPTION+679
C0
Bottom of stack

------------------------ ----------------- -----------------
Null Frame (U) 000007FD.BFFD54E0 FFFFFFFF.80002510 OTS$MOVE_C+00
270
Memory Stack Frame (U) 00000000.7ACC8810 FFFFFFFF.800FA160 SYSTEM_PRIMIT
IVES_MIN+FA160
Cannot display further call frames (corrupt unwind descriptor)

All of which tells me approximately nothing. But thanks for asking.

On 12/31/21 3:12 PM, Steven Schweda wrote:
>> Applying that patch kit seems to have stopped the runaway Apache
>> subprocesses, too.
>
> Wrong again.

I don't know much but it sounds a lot like this:

https://bz.apache.org/bugzilla/show_bug.cgi?id=63052

although that was supposedly fixed in 2.4.38.

The last thing in any of access_log, error_log, or ssl_request_log right
before it goes cpu-bound may (or may not) be of interest. Or maybe, in
your case, APACHE$SWS000F.LOG;1135.

Also, check for log files with version numbers of 32767 -- it might be
trying to open something that it can't.

SHOW DEV/FULL on the BG devices open by the process couldn't hurt.

Steven,

consider to force a process dump. This will allow analysis of the loop problem, if you can find someone with current listings and map files for the routines involved in the loop...

$ SET PROC/ID=<pid-of-looping-process>/DUMP=NOW

A .DMP file should be generated in the login-directory of the account, this process is running under. You can then 'statically' look at that dump with ANALYZE/SYSTEM dumpfile or ANALYZE/PROCES dumpfile.

According to your other recent entry, you have 2 RX systems. Are both running the same version of Apache etc. Are both system connected to the internet ?

Did the .log file show anything interesting ?

Volker.

Volker Halle schrieb am Samstag, 1. Januar 2022 um 08:03:23 UTC+1:

> You can then 'statically' look at that dump with ANALYZE/SYSTEM dumpfile or ANALYZE/PROCES dumpfile.

Sorry, should have been: ANALYZE/CRASH dumpfile.

Also consider to issue a SDA> READ/EXEC before looking at the stack. It might give better symbolization (i.e. show routine names instead of just execlet names).

Volker.

It's been a while, but I did find some time to try a process dump of
a few of the looping processes. I can't say that I learned much from
4000 lines of "Current Operating Stack (User)". I see a bunch of
(meaningless-to-me):

BUG$_LKBREFNEG+
BUG$_SECAUDEXH+
BUG$_NOTVCBUCB
BUG$_IVWSETLIST
BUG$_INCON_SCHED+
BUG$_NOTVVPVCB
BUG$_DECWINDOWS
BUG$_BADHOMEBLK

some:

MOD_SSL+

which suggested "HTTPS" to me,

some:

SYS$K_VERSION_19+

and one or two:

DCL+

which, if I caused it, might be related to a CGI program which uses
system() to run a command: "mcr tcpip$smtp_sff.exe [...]". (I had some
trouble with that (AH02429 errors) after I created a
TCPIP$SMTP_COMMON:TCPIP$SMTP_LOCAL_ALIASES.TXT file, which an
Apache-owned process couldn't read, but fixing that problem didn't help
with the looping APACHE$SWSxxxx processes.)

A process dump file and some ANAL/CRASH output can be found here, if
anyone is desperate for a good time:

http://aintinode.info/vsi/APACHE$HTTPD.DMP
http://aintinode.info/vsi/APACHE$HTTPD_sda.out

> A process dump file and some ANAL/CRASH output can be found here, if
> anyone is desperate for a good time:

Oops.

http://antinode.info/vsi/APACHE$HTTPD.DMP
http://antinode.info/vsi/APACHE$HTTPD_sda.out

Steven,

I'd start with SDA> SHOW CALL/SUMMARY. It should give you the list of routines called. If you compare this in multiple process dumps, you might spot some common routines towards the bottom of the call chain.

Did you collect PC values from the looping process ? SHOW PROC/CONT/ID=xxx with a log file of the terminal session (or SET HOST/LOG 0) should be able to collect PC values easily.

You could also use ANALYZE/PROCESS on that process dump. DBG> SHOW CALL would be the appropriate command.

Volker.

On Tuesday, April 26, 2022 at 5:34:32 AM UTC-4, Volker Halle wrote:
> Steven,
>
> I'd start with SDA> SHOW CALL/SUMMARY. It should give you the list of routines called. If you compare this in multiple process dumps, you might spot some common routines towards the bottom of the call chain.
>
> Did you collect PC values from the looping process ? SHOW PROC/CONT/ID=xxx with a log file of the terminal session (or SET HOST/LOG 0) should be able to collect PC values easily.
>
> You could also use ANALYZE/PROCESS on that process dump. DBG> SHOW CALL would be the appropriate command.
>
> Volker.

Another potential option is to create a file with a number of "show proc/register" lines in it.
Enter a "set process xxx" where xxx is the process that is looping followed by @filename (file created above). This will give you the register and PC information.
I have used this quite successfully when dealing with looping processes. This provides similar info to the show proc/cont but with less noise.

Dan

Steven Schweda schrieb am Dienstag, 26. April 2022 um 06:37:34 UTC+2:

> http://antinode.info/vsi/APACHE$HTTPD_sda.out

Steven,

when closely looking at that stack output, I see 3 Access Violations on the stack:

00000000.7ACC8EC8 0000000C.00000005 <<< the 32-bit signal array
00000000.7ACC8ED0 00D2A2F0.00000000 MOD_SSL+102F0
00000000.7ACC8ED8 0000001B.8406E2E0 LIBRTL+522E0
00000000.7ACC8EE0 00002604.00000005 CTL$C_CLIDATASZ+0089C <<< 64-bit Signal Array Argument Count
00000000.7ACC8EE8 00000000.0000000C <<< Access violation
00000000.7ACC8EF0 00000000.00000000 <<< reason mask
00000000.7ACC8EF8 0000454D.00D2A2F0 MOD_SSL+102F0 <<< failing virtual address
00000000.7ACC8F00 FFFFFFFF.8406E2E0 LIBRTL+522E0 <<< failing PC
00000000.7ACC8F08 00000000.0000001B <<< failing PS

Just search for '0000000C.00000005' in the output.

This matches the data in the Exception Frame summary.

Volker.

> Did you collect PC values from the looping process ? SHOW
> PROC/CONT/ID=xxx with a log file of the terminal session (or SET
> HOST/LOG 0) should be able to collect PC values easily.

Extracting the data is easier after "SET TERM /DEVI = LA120". A few
minutes' sample of "Current PC" lines should be here (unsorted, and
after "SORT /KEY = (POSI: 16, SIZE: 17)":

http://antinode.info/vsi/csws_loop_3.log_cpc
http://antinode.info/vsi/csws_loop_3.log_cpcs

There are some duplicates.

The whole log:

http://antinode.info/vsi/csws_loop_3.log

> when closely looking at that stack output, I see 3 Access Violations
> on the stack:

Not having looked at any of the actual code, I don't know what that
should be telling me.

Steven ,

all those PCs reported are system space PCs. Using SDA> MAP <address> you might get ideas, which routines are involved.

Access violations on the stack should not be 'normal' and can be indications of software problems futher down the stack. Did you try SDA> SHOW CALL/SUMMARY to have a look at the call stacks in the forced process dumps ?

If you have 2 dumps to compare from the same 'loop', look at the call stacks. You should probably find - working from the bottom of the stack - the same routine names and sequence of calls, then suddenly different routines. The loop may be in the top-most of those routines, before the calls diverge....

Volker.

Volker,

Running out of global sections will definitely make CSWS go berserk, easy enough to check
that possibility.

John

On 4/30/22 7:27 AM, Volker Halle via Info-vax wrote:
> Steven ,
>
> all those PCs reported are system space PCs. Using SDA> MAP <address> you might get ideas, which routines are involved.
>
> Access violations on the stack should not be 'normal' and can be indications of software problems futher down the stack. Did you try SDA> SHOW CALL/SUMMARY to have a look at the call stacks in the forced process dumps ?
>
> If you have 2 dumps to compare from the same 'loop', look at the call stacks. You should probably find - working from the bottom of the stack - the same routine names and sequence of calls, then suddenly different routines. The loop may be in the top-most of those routines, before the calls diverge...
>
> Volker.
> _______________________________________________
> Info-vax mailing list
> Info-vax@rbnsn.com
> http://rbnsn.com/mailman/listinfo/info-vax_rbnsn.com

> I'd start with SDA> SHOW CALL/SUMMARY. [...]

I might get around to it eventually.

> [...] It should give you the list of routines called. If you compare
> this in multiple process dumps, you might spot some common routines
> towards the bottom of the call chain.

I looked at SHOW CALLS in a set of four process dump files. As I
read these, starting from the bottom of the SHOW CALLS list, it got into
a pile of common SHARE$MOD_SSL_CODE2 stuff, and then diverged into
more/different SHARE$MOD_SSL_CODE2 stuff at one of two (close) places.

Then, after more/variable SHARE$MOD_SSL_CODE2 stuff, a
SHARE$DECC$SHR_CODE0, a SHARE$LIBRTL_CODE3, and an ACCVIO. Common stuff
follows:

[...]
[SHARE$MOD_SSL_CODE2 multiple, variable]
SHARE$MOD_SSL_CODE2 0000000000DCBB90 0000000000DCBB90 <
SHARE$MOD_SSL_CODE2 0000000000DC9CA0 0000000000DC9CA0 <
SHARE$MOD_SSL_CODE2 0000000000DAD9F0 0000000000DAD9F0
SHARE$MOD_SSL_CODE2 0000000000DA5B20 0000000000DA5B20
SHARE$MOD_SSL_CODE2 0000000000D539F0 0000000000D539F0
SHARE$MOD_SSL_CODE2 0000000000D55950 0000000000D55950
SHARE$APACHE$HTTPD_SHR_CODE2 0000000000053ED0 000000000009BED0
SHARE$MOD_SSL_CODE2 0000000000D35CE0 0000000000D35CE0
SHARE$APACHE$HTTPD_SHR_CODE2 00000000000C37B0 000000000010B7B0
SHARE$APACHE$HTTPD_SHR_CODE2 00000000000D4550 000000000011C550
SHARE$APACHE$HTTPD_SHR_CODE2 00000000000CBA60 0000000000113A60
SHARE$APACHE$HTTPD_SHR_CODE2 0000000000066B10 00000000000AEB10
*main main 72057 00000000000031A0 00000000000131A0
*main __main 71675 00000000000015E0 00000000000115E0
SHARE$PTHREAD$RTL_CODE2 000000000004AE80 FFFFFFFF844D4E80
SHARE$PTHREAD$RTL_CODE2 00000000000006C0 FFFFFFFF8448A6C0
FFFFFFFF80A40560 FFFFFFFF80A40560
DBG>

Highest lines in common:

SHARE$MOD_SSL_CODE2 0000000000DCBB90 0000000000DCBB90

csws_loop_2022-05-01c.calls line 59
csws_loop_2022-05-04a.calls line 70

SHARE$MOD_SSL_CODE2 0000000000DC9CA0 0000000000DC9CA0

csws_loop_2022-05-01c.calls line 60
csws_loop_2022-05-01d.calls line 83
csws_loop_2022-05-02a.calls line 80

Knowing nothing, I might start to suspect a problem in the mod_ssl
code.

Did anyone ever build a debug edition of this stuff?

The "SHOW CALLS" files are in the usual place:
http://antinode.info/vsi/csws_loop_2022-05-01c.calls
http://antinode.info/vsi/csws_loop_2022-05-01d.calls
http://antinode.info/vsi/csws_loop_2022-05-02a.calls
http://antinode.info/vsi/csws_loop_2022-05-04a.calls

For the record, the meaning of the following wasn't immediately
obvious to me:

%SDA-W-EXCLDATA, data excluded from dump due to insufficient privilege

ITS $ help /mess EXCLDATA
%MSGHLP-F-NOTFOUND, message not found in Help Message database

> Running out of global sections will definitely make CSWS go berserk,
> easy enough to check that possibility.

With a looping process:

ITS $ gbl
Global pages: 38158789, free: 37439616, contiguous: 37439488.
Global sections: 3072, free: 1467.

Without, it's about the same:

ITS $ gbl
Global pages: 38158789, free: 37439712, contiguous: 37439488.
Global sections: 3072, free: 1469.

Also, from time to time, it seems that "SET PROC /DUMP=NOW /ID =
xxxx" might create no .DMP file and/or not stop the process.

Steven,

if looking at your ANALYZE/PROCESS output, you'd know, who to ask (look at the source code tree name).

If you have a looping process, you could issue a couple of SDA> SHOW CALL/SUMMARY commands and compare the output. This might tell you, if the ACCVIOs are part of the original problem or if these are related to the STOP/ID/DUMP=NOW command.

While this may be an interesting exercise, only the support organization for this or these piece(s) of software will be able to further troubleshoot this problem effectively.

Volker.

Steven,

the following call is the lowest common call in ALL of your 4 SHOW CALL output files:

SHARE$MOD_SSL_CODE2 0000000000DC9CA0 0000000000DC9CA0

But without the source code and the linker map, what does it help you ? Except pointing to a possible problem in MOD_SSL code.

Volker.