Volker Halle wrote:
> Jouk,
>
> the data in the CURRENT IRP on DNA2: looks consistent:
>
> ...
> FFFFFFFF.829621D0 IRP$L_STS CCB_LOOKED_UP,FUNC,VIRT >>> 00008012
> ...
> FFFFFFFF.82962260 IRP$L_FUNC IO$_READPBLK >>> 0000000C
> ...
> FFFFFFFF.82962284 IRP$PS_CCB >>> 7FF7C400
> FFFFFFFF.82962288 IRP$L_QIO_P1 Buf addr >>> 01472110
> ...
> FFFFFFFF.82962290 IRP$L_QIO_P2 Length >>> 00007C00
> ...
> FFFFFFFF.82962298 IRP$L_QIO_P3 start LBN >>> 0004C01B
> ...
> FFFFFFFF.829622CC IRP$L_FPC 81D9F3E0 <<< what's at this address ?
> Try SDA> EXA/INS 81D9F3E0
>
> FFFFFFFF.829622F0 IRP$Q_RES_WAIT_STATE >>> 00000000.00000020 ???
> LIB.REQ: 'SCS Resource Wait State'
>
> This MAY be the key to this problem. Please force another crash after a hang. And look at the other (pending) IRPs on this device.
>

on the running system without problems yet:

FFFFFFFF.829B7D8C IRP$L_FPC 00010036
UCB$M_S
HD_SEQCMD_PEND+00036

FFFFFFFF.829B7DB0 IRP$Q_RES_WAIT_STATE 00000000.00000020

so the 00000000.00000020 looks normal

> SDA> EXA/INS 81D9F3E0
> %SDA-E-NOINSTRAN, cannot translate instruction
> SDA>

Jouk,

please try

SDA> EXA/INS @81D9F3E0

I'm interested into which code the FORK PC is pointing...

Volker.

> on the running system without problems yet:
>
> FFFFFFFF.829B7D8C IRP$L_FPC 00010036 UCB$M_SHD_SEQCMD_PEND+00036
>
> FFFFFFFF.829B7DB0 IRP$Q_RES_WAIT_STATE 00000000.00000020
>
> so the 00000000.00000020 looks normal

If there is no hung IRP, the Fork PC seems to contain the PID of the process, which has issued the current IO !

Wait for the next hang and then compare the fields of the current/hung IRP with the one in the previous forced crash.

Volker.

And if USB disk access is hung, the current IRP address in the running does not change - right ?

SDA> SHOW DEV DNA2
SDA> READ SYSDEF
SDA> format @(ucb+ucb$l_irp)

repeat this command. It will always show the SAME IRP with the SAME FPC and IRP$L_QIO_P3 (start address) ?

Volker.

Volker Halle wrote:
>> on the running system without problems yet:
>>
>> FFFFFFFF.829B7D8C IRP$L_FPC 00010036 UCB$M_SHD_SEQCMD_PEND+00036
>>
>> FFFFFFFF.829B7DB0 IRP$Q_RES_WAIT_STATE 00000000.00000020
>>
>> so the 00000000.00000020 looks normal
>
> If there is no hung IRP, the Fork PC seems to contain the PID of the process, which has issued the current IO !
>
> Wait for the next hang and then compare the fields of the current/hung IRP with the one in the previous forced crash.
>
> Volker.

From the next crash dump I find:

FFFFFFFF.82988F8C IRP$L_FPC 00020087
SYS$K_V
ERSION_08+00067

Volker Halle wrote:

> don't worry, it's a ring buffer. In the crash you would then only want to look at the entries for DNA2: (SDA output to file, then SEARCH).

Cyclic means on my machine only 2 hours recorded. So I will miss most of
the hangs, since I detect them after this period (especially when they
happen during the night)

Jouk

Joukj schrieb am Montag, 28. Februar 2022 um 10:58:12 UTC+1:

> From the next crash dump I find:
>
> FFFFFFFF.82988F8C IRP$L_FPC 00020087 SYS$K_VERSION_08+00067

This looks like an internal PID again, not a FORK PC.

If accesses to the USB disk seem to hang, issue a couple of those commands and see, if the IRP and FPC and IRP$L_QIO_P3 are always the same. That would prove, that the SAME IO operation is always current and is stuck.

SDA> SHOW DEV DNA2
SDA> READ SYSDEF

SDA> format @(ucb+ucb$l_irp)

Regarding the IO trace buffer: you could use a bigger buffer (SDA> IO START TRACE/BUFFER=n), only trace /DIO/NOBIO or write a batch job, which dumps the buffer contents to a file SDA> IO SHOW TRACE/DIO every couple of hours.

Volker.

Volker Halle wrote:
> Joukj schrieb am Montag, 28. Februar 2022 um 10:58:12 UTC+1:
>
>> From the next crash dump I find:
>>
>> FFFFFFFF.82988F8C IRP$L_FPC 00020087 SYS$K_VERSION_08+00067
>
> This looks like an internal PID again, not a FORK PC.
>
> If accesses to the USB disk seem to hang, issue a couple of those commands and see, if the IRP and FPC and IRP$L_QIO_P3 are always the same. That would prove, that the SAME IO operation is always current and is stuck.
>
> SDA> SHOW DEV DNA2
> SDA> READ SYSDEF
>
> SDA> format @(ucb+ucb$l_irp)
>

Correct stays the same on the hanging system

Now I get

FFFFFFFF.8299890C IRP$L_FPC 81DB3F50
SYS$DUD
RIVER+17B50

and

SDA> exa/ins @81DB3F50
%SDA-E-NOREAD, unable to access location 00000000.00003088

Jouk

> Correct stays the same on the hanging system
>
> Now I get
>
> FFFFFFFF.8299890C IRP$L_FPC 81DB3F50 SYS$DUDRIVER+17B50
>
> and
>
> SDA> exa/ins @81DB3F50
> %SDA-E-NOREAD, unable to access location 00000000.00003088
>

Now this does look like a fork-PC in DUDRIVER !

Try

SDA> EXA @81DB3F50 ! this looks like a procedure descriptor
SDA> EXA ! should be the routine address
SDA> EXA/INS @. ! first instruction in routine

Volker.

Volker Halle wrote:
>> Correct stays the same on the hanging system
>>
>> Now I get
>>
>> FFFFFFFF.8299890C IRP$L_FPC 81DB3F50 SYS$DUDRIVER+17B50
>>
>> and
>>
>> SDA> exa/ins @81DB3F50
>> %SDA-E-NOREAD, unable to access location 00000000.00003088
>>
>
> Now this does look like a fork-PC in DUDRIVER !
>
> Try
>
> SDA> EXA @81DB3F50 ! this looks like a procedure descriptor
> SDA> EXA ! should be the routine address
> SDA> EXA/INS @. ! first instruction in routine
>
> Volker.

Problem is that the IRP$L_FPC is varying. Now I get on the running
system with the USB hanging:

FFFFFFFF.82974FCC IRP$L_FPC 00000000

I'll try again at the next hang.

Jouk,

please also always check the QIO_P3 and RES_WAIT_STATE values.

If the underlying reason is some obscure resource wait state, did you ever run AUTOGEN on this system ?
Start with @SYS$UPDATE:AUTOGEN GETDATA GENPARAMS FEEDBACK and look at DIFFERENCES SYS$SYSTEM:AGEN$FEEDBACK.DAT

Volker.

Volker Halle wrote:
> Jouk,
>
> please also always check the QIO_P3 and RES_WAIT_STATE values.

Ok. It is now running again. So next time.

>
> If the underlying reason is some obscure resource wait state, did you ever run AUTOGEN on this system ?

of course, but the last one is some time ago.

> Start with @SYS$UPDATE:AUTOGEN GETDATA GENPARAMS FEEDBACK and look at DIFFERENCES SYS$SYSTEM:AGEN$FEEDBACK.DAT

Does this make sense? Since the last time I did run a autogen with
savparams. so this one runs with the same SYS$SYSTEM:AGEN$FEEDBACK.DAT
and a new one is not created (I cannot at the moment since the system is
not "up" for a long enough time to be representative for this system)

I will do a run with savparams when the system is up long enough.

Jouk

Jouk,

sorry, you're right @AUTOGEN SAVPARAMS GENPARAMS FEEDBACK

Volker.

Volker Halle wrote:
> Jouk,
>
> please also always check the QIO_P3 and RES_WAIT_STATE values.
>

Looks like "random" numbers to me, because now I get:

FFFFFFFF.829A68C0 IRP$L_IOQFL 82986580
FFFFFFFF.829A68C4 IRP$L_IOQBL 827554F8
UCB+000B8
FFFFFFFF.829A68C8 IRP$W_SIZE 0230
FFFFFFFF.829A68CA IRP$B_TYPE 0A
FFFFFFFF.829A68CB IRP$B_RMOD 47
FFFFFFFF.829A68CC IRP$L_PID 00010050
SYS$K_VERSION_16+00010
FFFFFFFF.829A68D0 IRP$L_ACB64X_OFFSET 00000028
IRP$Q_PARAM_0
FFFFFFFF.829A68D4 00000000
FFFFFFFF.829A68D8 IRP$L_ACB_FLAGS 00000012
IRP$Q_PARAM_1
FFFFFFFF.829A68DC IRP$L_THREAD_PID 00010050
SYS$K_VERSION_16+00010
FFFFFFFF.829A68E0 IRP$L_KAST 829361C0
IRP$L_MIRP
IRP$L_WIND
FFFFFFFF.829A68E4 IRP$L_UCB 82755440 UCB
FFFFFFFF.829A68E8 IRP$L_CTXB 00054620
SDA$SHARE+22620
IRP$L_IIRP_P0
IRP$L_SHD_IOFL
IRP$PQ_ACB64_AST
FFFFFFFF.829A68EC 00000000
FFFFFFFF.829A68F0 IRP$L_HRB 0269E928
IRP$L_IIRP_P1
IRP$L_MV_TMO
IRP$L_SHAD
IRP$Q_ACB64_ASTPRM
FFFFFFFF.829A68F4 00000000
FFFFFFFF.829A68F8 IRP$Q_USER_THREAD_ID 00000000.00000000
FFFFFFFF.829A6900 IRP$B_EFN 14
FFFFFFFF.829A6901 IRP$B_PRI 3C
FFFFFFFF.829A6902 IRP$B_CLN_INDX 00
FFFFFFFF.829A6903 IRP$B_WLG_FLAGS 00
FFFFFFFF.829A6904 IRP$L_CHAN 00000210
FFFFFFFF.829A6908 IRP$L_CLN_WLE 0269E940
IRP$L_IIRP_P2
IRP$PQ_IOSB
IRP$Q_PARAM_2
FFFFFFFF.829A690C 00000000
FFFFFFFF.829A6910 IRP$L_STS 00008012
IRP$L_STS_OVERLAY
IRP$Q_STATUS
IRP$R_STS_BITS
FFFFFFFF.829A6914 IRP$L_STS2 00000300
FFFFFFFF.829A6918 IRP$PQ_VA_PTE FFFFFEFC.00005488
FFFFFFFF.829A6920 IRP$L_SVAPTE 829A6948
IRP+00088
IRP$PS_BUFIO_PKT
FFFFFFFF.829A6924 IRP$L_BCNT 00007C00
FFFFFFFF.829A6928 IRP$L_BOFF 00000110
FFFFFFFF.829A692C IRP$L_ABOFF 00000110
IRP$L_OBOFF
FFFFFFFF.829A6930 IRP$L_EXTEND 00000000
FFFFFFFF.829A6934 IRP$PS_ALTDDT 00000000
IRP$PS_FDT_CONTEXT
FFFFFFFF.829A6938 IRP$R_DIOBM 00000004.00000000
FFFFFFFF.829A6940 00000008.1F790058
FFFFFFFF.829A6948 000038F1.0016FF09
FFFFFFFF.829A6950 00003E8C.0016FF09
FFFFFFFF.829A6958 00004007.0016FF09
FFFFFFFF.829A6960 00004988.0016FF09
FFFFFFFF.829A6968 00000000.00000000
... ...
FFFFFFFF.829A6990 IRP$L_IOST1 0825D820
IRP$L_MEDIA
FFFFFFFF.829A6994 IRP$B_CARCON 00
IRP$L_IOST2
IRP$L_TT_TERM
IRP$W_SHD_COPY_TYPE
IRP$W_SHD_VUN
FFFFFFFF.829A6995 00
FFFFFFFF.829A6996 IRP$W_SHD_DEV_TYPE 0000
IRP$W_SHD_MSCP_DISK_MODIFIER
FFFFFFFF.829A6998 IRP$L_ABCNT 00000000
IRP$Q_NT_PRVMSK
IRP$Q_STATION
IRP$Q_TT_STATE
FFFFFFFF.829A699C IRP$L_OBCNT 00007C00
FFFFFFFF.829A69A0 IRP$L_FUNC 0000000C
FFFFFFFF.829A69A4 00000000
FFFFFFFF.829A69A8 IRP$L_SEGVBN 0001CB61
SDA+1CB61
IRP$Q_SEGVBN
FFFFFFFF.829A69AC 00000000
FFFFFFFF.829A69B0 IRP$L_DIAGBUF 829A6948
IRP+00088
IRP$L_SCB_BUF
IRP$W_TT_PRMPT
FFFFFFFF.829A69B4 IRP$L_DCD_SRC_UCB 0039A30F
IRP$L_SEQNUM
FFFFFFFF.829A69B8 IRP$L_ARB 00000000
FFFFFFFF.829A69BC IRP$B_CPY_MODE 00
IRP$L_KEYDESC
IRP$L_WLE_PTR
FFFFFFFF.829A69BD 000000
FFFFFFFF.829A69C0 IRP$PS_KPB 828E3800
FFFFFFFF.829A69C4 IRP$PS_CCB 7FF7C400
FFFFFFFF.829A69C8 IRP$L_QIO_P1 01522110
IRP$Q_QIO_P1
FFFFFFFF.829A69CC 00000000
FFFFFFFF.829A69D0 IRP$L_QIO_P2 00007C00
IRP$Q_QIO_P2
FFFFFFFF.829A69D4 00000000
FFFFFFFF.829A69D8 IRP$L_QIO_P3 0001CB61
SDA+1CB61
IRP$Q_PARAM_3
IRP$Q_QIO_P3
FFFFFFFF.829A69DC 00000000
FFFFFFFF.829A69E0 IRP$L_QIO_P4 00000000
IRP$Q_QIO_P4
FFFFFFFF.829A69E4 00000000
FFFFFFFF.829A69E8 IRP$L_QIO_P5 00000000
IRP$Q_QIO_P5
FFFFFFFF.829A69EC 00000000
FFFFFFFF.829A69F0 IRP$L_QIO_P6 00000000
IRP$Q_QIO_P6
FFFFFFFF.829A69F4 00000000
FFFFFFFF.829A69F8 IRP$Q_LBN_64 00000000.00000000
FFFFFFFF.829A6A00 IRP$L_FQFL 82938C68
FFFFFFFF.829A6A04 IRP$L_FQBL 82938C68
FFFFFFFF.829A6A08 IRP$W_CDRPSIZE 0000
FFFFFFFF.829A6A0A IRP$B_CD_TYPE 02
FFFFFFFF.829A6A0B IRP$B_FLCK 20
FFFFFFFF.829A6A0C IRP$L_FPC 00010050
SYS$K_VERSION_16+00010
FFFFFFFF.829A6A10 IRP$Q_FR3 7AEE75F4.81D2FBD0
FFFFFFFF.829A6A18 IRP$Q_FR4 FFFFFFFF.825A87B8
FFFFFFFF.829A6A20 IRP$L_SAVD_RTN 81CE3620
EXE$QXQPPKT
FFFFFFFF.829A6A24 IRP$L_MSG_BUF 00000000
FFFFFFFF.829A6A28 IRP$L_RSPID 00000000
FFFFFFFF.829A6A2C IRP$L_CDT 82637FC0
FFFFFFFF.829A6A30 IRP$Q_RES_WAIT_STATE 00000000.00000000
FFFFFFFF.829A6A38 IRP$L_SCS_STALL_DATA 00000000
FFFFFFFF.829A6A3C IRP$L_RWCPTR 82638904
FFFFFFFF.829A6A40 IRP$L_BD_ADDR 00000000
FFFFFFFF.829A6A44 IRP$L_RBUN 00000000
FFFFFFFF.829A6A48 IRP$L_LBUFH_AD 00000000
FFFFFFFF.829A6A4C IRP$AR_PSB 828D9180
FFFFFFFF.829A6A50 IRP$B_SHD_PIO_CNT 00
IRP$L_ERASE_VBN
IRP$L_LBOFF
IRP$Q_ERASE_VBN
IRP$T_LBUFHNDL
FFFFFFFF.829A6A51 IRP$B_SHD_PIO_ACT 00
FFFFFFFF.829A6A52 IRP$B_SHD_PIO_FLAGS 00
FFFFFFFF.829A6A53 IRP$B_SHD_PIO_ERRCNT 00
FFFFFFFF.829A6A54 IRP$B_SHD_PIO_ERRINDEX 17
IRP$L_CDRPFL
IRP$L_RBUFH_AD
FFFFFFFF.829A6A55 IRP$B_SHD_PIO_ERRSEV 08
FFFFFFFF.829A6A56 87F0
FFFFFFFF.829A6A58 IRP$L_RBOFF F3A90008
IRP$Q_SHD_LOCK_FR0
FFFFFFFF.829A6A5C IRP$L_UBARSRCE 00000000
IRP$L_XCT_LEN
FFFFFFFF.829A6A60 IRP$L_DUTUFLAGS 00000000
IRP$Q_SHD_LOCK_FR1
FFFFFFFF.829A6A64 IRP$W_DUTUCNTR 0000
FFFFFFFF.829A6A66 IRP$W_ENDMSGSIZ 0000
FFFFFFFF.829A6A68 IRP$L_PDT 825A87B8
IRP$Q_SHD_LOCK_FR2
FFFFFFFF.829A6A6C IRP$L_WALK_CDDB 82637A00
FFFFFFFF.829A6A70 IRP$L_WALK_ALCLS 002CF174
IRP$Q_SHD_LOCK_FR4
FFFFFFFF.829A6A74 IRP$L_WALK_SVPC 00000000
FFFFFFFF.829A6A78 IRP$L_LB_CDDB 00000000
IRP$Q_SHD_LOCK_FR5
FFFFFFFF.829A6A7C 00000000
FFFFFFFF.829A6A80 IRP$L_DUTU_RSVD1 00000000
IRP$L_SHD_LOCK_FPC
FFFFFFFF.829A6A84 IRP$L_DUTU_RSVD2 00000000
IRP$L_SHD_PIO_ERROR
IRP$L_WBM_RMTSND_STS
FFFFFFFF.829A6A88 IRP$L_DUTU_RSVD3 00000000
IRP$L_SHD_PIO_LNK
IRP$L_WBM_REFCNT
FFFFFFFF.829A6A8C IRP$L_DUTU_RSVD4 00000000
IRP$L_SHDSPC
FFFFFFFF.829A6A90 IRP$L_SHD_CONTROL_IRP 00000000
FFFFFFFF.829A6A94 IRP$L_SHD_TEMP 00000000
FFFFFFFF.829A6A98 IRP$Q_SHD_SAVED_R1 00000000.00000000
FFFFFFFF.829A6AA0 IRP$Q_SHD_SAVED_R2 00000000.00000000
FFFFFFFF.829A6AA8 IRP$Q_SHD_SAVED_R4 00000000.00000000
FFFFFFFF.829A6AB0 IRP$L_SHD_SVD_CNT_IRP 00000000
FFFFFFFF.829A6AB4 IRP$L_SHD_SAVED_STATUS 00000000
FFFFFFFF.829A6AB8 IRP$L_SHD_WLG_MODE_FPC 00000000
FFFFFFFF.829A6ABC IRP$L_SHD_PERLKID 00000000
FFFFFFFF.829A6AC0 IRP$L_SHD_EXPEL_TIMER 00000000
FFFFFFFF.829A6AC4 IRP$L_SHD_EXPEL_FLAGS 00000008
FFFFFFFF.829A6AC8 IRP$L_SHD_EXPEL_MASK 00000000
FFFFFFFF.829A6ACC 00000000
FFFFFFFF.829A6AD0 IRP$Q_DATACHECKRETRYCOUNTER 00000000.00000000
IRP$Q_SHD_RESERV_Q8
FFFFFFFF.829A6AD8 IRP$Q_CURRENT_CLONE 00000000.00000000
IRP$Q_SHD_RESERV_Q9
FFFFFFFF.829A6AE0 IRP$Q_SHD_RESERV_Q10 00000000.00000000
FFFFFFFF.829A6AE8 IRP$B_SHD_FLAGS 00
FFFFFFFF.829A6AE9 00000000.000000
IRP$C_LENGTH

Jouk,

I wouldn't call this 'random numbers'. At least those values describing the IO are consistent:

FFFFFFFF.829A69C4 IRP$PS_CCB 7FF7C400 <<< channel control block address
FFFFFFFF.829A69C8 IRP$L_QIO_P1 01522110 <<< buffer address
FFFFFFFF.829A69D0 IRP$L_QIO_P2 00007C00 <<< length
FFFFFFFF.829A69D8 IRP$L_QIO_P3 0001CB61 <<< starting LBN

And if you look at the current IRP a couple of times (when access to the USB disk seems to 'hang'), those values are always the same, i.e. the current IRP does NOT change - correct ?

Volker.

Volker Halle wrote:
> Jouk,
>
> I wouldn't call this 'random numbers'. At least those values describing the IO are consistent:
>
> FFFFFFFF.829A69C4 IRP$PS_CCB 7FF7C400 <<< channel control block address
> FFFFFFFF.829A69C8 IRP$L_QIO_P1 01522110 <<< buffer address
> FFFFFFFF.829A69D0 IRP$L_QIO_P2 00007C00 <<< length
> FFFFFFFF.829A69D8 IRP$L_QIO_P3 0001CB61 <<< starting LBN
>

But the IRP$L_FPC is always different.

> And if you look at the current IRP a couple of times (when access to the USB disk seems to 'hang'), those values are always the same, i.e. the current IRP does NOT change - correct ?
>

After 2 hours I looked again and saw no changes in the IRP and all its
values.

Going to try the autogen now and restart the system

Jouk

Volker Halle wrote:
> Jouk,
>
> I wouldn't call this 'random numbers'. At least those values describing the IO are consistent:
>
> FFFFFFFF.829A69C4 IRP$PS_CCB 7FF7C400 <<< channel control block address
> FFFFFFFF.829A69C8 IRP$L_QIO_P1 01522110 <<< buffer address
> FFFFFFFF.829A69D0 IRP$L_QIO_P2 00007C00 <<< length
> FFFFFFFF.829A69D8 IRP$L_QIO_P3 0001CB61 <<< starting LBN
>
> And if you look at the current IRP a couple of times (when access to the USB disk seems to 'hang'), those values are always the same, i.e. the current IRP does NOT change - correct ?
>
> Volker.

More of the same at the next crash

FFFFFFFF.82954B00 IRP$PS_KPB 8297E240
FFFFFFFF.82954B04 IRP$PS_CCB 7FF7C3A0
FFFFFFFF.82954B08 IRP$L_QIO_P1 0098A110
IRP$Q_QIO_P1
FFFFFFFF.82954B0C 00000000
FFFFFFFF.82954B10 IRP$L_QIO_P2 00007C00
IRP$Q_QIO_P2
FFFFFFFF.82954B14 00000000
FFFFFFFF.82954B18 IRP$L_QIO_P3 000269C5
SDA+269C5
IRP$Q_PARAM_3
IRP$Q_QIO_P3
FFFFFFFF.82954B1C 00000000
FFFFFFFF.82954B20 IRP$L_QIO_P4 00000000
IRP$Q_QIO_P4
FFFFFFFF.82954B24 00000000
FFFFFFFF.82954B28 IRP$L_QIO_P5 00000000
IRP$Q_QIO_P5
FFFFFFFF.82954B2C 00000000
FFFFFFFF.82954B30 IRP$L_QIO_P6 00000000
IRP$Q_QIO_P6
FFFFFFFF.82954B34 00000000
FFFFFFFF.82954B38 IRP$Q_LBN_64 00000000.00000000
FFFFFFFF.82954B40 IRP$L_FQFL 00000000
FFFFFFFF.82954B40 IRP$L_FQFL 00000000
FFFFFFFF.82954B44 IRP$L_FQBL 00000000
FFFFFFFF.82954B48 IRP$W_CDRPSIZE 0000
FFFFFFFF.82954B4A IRP$B_CD_TYPE 00
FFFFFFFF.82954B4B IRP$B_FLCK 00
FFFFFFFF.82954B4C IRP$L_FPC 00000000
FFFFFFFF.82954B50 IRP$Q_FR3 00000000.00000000
FFFFFFFF.82954B58 IRP$Q_FR4 00000000.00000000
FFFFFFFF.82954B60 IRP$L_SAVD_RTN 00000000
FFFFFFFF.82954B64 IRP$L_MSG_BUF 00000000
FFFFFFFF.82954B68 IRP$L_RSPID 00000000
FFFFFFFF.82954B6C IRP$L_CDT 829337D0

Jouk

Volker Halle wrote:

>
> You may consider to start IO tracing ( see http://de.openvms.org/Spring2009/05-sda_extensions.html#IO ) in the runn9ing system, wait for the next hang, force a dump and then look at the IO trace in the dump...
>

I tried to switch on tracing again after the last restart of the
machine. But when I did, I saw no traces of operations on DNA2, while
I'm sure it is functioning. Then I did:

SDA> sh dev dna2

I/O data structures
-------------------
HOPAK$DNA2 WD My Passport 2626 UCB:
82706D80

Device status: 00021810 online,valid,unload,lcl_valid
Characteristics: 1C4D4008 dir,fod,shr,avl,mnt,elg,idv,odv,rnd
20010281 clu,srv,nnm,nlt,dtn
SUD Status 00000000

Owner UIC [000001,000004] Operation count 135411 ORB address
82707180
PID 00000000 Error count 0 DDB address
82706C80
Alloc. lock ID 150007EE Reference count 3 DDT address
81E331A0
Alloc. class 0 Online count 1 SUD address
826EE740
Class/Type 01/B6 BOFF 00000110 VCB address
828B3480
Def. buf. size 512 Byte count 00007C00 CRB address
82706D00
DEVDEPEND 88000101 SVAPTE 829A3CE0 I/O wait queue
82706E38
DEVDEPND2 00000000 SVPN 00006998
DEVDEPND3 00000000 DEVSTS 00000004
FLCK index 3A
DLCK address 82626700

I/O data structures
-------------------
*** I/O request queue is empty ***

--- Volume Control Block (VCB) 828B3480 ---

Volume: HOPAK3 Lock name: HOPAK3
Status: A0 extfid,system
Status2: 06 nocache,mountver
Status3: 00000032 struc_ods5,special_files,noxfccache

Mount count 1 Rel. volume 0 AQB address
826A6D00
Transactions 2 Max. files 16711679 RVT address
82706D80
Free blocks 2204107920 Rsvd. files 10 FCB queue
828B3080
Window size 7 Cluster size 16 Cache blk.
828B2580
Vol. lock ID 3C000908 Def. extend sz. 5
Block. lock ID 3D0005B6 Record size 0

I/O data structures
-------------------

--- ACP Queue Block (AQB) 826A6D00 ---

ACP requests are serviced by the eXtended Qio Processor (XQP)

Status: 14 defsys,xqioproc

Mount count 19 ACP type f11v2 Request queue
00000000
ACP class 0

*** ACP request queue is empty ***

So no IPR now?????? no I/O operations listed?????
I'm sure the USB-disk is working. I'm performing BACKUP and the
backup-save-set is still growing.

Then I checked the USB-DRIVE on the boot-member of the failing one. This
one is already working over 200 days without a hang. It shows in SDA the
same sort of output as shown above. So also no IPR and no I/O operations.

What is going on?????

At least I will let the backup run for now to see if it also hangs in
this "setup".

Jouk

On 2022-03-03, Joukj <joukj@hrem.nano.tudelft.nl> wrote:
> Volker Halle wrote:
>> Jouk,
>>
>> I wouldn't call this 'random numbers'. At least those values describing the IO are consistent:
>>
>> FFFFFFFF.829A69C4 IRP$PS_CCB 7FF7C400 <<< channel control block address
>> FFFFFFFF.829A69C8 IRP$L_QIO_P1 01522110 <<< buffer address
>> FFFFFFFF.829A69D0 IRP$L_QIO_P2 00007C00 <<< length
>> FFFFFFFF.829A69D8 IRP$L_QIO_P3 0001CB61 <<< starting LBN
>>
>> And if you look at the current IRP a couple of times (when access to the USB disk seems to 'hang'), those values are always the same, i.e. the current IRP does NOT change - correct ?
>>
>> Volker.
>
> More of the same at the next crash
>

[snip]

I've been following this for a while and maybe it's time for you and
Volker to try something else if the SDA USB tool previously mentioned
gives you this information. (I've never used USB on VMS, so I don't
know this.)

I'm assuming this is a standard EHCI controller, and that you are using
a USB high speed device, in which case have you tried looking at the
EHCI controller work queues for the device and seeing if the TD entries
are marked as completed or not ?

If the TD entries are pending, that could imply some hardware issue.

If they are all marked as completed (either with an error or not),
that could imply a driver issue.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 03/03/22 13:54, Simon Clubley wrote:
> On 2022-03-03, Joukj<joukj@hrem.nano.tudelft.nl> wrote:
>> Volker Halle wrote:
>>> Jouk,
>>>
>>> I wouldn't call this 'random numbers'. At least those values describing the IO are consistent:
>>>
>>> FFFFFFFF.829A69C4 IRP$PS_CCB 7FF7C400<<< channel control block address
>>> FFFFFFFF.829A69C8 IRP$L_QIO_P1 01522110<<< buffer address
>>> FFFFFFFF.829A69D0 IRP$L_QIO_P2 00007C00<<< length
>>> FFFFFFFF.829A69D8 IRP$L_QIO_P3 0001CB61<<< starting LBN
>>>
>>> And if you look at the current IRP a couple of times (when access to the USB disk seems to 'hang'), those values are always the same, i.e. the current IRP does NOT change - correct ?
>>>
>>> Volker.
>>
>> More of the same at the next crash
>>
>
> [snip]
>
> I've been following this for a while and maybe it's time for you and
> Volker to try something else if the SDA USB tool previously mentioned
> gives you this information. (I've never used USB on VMS, so I don't
> know this.)
>
> I'm assuming this is a standard EHCI controller, and that you are using
> a USB high speed device, in which case have you tried looking at the
> EHCI controller work queues for the device and seeing if the TD entries
> are marked as completed or not ?
>
> If the TD entries are pending, that could imply some hardware issue.
>
> If they are all marked as completed (either with an error or not),
> that could imply a driver issue.
>
> Simon.
>

I haven't been following this either, but usb sticks are consumer
electronics quality, with some having bugs in their firmware, or
not following interface specs. Some may work, others may not, but
other than certifed products, they are not professional quality
kit...

Chris.

Joukj schrieb am Donnerstag, 3. März 2022 um 11:02:20 UTC+1:
>
> Device status: 00021810 online,valid,unload,lcl_valid
> Characteristics: 1C4D4008 dir,fod,shr,avl,mnt,elg,idv,odv,rnd
> 20010281 clu,srv,nnm,nlt,dtn
> SUD Status 00000000
>
> ...
> I/O data structures
> -------------------
> *** I/O request queue is empty ***
>
Jouk,

you'll only see a 'current IRP', if the device is BUSY.

The IRP may be part of a bigger structure called CDRP, and if that one is stuck in some resource wait queue, so is the IRP.

I'll try the IO TRACE on our rx2600...

Volker.

Jouk,

OpenVMS I64 V8.2:

SDA> IO LOAD
SDA> IO START TRACE/DIO/NOBUFIO
SDA> SPAWN DIR DNA0:<*...>/OUT=NLA0:
SDA> IO SHOW TRACE
....
3-MAR 15:56:50.147379 00 84F21E80 0943DFDD 84C200C0 DNA0 DirIO readpblk FFFFFFFF.7D4C68E0 00000200 02000001
3-MAR 15:56:50.125368 00 84F10840 0943DFDB 84C200C0 DNA0 DirIO readpblk FFFFFFFF.7D4C68A8 00000200 02000001

So here it works.

Volker.

Only the SDA> USB SHOW OHCI UCB command may show useful information

Volker.

Volker Halle wrote:
> Jouk,
>
> OpenVMS I64 V8.2:
>
> SDA> IO LOAD
> SDA> IO START TRACE/DIO/NOBUFIO
> SDA> SPAWN DIR DNA0:<*...>/OUT=NLA0:
> SDA> IO SHOW TRACE
> ...
> 3-MAR 15:56:50.147379 00 84F21E80 0943DFDD 84C200C0 DNA0 DirIO readpblk FFFFFFFF.7D4C68E0 00000200 02000001
> 3-MAR 15:56:50.125368 00 84F10840 0943DFDB 84C200C0 DNA0 DirIO readpblk FFFFFFFF.7D4C68A8 00000200 02000001
>
> So here it works.
>

OK after a couple of ours I see some entries in the trace.
Now hope that when it hangs it does not overwrite the buffer again.

Jouk,

I've found an online copy of Ruth Goldenberg's 'VAX/VMS Troubleshooting' book. Page 92 ff talks about CDRPs (Class Driver Request Packet, which includes an IRP), which could be in different wait queues and has some commands, to try to locate those. The book is from 1985, so references VAX V3 and V4.

https://usermanual.wiki/m/dfc54808a6274253010331c408f6f5216a4dc1360555e4fbb42bc54921e3505f.pdf

Volker.