Hi all,

I have an 2 TB USB-disk attached to a DPWS500 running OpenVMS Alpha
V8.4-2L1. After a couple of hours to couple of days the process
reading/writing to the disk seems to hang on the disk access. When some
other process tries to access this disk in this state, it also hangs. A
normal reboot of the machine is not possible anymore since also the
dismount of the disk hangs. I have to reset the machine.

No errors are logged for the device.

What could be the problem and how can I avoid processes hanging on
access to this disk?

I already changed the USB-disk for a brand new one and I changed the
machine to which it is connected. I think I also tried a XP1000.

regards
Jouk

Jouk,

most likely a hardware and/or OpenVMS driver (DNDRIVER isn't it ?) issue.

Did you try a DISMOUNT/ABORT - this must be the first command to dismount the disk. If another DISMOUNT is already hanging, it won't help.

If you have a process hanging reading/writing to that disk, enter SDA, set context to that process and issue a SDA> SHOW PROC/CHAN - the DNcn: channel should be busy

Then also try SDA> SHOW DEV DNcn: to determine the state of the IO.

Instead of 'resetting' the system, try a forced crash at least once (CTRL-P >>> CRASH on the console). This would at least document the state of the system at that time. Then look at the dump with SDA, while the system is up and running again.

And as this is a VSI OpenVMS Alpha version, also try the VSI OpenVMS Forum.

Volker.

On 2022-02-17, Joukj <joukj@hrem.nano.tudelft.nl> wrote:
> Hi all,
>
> I have an 2 TB USB-disk attached to a DPWS500 running OpenVMS Alpha
> V8.4-2L1. After a couple of hours to couple of days the process
> reading/writing to the disk seems to hang on the disk access. When some
> other process tries to access this disk in this state, it also hangs. A
> normal reboot of the machine is not possible anymore since also the
> dismount of the disk hangs. I have to reset the machine.
>
> No errors are logged for the device.
>
> What could be the problem and how can I avoid processes hanging on
> access to this disk?
>

Is the drive going into some deep power down state that the VMS driver
cannot recover from ?

I am very surprised no errors are being logged. On Linux, if the USB
subsystem has any problems, permanent or transient, you _always_ end
up with detailed errors recorded.

Have you checked operator.log to see if anything ended up there instead ?

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

On 2/17/2022 4:15 AM, Joukj wrote:
> Hi all,
>
> I have an 2 TB USB-disk attached to a DPWS500 running OpenVMS Alpha V8.4-2L1.
> After a couple of hours to couple of days the process reading/writing to the
> disk seems to hang on the disk access. When some other process tries to access
> this disk in this state, it also hangs. A normal reboot of the machine is not
> possible anymore since also the dismount of the disk hangs. I have to reset the
> machine.
>
> No errors are logged for the device.
>
> What could be the problem and how can I avoid processes hanging on access to
> this disk?
>
> I already changed the USB-disk for a brand new one and I changed the machine to
> which it is connected. I think I also tried a XP1000.
>
>
> regards
> Jouk

I seem to recall that the USB implementation on older systems from DEC was less
than great. You may discover the problem with a system dump and SDA, but, then
what?

--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: davef@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486

On Thursday, February 17, 2022 at 2:07:29 PM UTC-5, Dave Froble wrote:
> On 2/17/2022 4:15 AM, Joukj wrote:
> > Hi all,
> >
> > I have an 2 TB USB-disk attached to a DPWS500 running OpenVMS Alpha V8.4-2L1.
> > After a couple of hours to couple of days the process reading/writing to the
> > disk seems to hang on the disk access. When some other process tries to access
> > this disk in this state, it also hangs. A normal reboot of the machine is not
> > possible anymore since also the dismount of the disk hangs. I have to reset the
> > machine.
> >
> > No errors are logged for the device.
> >
> > What could be the problem and how can I avoid processes hanging on access to
> > this disk?
> >
> > I already changed the USB-disk for a brand new one and I changed the machine to
> > which it is connected. I think I also tried a XP1000.
> >
> >
> > regards
> > Jouk
> I seem to recall that the USB implementation on older systems from DEC was less
> than great. You may discover the problem with a system dump and SDA, but, then
> what?
>
> --
> David Froble Tel: 724-529-0450
> Dave Froble Enterprises, Inc. E-Mail: da...@tsoft-inc.com
> DFE Ultralights, Inc.
> 170 Grimplin Road
> Vanderbilt, PA 15486

As with any "problem", knowing what exactly is wrong yields a potential solution. With a nebulous report of "hangs", there is nothing to go on here. Is the process waiting for an I/O completion that was missed or never occurred? An examination of the dump would answer this question. Perhaps there is a code path that results in a deadlock and that is what happened here. Who knows without additional investigation?

On 2/17/2022 2:27 PM, abrsvc wrote:
> On Thursday, February 17, 2022 at 2:07:29 PM UTC-5, Dave Froble wrote:
>> On 2/17/2022 4:15 AM, Joukj wrote:
>>> Hi all,
>>>
>>> I have an 2 TB USB-disk attached to a DPWS500 running OpenVMS Alpha V8.4-2L1.
>>> After a couple of hours to couple of days the process reading/writing to the
>>> disk seems to hang on the disk access. When some other process tries to access
>>> this disk in this state, it also hangs. A normal reboot of the machine is not
>>> possible anymore since also the dismount of the disk hangs. I have to reset the
>>> machine.
>>>
>>> No errors are logged for the device.
>>>
>>> What could be the problem and how can I avoid processes hanging on access to
>>> this disk?
>>>
>>> I already changed the USB-disk for a brand new one and I changed the machine to
>>> which it is connected. I think I also tried a XP1000.
>>>
>>>
>>> regards
>>> Jouk
>> I seem to recall that the USB implementation on older systems from DEC was less
>> than great. You may discover the problem with a system dump and SDA, but, then
>> what?
>>
>> --
>> David Froble Tel: 724-529-0450
>> Dave Froble Enterprises, Inc. E-Mail: da...@tsoft-inc.com
>> DFE Ultralights, Inc.
>> 170 Grimplin Road
>> Vanderbilt, PA 15486
>
> As with any "problem", knowing what exactly is wrong yields a potential solution. With a nebulous report of "hangs", there is nothing to go on here. Is the process waiting for an I/O completion that was missed or never occurred? An examination of the dump would answer this question. Perhaps there is a code path that results in a deadlock and that is what happened here. Who knows without additional investigation?
>

Correct Dan, but then what? Re-write the driver, and add to VMS? Perhaps a
non-USB SSD would be an easier "fix"?

It could also be the USB hardware.

--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: davef@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486

On Thursday, February 17, 2022 at 4:49:44 PM UTC-5, Dave Froble wrote:
> On 2/17/2022 2:27 PM, abrsvc wrote:
> > On Thursday, February 17, 2022 at 2:07:29 PM UTC-5, Dave Froble wrote:
> >> On 2/17/2022 4:15 AM, Joukj wrote:
> >>> Hi all,
> >>>
> >>> I have an 2 TB USB-disk attached to a DPWS500 running OpenVMS Alpha V8.4-2L1.
> >>> After a couple of hours to couple of days the process reading/writing to the
> >>> disk seems to hang on the disk access. When some other process tries to access
> >>> this disk in this state, it also hangs. A normal reboot of the machine is not
> >>> possible anymore since also the dismount of the disk hangs. I have to reset the
> >>> machine.
> >>>
> >>> No errors are logged for the device.
> >>>
> >>> What could be the problem and how can I avoid processes hanging on access to
> >>> this disk?
> >>>
> >>> I already changed the USB-disk for a brand new one and I changed the machine to
> >>> which it is connected. I think I also tried a XP1000.
> >>>
> >>>
> >>> regards
> >>> Jouk
> >> I seem to recall that the USB implementation on older systems from DEC was less
> >> than great. You may discover the problem with a system dump and SDA, but, then
> >> what?
> >>
> >> --
> >> David Froble Tel: 724-529-0450
> >> Dave Froble Enterprises, Inc. E-Mail: da...@tsoft-inc.com
> >> DFE Ultralights, Inc.
> >> 170 Grimplin Road
> >> Vanderbilt, PA 15486
> >
> > As with any "problem", knowing what exactly is wrong yields a potential solution. With a nebulous report of "hangs", there is nothing to go on here. Is the process waiting for an I/O completion that was missed or never occurred? An examination of the dump would answer this question. Perhaps there is a code path that results in a deadlock and that is what happened here. Who knows without additional investigation?
> >
> Correct Dan, but then what? Re-write the driver, and add to VMS? Perhaps a
> non-USB SSD would be an easier "fix"?
>
> It could also be the USB hardware.
> --
> David Froble Tel: 724-529-0450
> Dave Froble Enterprises, Inc. E-Mail: da...@tsoft-inc.com
> DFE Ultralights, Inc.
> 170 Grimplin Road
> Vanderbilt, PA 15486

That is true. But until the problem is understood, it is pure speculation on the solution. It could be a really simple fix. I recall a problem that was presented to me as a Cobol program that would "skip" a record in an indexed file using the START statement followed by a READ NEXT. Took almost a month to finally fix the problem that turned out to be a missing dot "." in the code for a CMPC3 instruction for Alpha. The problem would only be seen when the operands spanned an alpha page boundary and a few other things I no longer recall. Difficult to find, simple to fix. Could be the case here too.

On 2/17/22 3:15 AM, Joukj wrote:
> Hi all,
>
> I have an 2 TB USB-disk attached to a DPWS500 running OpenVMS Alpha
> V8.4-2L1. After a couple of hours to couple of days the process
> reading/writing to the disk seems to hang on the disk access. When some
> other process tries to access this disk in this state, it also hangs. A
> normal reboot of the machine is not possible anymore since also the
> dismount of the disk hangs. I have to reset the machine.
>
> No errors are logged for the device.

Logged where?

> What could be the problem and how can I avoid processes hanging on
> access to this disk?
>
> I already changed the USB-disk for a brand new one and I changed the
> machine to which it is connected. I think I also tried a XP1000.

Have you read the section in the system management utilities reference
manual (volume 2) on managing USB devices? Have you done any basic
investigation of what the system thinks about the device, with, say UCM
SHOW DEVICE or UCM SHOW EVENTS? Or just looked at
SYS$SYSTEM:USB$UCM_DEVICES.DAT to see how the device is configured?

Craig A. Berry wrote:
>
> On 2/17/22 3:15 AM, Joukj wrote:
>> Hi all,
>>
>> I have an 2 TB USB-disk attached to a DPWS500 running OpenVMS Alpha
>> V8.4-2L1. After a couple of hours to couple of days the process
>> reading/writing to the disk seems to hang on the disk access. When
>> some other process tries to access this disk in this state, it also
>> hangs. A normal reboot of the machine is not possible anymore since
>> also the dismount of the disk hangs. I have to reset the machine.
>>
>> No errors are logged for the device.
>
> Logged where?
>
>> What could be the problem and how can I avoid processes hanging on
>> access to this disk?
>>
>> I already changed the USB-disk for a brand new one and I changed the
>> machine to which it is connected. I think I also tried a XP1000.
>
> Have you read the section in the system management utilities reference
> manual (volume 2) on managing USB devices? Have you done any basic
> investigation of what the system thinks about the device, with, say UCM
> SHOW DEVICE or UCM SHOW EVENTS? Or just looked at
> SYS$SYSTEM:USB$UCM_DEVICES.DAT to see how the device is configured?

It is working now. Still waiting for the next hang

hopak-jj) ucm sh dev

Bus 0 (OHA0)
Root Hub - 2 port(s)
1.0.0.0.0.0 -> DNA2 - Western Digital My Passport 2626

I will repeat this when it actually hangs

UCM SHOW EVENTT shows only the normal messages at boot time, but nothing
at the time of the hang.

SYS$SYSTEM:USB$UCM_DEVICES.DAT: (currently the Western-digital is
connected, The seagate I used previously gave the same problems)

!
! ! *** DO NOT EDIT THIS FILE ***
! ! This file should only be modified by running the
! USB Configuration Manager Client program
(SYS$MANAGER:USB$UCM_CLIENT.EXE).
! !
! Manual editing of this file may result in making the system unusable.
! ! This file is installed as SYS$SPECIFIC:[SYSEXE]:USB$UCM_DEVICES.DAT.
!

DEVICE = "USB Keyboard Driver"
NAME = KBD
DRIVER = SYS$KBDDRIVER.EXE
BEGIN_PRIVATE
USB_CONFIG_TYPE = TAG
UNIT_NUMBER = 0
BEGIN_INTERFACE
HID_USAGE_DATA = 65542
END_INTERFACE
END_PRIVATE
END_DEVICE

DEVICE = "USB Mouse Driver"
NAME = MOU
DRIVER = SYS$MOUDRIVER.EXE
BEGIN_PRIVATE
USB_CONFIG_TYPE = TAG
UNIT_NUMBER = 0
BEGIN_INTERFACE
HID_USAGE_DATA = 65538
END_INTERFACE
END_PRIVATE
END_DEVICE

DEVICE = "USB Mass storage "
NAME = DNA
DRIVER = SYS$DNDRIVER.EXE
BEGIN_PRIVATE
USB_CONFIG_TYPE = INTERFACE
UNIT_NUMBER = 0
VENDOR_ID = 3010
PRODUCT_ID = 43814
RELEASE_NUMBER = 264
BUS_NUMBER = 0
PATH = 2.0.0.0.0.0
DEVICE_CLASS = 0
DEVICE_SUB_CLASS = 0
DEVICE_PROTOCOL = 0
NUMBER_OF_INTERFACES = 1
CONFIGURATION_VALUE = 1
NUMBER_OF_CONFIGURATIONS = 1
SERIAL_NUMBER = NA9J6MXJ
MANUFACTURER_STRING = Seagate
PRODUCT_STRING = BUP Slim RD
CONFIGURATION_NUMBER = 0
BEGIN_INTERFACE
INTERFACE_CLASS = 8
INTERFACE_SUB_CLASS = 6
INTERFACE_PROTOCOL = 80
END_INTERFACE
END_PRIVATE
END_DEVICE

DEVICE = "USB Mass storage "
NAME = DNA
DRIVER = SYS$DNDRIVER.EXE
BEGIN_PRIVATE
USB_CONFIG_TYPE = INTERFACE
UNIT_NUMBER = 1
VENDOR_ID = 4184
PRODUCT_ID = 4368
RELEASE_NUMBER = 4146
BUS_NUMBER = 0
PATH = 1.0.0.0.0.0
DEVICE_CLASS = 0
DEVICE_SUB_CLASS = 0
DEVICE_PROTOCOL = 0
NUMBER_OF_INTERFACES = 1
CONFIGURATION_VALUE = 1
NUMBER_OF_CONFIGURATIONS = 1
SERIAL_NUMBER = 574341563534393233383830
MANUFACTURER_STRING = Western Digital
PRODUCT_STRING = My Book 1110
CONFIGURATION_NUMBER = 0
BEGIN_INTERFACE
INTERFACE_CLASS = 8
INTERFACE_SUB_CLASS = 6
INTERFACE_PROTOCOL = 80
END_INTERFACE
END_PRIVATE
END_DEVICE

DEVICE = "USB Mass storage "
NAME = DNA
DRIVER = SYS$DNDRIVER.EXE
BEGIN_PRIVATE
USB_CONFIG_TYPE = INTERFACE
UNIT_NUMBER = 2
VENDOR_ID = 4184
PRODUCT_ID = 9766
RELEASE_NUMBER = 4145
BUS_NUMBER = 0
PATH = 1.0.0.0.0.0
DEVICE_CLASS = 0
DEVICE_SUB_CLASS = 0
DEVICE_PROTOCOL = 0
NUMBER_OF_INTERFACES = 1
CONFIGURATION_VALUE = 1
NUMBER_OF_CONFIGURATIONS = 1
SERIAL_NUMBER = 5758523245333146464C3655
MANUFACTURER_STRING = Western Digital
PRODUCT_STRING = My Passport 2626
CONFIGURATION_NUMBER = 0
BEGIN_INTERFACE
INTERFACE_CLASS = 8
INTERFACE_SUB_CLASS = 6
INTERFACE_PROTOCOL = 80
END_INTERFACE
END_PRIVATE
END_DEVICE

Simon Clubley wrote:
> On 2022-02-17, Joukj <joukj@hrem.nano.tudelft.nl> wrote:
>> Hi all,
>>
>
> I am very surprised no errors are being logged. On Linux, if the USB
> subsystem has any problems, permanent or transient, you _always_ end
> up with detailed errors recorded.
>
> Have you checked operator.log to see if anything ended up there instead ?
>
The operator log seems to be empty and the USB$UCM_EVENTS.LOG contains
only the normal messages at boot time.

Volker Halle wrote:
> Jouk,
>
> most likely a hardware and/or OpenVMS driver (DNDRIVER isn't it ?) issue.
>
> Did you try a DISMOUNT/ABORT - this must be the first command to dismount the disk. If another DISMOUNT is already hanging, it won't help.
>
> If you have a process hanging reading/writing to that disk, enter SDA, set context to that process and issue a SDA> SHOW PROC/CHAN - the DNcn: channel should be busy
>
> Then also try SDA> SHOW DEV DNcn: to determine the state of the IO.
>
> Instead of 'resetting' the system, try a forced crash at least once (CTRL-P >>> CRASH on the console). This would at least document the state of the system at that time. Then look at the dump with SDA, while the system is up and running again.
>
> And as this is a VSI OpenVMS Alpha version, also try the VSI OpenVMS Forum.
>
> Volker.

I will try the SDA next time it hangs. Then I'll try the DISMOUNT/ABORT.
I will try the "crash", but I'm not sure if on this system the .DMP is
large enough, so maybe I have to create it first.

Jouk,

make sure to set DUMPSTYLE=9 (selective and compressed) to minimize your dump file size requirements. It's a dynamic parameter, so check and eventually change it in the running system (using SYSGEN> WRITE ACTIVE after changing it).

Volker.

Volker Halle wrote:
> Jouk,
>
> most likely a hardware and/or OpenVMS driver (DNDRIVER isn't it ?) issue.
>
> Did you try a DISMOUNT/ABORT - this must be the first command to dismount the disk. If another DISMOUNT is already hanging, it won't help.

$ dism/abort dna2
%DISM-W-CANNOTDMT, DNA2 cannot be dismounted
%DISM-W-USERFILES, 2 user files open on volume

>
> If you have a process hanging reading/writing to that disk, enter SDA, set context to that process and issue a SDA> SHOW PROC/CHAN - the DNcn: channel should be busy

For the hanging process I get:

Channel CCB Window Status Device/file accessed
------- --- ------ ------ --------------------
01F0 7FF7C3C0 00000000 HOPAK$DNA2:

And then I get my SDA-process in RWAST

>

>
> Then also try SDA> SHOW DEV DNcn: to determine the state of the IO.

SDA> show dev dna2

I/O data structures
-------------------
HOPAK$DNA2 WD My Passport 2626 UCB:
82751A40

Device status: 00021910 online,bsy,valid,unload,lcl_valid
Characteristics: 1C4D4008 dir,fod,shr,avl,mnt,elg,idv,odv,rnd
20010281 clu,srv,nnm,nlt,dtn
SUD Status 00000000

Owner UIC [000001,000004] Operation count 1836467 ORB address
82751E40
PID 00000000 Error count 0 DDB address
8274C840
Alloc. lock ID 34000866 Reference count 4 DDT address
81E331A0
Alloc. class 0 Online count 1 SUD address
826C6B00
Class/Type 01/B6 BOFF 00000110 VCB address
828B18C0
Def. buf. size 512 Byte count 00007C00 CRB address
82751580
DEVDEPEND 88000101 SVAPTE 829715C8 IRP address
82971540
DEVDEPND2 00000000 SVPN 00006B5B Fork R3
82751600
DEVDEPND3 00000000 DEVSTS 00000004 Fork R4
82751A40
FLCK index 3A I/O wait queue
829950C0
DLCK address 826228C0
I/O data structures
-------------------
I/O request queue
-----------------

STATE IRP PID MODE CHAN FUNC WCB EFN AST
IOSB STATUS

C 82971540 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E4B8 8012
readpblk func,virtual

P 829950C0 00010049 K 0010 200C 00000000 128
FFFFFFFF.81D2FBD0 000000
00.7AEE7558 8002
readpblk func

P 8299D100 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E4E0 8012
readpblk func,virtual

Press RETURN for more.
/O data structures
-------------------
P 8297FAC0 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E508 8012
readpblk func,virtual

P 8297A300 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E530 8012
readpblk func,virtual

P 8298CC00 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E558 8012
readpblk func,virtual

P 82979340 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E580 8012
readpblk func,virtual

P 829A4840 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E5A8 8012
readpblk func,virtual

Press RETURN for more.
I/O data structures
-------------------

P 82991400 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E5D0 8012
readpblk func,virtual

P 82961E80 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E5F8 8012
readpblk func,virtual

P 8295AE00 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E620 8012
readpblk func,virtual

P 82997B80 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E648 8012
readpblk func,virtual

Press RETURN for more.
I/O data structures
-------------------
P 82967640 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E670 8012
readpblk func,virtual

P 8296CE00 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E698 8012
readpblk func,virtual

P 829A3AC0 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E6C0 8012
readpblk func,virtual

P 82979C40 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E6E8 8012
readpblk func,virtual

P 8295F600 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E710 8012
readpblk func,virtual

Press RETURN for more.
I/O data structures
-------------------

P 8297DB40 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E738 8012
readpblk func,virtual

P 8298DE00 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E760 8012
readpblk func,virtual

P 8298FB40 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E788 8012
readpblk func,virtual

P 82968F00 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269E7B0 8012
readpblk func,virtual

etc. etc. etc. (hundreds of them

SDA>

I/O data structures
-------------------

P 82999D40 0001004A U 0210 000C 8276D980 20
00000000.00054620 000000
00.0269AC08 8012
readpblk func,virtual

--- Volume Control Block (VCB) 828B18C0 ---

Volume: HOPAK3 Lock name: HOPAK3
Status: A0 extfid,system
Status2: 06 nocache,mountver
Status3: 00000032 struc_ods5,special_files,noxfccache

Mount count 1 Rel. volume 0 AQB address
826A8040
Transactions 3 Max. files 16711679 RVT address
82751A40
Free blocks 2209097424 Rsvd. files 10 FCB queue
828B1340
Window size 7 Cluster size 16 Cache blk.
828B1600
Vol. lock ID 3D000872 Def. extend sz. 5
Block. lock ID 36001577 Record size 0

Press RETURN for more.
SDA>

I/O data structures
-------------------

--- ACP Queue Block (AQB) 826A8040 ---

ACP requests are serviced by the eXtended Qio Processor (XQP)

Status: 14 defsys,xqioproc

Mount count 19 ACP type f11v2 Request queue
00000000
ACP class 0

*** ACP request queue is empty ***

Press RETURN for more.

Press RETURN for more.

I do not know if this indicates anything. Is this "I/O wait queue" OK?

> Instead of 'resetting' the system, try a forced crash at least once (CTRL-P >>> CRASH on the console). This would at least document the state of the system at that time. Then look at the dump with SDA, while the system is up and running again.
>
> And as this is a VSI OpenVMS Alpha version, also try the VSI OpenVMS Forum.
>
> Volker.

Jouk,

please force a crash. Then you can look at the static situation without SDA possibly going to RWAST ;-)

Device 'bsy' indicates, that the device has IOs pending and is probably waiting for an interrupt. Still: possible HW or driver problem.

There is a USB$SDA SDA extension, try SDA> USB HELP

Volker.

Volker Halle wrote:
> Jouk,
>
> please force a crash. Then you can look at the static situation without SDA possibly going to RWAST ;-)

This morning I did not succeed to crash teh system. Decwindows is on the
console and CRTL-F2 failed. Probably I need a LK keyboard (and not a
PC-keyboard as is connected now). I may still have one, but I'll have to
look where it is.
>
> Device 'bsy' indicates, that the device has IOs pending and is probably waiting for an interrupt. Still: possible HW or driver problem.
>

That's is logical. What I see is a hang of each process trying to
perform I/O on the USB-disk.

hmm.. I do not think it is a driver problem, since on the boot-member I
have attached another USB-drive (also 2GB) which already runs for 215
days since the last reboot. The machine(s) giving problems are all
satelites of this boot-member.

> There is a USB$SDA SDA extension, try SDA> USB HELP

I'll have a look at that.

Jouk

Jouk,

you can also force a crash from DCL - if you still have a logged-in terminal and the machine is not hung:

$ DEFINE OPC$NODUMP FALSE
$ MC OPCCRASH

Do you have a console window ? Are there any mount-verification messages ?

Volker.

Jouk,

the SDA USB extension won't probably help. It's mostly about all the USB related configuration data structures.

If the device would be waiting for an interrupt, the Device Status would have 'int,tim' set.

You could turn on IO tracing (see SDA> IO), but all you probably would get is the last IO started on the USB disk and not WHY it does not complete.

The current IRP is pointed to by the UCB, try to find out, what it is supposed to do and why it's not completing.

Volker.

Volker Halle wrote:
> Jouk,
>
> If you have a process hanging reading/writing to that disk, enter SDA, set context to that process and issue a SDA> SHOW PROC/CHAN - the DNcn: channel should be busy

Now using a SYSDUMP. I get:

.......
Process index: 0046 Name: BATCH_16 Extended PID: 2FA00146
--------------------------------------------------------------------

Channel CCB Window Status Device/file accessed
------- --- ------ ------ --------------------
0160 7FF7C2A0 826C1100 POLKA$DKA100:(1704,17558,0)
(section fi)
0170 7FF7C2C0 826BF100 POLKA$DKA100:(2025,10,0)
(section file)
0180 7FF7C2E0 82897180 POLKA$DKA100:(31988,2432,0)
0190 7FF7C300 826D1F40 POLKA$DKA100:(2232,10,0)
(section file)
01A0 7FF7C320 826D0F40 POLKA$DKA100:(2212,10,0)
(section file)
01B0 7FF7C340 828B7A40 POLKA$DKA100:(32058,349,0)
01C0 7FF7C360 82893840 POLKA$DKA100:(33280,36,0)
01D0 7FF7C380 827CF000 POLKA$DKA100:(51460,23,0)
(section file)
01E0 7FF7C3A0 826DA040 DNFS1:(18,17530,0)
01F0 7FF7C3C0 00000000 HOPAK$DNA2:
0210 7FF7C400 8276AAC0 Busy HOPAK$DNA2:(288,1,0)

Total number of open channels : 32.

The device is Busy.

>
> Then also try SDA> SHOW DEV DNcn: to determine the state of the IO.
>

This output looks similar to the one I created on the running system.

SHOW PROC gives:

SDA> show proc

Process index: 0046 Name: BATCH_16 Extended PID: 2FA00146
--------------------------------------------------------------------
Process status: 00044001 RES,BATCH,PHDRES
status2: 00000001 QUANTUM_RESCHED

PCB address 829324C0 JIB address 82943D40
PHD address 8CE50000 Swapfile disk address 00000000
KTB vector address 829327B0 HWPCB address FFFFFFFF.8CE50080
Callback vector address 00000000 Termination mailbox 0000
Master internal PID 00010046 Subprocess count 0
Creator extended PID 00000000 Creator internal PID 00000000
Previous CPU Id 00000000 Current CPU Id 00000000
Previous ASNSEQ 000000000000000E Previous ASN 0000000000000066
Initial process priority 3 # open files remaining 4987/5000
Delete pending count 0 Direct I/O count/limit 665/1024
UIC [00300,000300] Buffered I/O count/limit 10000/10000
Abs time of last event 01F54C81 BUFIO byte count/limit
2097952/2097952
# of threads 1 ASTs remaining 840/1200
Swapped copy of LEFC0 00000000 Timer entries remaining 399/400
Swapped copy of LEFC1 00000000 Active page table count 0
Global cluster 2 pointer 00000000 Process WS page count 4145

Press RETURN for more.
SDA>

Process index: 0046 Name: BATCH_16 Extended PID: 2FA00146
--------------------------------------------------------------------
Global cluster 3 pointer 00000000 Global WS page count 13
PCB Specific Spinlock 828DF780 Subprocesses in job 0

Press RETURN for more.
SDA>

Process index: 0046 Name: BATCH_16 Extended PID: 2FA00146
--------------------------------------------------------------------

Thread index: 0000
------------------
Current capabilities: System: 000C QUORUM,RUN
User: 0000.00000000
Permanent capabilities: System: 000C QUORUM,RUN
User: 0000.00000000

Current affinities: 00000000.00000000
Permanent affinities: 00000000.00000000
Permanent affinities: 00000000.00000000

Thread status: 00044001
status2: 00000001

KTB address 829324C0 HWPCB address FFFFFFFF.8CE50080
PKTA address 7FFEFF98 Callback vector address 00000000
Internal PID 00010046 Callback error 00000000
Extended PID 2FA00146 Current CPU id 00000000
State LEF Flags 00000000

Press RETURN for more.
SDA>

Process index: 0046 Name: BATCH_16 Extended PID: 2FA00146
--------------------------------------------------------------------
Base priority 3 Current priority 5
Waiting EF cluster 0 Event flag wait mask FFEFFFFF
CPU since last quantum 00000395 Mutex count 0
ASTs active NONE

SDA>
SDA>

Volker Halle wrote:
>
> Do you have a console window ? Are there any mount-verification messages ?
>
No mount-verification message observed.

Jouk

Jouk,

the process is busy waiting for an IO to finish on file HOPAK$DNA2:(288,1,0) <- this is the file ID.

SDA> SHOW DEV DNA2: should list the current IRP address

SDA> READ SYSDEF
SDA> FORMAT <irp address>

Is the console terminal 'broadcast enabled' ? The default for DECwindows may be to disable broadcasts. Any Mount-Verification messages in OPERATOR.LOG ?

You may consider to start IO tracing ( see http://de.openvms.org/Spring2009/05-sda_extensions.html#IO ) in the runn9ing system, wait for the next hang, force a dump and then look at the IO trace in the dump...

Volker.

Volker Halle wrote:

>
> the process is busy waiting for an IO to finish on file HOPAK$DNA2:(288,1,0) <- this is the file ID.

The file at which it hangs is always know to me. but it seems each time
another one. It looks like a "random" location

>
> SDA> SHOW DEV DNA2: should list the current IRP address

IRP address 82962180

>
> SDA> READ SYSDEF
> SDA> FORMAT <irp address>
>

SDA> READ SYSDEF
SDA> FORMAT 82962180
FFFFFFFF.82962180 IRP$L_IOQFL 82977D80
FFFFFFFF.82962184 IRP$L_IOQBL 827463F8
UCB+008
FFFFFFFF.82962188 IRP$W_SIZE 0230
FFFFFFFF.8296218A IRP$B_TYPE 0A
FFFFFFFF.8296218B IRP$B_RMOD 47
FFFFFFFF.8296218C IRP$L_PID 00010046
SYS$K_6
FFFFFFFF.82962190 IRP$L_ACB64X_OFFSET 00000028
IRP$Q_PARAM_0
FFFFFFFF.82962194 00000000
FFFFFFFF.82962198 IRP$L_ACB_FLAGS 00000012
IRP$Q_PARAM_1
FFFFFFFF.8296219C IRP$L_THREAD_PID 00010046
SYS$K_6
FFFFFFFF.829621A0 IRP$L_KAST 8276AAC0
IRP$L_MIRP
IRP$L_WIND
FFFFFFFF.829621A4 IRP$L_UCB 82746340 UCB
FFFFFFFF.829621A8 IRP$L_CTXB 00054620
OPCCRA0
IRP$L_IIRP_P0
IRP$L_SHD_IOFL
IRP$PQ_ACB64_AST
FFFFFFFF.829621AC 00000000
FFFFFFFF.829621B0 IRP$L_HRB 0269F040
IRP$L_IIRP_P1
IRP$L_MV_TMO
IRP$L_SHAD
IRP$Q_ACB64_ASTPRM
FFFFFFFF.829621B4 00000000
FFFFFFFF.829621B8 IRP$Q_USER_THREAD_ID 00000000.00000000
FFFFFFFF.829621C0 IRP$B_EFN 14
FFFFFFFF.829621C1 IRP$B_PRI 3C
FFFFFFFF.829621C2 IRP$B_CLN_INDX 00
FFFFFFFF.829621C3 IRP$B_WLG_FLAGS 00
FFFFFFFF.829621C4 IRP$L_CHAN 00000210
FFFFFFFF.829621C8 IRP$L_CLN_WLE 0269F058
IRP$L_IIRP_P2
IRP$PQ_IOSB
IRP$Q_PARAM_2
FFFFFFFF.829621CC 00000000
FFFFFFFF.829621D0 IRP$L_STS 00008012
IRP$L_STS_OVERLAY
IRP$Q_STATUS
IRP$R_STS_BITS
FFFFFFFF.829621D4 IRP$L_STS2 00000300
FFFFFFFF.829621D8 IRP$PQ_VA_PTE FFFFFEFC.000051C8
FFFFFFFF.829621E0 IRP$L_SVAPTE 82962208
IRP+008
IRP$PS_BUFIO_PKT
FFFFFFFF.829621E4 IRP$L_BCNT 00007C00
FFFFFFFF.829621E8 IRP$L_BOFF 00000110
FFFFFFFF.829621EC IRP$L_ABOFF 00000110
IRP$L_OBOFF
FFFFFFFF.829621F0 IRP$L_EXTEND 00000000
FFFFFFFF.829621F4 IRP$PS_ALTDDT 00000000
IRP$PS_FDT_CONTEXT
FFFFFFFF.829621F8 IRP$R_DIOBM 00000004.00000000
FFFFFFFF.82962200 00000008.1F790058
FFFFFFFF.82962208 00003DF8.0016FF09
FFFFFFFF.82962210 000035C2.0016FF09
FFFFFFFF.82962218 000035C1.0016FF09
FFFFFFFF.82962220 000056C2.0016FF09
FFFFFFFF.82962228 00000000.00000000
... ...
FFFFFFFF.82962250 IRP$L_IOST1 08B2A4FA
IRP$L_MEDIA
FFFFFFFF.82962254 IRP$B_CARCON 00
IRP$L_IOST2
IRP$L_TT_TERM
IRP$W_SHD_COPY_TYPE
IRP$W_SHD_VUN
FFFFFFFF.82962255 00
FFFFFFFF.82962256 IRP$W_SHD_DEV_TYPE 0000
IRP$W_SHD_MSCP_DISK_MODIFIER
FFFFFFFF.82962258 IRP$L_ABCNT 00000000
IRP$Q_NT_PRVMSK
IRP$Q_STATION
IRP$Q_TT_STATE
FFFFFFFF.8296225C IRP$L_OBCNT 00007C00
FFFFFFFF.82962260 IRP$L_FUNC 0000000C
FFFFFFFF.82962264 00000000
FFFFFFFF.82962268 IRP$L_SEGVBN 0004C01B
OPCCRAB
IRP$Q_SEGVBN
FFFFFFFF.8296226C 00000000
FFFFFFFF.82962270 IRP$L_DIAGBUF 82962208
IRP+008
IRP$L_SCB_BUF
IRP$W_TT_PRMPT
FFFFFFFF.82962274 IRP$L_DCD_SRC_UCB 017033A5
IRP$L_SEQNUM
FFFFFFFF.82962278 IRP$L_ARB 00000000
FFFFFFFF.8296227C IRP$B_CPY_MODE 00
IRP$L_KEYDESC
IRP$L_WLE_PTR
FFFFFFFF.8296227D 000000
FFFFFFFF.82962280 IRP$PS_KPB 82B821C0
FFFFFFFF.82962284 IRP$PS_CCB 7FF7C400
FFFFFFFF.82962288 IRP$L_QIO_P1 01472110
IRP$Q_QIO_P1
FFFFFFFF.8296228C 00000000
FFFFFFFF.82962290 IRP$L_QIO_P2 00007C00
IRP$Q_QIO_P2
IRP$Q_QIO_P2
FFFFFFFF.82962294 00000000
FFFFFFFF.82962298 IRP$L_QIO_P3 0004C01B
OPCCRAB
IRP$Q_PARAM_3
IRP$Q_QIO_P3
FFFFFFFF.8296229C 00000000
FFFFFFFF.829622A0 IRP$L_QIO_P4 00000000
IRP$Q_QIO_P4
FFFFFFFF.829622A4 00000000
FFFFFFFF.829622A8 IRP$L_QIO_P5 00000000
IRP$Q_QIO_P5
FFFFFFFF.829622AC 00000000
FFFFFFFF.829622B0 IRP$L_QIO_P6 00000000
IRP$Q_QIO_P6
FFFFFFFF.829622B4 00000000
FFFFFFFF.829622B8 IRP$Q_LBN_64 00000000.00000000
FFFFFFFF.829622C0 IRP$L_FQFL 00000000
FFFFFFFF.829622C4 IRP$L_FQBL 00000000
FFFFFFFF.829622C8 IRP$W_CDRPSIZE FEC0
FFFFFFFF.829622CA IRP$B_CD_TYPE 39
FFFFFFFF.829622CB IRP$B_FLCK 3A
FFFFFFFF.829622CC IRP$L_FPC 81D9F3E0
SCS$MS8
FFFFFFFF.829622D0 IRP$Q_FR3 FFFFFFFF.829AF9C0
FFFFFFFF.829622D8 IRP$Q_FR4 FFFFFFFF.825A8BB8
FFFFFFFF.829622E0 IRP$L_SAVD_RTN 00000000
FFFFFFFF.829622E4 IRP$L_MSG_BUF 00000000
FFFFFFFF.829622E8 IRP$L_RSPID 00000000
FFFFFFFF.829622EC IRP$L_CDT 8276BD90
FFFFFFFF.829622F0 IRP$Q_RES_WAIT_STATE 00000000.00000020
FFFFFFFF.829622F8 IRP$L_SCS_STALL_DATA 00000000
FFFFFFFF.829622FC IRP$L_RWCPTR 00000000
FFFFFFFF.82962300 IRP$L_BD_ADDR 00000000
FFFFFFFF.82962304 IRP$L_RBUN 00000000
FFFFFFFF.82962308 IRP$L_LBUFH_AD 00000000
FFFFFFFF.8296230C IRP$AR_PSB 8289EA40
FFFFFFFF.82962310 IRP$B_SHD_PIO_CNT 00
IRP$L_ERASE_VBN
IRP$L_LBOFF
IRP$Q_ERASE_VBN
IRP$T_LBUFHNDL
FFFFFFFF.82962311 IRP$B_SHD_PIO_ACT 00
FFFFFFFF.82962312 IRP$B_SHD_PIO_FLAGS 00
FFFFFFFF.82962313 IRP$B_SHD_PIO_ERRCNT 00
FFFFFFFF.82962314 IRP$B_SHD_PIO_ERRINDEX 00
IRP$L_CDRPFL
IRP$L_RBUFH_AD
FFFFFFFF.82962315 IRP$B_SHD_PIO_ERRSEV 00
FFFFFFFF.82962316 0000
FFFFFFFF.82962318 IRP$L_RBOFF 00000000
IRP$Q_SHD_LOCK_FR0
FFFFFFFF.8296231C IRP$L_UBARSRCE 00000000
IRP$L_XCT_LEN
FFFFFFFF.82962320 IRP$L_DUTUFLAGS 00000000
IRP$Q_SHD_LOCK_FR1
FFFFFFFF.82962324 IRP$W_DUTUCNTR BD80
FFFFFFFF.82962326 IRP$W_ENDMSGSIZ 8276
FFFFFFFF.82962328 IRP$L_PDT 00000000
IRP$Q_SHD_LOCK_FR2
FFFFFFFF.8296232C IRP$L_WALK_CDDB 00000000
FFFFFFFF.82962330 IRP$L_WALK_ALCLS 00000000
IRP$Q_SHD_LOCK_FR4
FFFFFFFF.82962334 IRP$L_WALK_SVPC 00000000
FFFFFFFF.82962338 IRP$L_LB_CDDB 00000000
IRP$Q_SHD_LOCK_FR5
FFFFFFFF.8296233C 00000000
FFFFFFFF.82962340 IRP$L_DUTU_RSVD1 00000000
IRP$L_SHD_LOCK_FPC
FFFFFFFF.82962344 IRP$L_DUTU_RSVD2 00000000
IRP$L_SHD_PIO_ERROR
IRP$L_WBM_RMTSND_STS
FFFFFFFF.82962348 IRP$L_DUTU_RSVD3 00000000
IRP$L_SHD_PIO_LNK
IRP$L_WBM_REFCNT
FFFFFFFF.8296234C IRP$L_DUTU_RSVD4 00000000
IRP$L_SHDSPC
FFFFFFFF.82962350 IRP$L_SHD_CONTROL_IRP 00000000
FFFFFFFF.82962354 IRP$L_SHD_TEMP 00000000
FFFFFFFF.82962358 IRP$Q_SHD_SAVED_R1 00000000.00000000
FFFFFFFF.82962360 IRP$Q_SHD_SAVED_R2 00000000.00000000
FFFFFFFF.82962368 IRP$Q_SHD_SAVED_R4 00000000.00000000
FFFFFFFF.82962370 IRP$L_SHD_SVD_CNT_IRP 00000000
FFFFFFFF.82962374 IRP$L_SHD_SAVED_STATUS 00000000
FFFFFFFF.82962378 IRP$L_SHD_WLG_MODE_FPC 00000000
FFFFFFFF.8296237C IRP$L_SHD_PERLKID 00000000
FFFFFFFF.82962380 IRP$L_SHD_EXPEL_TIMER 00000000
FFFFFFFF.82962384 IRP$L_SHD_EXPEL_FLAGS 00000008
FFFFFFFF.82962388 IRP$L_SHD_EXPEL_MASK 00000000
FFFFFFFF.8296238C 00000000
FFFFFFFF.82962390 IRP$Q_DATACHECKRETRYCOUNTER 00000000.00000000
IRP$Q_SHD_RESERV_Q8
FFFFFFFF.82962398 IRP$Q_CURRENT_CLONE 00000000.00000000
IRP$Q_SHD_RESERV_Q9
FFFFFFFF.829623A0 IRP$Q_SHD_RESERV_Q10 00000000.00000000
FFFFFFFF.829623A8 IRP$B_SHD_FLAGS 00
FFFFFFFF.829623A9 00000000.000000
IRP$C_LENGTH

Volker Halle wrote:

> You may consider to start IO tracing ( see http://de.openvms.org/Spring2009/05-sda_extensions.html#IO ) in the runn9ing system, wait for the next hang, force a dump and then look at the IO trace in the dump...

is it possible to trace only one device? I just started it and after
just 1 minute I already got tons of useless data on other devices. It
will be maybe too huge after a couple of days.

Jouk

Jouk,

the data in the CURRENT IRP on DNA2: looks consistent:

....
FFFFFFFF.829621D0 IRP$L_STS CCB_LOOKED_UP,FUNC,VIRT >>> 00008012
....
FFFFFFFF.82962260 IRP$L_FUNC IO$_READPBLK >>> 0000000C
....
FFFFFFFF.82962284 IRP$PS_CCB >>> 7FF7C400
FFFFFFFF.82962288 IRP$L_QIO_P1 Buf addr >>> 01472110
....
FFFFFFFF.82962290 IRP$L_QIO_P2 Length >>> 00007C00
....
FFFFFFFF.82962298 IRP$L_QIO_P3 start LBN >>> 0004C01B
....
FFFFFFFF.829622CC IRP$L_FPC 81D9F3E0 <<< what's at this address ?
Try SDA> EXA/INS 81D9F3E0

FFFFFFFF.829622F0 IRP$Q_RES_WAIT_STATE >>> 00000000.00000020 ???
LIB.REQ: 'SCS Resource Wait State'

This MAY be the key to this problem. Please force another crash after a hang. And look at the other (pending) IRPs on this device.

Volker.

Jouk,

don't worry, it's a ring buffer. In the crash you would then only want to look at the entries for DNA2: (SDA output to file, then SEARCH).

Volker.

Volker Halle wrote:
> Jouk,
>
> the data in the CURRENT IRP on DNA2: looks consistent:
>
> ...
> FFFFFFFF.829621D0 IRP$L_STS CCB_LOOKED_UP,FUNC,VIRT >>> 00008012
> ...
> FFFFFFFF.82962260 IRP$L_FUNC IO$_READPBLK >>> 0000000C
> ...
> FFFFFFFF.82962284 IRP$PS_CCB >>> 7FF7C400
> FFFFFFFF.82962288 IRP$L_QIO_P1 Buf addr >>> 01472110
> ...
> FFFFFFFF.82962290 IRP$L_QIO_P2 Length >>> 00007C00
> ...
> FFFFFFFF.82962298 IRP$L_QIO_P3 start LBN >>> 0004C01B
> ...
> FFFFFFFF.829622CC IRP$L_FPC 81D9F3E0 <<< what's at this address ?
> Try SDA> EXA/INS 81D9F3E0

SDA> EXA/INS 81D9F3E0
%SDA-E-NOINSTRAN, cannot translate instruction
SDA>

>
> FFFFFFFF.829622F0 IRP$Q_RES_WAIT_STATE >>> 00000000.00000020 ???
> LIB.REQ: 'SCS Resource Wait State'
>
> This MAY be the key to this problem. Please force another crash after a hang. And look at the other (pending) IRPs on this device.
>
> Volker.