Is there any way to research the sequence of lock/unlock request for a particular structure? I do not think this is captured in the audit, looking at the audit record types in "Record Type Mnemonics" in the "Enterprise Database Server Utilities Operations Guide".
By the way, is a more detailed explanation of those record types available anywhere else?

-------- Original Message --------
Subject: How to research DMSII lock/unlock requests
From: Luke Numrych <l.numrych@gmail.com>
To:
Date: Thu Jun 24 2021 09:31:26 GMT-0700 (Pacific Daylight Time)

> Is there any way to research the sequence of lock/unlock request for a particular structure? I do not think this is captured in the audit, looking at the audit record types in "Record Type Mnemonics" in the "Enterprise Database Server Utilities Operations Guide".
> By the way, is a more detailed explanation of those record types available anywhere else?
>

Record lock and unlock requests are not audited. The purpose of audit
files is to restore the data base to a consistent state after a
transaction abort, a DMSII or system restart, or reloading one or more
structures from backups. Locking does not participate in any of that.

By "Record Type Mnemonics" I assume you are referring to the table in
the section of that manual on PRINTAUDIT, "Selecting Records by Record
Type". I don't know of a better reference. Most of the record types have
to do with internal DMSII structure manipulations that would not
normally be of interest at the application program level. The record
types you would normally be most interested in (especially for Standard
Data Sets) are:

BIO
BTR
CCD
DSC
DSD
DSM (note this has both before and after images)
ETR
LGRR
STRDC

You might want to take a look at the sections in the same manual on
"Database Events Management" and "Logging Data Access". I've never used
these, but it appears to you can monitor most DM verbs, including LOCK
and FREE. I don't see ENDTRANSACTION or ABORTTRANSACTION listed, though,
noting that if your data base has INDEPENDENTTRANS set, all locked
records are implicitly freed when a transaction terminates. Transaction
boundaries (BTR, ETR) are written to the audit trail, however, so you
may be able to merge those entries with the event log entries.

You didn't mention your purpose in researching lock/unlock requests. We
might be able to give you better answers if you could describe what the
problem is or what you're really trying to do.

Paul

On Sunday, June 27, 2021 at 4:57:29 PM UTC-5, Paul Kimpel wrote:
> -------- Original Message --------
> Subject: How to research DMSII lock/unlock requests
> From: Luke Numrych <l.nu...@gmail.com>
> To:
> Date: Thu Jun 24 2021 09:31:26 GMT-0700 (Pacific Daylight Time)
>
> > Is there any way to research the sequence of lock/unlock request for a particular structure? I do not think this is captured in the audit, looking at the audit record types in "Record Type Mnemonics" in the "Enterprise Database Server Utilities Operations Guide".
> > By the way, is a more detailed explanation of those record types available anywhere else?
> >
> Record lock and unlock requests are not audited. The purpose of audit
> files is to restore the data base to a consistent state after a
> transaction abort, a DMSII or system restart, or reloading one or more
> structures from backups. Locking does not participate in any of that.
>
> By "Record Type Mnemonics" I assume you are referring to the table in
> the section of that manual on PRINTAUDIT, "Selecting Records by Record
> Type". I don't know of a better reference. Most of the record types have
> to do with internal DMSII structure manipulations that would not
> normally be of interest at the application program level. The record
> types you would normally be most interested in (especially for Standard
> Data Sets) are:
>
> BIO
> BTR
> CCD
> DSC
> DSD
> DSM (note this has both before and after images)
> ETR
> LGRR
> STRDC
>
> You might want to take a look at the sections in the same manual on
> "Database Events Management" and "Logging Data Access". I've never used
> these, but it appears to you can monitor most DM verbs, including LOCK
> and FREE. I don't see ENDTRANSACTION or ABORTTRANSACTION listed, though,
> noting that if your data base has INDEPENDENTTRANS set, all locked
> records are implicitly freed when a transaction terminates. Transaction
> boundaries (BTR, ETR) are written to the audit trail, however, so you
> may be able to merge those entries with the event log entries.
>
> You didn't mention your purpose in researching lock/unlock requests. We
> might be able to give you better answers if you could describe what the
> problem is or what you're really trying to do.
>
> Paul
Hi Paul - thanks for your reply.
I did not think lock/unlock request were part of the audit - as you pointed out, they would be meaningless for recovery, and that is what AUDITs are for. As I am new to DMSII; however, I felt the need to verify.
By the way - most database management systems call such mechanisms "transaction logs". Is there a story behind why DMSII calls them AUDITs? Just curious - I find that knowing the reasons "why" helps me become familiar with a system.

Thank you for pointing me at the other sections in the manual, I will definitely go through them.

As far as the reason why I was looking for lock/unlock requests: we had a database access failure that manifested itself as program receiving a NOTLOCKED 1 exception. It happened to existing code that has been running for ages (and properly locking/unlocking structures); therefore, the database had to be at fault ;). We just installed a new DMSII release on the system, so that could potentially have been the problem... I was asked by the development team to investigate. Since I could not figure a way to track lock/unlock requests, I decided to use the proven RTFM method. Through application of it, I developed a theory that the most likely cause was not that DMSII was suddenly losing locks put on structures by programs, and that the NOTLOCKED exception was in fact a secondary failure, the primary failure being a DEADLOCK. Which then caused DMSII to free all locks owned by the victim process. Because the victim process was ignoring the DEADLOCK exception (well, "eating" is more like it - it did not display the exception or give any other hint that it was happening), it didn't bother re-locking structures before attempting to retry the original transaction - which was causing the NOTLOCKED 1 exception.

My theory was proven true once code was added to the application to notify of the DEADLOCK exception and the development team has now updated the application to handle the DEADLOCK exception properly.

-------- Original Message --------
Subject: Re: How to research DMSII lock/unlock requests
From: Luke Numrych <l.numrych@gmail.com>
To:
Date: Wed Jun 30 2021 22:16:33 GMT-0700 (Pacific Daylight Time)
> On Sunday, June 27, 2021 at 4:57:29 PM UTC-5, Paul Kimpel wrote:
>> -------- Original Message --------
>> Subject: How to research DMSII lock/unlock requests
>> From: Luke Numrych <l.nu...@gmail.com>
>> To:
>> Date: Thu Jun 24 2021 09:31:26 GMT-0700 (Pacific Daylight Time)
>>
>>> Is there any way to research the sequence of lock/unlock request for a particular structure? I do not think this is captured in the audit, looking at the audit record types in "Record Type Mnemonics" in the "Enterprise Database Server Utilities Operations Guide".
>>> By the way, is a more detailed explanation of those record types available anywhere else?
>>>
>> Record lock and unlock requests are not audited. The purpose of audit
>> files is to restore the data base to a consistent state after a
>> transaction abort, a DMSII or system restart, or reloading one or more
>> structures from backups. Locking does not participate in any of that.
>>
>> By "Record Type Mnemonics" I assume you are referring to the table in
>> the section of that manual on PRINTAUDIT, "Selecting Records by Record
>> Type". I don't know of a better reference. Most of the record types have
>> to do with internal DMSII structure manipulations that would not
>> normally be of interest at the application program level. The record
>> types you would normally be most interested in (especially for Standard
>> Data Sets) are:
>>
>> BIO
>> BTR
>> CCD
>> DSC
>> DSD
>> DSM (note this has both before and after images)
>> ETR
>> LGRR
>> STRDC
>>
>> You might want to take a look at the sections in the same manual on
>> "Database Events Management" and "Logging Data Access". I've never used
>> these, but it appears to you can monitor most DM verbs, including LOCK
>> and FREE. I don't see ENDTRANSACTION or ABORTTRANSACTION listed, though,
>> noting that if your data base has INDEPENDENTTRANS set, all locked
>> records are implicitly freed when a transaction terminates. Transaction
>> boundaries (BTR, ETR) are written to the audit trail, however, so you
>> may be able to merge those entries with the event log entries.
>>
>> You didn't mention your purpose in researching lock/unlock requests. We
>> might be able to give you better answers if you could describe what the
>> problem is or what you're really trying to do.
>>
>> Paul
> Hi Paul - thanks for your reply.
> I did not think lock/unlock request were part of the audit - as you pointed out, they would be meaningless for recovery, and that is what AUDITs are for. As I am new to DMSII; however, I felt the need to verify.
> By the way - most database management systems call such mechanisms "transaction logs". Is there a story behind why DMSII calls them AUDITs? Just curious - I find that knowing the reasons "why" helps me become familiar with a system.
>
> Thank you for pointing me at the other sections in the manual, I will definitely go through them.
>
> As far as the reason why I was looking for lock/unlock requests: we had a database access failure that manifested itself as program receiving a NOTLOCKED 1 exception. It happened to existing code that has been running for ages (and properly locking/unlocking structures); therefore, the database had to be at fault ;). We just installed a new DMSII release on the system, so that could potentially have been the problem... I was asked by the development team to investigate. Since I could not figure a way to track lock/unlock requests, I decided to use the proven RTFM method. Through application of it, I developed a theory that the most likely cause was not that DMSII was suddenly losing locks put on structures by programs, and that the NOTLOCKED exception was in fact a secondary failure, the primary failure being a DEADLOCK. Which then caused DMSII to free all locks owned by the victim process. Because the victim process was ignoring the DEADLOCK exception (well, "eating" is more like it - it did not display the exception or give any other hint that it was happening), it didn't bother re-locking structures before attempting to retry the original transaction - which was causing the NOTLOCKED 1 exception.
>
> My theory was proven true once code was added to the application to notify of the DEADLOCK exception and the development team has now updated the application to handle the DEADLOCK exception properly.
As to "AUDITs", the term is actually "audit trail". Historically, to
"maintain an audit trail" is to record all of the activities that apply
changes to a record system. The term probably comes from accounting,
which has obsessed over maintaining reliable, verifiable records for
centuries.
Before centralized data base systems were used, "audit trail" was a
common term for a file or a set of files an application would create to
record activities against the (often tape-based) master files for the
application. It's not a Unisys/Burroughs-specific term at all. "Audit
trail", "journal", and "transaction log" all mean the same thing in the
context of data base systems.
The predecessor to DMSII, DM6700 (early 1970s), used the term "audit
trail" for its recovery file, and I'm certain that simply got carried
forward into DMSII.
Glad to know you got your NOTLOCKED problem resolved. It's just one more
example of why you need to handle (or at least report) all DMSII exceptions.
Paul

On Friday, July 9, 2021 at 7:05:31 AM UTC-7, Paul Kimpel wrote:

> As to "AUDITs", the term is actually "audit trail". Historically, to
> "maintain an audit trail" is to record all of the activities that apply
> changes to a record system. The term probably comes from accounting,
> which has obsessed over maintaining reliable, verifiable records for
> centuries.

Modern financial compliance auditing and security auditing also wants to know about access, not just change. There is a case for the DMSII audit files to be expanded to include LOCK and FIND verbs, too. Yes, the database open is recorded in the sumlog (if the correct logging options are set), but that does not show what records were accessed.

-------- Original Message --------
Subject: Re: How to research DMSII lock/unlock requests
From: mpe...@gmail.com <mperew@gmail.com>
To:
Date: Fri Jul 09 2021 08:43:21 GMT-0700 (Pacific Daylight Time)

> On Friday, July 9, 2021 at 7:05:31 AM UTC-7, Paul Kimpel wrote:
>
>> As to "AUDITs", the term is actually "audit trail". Historically, to
>> "maintain an audit trail" is to record all of the activities that apply
>> changes to a record system. The term probably comes from accounting,
>> which has obsessed over maintaining reliable, verifiable records for
>> centuries.
>
> Modern financial compliance auditing and security auditing also wants to know about access, not just change. There is a case for the DMSII audit files to be expanded to include LOCK and FIND verbs, too. Yes, the database open is recorded in the sumlog (if the correct logging options are set), but that does not show what records were accessed.
>

I agree that there is a need for DMSII to support tracking of read
access, but recording user access for either read or write is not the
function of the audit trail. Its purpose is to restore the data base to
a consistent state after a crash, a data rebuild, or a rollback. People
certainly use it for application-level issues like monitoring or
auditing updates, but that's not what it's for.

Another concern with adding read access logging to the audit trail would
be performance and resource consumption. Bandwidth to the audit trail is
already an issue for high-throughput data bases, or else we wouldn't
have partitioned audits. Read access logging would compound that issue
many times over.

A third concern is that even if you added read access logging to the
audit trail, in the general case that wouldn't tell you anything about
the user or device doing the read. In a COMS environment, for example,
the user that DMSII sees is that of the COMS TP, not the user submitting
the transaction. That information is normally available to the TP, but
presently there isn't any interface that passes that to DMSII.

A fourth, perhaps lesser, concern is that the audit trail is not a
linear record. There are times that it backs up and erases data that was
previously written. That could be a small number of records in the case
of an abort or halt/load recovery, or a whole lot of records in the case
of a rebuild or a rollback.

So while there's a real need for low-level access logging, I don't think
the audit trail is the place to record it. I suggest that there needs to
be an entirely separate mechanism, perhaps integrated with statistics
gathering, that could be optimized for time-linear recording of both
read and write access, and include the necessary task and end-user
identifiers.

Paul