It seems as if directory blocks containing zero file entry records, and
subrecords, tend to accumulate in active directories. Can only assume
these arise when multi-multi-multiversion consecutive file names are
deleted from the directory. Recently encountered 32 consecutive empty
blocks at which my code sanity checked.

Quick solution; create an equivalent directory and copy from the
original to the new. Problem Solvered. Assume a backup-restore would
accomplish similar, etc.

Questions:

Is this expected directory file behaviour?

Do these empty blocks continue to accumulate, only to be reused should
in-order file names be created?

Do extensive empty directory blocks represent tangible overhead to the
(persumably) XQP? (My code sanity checked at 32 but who knows exactly
how many there really were.)

Are their tools to measure such directory file efficiency (shall we say)
and to "compress" such files (apart from backup-restore).

TIA, Mark.

PS. I do recall descriptions of the FILES-11 directory internals being
very simple-minded and inefficient.

PPS. My entire technical hard-copy collection, including such VMS tomes
as McCoy's File System Internals, in a fit of pique, went into a recycle
bin some years ago. Moral of the story; don't let line-management get
under your skin.

--
Anyone, who using social-media, forms an opinion regarding anything
other than the relative cuteness of this or that puppy-dog, needs
seriously to examine their critical thinking.

On Saturday, September 24, 2022 at 1:31:31 PM UTC-4, Mark Daniel wrote:
> It seems as if directory blocks containing zero file entry records, and
> subrecords, tend to accumulate in active directories. Can only assume
> these arise when multi-multi-multiversion consecutive file names are
> deleted from the directory. Recently encountered 32 consecutive empty
> blocks at which my code sanity checked.
>
> Quick solution; create an equivalent directory and copy from the
> original to the new. Problem Solvered. Assume a backup-restore would
> accomplish similar, etc.
>
> Questions:
>
> Is this expected directory file behaviour?
>
> Do these empty blocks continue to accumulate, only to be reused should
> in-order file names be created?
>
> Do extensive empty directory blocks represent tangible overhead to the
> (persumably) XQP? (My code sanity checked at 32 but who knows exactly
> how many there really were.)
>
> Are their tools to measure such directory file efficiency (shall we say)
> and to "compress" such files (apart from backup-restore).
>
> TIA, Mark.
>
> PS. I do recall descriptions of the FILES-11 directory internals being
> very simple-minded and inefficient.
>
> PPS. My entire technical hard-copy collection, including such VMS tomes
> as McCoy's File System Internals, in a fit of pique, went into a recycle
> bin some years ago. Moral of the story; don't let line-management get
> under your skin.
>
> --
> Anyone, who using social-media, forms an opinion regarding anything
> other than the relative cuteness of this or that puppy-dog, needs
> seriously to examine their critical thinking.

It is my understanding that empty blocks will cause a shuffle of the entries that follow filling the block*. Empty blocks should only exist at the end of the directory file. Since directories are/must be contiguous, these trailing blocks will be used as space is required. I don't believe that the extra allocated blocks is a performance hit.

* This is why deleting large numbers of files in reverse order is so much faster as all that shuffling is avoided.

Dan

On 2022-09-24 19:42, abrsvc wrote:
> On Saturday, September 24, 2022 at 1:31:31 PM UTC-4, Mark Daniel wrote:
>> It seems as if directory blocks containing zero file entry records, and
>> subrecords, tend to accumulate in active directories. Can only assume
>> these arise when multi-multi-multiversion consecutive file names are
>> deleted from the directory. Recently encountered 32 consecutive empty
>> blocks at which my code sanity checked.
>>
>> Quick solution; create an equivalent directory and copy from the
>> original to the new. Problem Solvered. Assume a backup-restore would
>> accomplish similar, etc.
>>
>> Questions:
>>
>> Is this expected directory file behaviour?
>>
>> Do these empty blocks continue to accumulate, only to be reused should
>> in-order file names be created?
>>
>> Do extensive empty directory blocks represent tangible overhead to the
>> (persumably) XQP? (My code sanity checked at 32 but who knows exactly
>> how many there really were.)
>>
>> Are their tools to measure such directory file efficiency (shall we say)
>> and to "compress" such files (apart from backup-restore).
>>
>> TIA, Mark.
>>
>> PS. I do recall descriptions of the FILES-11 directory internals being
>> very simple-minded and inefficient.
>>
>> PPS. My entire technical hard-copy collection, including such VMS tomes
>> as McCoy's File System Internals, in a fit of pique, went into a recycle
>> bin some years ago. Moral of the story; don't let line-management get
>> under your skin.
>>
>> --
>> Anyone, who using social-media, forms an opinion regarding anything
>> other than the relative cuteness of this or that puppy-dog, needs
>> seriously to examine their critical thinking.
>
> It is my understanding that empty blocks will cause a shuffle of the entries that follow filling the block*. Empty blocks should only exist at the end of the directory file. Since directories are/must be contiguous, these trailing blocks will be used as space is required. I don't believe that the extra allocated blocks is a performance hit.
>
> * This is why deleting large numbers of files in reverse order is so much faster as all that shuffling is avoided.

What? Does directory files really need to be contiguous? I would be very
surprised by that. But directories in VMS are always sorted. I can see
that a delete might leave an empty space, but chances are high that when
you enter something new, it would need resorting and shuffling the
directory, so I wouldn't expect you to constantly see large empty spaces
in a directory under VMS.
Or ODS-2 I should say. Under ODS-1, you can have large empty chunks in a
directory, since it does not keep files sorted in the first place. ODS-1
also definitely do not require directories to be contiguous.

I would suspect ODS-5 to be rather similar to ODS-2 here, but don't know
for sure.

Johnny

I'll just refine my query in the bright light of the new day.

Dan has clarified the "Empty blocks should only exist at the end of the
directory file". I seem to recall this shuffle myself. Can someone
point to, or reference a resource which confirms this definitely. This
would mean my code could quit after the first empty block encountered
without continuing on to EOF. This also would imply that without
intervention, directory files will not shrink, only grow in size.

PS. There are a number of resources out there that do not seem to
clarify the issue of these zeroed blocks. (Links posted as quotations
to stop them being broken.)

> http://www.bitsavers.org/pdf/dec/vax/vms/training/EY-F575E-DP_VMS_File_System_Internals_1990.pdf
> https://web-docs.gsi.de/~kraemer/COLLECTION/VMS/ods2.txt
> https://groups.google.com/g/comp.os.vms/c/SU2YvPOeowo

and an interesting (but not immediately relevant) first-person recollection

> https://www.youtube.com/watch?v=b0YenelPw-Y

On 25/9/2022 3:01 am, Mark Daniel wrote:
> It seems as if directory blocks containing zero file entry records, and
> subrecords, tend to accumulate in active directories.  Can only assume
> these arise when multi-multi-multiversion consecutive file names are
> deleted from the directory.  Recently encountered 32 consecutive empty
> blocks at which my code sanity checked.
>
> Quick solution; create an equivalent directory and copy from the
> original to the new.  Problem Solvered.  Assume a backup-restore would
> accomplish similar, etc.
>
> Questions:
>
> Is this expected directory file behaviour?
>
> Do these empty blocks continue to accumulate, only to be reused should
> in-order file names be created?
>
> Do extensive empty directory blocks represent tangible overhead to the
> (persumably) XQP?  (My code sanity checked at 32 but who knows exactly
> how many there really were.)
>
> Are their tools to measure such directory file efficiency (shall we say)
> and to "compress" such files (apart from backup-restore).
>
> TIA, Mark.
>
> PS. I do recall descriptions of the FILES-11 directory internals being
> very simple-minded and inefficient.
>
> PPS. My entire technical hard-copy collection, including such VMS tomes
> as McCoy's File System Internals, in a fit of pique, went into a recycle
> bin some years ago.  Moral of the story; don't let line-management get
> under your skin.

--
Anyone, who using social-media, forms an opinion regarding anything
other than the relative cuteness of this or that puppy-dog, needs
seriously to examine their critical thinking.

On Saturday, September 24, 2022 at 7:47:38 PM UTC-4, Mark Daniel wrote:
> I'll just refine my query in the bright light of the new day.
>
> Dan has clarified the "Empty blocks should only exist at the end of the
> directory file". I seem to recall this shuffle myself. Can someone
> point to, or reference a resource which confirms this definitely. This
> would mean my code could quit after the first empty block encountered
> without continuing on to EOF. This also would imply that without
> intervention, directory files will not shrink, only grow in size.
>
> PS. There are a number of resources out there that do not seem to
> clarify the issue of these zeroed blocks. (Links posted as quotations
> to stop them being broken.)
>
> > http://www.bitsavers.org/pdf/dec/vax/vms/training/EY-F575E-DP_VMS_File_System_Internals_1990.pdf
> > https://web-docs.gsi.de/~kraemer/COLLECTION/VMS/ods2.txt
> > https://groups.google.com/g/comp.os.vms/c/SU2YvPOeowo
>
> and an interesting (but not immediately relevant) first-person recollection
>
> > https://www.youtube.com/watch?v=b0YenelPw-Y
> On 25/9/2022 3:01 am, Mark Daniel wrote:
> > It seems as if directory blocks containing zero file entry records, and
> > subrecords, tend to accumulate in active directories. Can only assume
> > these arise when multi-multi-multiversion consecutive file names are
> > deleted from the directory. Recently encountered 32 consecutive empty
> > blocks at which my code sanity checked.
> >
> > Quick solution; create an equivalent directory and copy from the
> > original to the new. Problem Solvered. Assume a backup-restore would
> > accomplish similar, etc.
> >
> > Questions:
> >
> > Is this expected directory file behaviour?
> >
> > Do these empty blocks continue to accumulate, only to be reused should
> > in-order file names be created?
> >
> > Do extensive empty directory blocks represent tangible overhead to the
> > (persumably) XQP? (My code sanity checked at 32 but who knows exactly
> > how many there really were.)
> >
> > Are their tools to measure such directory file efficiency (shall we say)
> > and to "compress" such files (apart from backup-restore).
> >
> > TIA, Mark.
> >
> > PS. I do recall descriptions of the FILES-11 directory internals being
> > very simple-minded and inefficient.
> >
> > PPS. My entire technical hard-copy collection, including such VMS tomes
> > as McCoy's File System Internals, in a fit of pique, went into a recycle
> > bin some years ago. Moral of the story; don't let line-management get
> > under your skin.
>
> --
> Anyone, who using social-media, forms an opinion regarding anything
> other than the relative cuteness of this or that puppy-dog, needs
> seriously to examine their critical thinking.
To be honest, I don't know if the contiguous requirement is still valid. I got that from the Mccoy book. I will check further and look for the "end" record for the entire directory file. I assume that there is one.

Dan

Mark,

does your utility read until the 'End of file block' or the 'Highest block' of the directory file ?

Free blocks should only be at the end of the directory file - after the EFBLK.

The logical end of the directory would be the last 'directory record' in the 'End of file block' of the directory file, i.e. a directory record with DIR$W_SIZE=-1 marks the end of directory records in the current block and therefore the end of the directory in the 'End of file block'

$ DUMP/DIR directory_file.DIR should show the last record as: 'xxx End of records (-1)' in the EOF block.

The function to 'shuffle' a directory (extend or compress) is called SHUFFLE_DIR, found in module [F11X]SHFDIR

Directory files are and must be contiguous. DUMP/HEADER directory.DIR shows:

File characteristics: Contiguous, Directory

This allows more efficient reads/writes and prevents window turns, when already executing inside the XQP.

Volker.

Additions:

When compressing a directory file, EFBLK will be updated, but the blocks between EFBLK+1 and the highest block allocated will not be overwritten with zeroes, even if 'file high-water marking' is set.

$ CREATE/DIR/ALLOC=nnn will create .DIR file with zeroed blocks after EFBLK (if 'file high-water marking' is set).

Mark, what is your definition of 'empty blocks' - all zeroes ?

Volker.

Op zaterdag 24 september 2022 om 19:31:31 UTC+2 schreef Mark Daniel:
> It seems as if directory blocks containing zero file entry records, and
> subrecords, tend to accumulate in active directories. Can only assume
> these arise when multi-multi-multiversion consecutive file names are
> deleted from the directory. Recently encountered 32 consecutive empty
> blocks at which my code sanity checked.
>
> Quick solution; create an equivalent directory and copy from the
> original to the new. Problem Solvered. Assume a backup-restore would
> accomplish similar, etc.
>
> Questions:
>
> Is this expected directory file behaviour?
>
> Do these empty blocks continue to accumulate, only to be reused should
> in-order file names be created?
>
> Do extensive empty directory blocks represent tangible overhead to the
> (persumably) XQP? (My code sanity checked at 32 but who knows exactly
> how many there really were.)
>
> Are their tools to measure such directory file efficiency (shall we say)
> and to "compress" such files (apart from backup-restore).
>
> TIA, Mark.
>
> PS. I do recall descriptions of the FILES-11 directory internals being
> very simple-minded and inefficient.
>
> PPS. My entire technical hard-copy collection, including such VMS tomes
> as McCoy's File System Internals, in a fit of pique, went into a recycle
> bin some years ago. Moral of the story; don't let line-management get
> under your skin.
>
> --
> Anyone, who using social-media, forms an opinion regarding anything
> other than the relative cuteness of this or that puppy-dog, needs
> seriously to examine their critical thinking.

You can install dfu (https://www.digiater.nl/dfu.html). This utility has an option to compress directory files.

Oswald

The DFU documentation discusses directory compression in

5.3.2 Discussion of directory compression

$ SET FILE/TRUNCATE on a .DIR file also works, but only truncates the blocks after EFBLK.

DFU can also compress sparsely filled directory blocks.

Volker.

On 25/9/2022 5:01 pm, Volker Halle wrote:
> Mark,

Thanks for the detailed response Volker.

> does your utility read until the 'End of file block' or the 'Highest block' of the directory file ?

It uses a "generic" file read function as "raw" blocks into memory. It
reads the exact number of bytes indicated by the file header.

if (odsptr->XabFhc.xab$l_ebk <= 1)
SizeInBytes = odsptr->XabFhc.xab$w_ffb;
else
SizeInBytes = ((odsptr->XabFhc.xab$l_ebk-1) << 9) +
odsptr->XabFhc.xab$w_ffb;
/* round-up ensure we have a full complement of blocks allocated */
odsptr->DataPtr = (char*)VmGet (((SizeInBytes/512)+1)*512);
odsptr->DataLength = 0;

odsptr->Rab.rab$l_ubf = odsptr->DataPtr;
for (;;)
{
if (SizeInBytes > 512*127)
odsptr->Rab.rab$w_usz = 512*127;
else
if (SizeInBytes > 0)
odsptr->Rab.rab$w_usz = SizeInBytes;
else
break;

status = sys$read (&odsptr->Rab, 0, 0);
if (VMSnok (status)) break;

odsptr->DataLength += odsptr->Rab.rab$w_rsz;
odsptr->Rab.rab$l_ubf += odsptr->Rab.rab$w_rsz;
SizeInBytes -= odsptr->Rab.rab$w_rsz;
}

sys$close (&odsptr->Fab, 0, 0);

So I guess to the highest block.

> Free blocks should only be at the end of the directory file - after the EFBLK.
>
> The logical end of the directory would be the last 'directory record' in the 'End of file block' of the directory file, i.e. a directory record with DIR$W_SIZE=-1 marks the end of directory records in the current block and therefore the end of the directory in the 'End of file block'

Then it looks like the code reads the size of the file bytes and so
scans past the DIR$W_SIZE=-1 in the end of file block and on to the end
of the file itself.

Having been emptied by file name deletions mid-file and then
(effectively) "shuffled" to the end of the file these should all contain
DIR$W_SIZE=-1 and therefore qualify as empty blocks, as per your later
post that included ...

> When compressing a directory file, EFBLK will be updated, but the blocks between EFBLK+1 and the highest block allocated will not be overwritten with zeroes, even if 'file high-water marking' is set.

> $ DUMP/DIR directory_file.DIR should show the last record as: 'xxx End of records (-1)' in the EOF block.
>
> The function to 'shuffle' a directory (extend or compress) is called SHUFFLE_DIR, found in module [F11X]SHFDIR

:-) Seems all engineering types go for unimaginative (but
understandable) nomenclature.

It also indicates that any accessible directory file should not contain
embedded empty blocks as they should all have been shuffled to the end.

> Directory files are and must be contiguous. DUMP/HEADER directory.DIR shows:

Still a design consideration then.

> File characteristics: Contiguous, Directory
>
> This allows more efficient reads/writes and prevents window turns, when already executing inside the XQP.
>
> Volker.

> Mark, what is your definition of 'empty blocks' - all zeroes ?

No, not zeroes.

/* if not end of records (in this block) sentinal */
if (rlen && rlen != (uint)0xffff && rlen < 508)
{
/* records are always on even byte (word) boundaries */
rzptr = rptr + rlen;
if ((int)rzptr & 1) rzptr++;

/* break from loop to process record */
EmptyBlockCount = 0;
break;
}
else
if (rlen && rlen != (uint)0xffff && rlen > 508)
{
status = SS$_BUGCHECK;
8< snip 8<
/* sanity check report */
break;
}
else
if (EmptyBlockCount++ > 16)
{
/* empty blocks should only occur at end of directory */
break;
}

The |if (EmptyBlockCount++ > 16)| is recently revised code based on the
strong feeling from c.o.v. posts all "empty blocks" from the "shuffle"
should be at the trailing end of the directory file. This used to be a
sanity check because I was originally unsure of the role of these.

I should add that this code has been in use for some years and apart
from a handful of (corrected) bugs has been (appearing to) perform well.

--
Anyone, who using social-media, forms an opinion regarding anything
other than the relative cuteness of this or that puppy-dog, needs
seriously to examine their critical thinking.

Mark,

if your program reports an error again, you should then carefully examine the directory file in question with

$ DUMP/DIR/HEADER directory-filename.DIR/OUT=temp_file

> So I guess to the highest block.

No, the code reads until XabFhc.xab$l_ebk, which is the EFBLK (end-of-file block). And therefore should not find 'empty blocks'.

Test for yourself: create a directory, populate it with a couple of files until EFBLK becomes > 1, then delete all files in that directory.
Use DUMP/HEADER to find the start LBN of the directory file and then DUMP/BLOCKS=(START=start_lbn,SIZE=2) disk:
You'll see, that the blocks beyond VBN 1 of the directory file contain 'old directory data', not zeros and they don't start with DIR$W_SIZE=FFFF

> > File characteristics: Contiguous, Directory
> >
> > This allows more efficient reads/writes and prevents window turns, when already executing inside the XQP.
> Still a design consideration then.

Please read the second half of that sentence: I strongly believe you can't do a window turn (which involves the XQP), if you are already in the XQP executing a directory operation. So directory files must be contiguous.

Once you hit this problem again, feel free to provide some more details.

Volker.

On Sunday, September 25, 2022 at 10:27:19 AM UTC-4, Volker Halle wrote:
> Please read the second half of that sentence: I strongly believe you can't do a window turn (which involves the XQP), if you are already in the XQP executing a directory operation. So directory files must be contiguous.
>

There is a middle ground, such as what happens with the pager. The page file doesn't have to be contiguous,
but the file can't have extension headers (i.e. all retrieval pointers must fit in the primary header). If the header
for the directory file is cached in memory, a window turn won't require I/O.

On Sunday, September 25, 2022 at 1:59:55 PM UTC-4, osuv...@gmail.com wrote:
> On Sunday, September 25, 2022 at 10:27:19 AM UTC-4, Volker Halle wrote:
> > Please read the second half of that sentence: I strongly believe you can't do a window turn (which involves the XQP), if you are already in the XQP executing a directory operation. So directory files must be contiguous.
> >
> There is a middle ground, such as what happens with the pager. The page file doesn't have to be contiguous,
> but the file can't have extension headers (i.e. all retrieval pointers must fit in the primary header). If the header
> for the directory file is cached in memory, a window turn won't require I/O.
Mark,

Am in a bit of a hurry, so I will not go into details.

However, COPY is definitely not needed. RENAME is quite enough, preferably in the advantageous order to not create repeated shuffles.

I would have to go digging, but it is also advantageous to create a directory of the desired size, and not just use the default. That way, there is no need to grow the directory and incur the resulting copies.

- Bob Gezelter, http://www.rlgsc.com

On 25/9/2022 5:01 pm, Volker Halle wrote:
> Mark,
>
> does your utility read until the 'End of file block' or the 'Highest block' of the directory file ?

I responded as 'highest block' but as you pointed out it only read to
the final byte in the file, not to the end-of-block or allocation.

> Free blocks should only be at the end of the directory file - after the EFBLK.
>
> The logical end of the directory would be the last 'directory record' in the 'End of file block'of the directory file, i.e. a directory record with DIR$W_SIZE=-1 marks the end of directory records in the current block and therefore the end of the directory in the 'End of file block'

This is the bit I missed!!

The "DIR$W_SIZE=-1 marks the end of directory records in the
current block and therefore the end of the directory in the
'End of file block'

After reading the end of record (-1) the code did not check whether this
was the EFBLK. It "continue"d (literally) on its merry way trying to
read more records.

/* continue to parse if not at end of content */
if (rptr < dzptr) continue;

/* end of records (file names) */
odsptr->DirectTheLength =
odsptr->DirectTheRecord =
odsptr->DirectSubRecord = 0;
if (odsptr->DirectWildcard[0])
status = RMS$_NMF;
else
status = RMS$_FNF;
break;

The code now explicitly tests for EBLK (in its own inimitable fashion).

if (rlen == (uint)0xffff && dzptr - rptr <= 512)
{
/* end of records in end file block */
odsptr->DirectTheLength =
odsptr->DirectTheRecord =
odsptr->DirectSubRecord = 0;
if (odsptr->DirectWildcard[0])
status = RMS$_NMF;
else
status = RMS$_FNF;
break;
}

The |EmtypBlockCount| was always a misnomer, perhaps better as
|EmptyRecordCount| because that is what it was ticking over. But still
a kludge and a result of missing the EFBLK factor. The kludge worked
until it didn't.

The generic file read originally referred to, does allocate what it
considers a whole number of blocks (of zeroed memory), from an earlier post

> /* round-up ensure we have a full complement of blocks allocated */
> odsptr->DataPtr = (char*)VmGet (((SizeInBytes/512)+1)*512);

which would (perhaps) explain the variable number of "empty records"
(less misleading name for the variable) following the end-of-records
sentinel in the end-of-file block.

> $ DUMP/DIR directory_file.DIR should show the last record as: 'xxx End of records (-1)' in the EOF block.
>
> The function to 'shuffle' a directory (extend or compress) is called SHUFFLE_DIR, found in module [F11X]SHFDIR
>
> Directory files are and must be contiguous. DUMP/HEADER directory.DIR shows:
>
> File characteristics: Contiguous, Directory
>
> This allows more efficient reads/writes and prevents window turns, when already executing inside the XQP.
>
> Volker.

--
Anyone, who using social-media, forms an opinion regarding anything
other than the relative cuteness of this or that puppy-dog, needs
seriously to examine their critical thinking.

On 25/9/2022 11:57 pm, Volker Halle wrote:
> Mark,
>
> if your program reports an error again, you should then carefully examine the directory file in question with
>
> $ DUMP/DIR/HEADER directory-filename.DIR/OUT=temp_file
>
>> So I guess to the highest block.
>
> No, the code reads until XabFhc.xab$l_ebk, which is the EFBLK (end-of-file block). And therefore should not find 'empty blocks'.

As noted in my previous post, yes, indeed. I missed the end-of-record
in the end-of-file-block essential element and went on parsing "empty
records" (I believe) in the zeroed memory.

> Test for yourself: create a directory, populate it with a couple of files until EFBLK becomes > 1, then delete all files in that directory.

I have performed multiple iterations of this scenario and am currently
using it as a test setup. Can't explain why initially I felt so
flummoxed in setting up a test scenario. Thanks for all the DUMP tools.
Don't believe I've used DUMP for file system work in at least 25 years.

> Use DUMP/HEADER to find the start LBN of the directory file and then DUMP/BLOCKS=(START=start_lbn,SIZE=2) disk:
> You'll see, that the blocks beyond VBN 1 of the directory file contain 'old directory data', not zeros and they don't start with DIR$W_SIZE=FFFF

$ DUMP/BLOCKS=(START=n,COUNT=n) DKA100:

confirms your statement.

>>> File characteristics: Contiguous, Directory
>>>
>>> This allows more efficient reads/writes and prevents window turns, when already executing inside the XQP.
>> Still a design consideration then.

I merely meant that directory files (still) must be contiguous.

> Please read the second half of that sentence: I strongly believe you can't do a window turn (which involves the XQP), if you are already in the XQP executing a directory operation. So directory files must be contiguous.
>
> Once you hit this problem again, feel free to provide some more details.
>
> Volker.

Thanks to all who contributed to this thread.

--
Anyone, who using social-media, forms an opinion regarding anything
other than the relative cuteness of this or that puppy-dog, needs
seriously to examine their critical thinking.

On 2022-09-25 15:00, Mark Daniel wrote:
> On 25/9/2022 5:01 pm, Volker Halle wrote:
>> Mark,
>
> Thanks for the detailed response Volker.
>
>> does your utility read until the 'End of file block' or the 'Highest
>> block' of the directory file ?
>
> It uses a "generic" file read function as "raw" blocks into memory.  It
> reads the exact number of bytes indicated by the file header.
>
>    if (odsptr->XabFhc.xab$l_ebk <= 1)
>       SizeInBytes = odsptr->XabFhc.xab$w_ffb;
>    else
>       SizeInBytes = ((odsptr->XabFhc.xab$l_ebk-1) << 9) +
>                     odsptr->XabFhc.xab$w_ffb;

Are directory files in ODS-2 using fixed size records? Because otherwise
you have a problem here that you cannot compute how many bytes you have
in a file like this.

The ffb tells where the first free byte is, so that appends to a file
can be done. However, the number of bytes you actually have in a file
does not include the record lengths of variable sized records, nor the
padding for odd length records. But these things only happen on variable
record length files, which is why I ask.
In ODS-1, the records are fixed length, but then again, ODS-1 filenames
are very much more limited, and the directory format is much simpler.

Johnny

Johnny,

you'll find all you want to know about OpenVMS directory structures in

http://www.bitsavers.org/pdf/dec/vax/vms/training/EY-F575E-DP_VMS_File_System_Internals_1990.pdf

page 53 onwards: 2.4 Basic Concept of a Directory

Volker.

osuv...@gmail.com schrieb am Sonntag, 25. September 2022 um 19:59:55 UTC+2:
> On Sunday, September 25, 2022 at 10:27:19 AM UTC-4, Volker Halle wrote:
> > Please read the second half of that sentence: I strongly believe you can't do a window turn (which involves the XQP), if you are already in the XQP executing a directory operation. So directory files must be contiguous.
> >
> There is a middle ground, such as what happens with the pager. The page file doesn't have to be contiguous,
> but the file can't have extension headers (i.e. all retrieval pointers must fit in the primary header). If the header
> for the directory file is cached in memory, a window turn won't require I/O.

Even if a directory would only be allowed to have a primary file header, you would still have to do a window turn, if there are more than ACP_WINDOW (default: 7) retrieval pointers. When handling directory expansion/compression the XQP is already operating in XQP 'secondary context' and I bet, that the XQP cannot invoke a window turn in that context.

Volker.

On 9/26/2022 4:42 AM, Johnny Billquist wrote:
> On 2022-09-25 15:00, Mark Daniel wrote:
>> On 25/9/2022 5:01 pm, Volker Halle wrote:
>>> does your utility read until the 'End of file block' or the 'Highest
>>> block' of the directory file ?
>>
>> It uses a "generic" file read function as "raw" blocks into memory.
>> It reads the exact number of bytes indicated by the file header.
>>
>>     if (odsptr->XabFhc.xab$l_ebk <= 1)
>>        SizeInBytes = odsptr->XabFhc.xab$w_ffb;
>>     else
>>        SizeInBytes = ((odsptr->XabFhc.xab$l_ebk-1) << 9) +
>>                      odsptr->XabFhc.xab$w_ffb;
>
> Are directory files in ODS-2 using fixed size records? Because otherwise
> you have a problem here that you cannot compute how many bytes you have
> in a file like this.
>
> The ffb tells where the first free byte is, so that appends to a file
> can be done. However, the number of bytes you actually have in a file
> does not include the record lengths of variable sized records, nor the
> padding for odd length records. But these things only happen on variable
> record length files, which is why I ask.

Fixed length files are also padded to even number bytes.

But the "standard" FIX 512 is even.

Arne

On 2022-09-25, Mark Daniel <mark.daniel@wasd.vsm.com.au> wrote:
>
> odsptr->Rab.rab$l_ubf = odsptr->DataPtr;
> for (;;)
> {
> if (SizeInBytes > 512*127)
> odsptr->Rab.rab$w_usz = 512*127;
> else
> if (SizeInBytes > 0)
> odsptr->Rab.rab$w_usz = SizeInBytes;
> else
> break;
>

I have a style question about this part of Mark's code and was wondering
if this was a common style, or if you always made liberal use of braces
to help draw attention to the nesting going on here and to help avoid
some later editing mistakes.

For example, I would write this code section as something like this:

if (SizeInBytes > 512*127)
{
odsptr->Rab.rab$w_usz = 512*127;
}
else
{
if (SizeInBytes > 0)
{
odsptr->Rab.rab$w_usz = SizeInBytes;
}
else
{
break;
}
}

(and yes, wrapping single statements in braces is also deliberate here.)

For me, I find the liberal use of braces helps to draw attention to the
code structure, is more readable in general (at least to me), and helps
avoid some mistakes during later editing.

Just wondering which is the more common style in the C code you have seen.

> status = sys$read (&odsptr->Rab, 0, 0);
> if (VMSnok (status)) break;
>
> odsptr->DataLength += odsptr->Rab.rab$w_rsz;
> odsptr->Rab.rab$l_ubf += odsptr->Rab.rab$w_rsz;
> SizeInBytes -= odsptr->Rab.rab$w_rsz;
> }
>
> sys$close (&odsptr->Fab, 0, 0);

Simon.

PS: Yes, Simon's a Whitesmiths fan. Other indentation styles are available. :-)

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

> For me, I find the liberal use of braces helps to draw attention to the
> code structure, is more readable in general (at least to me), and helps
> avoid some mistakes during later editing.

Wrapping even single statement loop/conditional/... blocks in braces
avoids misconstruing an indented statement as inside the block because
it shares indent level. It's recommended by some style / code quality
guides.

There are lots of styles, and about lots^2 internet wars over which is
best. Best not to start the ^3 round. :)

De

On 9/27/2022 1:56 PM, Simon Clubley wrote:
> On 2022-09-25, Mark Daniel <mark.daniel@wasd.vsm.com.au> wrote:
>>
>> odsptr->Rab.rab$l_ubf = odsptr->DataPtr;
>> for (;;)
>> {
>> if (SizeInBytes > 512*127)
>> odsptr->Rab.rab$w_usz = 512*127;
>> else
>> if (SizeInBytes > 0)
>> odsptr->Rab.rab$w_usz = SizeInBytes;
>> else
>> break;
>>
>
> I have a style question about this part of Mark's code and was wondering
> if this was a common style, or if you always made liberal use of braces
> to help draw attention to the nesting going on here and to help avoid
> some later editing mistakes.
>
> For example, I would write this code section as something like this:
>
> if (SizeInBytes > 512*127)
> {
> odsptr->Rab.rab$w_usz = 512*127;
> }
> else
> {
> if (SizeInBytes > 0)
> {
> odsptr->Rab.rab$w_usz = SizeInBytes;
> }
> else
> {
> break;
> }
> }
>
> (and yes, wrapping single statements in braces is also deliberate here.)
>
> For me, I find the liberal use of braces helps to draw attention to the
> code structure, is more readable in general (at least to me), and helps
> avoid some mistakes during later editing.
>
> Just wondering which is the more common style in the C code you have seen.

For both whether to use block construct for single statement and
for indentation style the right answer is whatever style is agreed on
in the project.

:-)

If I were in the project meeting discussing these topics then I would
argue for using block construct for single statements.

But honestly then I sometimes I forget myself.

Arne

PS: For braces I would vote Allman for C/C++, but different for other
languages.