I currently have a simple shell script along the lines of:

find "$1" -type d \
-exec test -f {}/SENTINEL-1 \; \
-exec test -f {}/SENTINEL-2 \; \
-print

The idea is to list directories that satisfy the condition of having two
exactly named files within them. This code works, but it's not exactly
fast (two execs for each matched directory), and it doesn't scale well:
three files or has one but NOT the other, has either one of them, has
sub-directories of particular names etc, all require rewriting the whole
command. The way I use this tool, it needs to examine both immediate
subdirectories of "$1" and one level further.

Before I write myself a custom tool, is there a better way of doing this
sort of thing? In my case, the directories are all small, with a median
of under ten items, so spped of readdir() versus stat() isn't a big
concern (and I'd be comparing the speed to find).

I already have a Perl script that might be a candidate for replacement
if there is a better way. That script does not need to recurse, so it
doesn't use find (or the Perl equivilent module). It takes a list of
directories and prints the ones that only have files matching a couple of
globs, specifically (case insenitive) *.txt and *.html. Any directory
that has a .jpg or other non-txt, non-html suffix, an or a file with no
suffix, is not printed.

For those wondering, the commonality is both of these uses are for
automatic clean-up of directories created by other programs. Directories
that match are safe to delete without further inspection (subject to
race conditions).

Elijah
------
has this set of data been processed? and backed up?

On Mon, 10 May 2021 22:13:50 +0000 (UTC)
Eli the Bearded <*@eli.users.panix.com> wrote:
> I currently have a simple shell script along the lines of:
>
> find "$1" -type d \
> -exec test -f {}/SENTINEL-1 \; \
> -exec test -f {}/SENTINEL-2 \; \
> -print
>
> The idea is to list directories that satisfy the condition of having two
> exactly named files within them. This code works, but it's not exactly
> fast (two execs for each matched directory), and it doesn't scale well:
> three files or has one but NOT the other, has either one of them, has
> sub-directories of particular names etc, all require rewriting the whole
> command.

Perhaps

find "$1" -type d -printf '[ -f %p/SENTINEL-1 -a -f %p/SENTINEL-2 ] && echo %p\n' | sh

You would need to do minimal rewriting for the other uses you mention. But
it won't work if the file names may contain characters which are special
to the shell.

> The way I use this tool, it needs to examine both immediate
> subdirectories of "$1" and one level further.

I would need to see an example to be clear what this means.

Op 10-05-21 om 23:13 schreef Eli the Bearded:
> I currently have a simple shell script along the lines of:
>
> find "$1" -type d \
> -exec test -f {}/SENTINEL-1 \; \
> -exec test -f {}/SENTINEL-2 \; \
> -print
>
> The idea is to list directories that satisfy the condition of having two
> exactly named files within them. This code works, but it's not exactly
> fast (two execs for each matched directory), and it doesn't scale well:

To do that sort of thing well with the standard shell and utilities,
your only recourse is to make 'find' invoke another shell that does the
work in a separate script. That script could be in a file or passed to
the shell via the -c option, as below.

find "$1" -type d -exec sh -c '
for dir do
if test -f "$dir/SENTINEL-1" \
&& test -f "$dir/SENTINEL-2"
then
printf %s\\n "$dir"
fi
done
' dummy {} +

The {} + (instead of {} \;) saves up all the arguments and passes as
many of them as possible to the sh invocation. This feature is in POSIX
and is widespread now (although still unfortunately quite broken in
BusyBox 'find').

The above should work for your stated purpose, but if you want to do
anything more complicated, you're quickly going to run into limitations.
For instance, you can't set any variables in the child shell and expect
to use them outside of it, nor can the separate script use any of your
shell functions.

These limitations are why I integrated the 'find' utility into the shell
as one of the modular loop constructs in my shell library, modernish,
which runs on nearly every POSIX shell and OS (see my .sig for the
link). As far as I know, this is a unique feature that does not exist
anywhere else. In modernish, you can do:

.. modernish
use var/loop/find
found=0
LOOP find dir in "$1" -type d
DO
if test -f "$dir/SENTINEL-1" \
&& test -f "$dir/SENTINEL-2"
then
printf 'Found: %s\n' "$dir"
found=$((found+1))
fi
DONE
printf 'Found %d matching directories\n' "$found"

The loop construct invokes 'find' in the background and feeds the
results into the shell in the specified variable (in this case, dir).
This gives you robust processing for free; even newlines in file names
are no problem (provided you quote your variables).

The counter in the example is just to demonstrate that variables survive
the loop as you would expect. The loop body runs in the main shell
environment and can do anything you want without limitations. Internally
it's just a 'while read && eval; do ... done' loop, connected to a file
descriptor that maintains a pipe to the 'find' utility. So things like
'continue' and 'break' also work as expected; the pipe is automatically
broken if the loop is exited for any reason, terminating the 'find'
background process. If you're curious about how it works, just look at
the library code -- it's well commented.

If you'll indulge me a little more, below is functionally identical
example that uses a few more shell language modernisation features in
modernish (see notes below):

.. modernish
use safe # remove most quoting headaches
use var/loop/find
dirsfound=0
LOOP find dir in $1 -type d
DO
if is present $dir/SENTINEL-1 \
&& is present $dir/SENTINEL-2
then
putln "Found: $dir"
let dirsfound+=1
fi
DONE
putln "Found $dirsfound directories matching criteria"

Notes:
- The safe mode turns off global split/glob to remove most quoting
headaches. With split and glob gone, only empty removal (meaning: an
unquoted empty variable is removed and will not become an empty
argument) remains as a potential issue.
- The is() function is part of a more readable alternative to the
horrible 'test' botch of a command that, unlike 'test', deals correctly
with empty removal. Use 'is present' to see if there's anything there,
'is reg' to test for a regular file, 'is -L reg' to test for a regular
file or symlink to a regular file.
- putln() avoids echo's portability pitfalls without needing to resort
to printf. Each argument is printed on a separate line. There's also
put() which prints space-separated arguments without a newline.
- the 'let' arithmetic command is provided as a shell function on any
shell that doesn't have a bash/ksh/zsh-style 'let' command built in.

--
|| modernish -- harness the shell
|| https://github.com/modernish/modernish
||
|| KornShell lives!
|| https://github.com/ksh93/ksh

On 5/10/2021 5:13 PM, Eli the Bearded wrote:
> I currently have a simple shell script along the lines of:
>
> find "$1" -type d \
> -exec test -f {}/SENTINEL-1 \; \
> -exec test -f {}/SENTINEL-2 \; \
> -print
>
> The idea is to list directories that satisfy the condition of having two
> exactly named files within them.

FWIW I'd just do something like (untested):

find "$1" -type f |
awk -F'/' '
{ dir=$(NF-1) }
dir != prev { prt(); prev=dir}
{ files[$NF] }
END { prt() }
function prt() {
if ( ("SENTINEL-1" in files) && ("SENTINEL-2" in files) ) {
print prev
}
delete files
}
'

so the `find` just prints a list of paths to files, awk is only called
once and will be extremely efficient so it scales well, it's only using
mandatory POSIX tools so it'll work on any Unix box, and you can
trivially write whatever condition you like in the `prt()` function or
write a generalized prt() function such that you can pass a variable to
specify different types of comparison and/or a list of files to test for.

Regards

Ed.

This code works, but it's not exactly
> fast (two execs for each matched directory), and it doesn't scale well:
> three files ;or has one but NOT the other, has either one of them, has
> sub-directories of particular names etc, all require rewriting the whole
> command. The way I use this tool, it needs to examine both immediate
> subdirectories of "$1" and one level further.
>
> Before I write myself a custom tool, is there a better way of doing this
> sort of thing? In my case, the directories are all small, with a median
> of under ten items, so spped of readdir() versus stat() isn't a big
> concern (and I'd be comparing the speed to find).
>
> I already have a Perl script that might be a candidate for replacement
> if there is a better way. That script does not need to recurse, so it
> doesn't use find (or the Perl equivilent module). It takes a list of
> directories and prints the ones that only have files matching a couple of
> globs, specifically (case insenitive) *.txt and *.html. Any directory
> that has a .jpg or other non-txt, non-html suffix, an or a file with no
> suffix, is not printed.
>
> For those wondering, the commonality is both of these uses are for
> automatic clean-up of directories created by other programs. Directories
> that match are safe to delete without further inspection (subject to
> race conditions).
>
> Elijah
> ------
> has this set of data been processed? and backed up?
>

Eli the Bearded <*@eli.users.panix.com> writes:

> find "$1" -type d \
> -exec test -f {}/SENTINEL-1 \; \
> -exec test -f {}/SENTINEL-2 \; \
> -print

What about

find "$1" -type f -maxdepth 2 \
| sed -n '/\/SENTINEL-[12]$/s/\/SENTINEL-[12]$//p' \
| uniq -c \
| awk '$1 == 2 {print $2}'

?

Best regards

Axel

On 5/13/2021 7:56 AM, Axel Reichert wrote:
> Eli the Bearded <*@eli.users.panix.com> writes:
>
>> find "$1" -type d \
>> -exec test -f {}/SENTINEL-1 \; \
>> -exec test -f {}/SENTINEL-2 \; \
>> -print
>
> What about
>
> find "$1" -type f -maxdepth 2 \
> | sed -n '/\/SENTINEL-[12]$/s/\/SENTINEL-[12]$//p' \

You don't need to match the regexp twice, that sed command can be
written as just:

sed -n 's/\/SENTINEL-[12]$//p'

You'd obviously have to be careful if your file names contained regexp
metachars or, with the awk below, if your directory names contained any
spaces.

Ed.

> | uniq -c \
> | awk '$1 == 2 {print $2}'
>
> ?
>
> Best regards
>
> Axel
>

On 5/13/2021 7:20 AM, Ed Morton wrote:
> On 5/10/2021 5:13 PM, Eli the Bearded wrote:
>> I currently have a simple shell script along the lines of:
>>
>>     find "$1" -type d \
>>         -exec test -f {}/SENTINEL-1 \; \
>>         -exec test -f {}/SENTINEL-2 \; \
>>         -print
>>
>> The idea is to list directories that satisfy the condition of having two
>> exactly named files within them.
>
> FWIW I'd just do something like (untested):
>
> find "$1" -type f |
> awk -F'/' '

In the unlikely event that your file/directory names can contain
newlines then you'd use GNU find with `-print0` and GNU awk with
`BEGIN{RS=ORS="\0"}` to use nul-terminated instead of newline-terminated
paths.

Ed.

>     { dir=$(NF-1) }
>     dir != prev { prt(); prev=dir}
>     { files[$NF] }
>     END { prt() }
>     function prt() {
>         if ( ("SENTINEL-1" in files) && ("SENTINEL-2" in files) ) {
>             print prev
>         }
>         delete files
>     }
> '
>
> so the `find` just prints a list of paths to files, awk is only called
> once and will be extremely efficient so it scales well, it's only using
> mandatory POSIX tools so it'll work on any Unix box, and you can
> trivially write whatever condition you like in the `prt()` function or
> write a generalized prt() function such that you can pass a variable to
> specify different types of comparison and/or a list of files to test for.
>
> Regards
>
>     Ed.
>
>  This code works, but it's not exactly
>> fast (two execs for each matched directory), and it doesn't scale well:
>> three files ;or has one but NOT the other, has either one of them, has
>> sub-directories of particular names etc, all require rewriting the whole
>> command. The way I use this tool, it needs to examine both immediate
>> subdirectories of "$1" and one level further.
>>
>> Before I write myself a custom tool, is there a better way of doing this
>> sort of thing? In my case, the directories are all small, with a median
>> of under ten items, so spped of readdir() versus stat() isn't a big
>> concern (and I'd be comparing the speed to find).
>>
>> I already have a Perl script that might be a candidate for replacement
>> if there is a better way. That script does not need to recurse, so it
>> doesn't use find (or the Perl equivilent module). It takes a list of
>> directories and prints the ones that only have files matching a couple of
>> globs, specifically (case insenitive) *.txt and *.html. Any directory
>> that has a .jpg or other non-txt, non-html suffix, an or a file with no
>> suffix, is not printed.
>>
>> For those wondering, the commonality is both of these uses are for
>> automatic clean-up of directories created by other programs. Directories
>> that match are safe to delete without further inspection (subject to
>> race conditions).
>>
>> Elijah
>> ------
>> has this set of data been processed? and backed up?
>>
>

>>>>> "EtB" == Eli the Bearded <*@eli.users.panix.com> writes:

EtB> I currently have a simple shell script along the lines of:
EtB> find "$1" -type d \
EtB> -exec test -f {}/SENTINEL-1 \; \
EtB> -exec test -f {}/SENTINEL-2 \; \
EtB> -print

Eli! Have you forgotten your Perl?

find2perl "$1" -type -d \
-eval '-f "SENTINEL-1" and -f "SENTINEL-2"' -print | perl

Done. And if you cache the program, it's just one Perl process. :)

print "Just another Perl hacker,"; # the original
--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Dart/Flutter consulting, Technical writing, Comedy, etc. etc.
Still trying to think of something clever for the fourth line of this .sig

In comp.unix.shell, Randal L. Schwartz <merlyn@stonehenge.com> wrote:
> >>>>> "EtB" == Eli the Bearded <*@eli.users.panix.com> writes:
> EtB> I currently have a simple shell script along the lines of:
> EtB> find "$1" -type d \
> EtB> -exec test -f {}/SENTINEL-1 \; \
> EtB> -exec test -f {}/SENTINEL-2 \; \
> EtB> -print
>
> Eli! Have you forgotten your Perl?

No, I "use File::Find" for other finding needs.

> find2perl "$1" -type -d \
> -eval '-f "SENTINEL-1" and -f "SENTINEL-2"' -print | perl

But I have forgotten "find2perl". That looks a lot better for "one"
liners.

> Done. And if you cache the program, it's just one Perl process. :)

I expect I will do that.

Elijah
------
likes this suggestion

Ed Morton <mortonspam@gmail.com> writes:

> On 5/13/2021 7:56 AM, Axel Reichert wrote:
>>
>> find "$1" -type f -maxdepth 2 \
>> | sed -n '/\/SENTINEL-[12]$/s/\/SENTINEL-[12]$//p' \
>
> You don't need to match the regexp twice, that sed command can be
> written as just:
>
> sed -n 's/\/SENTINEL-[12]$//p'

My bad, thanks for pointing this out. This redundancy crept in while
playing around with various approaches.

> You'd obviously have to be careful if your file names contained regexp
> metachars or, with the awk below, if your directory names contained
> any spaces.

Yep, that's clear.

Axel

On 2021-05-13, Randal L. Schwartz <merlyn@stonehenge.com> wrote:

> Eli! Have you forgotten your Perl?
>
> find2perl "$1" -type -d \
> -eval '-f "SENTINEL-1" and -f "SENTINEL-2"' -print | perl

find2perl is actually no longer part of perl.
You have to get it from CPAN (App-find2perl).

--
Christian "naddy" Weisgerber naddy@mips.inka.de

>>>>> "Christian" == Christian Weisgerber <naddy@mips.inka.de> writes:

Christian> find2perl is actually no longer part of perl.
Christian> You have to get it from CPAN (App-find2perl).

Nooooooo! First CGI.pm, and now this? Who's in charge here? :)

print "Just another Perl hacker,"; # the original

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Dart/Flutter consulting, Technical writing, Comedy, etc. etc.
Still trying to think of something clever for the fourth line of this .sig

In article <slrns9r8ip.2c69.naddy@lorvorc.mips.inka.de>,
Christian Weisgerber <naddy@mips.inka.de> wrote:
>On 2021-05-13, Randal L. Schwartz <merlyn@stonehenge.com> wrote:
>
>> Eli! Have you forgotten your Perl?
>>
>> find2perl "$1" -type -d \
>> -eval '-f "SENTINEL-1" and -f "SENTINEL-2"' -print | perl
>
>find2perl is actually no longer part of perl.
>You have to get it from CPAN (App-find2perl).

Indeed. I just tried running it - and, when that failed - trying to
install it - on a current Debian-ish system, and found that both failed -
i.e., it is no longer installed by default and it is no longer in the repos.

So, for the benefit of those of us who know next-to-nothing about Perl (and
prefer to remain that way), how (in simplest terms) do I go about
installing find2perl on this particular system?

Also, any idea why it was taken out (why it is no longer part of Perl) ?

--
"Women should not be enlightened or educated in any way. They should be
segregated because they are the cause of unholy erections in holy men.

-- Saint Augustine (354-430) --

gazelle@shell.xmission.com (Kenny McCormack) writes:

> In article <slrns9r8ip.2c69.naddy@lorvorc.mips.inka.de>,
> Christian Weisgerber <naddy@mips.inka.de> wrote:
>>On 2021-05-13, Randal L. Schwartz <merlyn@stonehenge.com> wrote:
>>
>>> Eli! Have you forgotten your Perl?
>>>
>>> find2perl "$1" -type -d \
>>> -eval '-f "SENTINEL-1" and -f "SENTINEL-2"' -print | perl
>>
>>find2perl is actually no longer part of perl.
>>You have to get it from CPAN (App-find2perl).
>
> Indeed. I just tried running it - and, when that failed - trying to
> install it - on a current Debian-ish system, and found that both failed -
> i.e., it is no longer installed by default and it is no longer in the repos.
>
> So, for the benefit of those of us who know next-to-nothing about Perl (and
> prefer to remain that way), how (in simplest terms) do I go about
> installing find2perl on this particular system?
>
> Also, any idea why it was taken out (why it is no longer part of Perl) ?

Fedora here.

dnf install perl-App-find2perl

Fed it a find command, it generated Perl.

--
Dan Espen

On 2021-05-15, Kenny McCormack <gazelle@shell.xmission.com> wrote:

>>find2perl is actually no longer part of perl.
>
> So, for the benefit of those of us who know next-to-nothing about Perl (and
> prefer to remain that way), how (in simplest terms) do I go about
> installing find2perl on this particular system?

dh-make-perl --build --cpan App::find2perl
sudo dpkg -i *find2perl*.deb

Or so says
https://wiki.debian.org/PerlFAQ

> Also, any idea why it was taken out (why it is no longer part of Perl) ?

The s2p (sed-to-perl) and a2p (awk-to-perl) translators were removed
because they were ancient cruft that had outlived its usefulness,
and find2perl was in the same bathtub^Wdirectory.
https://groups.google.com/g/perl.perl5.porters/c/3RBKnlPaSmA

--
Christian "naddy" Weisgerber naddy@mips.inka.de

>>>>> "Kenny" == Kenny McCormack <gazelle@shell.xmission.com> writes:

Kenny> So, for the benefit of those of us who know next-to-nothing about Perl (and
Kenny> prefer to remain that way), how (in simplest terms) do I go about
Kenny> installing find2perl on this particular system?

Kenny> Also, any idea why it was taken out (why it is no longer part of
Kenny> Perl) ?

There's also my File::Finder in the CPAN which can build the equivalent
of nearly any find(1) expression using method chaining.

In this case, it'd be:

use File::Finder;

File::Finder
->type('d')
->eval('-f "SENTINEL-1" and -f "SENTINEL-2"')
->print
->in(@ARGV);

Or instead of printing them, you can gather them as an array:

use File::Finder;

my @list = File::Finder
->type('d')
->eval('-f "SENTINEL-1" and -f "SENTINEL-2"')
->in(@ARGV);

File::Finder was fun to write. And I essentially used a FORTH-like
"builds...does" strategy for the method chaining. And the expression
evaluator supports and/or/not/parens. So for something in find(1) like:

-condition -and -not \( -this -or -that \)

you could write it as:

condition->and->not->left->this->or->that->right

print "Just another Perl hacker,"; # the original

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Dart/Flutter consulting, Technical writing, Comedy, etc. etc.
Still trying to think of something clever for the fourth line of this .sig

In article <slrnsa0b2l.16m8.naddy@lorvorc.mips.inka.de>,
Christian Weisgerber <naddy@mips.inka.de> wrote:
>On 2021-05-15, Kenny McCormack <gazelle@shell.xmission.com> wrote:
>
>>>find2perl is actually no longer part of perl.
>>
>> So, for the benefit of those of us who know next-to-nothing about Perl (and
>> prefer to remain that way), how (in simplest terms) do I go about
>> installing find2perl on this particular system?
>
>dh-make-perl --build --cpan App::find2perl

Interesting. Alas, it didn't work.

First, I had to "apt install dh-make-perl". That installed a crapload of
packages. And when I say crapload, I mean crapload - 104 of them, to be
exact.

Once that was installed, I tried running the above command (as a user
[non-root] account). It then figured out that it had not been run before,
so it did another route of mega-installing of stuff from here, there, and
everywhere. At some points during this process, it would seem to hang,
producing no output and consuming 100% CPU - doing God-knows-what.

But, fortunately it would seem, no error messages (yet).

Finally, that part ended, and it got to the business at hand - of actually
building the .deb file. That failed with some obscure error messages and
the whole process aborted. Just for safety, I tried it again, but with the
same result.

I think I've run into this before Perl, where it just seems to take on a
gargantuan task that it isn't quite capable of finishing. I've also seen
it with a similar product called "Python". Very similar...

Anyway, fortunately, both of the following are true:

1) This was always just an academic project for me. I'm perfectly
happy using ordinary "find" - seems more reliable, although I have
been impressed with how well find2perl actually does work (if you
can get it installed...)

2) I *was* able to take the script (/usr/bin/find2perl) from another
system (*) and copy it over to the new system (**) and it works
fine, so problem solved (more or less).

Still curious why it didn't work, but not enough so to spend any more time
on it...

(*) An earlier version of the OS, where find2perl *was* part of Perl.

(**) A newer version of the same OS, where find2perl is not part of Perl.

>sudo dpkg -i *find2perl*.deb
>
>Or so says
>https://wiki.debian.org/PerlFAQ

--
That's the Trump playbook. Every action by Trump or his supporters can
be categorized as one (or more) of:

outrageous, incompetent, or mentally ill.