On Sat, 14 Aug 2021 20:53:08 -0000 (UTC)
Martin Gregorie <martin@mydomain.invalid> wrote:
> On Sat, 14 Aug 2021 12:57:09 -0700, Mike Easter wrote:
> > The commands were designed to be very powerful; the greater the power
> > the greater the 'responsibility'. Sometimes when one sees a man result
> > he feels like he is drowning.
> >
> The other thing that needs to be taught is that each command is designed
> to do one thing and to do it well, which is why there are so *many*
> commands. hence the need to understand 'apropos', or better,
>
> apropos 'action name' | less
>
> as an aid to finding the command they want and than
>
> man commandname
>
> to see how to use it.
>
> > Of course, being thrown into the water and needing to swim back out is
> > one way to learn to swim :-)
> >
> .. and sadly there isn't a lot else. If the student is capable of using
> command lines on another system, then something like 'Linux in a
> Nutshell' may be helpful, but that's about the only decent book I know
> unless there's a 'Linux for Dummies' available and they're not put off by
> the title.

If I go on amazon and search for "Linux command line" I see many books and
they tend to have high rating average. Either you don't consider them decent
or you haven't performed any such search in a long time even just for
curiosity.

--
vlaho.ninja/prog

Martin Gregorie <martin@mydomain.invalid> writes:
> William Unruh wrote:
>
>> Says someone, apparently, who has never looked at the command "find", or
>> may other commands. billions of different option combinations, some of
>> which work, others of which do not.
>
> Quite right: I don't use it because 'locate' is *much* faster and easier
> to use, especially if updatedb is run overnight by a cronjob

Bad comparison, since it doesn’t do the same thing.

--
https://www.greenend.org.uk/rjk/

Martin Gregorie wrote:
> On Sat, 14 Aug 2021 22:52:08 +0000, William Unruh wrote:
>
>> Says someone, apparently, who has never looked at the command "find", or
>> may other commands. billions of different option combinations, some of
>> which work, others of which do not.
>>
> Quite right: I don't use it because 'locate' is *much* faster and easier
> to use, especially if updatedb is run overnight by a cronjob
>
> Similarly, 'apropos' is nearly as fast 'locate' since it only has to scan
> the contents of /usr/share/man/* - and in addition, because its scanning
> the first line of each manpage, it also matches words of phrases
> describing what a program does, so often searching on a word or phrase
> describing what a program does means you a suitable program without
> knowing its name:
>
> $ apropos 'free space'
> e2freefrag (8) - report free space fragmentation information
> xfs_spaceman (8) - show free space information about an XFS filesystem
>
> $ apropos 'space used'
> space used: nothing appropriate.
> $ apropos 'space usage'
> df (1) - report file system disk space usage
> du (1) - estimate file space usage
> du (1p) - estimate file space usage
>
> ... and its no use complaining about how well or badly a manpage is
> written: often the only way to fit would be to submit a manpage patch.
>
> Yes, I know some Linux manpages are pretty bad. However, others (those
> for bash, sort and awk to name but a few) are excellent and most are
> usable.

This is why you keep a "notes" file.

find /media/somedisk -type d -exec ls -al -1 -d {} + > directories.txt
find /media/somedisk -type f -exec ls -al -1 {} + > filelist.txt

Any time you put some effort into crafting one, you
record it for the future.

./ffmpeg -hwaccel nvdec -i "fedora.mkv" -y -acodec aac -vcodec h264_nvenc -crf 23 "output2.mp4"

16.3x speed, 488FPS

Part of the fun is making them cryptic, so you can't understand them later.

Paul

On 13/08/2021 20:28, Java Jive wrote:
> I have the following lines in a shell script ...
>
> while [ -n "${LINE}" ]
>     do
>         if [ -n "${LINE} ]
>             then
>                 # Do processing
>         fi
>     done < "${DATA}"
>
> .... and this works fine for all but two lines in the data file, which
> contain accented characters. A file erroneously named with an e acute
> needs to be renamed to have an e grave, and a filename containing an e
> umlaut needs to be moved to a new location and given a new name.

Uggghhh! The reason for this disgust will become clear shortly!

This is a follow up question about character encodings ...

Previously I have released to my family two versions of the same archive
of family documents going back to the reign of Queen Anne, some items
possibly a little earlier. These documents were scanned (1o for
original scan) and then put through four possible stages of post-processing:
2n Contrast 'normalised' using pnnorm
3t Textcleaned
4nt n followed by 3
5tn t followed by n

For each document, the best result was copied into the main archive,
while the above preprocessing stages were left in an '_all'
sub-directory structure, with five subdirectories named as above, each
of which having beneath it a directory tree mirroring the main archive.

The main version of the archive, which most family members seem to have
downloaded, only included the main archive and didn't include the _all
subdirectory with all the pre-processing results, the full version
included this directory. IIRC, the former was compressed by WinZip from
the archive as it existed on a Windows PC at the time, but WinZip threw
a wobbly over the size of the full archive, so for that I had to use 7zip.

Now the crunch, when I unzip these on a Linux machine, I see different
bastardisations of accented characters. So, for example where the full
7zip archive when extracted shows an e acute correctly in both a console
and a file manager listing ...
"Chat Botté, Le" [e is correctly acute]
.... (if you're wondering, a French children's picture book version of
apparently 'Puss In Boots'), while with the WinZip main archive a
console listing shows a very odd character sequence instead of the e
acute ...
"Chat Bott'$'\302\202'', Le"
.... and a file manager listing has a graphic character resembling a 2x2
matrix, concerning which note that while \302 octal = \xC2 hex, and
\202 octal = \x82 hex, only the second of these and not the first
appears in the symbol:
|00|
|82|

My problem is that I can't find a search term to trap this strange
character to correct it, for example the following, and a few similar
that I've tried, don't work because they don't find the directory:
mv "Chat Bott'$'\302\202'', Le" "Chat Botté, Le"
mv Chat\ Bott\'$\'\\302\\202\'\',\ Le "Chat Botté, Le"

I could use a glob wildcard character such as '?', but currently all the
filenames are within quotes, where globbing doesn't seem to work, and it
would be a hell of a business removing the quotes, because many names in
the archive use many characters that would each need to be anticipated
and escaped for in an unquoted filename, such as spaces, ampersands,
brackets, etc.

Can anyone suggest a sequence that will find the file, when put inside
quotes as the filename in the controlling data file mentioned previously
in the thread, so that it can just be treated like all the other lines?
As someone here suggested the data file is now stored as UTF-8 rather
than ANSI as it was formerly, and some example lines are given below in
a form for easier readability in a ng - in reality the fields are tab
separated but here are separated by double spacing and have been further
abbreviated to keep them from wrapping; leading symbols such as '+' and
'=' have special meanings for the program doing the work; and, yes, the
commands are basically DOS commands which for Linux are translated to
their bash equivalents:

=ATTRIB -R "./F H/Close/Sts Mary & John Churchyard Monuments.pdf"
=RD "./F H /_all/1o/Blessig & Heyder"
REN "./Chat Bott'$'\302\202'', Le" "Chat Botté, Le"
MOVE "./Photo - D & M Close.png" "./Photos/D & M Close.png"
[etc]

--

Fake news kills!

I may be contacted via the contact address given on my website:
www.macfh.co.uk

On Sun, 15 Aug 2021 08:37:45 +0100, Richard Kettlewell wrote:

> Martin Gregorie <martin@mydomain.invalid> writes:
>> William Unruh wrote:
>>
>>> Says someone, apparently, who has never looked at the command "find",
>>> or may other commands. billions of different option combinations, some
>>> of which work, others of which do not.
>>
>> Quite right: I don't use it because 'locate' is *much* faster and
>> easier to use, especially if updatedb is run overnight by a cronjob
>
> Bad comparison, since it doesn’t do the same thing.

Both look for filenames, but 'find' can be restricted to a directory
structure - thats about the difference I can see in a quick manpage scan.

--
--
Martin | martin at
Gregorie | gregorie dot org

On Sun, 15 Aug 2021 05:23:18 +0000, Spiros Bousbouras wrote:

> On Sat, 14 Aug 2021 20:53:08 -0000 (UTC)
> Martin Gregorie <martin@mydomain.invalid> wrote:
>> On Sat, 14 Aug 2021 12:57:09 -0700, Mike Easter wrote:
>> > The commands were designed to be very powerful; the greater the power
>> > the greater the 'responsibility'. Sometimes when one sees a man
>> > result he feels like he is drowning.
>> >
>> The other thing that needs to be taught is that each command is
>> designed to do one thing and to do it well, which is why there are so
>> *many* commands. hence the need to understand 'apropos', or better,
>>
>> apropos 'action name' | less
>>
>> as an aid to finding the command they want and than
>>
>> man commandname
>>
>> to see how to use it.
>>
>> > Of course, being thrown into the water and needing to swim back out
>> > is one way to learn to swim :-)
>> >
>> .. and sadly there isn't a lot else. If the student is capable of using
>> command lines on another system, then something like 'Linux in a
>> Nutshell' may be helpful, but that's about the only decent book I know
>> unless there's a 'Linux for Dummies' available and they're not put off
>> by the title.
>
> If I go on amazon and search for "Linux command line" I see many books
> and they tend to have high rating average. Either you don't consider
> them decent or you haven't performed any such search in a long time even
> just for curiosity.

Back in 2003 when I was moving my home systems from OS9/68000 to Linux I
got a copy of 'UNIX in Nutshell', which contained most of what I needed'
except for sysadmin stuff - and I couldn't find anything useful for
system administration and hor the kernel is organised until I settled on
'Debian Reference':

https://www.debian.org/doc/manuals/debian-reference/

Since then I've added Fedora documentation:
https://fedoraproject.org/wiki/Category:Documentation?rd=Docs

and stuff on Systemd

https://www.freedesktop.org/software/systemd/man/systemd.html#

--
--
Martin | martin at
Gregorie | gregorie dot org

Martin Gregorie <martin@mydomain.invalid> writes:
> Richard Kettlewell wrote:
>> Martin Gregorie <martin@mydomain.invalid> writes:
>>> William Unruh wrote:

>>>> Says someone, apparently, who has never looked at the command
>>>> "find", or may other commands. billions of different option
>>>> combinations, some of which work, others of which do not.
>>>
>>> Quite right: I don't use it because 'locate' is *much* faster and
>>> easier to use, especially if updatedb is run overnight by a cronjob
>>
>> Bad comparison, since it doesn’t do the same thing.
>
> Both look for filenames,

No. I mean, that’s one thing find can do, but it’s nowhere near all of
it.

> but 'find' can be restricted to a directory structure - thats about
> the difference I can see in a quick manpage scan.

find has numerous ways of selecting which files to act on and what to do
with them. It also differs more fundamentally in that it traverses the
real directory structure, rather than an incomplete snapshot made at
some point in the past.

--
https://www.greenend.org.uk/rjk/

On Sun, 15 Aug 2021 12:57:24 +0100
Java Jive <java@evij.com.invalid> wrote:
> Now the crunch, when I unzip these on a Linux machine, I see different
> bastardisations of accented characters. So, for example where the full
> 7zip archive when extracted shows an e acute correctly in both a console
> and a file manager listing ...
> "Chat Botté, Le" [e is correctly acute]
> ... (if you're wondering, a French children's picture book version of
> apparently 'Puss In Boots'), while with the WinZip main archive a
> console listing shows a very odd character sequence instead of the e
> acute ...
> "Chat Bott'$'\302\202'', Le"
> ... and a file manager listing has a graphic character resembling a 2x2
> matrix, concerning which note that while \302 octal = \xC2 hex, and
> \202 octal = \x82 hex, only the second of these and not the first
> appears in the symbol:
> |00|
> |82|

You aren't going to get anywhere with using high level tools for this. You
need to go low level and see the values of the actual bytes in the filenames.
So for example something like

ls *Chat* | od -A n -t x1

which will show the bytes in hexadecimal.

> My problem is that I can't find a search term to trap this strange
> character to correct it, for example the following, and a few similar
> that I've tried, don't work because they don't find the directory:
> mv "Chat Bott'$'\302\202'', Le" "Chat Botté, Le"
> mv Chat\ Bott\'$\'\\302\\202\'\',\ Le "Chat Botté, Le"

What directory ? Your post says that some files have strange names. Do also
some directories have strange names ? In any case , the commands above do not
show a directory separator.

--
Who is the poster boy for posters ?

On 14.08.2021 at 22:52, William Unruh scribbled:

> On 2021-08-14, Martin Gregorie <martin@mydomain.invalid> wrote:
>
> > apropos 'action name' | less
> >
> > as an aid to finding the command they want and than
> >
> > man commandname
>
> Unfortunately man is really good for reminding someone what various
> things mean, but is pretty bad at teaching anyone how to use the
> command. A much much larger section for each man page with explicity
> examples would go a long way to making "man" useful for newbies (and
> oldies as well).

True, but this is what the GNU "info" command was (supposedly) intended
for. Well, that, and a more markup-style browsing experience, with
embedded links to other locally installed pages.

But then again, not all distributions install "info", and conversely,
I have already come across a distribution -- I seem to remember that it
was an earlier release of either PCLinuxOS or Mageia, but I'm not sure
anymore -- that installed the "info" pages by default but not the "man"
pages. And that was odd, because every UNIX system should at the very
least have "man" installed.

On my Manjaro system here, I have both "man" and "info" installed. The
latter did not come installed by default, but I either way find myself
looking far more at "man" pages than at "info" pages, as the "info"
system is actually quite bulky and a bit overkill -- sort of like the
difference between the simplicity of how the original GRUB worked and
the complexity of how GRUB2 works.

--
With respect,
= Aragorn =

On 15/08/2021 13:58, Spiros Bousbouras wrote:
>
> On Sun, 15 Aug 2021 12:57:24 +0100
> Java Jive <java@evij.com.invalid> wrote:
>>
>> Now the crunch, when I unzip these on a Linux machine, I see different
>> bastardisations of accented characters. So, for example where the full
>> 7zip archive when extracted shows an e acute correctly in both a console
>> and a file manager listing ...
>> "Chat Botté, Le" [e is correctly acute]
>> ... (if you're wondering, a French children's picture book version of
>> apparently 'Puss In Boots'), while with the WinZip main archive a
>> console listing shows a very odd character sequence instead of the e
>> acute ...
>> "Chat Bott'$'\302\202'', Le"
>> ... and a file manager listing has a graphic character resembling a 2x2
>> matrix, concerning which note that while \302 octal = \xC2 hex, and
>> \202 octal = \x82 hex, only the second of these and not the first
>> appears in the symbol:
>> |00|
>> |82|
>
> You aren't going to get anywhere with using high level tools for this. You
> need to go low level and see the values of the actual bytes in the filenames.
> So for example something like
>
> ls *Chat* | od -A n -t x1
>
> which will show the bytes in hexadecimal.

Thanks again, will look into that.

>> My problem is that I can't find a search term to trap this strange
>> character to correct it, for example the following, and a few similar
>> that I've tried, don't work because they don't find the directory:
>> mv "Chat Bott'$'\302\202'', Le" "Chat Botté, Le"
>> mv Chat\ Bott\'$\'\\302\\202\'\',\ Le "Chat Botté, Le"
>
> What directory ? Your post says that some files have strange names. Do also
> some directories have strange names ? In any case , the commands above do not
> show a directory separator.

As part of my manual investigations of the problem, I changed to the
directory of which the problem directory is a direct sub-directory, to
allow experimentation without having to type tediously extended pathnames.

--

Fake news kills!

I may be contacted via the contact address given on my website:
www.macfh.co.uk

On 15/08/2021 10:51 am, Paul wrote:
> Martin Gregorie wrote:
>> On Sat, 14 Aug 2021 22:52:08 +0000, William Unruh wrote:
>>
>>> Says someone, apparently, who has never looked at the command "find", or
>>> may other commands. billions of different option combinations, some of
>>> which work, others of which do not.
>>>
>> Quite right: I don't use it because 'locate' is *much* faster and
>> easier to use, especially if updatedb is run overnight by a cronjob
>>
>> Similarly, 'apropos' is nearly as fast 'locate' since it only has to
>> scan the contents of /usr/share/man/* - and in addition, because its
>> scanning the first line of each manpage, it also matches words of
>> phrases describing what a program does, so often searching on a word
>> or phrase describing what a program does means you a suitable program
>> without knowing its name:
>>
>> $ apropos 'free space'
>> e2freefrag (8) - report free space fragmentation information
>> xfs_spaceman (8) - show free space information about an XFS
>> filesystem
>>
>> $ apropos 'space used'
>> space used: nothing appropriate.
>> $ apropos 'space usage'
>> df (1) - report file system disk space usage
>> du (1) - estimate file space usage
>> du (1p) - estimate file space usage
>>
>> ... and its no use complaining about how well or badly a manpage is
>> written: often the only way to fit would be to submit a manpage patch.
>> Yes, I know some Linux manpages are pretty bad. However, others (those
>> for bash, sort and awk to name but a few) are excellent and most are
>> usable.
>
> This is why you keep a "notes" file.
>
> find /media/somedisk -type d -exec ls -al -1 -d {} + > directories.txt
> find /media/somedisk -type f -exec ls -al -1 {} + > filelist.txt
>
> Any time you put some effort into crafting one, you
> record it for the future.
>
> ./ffmpeg -hwaccel nvdec -i "fedora.mkv" -y -acodec aac -vcodec
> h264_nvenc -crf 23 "output2.mp4"
>
> 16.3x speed, 488FPS
>
> Part of the fun is making them cryptic, so you can't understand them later.
>
> Paul
>

Why not:
find /media/somedisk -type f -ls > filelist.txt

Saves a process (or two)

--
Chris Elvidge
England

On 15/08/2021 16:00, Java Jive wrote:
>
> On 15/08/2021 13:58, Spiros Bousbouras wrote:
>>
>> You aren't going to get anywhere with using high level tools for this.
>> You
>> need to go low level and see the values of the actual bytes in the
>> filenames.
>> So for example something like
>>
>>      ls *Chat* | od -A n -t x1
>>
>> which will show the bytes in hexadecimal.
>
> Thanks again, will look into that.

As I suspected from the octal, it's C2 82 ...

For example the first line is (though really I need a fixed font to show
this):
43 68 61 74 20 42 6f 74 74 c2 82 2c 20 4c 65 20
C h a t B o t t , L e

--

Fake news kills!

I may be contacted via the contact address given on my website:
www.macfh.co.uk

Le 15-08-2021, Martin Gregorie <martin@mydomain.invalid> a écrit :
> On Sun, 15 Aug 2021 08:37:45 +0100, Richard Kettlewell wrote:
>
>> Martin Gregorie <martin@mydomain.invalid> writes:
>>> William Unruh wrote:
>>>
>>>> Says someone, apparently, who has never looked at the command "find",
>>>> or may other commands. billions of different option combinations, some
>>>> of which work, others of which do not.
>>>
>>> Quite right: I don't use it because 'locate' is *much* faster and
>>> easier to use, especially if updatedb is run overnight by a cronjob
>>
>> Bad comparison, since it doesn’t do the same thing.
>
> Both look for filenames, but 'find' can be restricted to a directory
> structure - thats about the difference I can see in a quick manpage scan.

Your search looks very quick. I don't use locate but I found its
manpage on Internet. From it, I don't see how to use locate to know
which files have been created recently. For example, I see a lot of
things find can do and not locate.

You can use find to look for files in an usb stick too. I guess running
updatedb on an usb stick may become ugly.

--
Si vous avez du temps à perdre :
https://scarpet42.gitlab.io

Java Jive wrote:
> On 13/08/2021 20:28, Java Jive wrote:
>> I have the following lines in a shell script ...
>>
>> while [ -n "${LINE}" ]
>> do
>> if [ -n "${LINE} ]
>> then
>> # Do processing
>> fi
>> done < "${DATA}"
>>
>> .... and this works fine for all but two lines in the data file, which
>> contain accented characters. A file erroneously named with an e acute
>> needs to be renamed to have an e grave, and a filename containing an e
>> umlaut needs to be moved to a new location and given a new name.
>
> Uggghhh! The reason for this disgust will become clear shortly!
>
> This is a follow up question about character encodings ...
>
> Previously I have released to my family two versions of the same archive
> of family documents going back to the reign of Queen Anne, some items
> possibly a little earlier. These documents were scanned (1o for
> original scan) and then put through four possible stages of
> post-processing:
> 2n Contrast 'normalised' using pnnorm
> 3t Textcleaned
> 4nt n followed by 3
> 5tn t followed by n
>
> For each document, the best result was copied into the main archive,
> while the above preprocessing stages were left in an '_all'
> sub-directory structure, with five subdirectories named as above, each
> of which having beneath it a directory tree mirroring the main archive.
>
> The main version of the archive, which most family members seem to have
> downloaded, only included the main archive and didn't include the _all
> subdirectory with all the pre-processing results, the full version
> included this directory. IIRC, the former was compressed by WinZip from
> the archive as it existed on a Windows PC at the time, but WinZip threw
> a wobbly over the size of the full archive, so for that I had to use 7zip.
>
> Now the crunch, when I unzip these on a Linux machine, I see different
> bastardisations of accented characters. So, for example where the full
> 7zip archive when extracted shows an e acute correctly in both a console
> and a file manager listing ...
> "Chat Botté, Le" [e is correctly acute]
> ... (if you're wondering, a French children's picture book version of
> apparently 'Puss In Boots'), while with the WinZip main archive a
> console listing shows a very odd character sequence instead of the e
> acute ...
> "Chat Bott'$'\302\202'', Le"
> ... and a file manager listing has a graphic character resembling a 2x2
> matrix, concerning which note that while \302 octal = \xC2 hex, and
> \202 octal = \x82 hex, only the second of these and not the first
> appears in the symbol:
> |00|
> |82|
>
> My problem is that I can't find a search term to trap this strange
> character to correct it, for example the following, and a few similar
> that I've tried, don't work because they don't find the directory:
> mv "Chat Bott'$'\302\202'', Le" "Chat Botté, Le"
> mv Chat\ Bott\'$\'\\302\\202\'\',\ Le "Chat Botté, Le"
>
> I could use a glob wildcard character such as '?', but currently all the
> filenames are within quotes, where globbing doesn't seem to work, and it
> would be a hell of a business removing the quotes, because many names in
> the archive use many characters that would each need to be anticipated
> and escaped for in an unquoted filename, such as spaces, ampersands,
> brackets, etc.
>
> Can anyone suggest a sequence that will find the file, when put inside
> quotes as the filename in the controlling data file mentioned previously
> in the thread, so that it can just be treated like all the other lines?
> As someone here suggested the data file is now stored as UTF-8 rather
> than ANSI as it was formerly, and some example lines are given below in
> a form for easier readability in a ng - in reality the fields are tab
> separated but here are separated by double spacing and have been further
> abbreviated to keep them from wrapping; leading symbols such as '+' and
> '=' have special meanings for the program doing the work; and, yes, the
> commands are basically DOS commands which for Linux are translated to
> their bash equivalents:
>
> =ATTRIB -R "./F H/Close/Sts Mary & John Churchyard Monuments.pdf"
> =RD "./F H /_all/1o/Blessig & Heyder"
> REN "./Chat Bott'$'\302\202'', Le" "Chat Botté, Le"
> MOVE "./Photo - D & M Close.png" "./Photos/D & M Close.png"
> [etc]
>

https://stackoverflow.com/questions/4177783/xc3-xa9-and-other-codes/4177813#4177813

It looks like perhaps this "text string" for the filename,
went through some web encoding at some point. With a hex
editor, I can change C3 A9 to E9 hex, and the character in
the hex editor (on the right hand side) looks visually correct.

https://i.postimg.cc/TP57bLD9/C3-A9-to-E9.gif

You could do such an operation, in Perl, right on the
file system.

*********************** rename2.ps *************************
printf("this is a test\n");

$start = "Chat Bott";
$finish = ", Le";
$naughty1 = <\x{C3}\x{A9}> ;
$naughty2 = <\x{E9}> ;

$x = $start.$finish ;
$y = $start.$naughty1.$finish ;
$z = $start.$naughty2.$finish ;

open(OUT, ">>$x") || die("Cannot create X");
close(OUT);

open(OUT, ">>$y") || die("Cannot create Y");
close(OUT);

open(OUT, ">>$z") || die("Cannot create Z");
close(OUT);

use Cwd;

$c = getcwd ;

printf("Making a mess in %s\n", $c );

#rename( $y , $z );

exit(0);
*********************** end of rename2.ps *************************

I ran this in Windows 11, by double-clicking the file. I
could not run it using one of their terminals. I just thought
it was mildly amusing as to what the filenames looked like.

The idea of the script above, is you run it multiple times,
commenting out a line here or there, while you do your tests.
For example, comment out the creation of file $z and
enable the rename(y,z) command near the bottom, to see
if the created $y can be renamed to the presumed operational $z value.

https://i.postimg.cc/gksLyGFL/rename2-output.gif [Picture]

So far, I only tested it as copy/pasted above. I haven't
tested the rename.

Then, you'd need to pick up a recursive tree ("find-next-file")
type pattern, and look for a filename with $naughty1 in it,
and rename it somehow. Maybe something like one of the
examples here. You would probably need to look for a
substring of $naughty1, in the filenames returned.

https://stackoverflow.com/questions/5089680/how-to-find-files-folders-recursively-in-perl-script

File renaming, is the only thing I've done with Perl :-)
I'll never be a Perl person I guess.

Paul

Chris Elvidge wrote:
> On 15/08/2021 10:51 am, Paul wrote:
>> Martin Gregorie wrote:
>>> On Sat, 14 Aug 2021 22:52:08 +0000, William Unruh wrote:
>>>
>>>> Says someone, apparently, who has never looked at the command
>>>> "find", or
>>>> may other commands. billions of different option combinations, some of
>>>> which work, others of which do not.
>>>>
>>> Quite right: I don't use it because 'locate' is *much* faster and
>>> easier to use, especially if updatedb is run overnight by a cronjob
>>>
>>> Similarly, 'apropos' is nearly as fast 'locate' since it only has to
>>> scan the contents of /usr/share/man/* - and in addition, because its
>>> scanning the first line of each manpage, it also matches words of
>>> phrases describing what a program does, so often searching on a word
>>> or phrase describing what a program does means you a suitable program
>>> without knowing its name:
>>>
>>> $ apropos 'free space'
>>> e2freefrag (8) - report free space fragmentation information
>>> xfs_spaceman (8) - show free space information about an XFS
>>> filesystem
>>>
>>> $ apropos 'space used'
>>> space used: nothing appropriate.
>>> $ apropos 'space usage'
>>> df (1) - report file system disk space usage
>>> du (1) - estimate file space usage
>>> du (1p) - estimate file space usage
>>>
>>> ... and its no use complaining about how well or badly a manpage is
>>> written: often the only way to fit would be to submit a manpage patch.
>>> Yes, I know some Linux manpages are pretty bad. However, others
>>> (those for bash, sort and awk to name but a few) are excellent and
>>> most are usable.
>>
>> This is why you keep a "notes" file.
>>
>> find /media/somedisk -type d -exec ls -al -1 -d {} + >
>> directories.txt
>> find /media/somedisk -type f -exec ls -al -1 {} + > filelist.txt
>>
>> Any time you put some effort into crafting one, you
>> record it for the future.
>>
>> ./ffmpeg -hwaccel nvdec -i "fedora.mkv" -y -acodec aac -vcodec
>> h264_nvenc -crf 23 "output2.mp4"
>>
>> 16.3x speed, 488FPS
>>
>> Part of the fun is making them cryptic, so you can't understand them
>> later.
>>
>> Paul
>>
>
> Why not:
> find /media/somedisk -type f -ls > filelist.txt
>
> Saves a process (or two)

At the time, this was a speed optimization.

I have no idea today, whether it's a good idea or not,
but I still use that.

It's also possible it might blow up on a pathological
file tree of some sort ("line too long").

Paul

Paul wrote:

>
> I ran this in Windows 11

Now, before everyone gets on my case about where I ran it,
I needed to be able to see what the users see when they
unpack their 7zip in Windows, and whether the filename
looks as intended.

If I did the test purely in Linux, against an NTFS file
system, who knows whether the text string display would
look just like it does on Windows. I'm not a character
set expert and cannot predict what those look like on
the Linux side. It's unlikely at the moment, that
Linux will even mount that file system (MFTMIRR) :-/ Thanks
to Microsoft. Only Fedora could mount it without whining.

It's hardly easy to do anything in a heterogenous
environment now. Like pulling teeth with dull pliers.

Paul

On 2021-08-15, Martin Gregorie <martin@mydomain.invalid> wrote:
> On Sun, 15 Aug 2021 08:37:45 +0100, Richard Kettlewell wrote:
>
>> Martin Gregorie <martin@mydomain.invalid> writes:
>>> William Unruh wrote:
>>>
>>>> Says someone, apparently, who has never looked at the command "find",
>>>> or may other commands. billions of different option combinations, some
>>>> of which work, others of which do not.
>>>
>>> Quite right: I don't use it because 'locate' is *much* faster and
>>> easier to use, especially if updatedb is run overnight by a cronjob
>>
>> Bad comparison, since it doesn’t do the same thing.
>
> Both look for filenames, but 'find' can be restricted to a directory
> structure - thats about the difference I can see in a quick manpage scan.

find can search accoding to user, group, permission, age, size, type ...
find can launch other commands, and delete files.

locate only looks at filenames.

--
Jasen.

On 15/08/2021 20.33, Paul wrote:

> If I did the test purely in Linux, against an NTFS file
> system, who knows whether the text string display would
> look just like it does on Windows. I'm not a character
> set expert and cannot predict what those look like on
> the Linux side.

As long the systems has the same charset, then there shouldn't be any
differences, this do not just apply to Linux but other operating systems
as microsoft windows.

> It's unlikely at the moment, that
> Linux will even mount that file system (MFTMIRR) :-/ Thanks
> to Microsoft. Only Fedora could mount it without whining.

Much depends on the ntfs module loaded, the current in kernel ntfs
support is crappy and still used by some distributions, but most do have
support for the ntfs-3g driver, just you may install it manually.
The good news is that this driver will be in the kernel in a near future.

Mounting BitLock encrypted file systems can also be done on the Linux,
just in case you need to access files from your work computers harddrive.

Had been nice to see an in kernel exFat support too, but I doubt
microsoft has need of that in their Linux distributions, so I doubt they
will provide a driver.

--

//Aho

Il 15/08/2021 13:57, Java Jive ha scritto:
> On 13/08/2021 20:28, Java Jive wrote:
>> I have the following lines in a shell script ...
>>
>> while [ -n "${LINE}" ]
>>      do
>>          if [ -n "${LINE} ]
>>              then
>>                  # Do processing
>>          fi
>>      done < "${DATA}"
>>
>> .... and this works fine for all but two lines in the data file, which
>> contain accented characters. A file erroneously named with an e acute
>> needs to be renamed to have an e grave, and a filename containing an e
>> umlaut needs to be moved to a new location and given a new name.
>
> Uggghhh!  The reason for this disgust will become clear shortly!
>
> This is a follow up question about character encodings ...
>
> Previously I have released to my family two versions of the same archive
> of family documents going back to the reign of Queen Anne, some items
> possibly a little earlier.  These documents were scanned (1o for
> original scan) and then put through four possible stages of
> post-processing:
>     2n    Contrast 'normalised' using pnnorm
>     3t    Textcleaned
>     4nt    n followed by 3
>     5tn    t followed by n
>
> For each document, the best result was copied into the main archive,
> while the above preprocessing stages were left in an '_all'
> sub-directory structure, with five subdirectories named as above, each
> of which having beneath it a directory tree mirroring the main archive.
>
> The main version of the archive, which most family members seem to have
> downloaded, only included the main archive and didn't include the _all
> subdirectory with all the pre-processing results, the full version
> included this directory.  IIRC, the former was compressed by WinZip from
> the archive as it existed on a Windows PC at the time, but WinZip threw
> a wobbly over the size of the full archive, so for that I had to use 7zip.
>
> Now the crunch, when I unzip these on a Linux machine, I see different
> bastardisations of accented characters.  So, for example where the full
> 7zip archive when extracted shows an e acute correctly in both a console
> and a file manager listing ...
>     "Chat Botté, Le"    [e is correctly acute]
> ... (if you're wondering, a French children's picture book version of
> apparently 'Puss In Boots'), while with the WinZip main archive a
> console listing shows a very odd character sequence instead of the e
> acute ...
>     "Chat Bott'$'\302\202'', Le"
> ... and a file manager listing has a graphic character resembling a 2x2
> matrix, concerning which note that while \302 octal = \xC2 hex,  and
> \202 octal = \x82 hex, only the second of these and not the first
> appears in the symbol:
>     |00|
>     |82|
>
> My problem is that I can't find a search term to trap this strange
> character to correct it, for example the following, and a few similar
> that I've tried, don't work because they don't find the directory:
>     mv "Chat Bott'$'\302\202'', Le"    "Chat Botté, Le"
>     mv Chat\ Bott\'$\'\\302\\202\'\',\ Le "Chat Botté, Le"
>
> I could use a glob wildcard character such as '?', but currently all the
> filenames are within quotes, where globbing doesn't seem to work, and it
> would be a hell of a business removing the quotes, because many names in
> the archive use many characters that would each need to be anticipated
> and escaped for in an unquoted filename, such as spaces, ampersands,
> brackets, etc.
>
> Can anyone suggest a sequence that will find the file, when put inside
> quotes as the filename in the controlling data file mentioned previously
> in the thread, so that it can just be treated like all the other lines?
> As someone here suggested the data file is now stored as UTF-8 rather
> than ANSI as it was formerly, and some example lines are given below in
> a form for easier readability in a ng  -  in reality the fields are tab
> separated but here are separated by double spacing and have been further
> abbreviated to keep them from wrapping; leading symbols such as '+' and
> '=' have special meanings for the program doing the work; and, yes, the
> commands are basically DOS commands which for Linux are translated to
> their bash equivalents:
>
> =ATTRIB -R  "./F H/Close/Sts Mary & John Churchyard Monuments.pdf"
> =RD  "./F H /_all/1o/Blessig & Heyder"
> REN  "./Chat Bott'$'\302\202'', Le"  "Chat Botté, Le"
> MOVE  "./Photo - D & M Close.png" "./Photos/D & M Close.png"
> [etc]
>

Hi,
you could use the find command looking for filenames as a regular
expression, then use the command you need on them.
In this example I search for files with the extension ".o", display the
name with the command 'echo' and display it again converted to
uppercase:

find . -iregex ".*\.o$" -exec bash -c "echo -n original: {} && echo \"
modified: {}\" | tr [a-z] [A-Z]}" \;

There should be everything you need.

cheers

On 15/08/2021 22:27, jak wrote:
>
> Il 15/08/2021 13:57, Java Jive ha scritto:
>>
>> Can anyone suggest a sequence that will find the file, when put inside
>> quotes as the filename in the controlling data file mentioned
>> previously in the thread, so that it can just be treated like all the
>> other lines? As someone here suggested the data file is now stored as
>> UTF-8 rather than ANSI as it was formerly, and some example lines are
>> given below in a form for easier readability in a ng  -  in reality
>> the fields are tab separated but here are separated by double spacing
>> and have been further abbreviated to keep them from wrapping; leading
>> symbols such as '+' and '=' have special meanings for the program
>> doing the work; and, yes, the commands are basically DOS commands
>> which for Linux are translated to their bash equivalents:
>>
>> =ATTRIB -R  "./F H/Close/Sts Mary & John Churchyard Monuments.pdf"
>> =RD  "./F H /_all/1o/Blessig & Heyder"
>> REN  "./Chat Bott'$'\302\202'', Le"  "Chat Botté, Le"
>> MOVE  "./Photo - D & M Close.png" "./Photos/D & M Close.png"
>> [etc]
>>
>
> Hi,
> you could use the find command looking for filenames as a regular
> expression, then use the command you need on them.
> In this example I search for files with the extension ".o", display the
> name with the command 'echo' and display it again converted to
> uppercase:
>
>  find . -iregex ".*\.o$" -exec bash -c "echo -n original: {} && echo \"
>    modified: {}\" | tr [a-z] [A-Z]}" \;
>
> There should be everything you need.

Thanks but no, that doesn't work. I had considered, before the script
works through the data file, of running a pre-process to find and rename
all these characters, but neither find nor ls will actually find the
erroneous characters *DIRECTLY*. The best either can do is find the
characters either side, but that means I have to know in advance where
all the problems are, and I'm not sure yet that I do. Really, if I'm
going to go down that road, I need a way of searching the entire archive
structure directly for affected files and renaming them, as a separate
process from working through the data file.

So, for example, this works because I'm specifying and finding the
neighbouring characters of one known instance, not because ls is finding
the oddball characters directly ...
ls Chat\ Bott?,\ Le | sed 's~\xc2\x82~é~g'
.... whereas these don't, with neither single nor double backslashes nor
various other combinations that I've tried, because neither find nor ls
seem able to find the oddball characters directly:
find . -regex ".*\\xc2\\x82.*"
ls -R *\\xc2\\x82*
ls -R *'$'\\302\\202''*

--

Fake news kills!

I may be contacted via the contact address given on my website:
www.macfh.co.uk

On Mon, 16 Aug 2021 10:23:23 +0100
Java Jive <java@evij.com.invalid> wrote:
> So, for example, this works because I'm specifying and finding the
> neighbouring characters of one known instance, not because ls is finding
> the oddball characters directly ...
> ls Chat\ Bott?,\ Le | sed 's~\xc2\x82~é~g'
> ... whereas these don't, with neither single nor double backslashes nor
> various other combinations that I've tried, because neither find nor ls
> seem able to find the oddball characters directly:
> find . -regex ".*\\xc2\\x82.*"
> ls -R *\\xc2\\x82*
> ls -R *'$'\\302\\202''*

Try ls -R *$'\302\202'*

On 16/08/2021 13:47, Spiros Bousbouras wrote:
> On Mon, 16 Aug 2021 10:23:23 +0100
> Java Jive <java@evij.com.invalid> wrote:
>> So, for example, this works because I'm specifying and finding the
>> neighbouring characters of one known instance, not because ls is finding
>> the oddball characters directly ...
>> ls Chat\ Bott?,\ Le | sed 's~\xc2\x82~é~g'
>> ... whereas these don't, with neither single nor double backslashes nor
>> various other combinations that I've tried, because neither find nor ls
>> seem able to find the oddball characters directly:
>> find . -regex ".*\\xc2\\x82.*"
>> ls -R *\\xc2\\x82*
>> ls -R *'$'\\302\\202''*
>
> Try ls -R *$'\302\202'*

No luck with that either ...
ls: cannot access '*'$'\302\202''*': No such file or directory

I think the trouble with all these methods is that they are specifying a
succession of two characters, where as far as unicode is concerned the
oddballs are single characters, so I fear they will never match, no
matter what magic incantation is used.

So I've been looking at putting in the following as a hack around. It's
designed to search for wildcards in the file name coming from the data
file, and if one is found, escape all the other 'dodgy' characters in it
and use it without quotes, but, although everything *LOOKS* as though it
should work, it gives an error message at the final file testing if
statement:

Before, works except for filenames containing wildcard characters:

if [ -n "${Debug}" ]
then
echo "CE3 = ${CE3}"
fi
if [ "${CE3/./}" != "${CE3}" ]
then
# Is file spec
if [ ! -f "${CE3}" ] # Note quotes
then
if [ -n "${Debug}" ]
then
echo "WARNING - File '${CE3}' does not exist!"
fi
Result=1
CE2=""
fi
else
# Is path spec
if [ ! -d "${CE3}" ] # Note quotes
then
if [ -n "${Debug}" ]
then
echo "WARNING - Directory '${CE3}' does not exist!"
fi
Result=1
CE2=""
fi
fi

After, try to escape wildcard containing filenames:

# Need to remove any enclosing quotes at this stage
while [ "${CE3:0:1}" == "'" ] && [ "${CE3: -1:1}" == "'" ]
do
CE3="${CE3:1:${#CE3}-2}"
done
while [ "${CE3:0:1}" == "\"" ] && [ "${CE3: -1:1}" == "\"" ]
do
CE3="${CE3:1:${#CE3}-2}"
done
# Check for wildcard chars
if [ "${CE3/\?/}" != "${CE3}" ] || [ "${CE3/\*/}" != "${CE3}" ]
then
# Wildcards, cannot quote, so escape difficult characters
CE3=$(echo "${CE3}" | sed "s~\([ #&'(),;-]}\)~\\\\\1~g")
else
# No wildcards, enclose in quotes
CE3="\"${CE3}\""
fi
if [ -n "${Debug}" ]
then
echo "CE3 = ${CE3}"
# Example output here seems correct, for example ...
# CE3 = ... /Newscuttings\ \-\ Wedding\ Of\ Zo?\ <Surname>.png
fi
if [ "${CE3/./}" != "${CE3}" ]
then
# Is file spec
if [ ! -f ${CE3} ] # Note no quotes
# ... but errors here: <scriptname>: line <num>: [: too many arguments
then
if [ -n "${Debug}" ]
then
echo "WARNING - File '${CE3}' does not exist!"
fi
Result=1
CE2=""
fi
else
# Is path spec
if [ ! -d ${CE3} ] # Note no quotes
# ... and here: <scriptname>: line <num>: [: too many arguments
then
if [ -n "${Debug}" ]
then
echo "WARNING - Directory '${CE3}' does not exist!"
fi
Result=1
CE2=""
fi
fi

--

Fake news kills!

I may be contacted via the contact address given on my website:
www.macfh.co.uk

On Mon, 16 Aug 2021 15:33:15 +0100, Java Jive wrote:

> No luck with that either ...
> ls: cannot access '*'$'\302\202''*': No such file or directory
>
Might be worth writing a noddy Java program to see if it can resolve your
problem character codes.

The Java 'char' primitive can hold multibyte character values. and the
Character() class provides methods to recognise character types, lengths,
and non-Unicode characters.

--
--
Martin | martin at
Gregorie | gregorie dot org

On 16/08/2021 16:58, Martin Gregorie wrote:
> On Mon, 16 Aug 2021 15:33:15 +0100, Java Jive wrote:
>
>> No luck with that either ...
>> ls: cannot access '*'$'\302\202''*': No such file or directory
>>
> Might be worth writing a noddy Java program to see if it can resolve your
> problem character codes.
>
> The Java 'char' primitive can hold multibyte character values. and the
> Character() class provides methods to recognise character types, lengths,
> and non-Unicode characters.

But I can't be sure that any of the target machines will have Java,
Perl, or Python installed. This has to be achieved with what will
normally be installed on a Linux or MacOS box.

--

Fake news kills!

I may be contacted via the contact address given on my website:
www.macfh.co.uk

Java Jive wrote:

> console listing shows a very odd character sequence instead of the e
> acute ...
>     "Chat Bott'$'\302\202'', Le"

Are you sure the filename is exactly as you say/think? What does

ls -b

show?