I've been using dd for nearly 30 years and I've tended to think of it as
_the_ way to read from and write to block special device files[*]. This
was an interesting read for me:

Useless Use of dd
https://www.vidarholen.net/contents/blog/?p=479

[*] And for some of the cases, it still is the best tool. For me that's
included
1. Reading partitions off a disk armed with knowing the partition
table but not having the OS recongize the partition table. I
did that for Mac System 7 disks (not floppies) to then use the
partition images with Basilisk II.
2. Rapid wiping of partition table prior to reformatting (eg
dd if=/dev/zero of=/dev/to-wipe bs=1M count=10
)
And non-device file uses:
3. Trivial encryption to bypass email / web upload content
inspection, `conv=swab` which is as secure and easily
reversible as rot13, but is also binary safe.
4. Extracting sections of huge log files for easier inspection
with the iseek option.

Elijah
------
article is not new and might be old hat, but still seemed pertinant

At Wed, 23 Jun 2021 20:24:14 -0000 (UTC) Eli the Bearded <*@eli.users.panix.com> wrote:

>
> I've been using dd for nearly 30 years and I've tended to think of it as
> _the_ way to read from and write to block special device files[*]. This
> was an interesting read for me:
>
> Useless Use of dd
> https://www.vidarholen.net/contents/blog/?p=479
>
> [*] And for some of the cases, it still is the best tool. For me that's
> included
> 1. Reading partitions off a disk armed with knowing the partition
> table but not having the OS recongize the partition table. I
> did that for Mac System 7 disks (not floppies) to then use the
> partition images with Basilisk II.
> 2. Rapid wiping of partition table prior to reformatting (eg
> dd if=/dev/zero of=/dev/to-wipe bs=1M count=10
> )
> And non-device file uses:
> 3. Trivial encryption to bypass email / web upload content
> inspection, `conv=swab` which is as secure and easily
> reversible as rot13, but is also binary safe.
> 4. Extracting sections of huge log files for easier inspection
> with the iseek option.
>
> Elijah
> ------
> article is not new and might be old hat, but still seemed pertinant

And here is another "novel" use of dd (I do this quite often):

tar czf - -C some/directory/path . | ssh someotherhost dd of=some_directory_path.tar.bz

OR

ssh someotherhost dd if=some_directory_path.tar.bz | tar xf - -C someplace/new

>
>

--
Robert Heller -- Cell: 413-658-7953 GV: 978-633-5364
Deepwoods Software -- Custom Software Services
http://www.deepsoft.com/ -- Linux Administration Services
heller@deepsoft.com -- Webhosting Services

Eli the Bearded <*@eli.users.panix.com> writes:
> I've been using dd for nearly 30 years and I've tended to think of it as
> _the_ way to read from and write to block special device files[*]. This
> was an interesting read for me:
>
> Useless Use of dd
> https://www.vidarholen.net/contents/blog/?p=479

It’s been unnecessary at least since 1979, see e.g.
https://www.freebsd.org/cgi/man.cgi?query=hp&apropos=0&sektion=0&manpath=Unix+Seventh+Edition&arch=default&format=html

(also mentions raw devices, in which dd or something else which either
gets the right block size, or can be told to do so, must be used.)

--
https://www.greenend.org.uk/rjk/

Robert Heller <heller@deepsoft.com> writes:
> And here is another "novel" use of dd (I do this quite often):
>
> tar czf - -C some/directory/path . | ssh someotherhost dd
> of=some_directory_path.tar.bz
>
> OR
>
> ssh someotherhost dd if=some_directory_path.tar.bz | tar xf - -C someplace/new

You can use ‘cat’ for both, and part of the point of the article is that
it may be faster to do so.

--
https://www.greenend.org.uk/rjk/

In article <eli$2106231624@qaz.wtf>,
Eli the Bearded <*@eli.users.panix.com> wrote:
>I've been using dd for nearly 30 years and I've tended to think of it as
>_the_ way to read from and write to block special device files[*]. This
>was an interesting read for me:
>
>Useless Use of dd
>https://www.vidarholen.net/contents/blog/?p=479
>
>[*] And for some of the cases, it still is the best tool. For me that's
> included
> 1. Reading partitions off a disk armed with knowing the partition
> table but not having the OS recongize the partition table. I
> did that for Mac System 7 disks (not floppies) to then use the
> partition images with Basilisk II.
> 2. Rapid wiping of partition table prior to reformatting (eg
> dd if=/dev/zero of=/dev/to-wipe bs=1M count=10
> )
> And non-device file uses:
> 3. Trivial encryption to bypass email / web upload content
> inspection, `conv=swab` which is as secure and easily
> reversible as rot13, but is also binary safe.
> 4. Extracting sections of huge log files for easier inspection
> with the iseek option.
>

You must be senile.

>Elijah
>------
>article is not new and might be old hat, but still seemed pertinant

--
Member - Liberal International This is doctor@@nl2k.ab.ca Ici doctor@@nl2k.ab.ca
Yahweh, Queen & country!Never Satan President Republic!Beware AntiChrist rising!
Look at Psalms 14 and 53 on Atheism https://www.empire.kred/ROOTNK?t=94a1f39b
Changing the dictionary does not move us forward. -unknown Beware https://mindspring.com

Robert Heller <hel...@deepsoft.com> [RH]:
RH> tar czf - -C some/directory/path . | ssh someotherhost dd
RH> of=some_directory_path.tar.bz

And if dd is not readily available (e.g. some low end, firmware
storage space deficient, linux based IoT systems and
modems/routers/set-top boxes)

<from >to cp /dev/stdin /dev/stdout

is practically equivalent to

dd if=from of=to

It can be useful for making automated remote backup copies of their
settings and SD card contents. E.g. one can periodically poll
and get any new messages in their voice mailbox, encrypt them and
upload them to the cloud of their choice, without using the dedicated
cloud based service of their "smart" answering machine.

Nomen Nescio <nobody@dizum.com> writes:

> Robert Heller <hel...@deepsoft.com> [RH]:
> RH> tar czf - -C some/directory/path . | ssh someotherhost dd
> RH> of=some_directory_path.tar.bz
>
> And if dd is not readily available (e.g. some low end, firmware
> storage space deficient, linux based IoT systems and
> modems/routers/set-top boxes)
>
> <from >to cp /dev/stdin /dev/stdout

Why would you not just use cat? Is there some kind of contest for
finding the most roundabout way to copy bytes from A to B? If so what is
the prize?

--
https://www.greenend.org.uk/rjk/

At Thu, 24 Jun 2021 16:03:02 +0100 Richard Kettlewell <invalid@invalid.invalid> wrote:

>
> Nomen Nescio <nobody@dizum.com> writes:
>
> > Robert Heller <hel...@deepsoft.com> [RH]:
> > RH> tar czf - -C some/directory/path . | ssh someotherhost dd
> > RH> of=some_directory_path.tar.bz
> >
> > And if dd is not readily available (e.g. some low end, firmware
> > storage space deficient, linux based IoT systems and
> > modems/routers/set-top boxes)
> >
> > <from >to cp /dev/stdin /dev/stdout
>
> Why would you not just use cat? Is there some kind of contest for
> finding the most roundabout way to copy bytes from A to B? If so what is
> the prize?

Cat does not take parameter for its output -- it depends on the shell to
manage that.

This command:

tar czf - -C some/directory/path . | ssh someotherhost cat >some_directory_path.tar.bz

is effectively the same as:

tar czf some_directory_path.tar.bz -C some/directory/path .

except that the tar file makes a *round trip* over the network.

Yes you could do this:

tar czf - -C some/directory/path . | ssh someotherhost sh -c '"cat >some_directory_path.tar.bz"'

But that is really cumbersome and involves quoting hell...

(going the other way does work, but using a symetrial set of commands is neat
and elegant.)

--
Robert Heller -- Cell: 413-658-7953 GV: 978-633-5364
Deepwoods Software -- Custom Software Services
http://www.deepsoft.com/ -- Linux Administration Services
heller@deepsoft.com -- Webhosting Services

In comp.os.linux.misc, Robert Heller <heller@deepsoft.com> wrote:
> Yes you could do this:
>
> tar czf - -C some/directory/path . | ssh someotherhost sh -c '"cat
> >some_directory_path.tar.bz"'
>
> But that is really cumbersome and involves quoting hell...

I do tar over ssh all the time and I believe you are overthinking the
quoting needs.

For moving a directory of text files:

tar cf - directory/ | gzip -3 | ssh somwhere "cd /base && tar xzf -"

(The -3 was at least at one time measured to be a good choice balancing
time spent compressing and time savings in bandwidth.)

For just archiving remotely:

tar cf - directory/ | bzip2 | ssh somwhere "cat > directory.tbz2"

And for extracting from remote:

ssh somewhere "cat directory.tbz2" | tar xaf -

I can see using a 'dd' as a way to specify a filename _IF_ you also
needed to throw a sudo in the archiving remotely case.

Elijah
------
can see, but hasn't ever needed to

At Thu, 24 Jun 2021 17:30:34 -0000 (UTC) Eli the Bearded <*@eli.users.panix.com> wrote:

>
> In comp.os.linux.misc, Robert Heller <heller@deepsoft.com> wrote:
> > Yes you could do this:
> >
> > tar czf - -C some/directory/path . | ssh someotherhost sh -c '"cat
> > >some_directory_path.tar.bz"'
> >
> > But that is really cumbersome and involves quoting hell...
>
> I do tar over ssh all the time and I believe you are overthinking the
> quoting needs.
>
> For moving a directory of text files:
>
> tar cf - directory/ | gzip -3 | ssh somwhere "cd /base && tar xzf -"

For this I just use rsync...

>
> (The -3 was at least at one time measured to be a good choice balancing
> time spent compressing and time savings in bandwidth.)
>
> For just archiving remotely:
>
> tar cf - directory/ | bzip2 | ssh somwhere "cat > directory.tbz2"
>
> And for extracting from remote:
>
> ssh somewhere "cat directory.tbz2" | tar xaf -
>
> I can see using a 'dd' as a way to specify a filename _IF_ you also
> needed to throw a sudo in the archiving remotely case.
>

I just find dd easier than cat, because that way I don't even need to think
about quoting at all.

> Elijah
> ------
> can see, but hasn't ever needed to
>
>

--
Robert Heller -- Cell: 413-658-7953 GV: 978-633-5364
Deepwoods Software -- Custom Software Services
http://www.deepsoft.com/ -- Linux Administration Services
heller@deepsoft.com -- Webhosting Services

Robert Heller <heller@deepsoft.com> writes:
> Yes you could do this:
>
> tar czf - -C some/directory/path . | ssh someotherhost sh -c '"cat
> >some_directory_path.tar.bz"'
>
> But that is really cumbersome and involves quoting hell...

It can be much simpler:

tar -czf -C some/directory/path . | ssh host 'cat > path.tar.bz'

--
https://www.greenend.org.uk/rjk/

In comp.os.linux.misc, Robert Heller <heller@deepsoft.com> wrote:
> Eli the Bearded <*@eli.users.panix.com> wrote:
>> For moving a directory of text files:
>> tar cf - directory/ | gzip -3 | ssh somwhere "cd /base && tar xzf -"
> For this I just use rsync...

I can never remember the exact way to run rsync, so I typically only use
it when I need rsync specific features (like incremental updates to a
remote copy). For tar / cpio and ssh I just know the options from years
of use.

Elijah
------
moves complete things more often than incremental things

At Thu, 24 Jun 2021 22:46:08 -0000 (UTC) Eli the Bearded <*@eli.users.panix.com> wrote:

>
> In comp.os.linux.misc, Robert Heller <heller@deepsoft.com> wrote:
> > Eli the Bearded <*@eli.users.panix.com> wrote:
> >> For moving a directory of text files:
> >> tar cf - directory/ | gzip -3 | ssh somwhere "cd /base && tar xzf -"
> > For this I just use rsync...
>
> I can never remember the exact way to run rsync, so I typically only use
> it when I need rsync specific features (like incremental updates to a
> remote copy). For tar / cpio and ssh I just know the options from years
> of use.

rsync -a sourcedir/ host:destdir/

To make a *new* dir, leave off the trailing slash on sourcedir.

'host:' can be swapped to the source side.

What I have described covers like 90%+ of the time.

>
> Elijah
> ------
> moves complete things more often than incremental things
>
>

--
Robert Heller -- Cell: 413-658-7953 GV: 978-633-5364
Deepwoods Software -- Custom Software Services
http://www.deepsoft.com/ -- Linux Administration Services
heller@deepsoft.com -- Webhosting Services

On 25.06.21 00:46, Eli the Bearded wrote:
> In comp.os.linux.misc, Robert Heller <heller@deepsoft.com> wrote:
>> Eli the Bearded <*@eli.users.panix.com> wrote:
>>> For moving a directory of text files:
>>> tar cf - directory/ | gzip -3 | ssh somwhere "cd /base && tar xzf -"
>> For this I just use rsync...
>
> I can never remember the exact way to run rsync, so I typically only use
> it when I need rsync specific features (like incremental updates to a
> remote copy). For tar / cpio and ssh I just know the options from years
> of use.

I'm the exact other way around: I remember rsync parmeters and not tar
ones. You know why? Because I use it more often. So it all boils down to
which one you prefer (or learned first), you remember that and all
others seem more difficult. Sometimes there is indeed a better way to do
things, but most times it's just a matter of preference, as long as the
task gets done in a reasonable time.

It's still good to share experience and command examples on the group
though.

Andrea

>
> Elijah
> ------
> moves complete things more often than incremental things
>

On Wed, 23 Jun 2021 20:24:14 -0000 (UTC), Eli the Bearded
<*@eli.users.panix.com> wrote:

>I've been using dd for nearly 30 years and I've tended to think of it as
>_the_ way to read from and write to block special device files[*]. This
>was an interesting read for me:
>
>Useless Use of dd
>https://www.vidarholen.net/contents/blog/?p=479
>
>[*] And for some of the cases, it still is the best tool. For me that's
> included
> 1. Reading partitions off a disk armed with knowing the partition
> table but not having the OS recongize the partition table. I
> did that for Mac System 7 disks (not floppies) to then use the
> partition images with Basilisk II.
> 2. Rapid wiping of partition table prior to reformatting (eg
> dd if=/dev/zero of=/dev/to-wipe bs=1M count=10
> )
> And non-device file uses:
> 3. Trivial encryption to bypass email / web upload content
> inspection, `conv=swab` which is as secure and easily
> reversible as rot13, but is also binary safe.
> 4. Extracting sections of huge log files for easier inspection
> with the iseek option.
>
>Elijah
>------
>article is not new and might be old hat, but still seemed pertinant

?Que?

'dd' is good for what 'dd' is good for - no more, no less.

It will NOT fix your mail/file encryption issues. It CAN
"wipe" your old files if you use /dev/urandom or /dev/zero
as the 'source', though not against State-level probers.
Use BleachBit or multiple 'dd' passes for that. Takes a
LONG time. Better yet, disassemble your drives and
physically destroy the pretty silver platters ...... they
do make nice mobiles though ........

E-drives - you have to put a torch, or maybe a stun-gun,
to the things. The leveling algos make them basically
imposssible to erase securely. Buy a surplus neon sign
transformer, put the e-drive between the wires ....

Try to inspect "wiped" drives with 'dd' - output a number
of blocks to a file and LOOK at it with 'ghex' or something.
Anything intelligible left ? Reformatting leaves almost ALL
the old data intact - 'dd' can grab it.

"Encrypted drives" have plusses and minuses. Encryption
makes everything slower, of course. It also makes it harder
to move said drives to other machines in case of emergency.
Finally, encryption is touchy - a single bit error might make
ALL the info un-recoverable. Per-file encryption is safer, but
not as handy.

Toto <Q274@buzz.net> writes:
> It will NOT fix your mail/file encryption issues. It CAN
> "wipe" your old files if you use /dev/urandom or /dev/zero
> as the 'source', though not against State-level probers.
> Use BleachBit or multiple 'dd' passes for that. Takes a
> LONG time. Better yet, disassemble your drives and
> physically destroy the pretty silver platters ...... they
> do make nice mobiles though ........

One the one hand I’m skeptical that anyone can recover zeroized data
from a modern hard disk surface.

On the other hand:

1) You can’t zeroize a drive that has failed or been lost/stolen. (You
can’t physically destroy them in the latter cases, either.)

2) You can only zeroize physical sectors that are still addressable. On
a hard disk that’s most of them but on an SSD or virtual storage it’s
a trickier question.

My personal solution is OS-level encryption of ordinary drives.

> "Encrypted drives" have plusses and minuses. Encryption makes
> everything slower, of course. It also makes it harder to move said
> drives to other machines in case of emergency. Finally, encryption
> is touchy - a single bit error might make ALL the info
> un-recoverable. [...]

Backups.

--
https://www.greenend.org.uk/rjk/

On 27.06.21 11:05, Richard Kettlewell wrote:

> My personal solution is OS-level encryption of ordinary drives.
>
How do you do that? Do you use a software like VeraCrypt or a built in
program in Linux? On a version of Ubuntu I was asked by the installation
if I wanted to encrypt my /home folder, but recently I don't get that
question any more. I guess it's still possible and I could look for a
solution online, but that was specifically for the /home folder. Do you
encrypt the operating system itself too? I remember having read that one
of the possible problems with that is that if you forget the password or
for some reason are unable to unencrypt the folder, than the data there
wouldn't be accessible and one of the thing that was recommended was to
move your ssh keys out of the /home folder, so that you would still have
access to the computer via ssh if something goes wrong. What happens if
you encrypt the whole thing? Any other suggestion to the matter?

On Sun, 27 Jun 2021 13:42:18 -0400, Andrea Croci <andrea.croci@gmx.de> wrote:

> On 27.06.21 11:05, Richard Kettlewell wrote:
>
>> My personal solution is OS-level encryption of ordinary drives.
>>
> How do you do that? Do you use a software like VeraCrypt or a built in
> program in Linux? On a version of Ubuntu I was asked by the installation
> if I wanted to encrypt my /home folder, but recently I don't get that
> question any more. I guess it's still possible and I could look for a
> solution online, but that was specifically for the /home folder. Do you
> encrypt the operating system itself too? I remember having read that one
> of the possible problems with that is that if you forget the password or
> for some reason are unable to unencrypt the folder, than the data there
> wouldn't be accessible and one of the thing that was recommended was to
> move your ssh keys out of the /home folder, so that you would still have
> access to the computer via ssh if something goes wrong. What happens if
> you encrypt the whole thing? Any other suggestion to the matter?

I don't use Ubuntu, but do have some experience with using encrypted file systems.

https://gitlab.com/cryptsetup/cryptsetup/ is what the Mageia installer uses.

Of course you lose access to the data if you forget the passphrase. That's the
whole purpose of using encryption, to ensure a stranger can not access the data.

The cryptsetup program does not store the key on disk. The key is generated using
the encryption passphrase and a hash program. The ssh program has nothing to do with
disk encryption.

The ssh keys used for remote system access, like all important data should be backed
up on external devices, in case of drive failure or device theft.

The installer likely does not show the option to encrypt /home if you're reusing
existing partitions or upgrading, since the partition must be re-formatted when
changing from normal to encrypted.

Regards, Dave Hodgins

--
Change dwhodgins@nomail.afraid.org to davidwhodgins@teksavvy.com for
email replies.

Andrea Croci <andrea.croci@gmx.de> writes:
> On 27.06.21 11:05, Richard Kettlewell wrote:
>
>> My personal solution is OS-level encryption of ordinary drives.
>
> How do you do that? Do you use a software like VeraCrypt or a built in
> program in Linux?

On Linux I use cryptsetup with LUKS.

> On a version of Ubuntu I was asked by the installation if I wanted to
> encrypt my /home folder, but recently I don't get that question any
> more. I guess it's still possible and I could look for a solution
> online, but that was specifically for the /home folder. Do you encrypt
> the operating system itself too?

At present I only encrypt backup drives and (non-Linux) laptops, which
are the things most likely to go missing. The rest can wait for another
day...

> I remember having read that one of the possible problems with that is
> that if you forget the password or for some reason are unable to
> unencrypt the folder, than the data there wouldn't be accessible and
> one of the thing that was recommended was to move your ssh keys out of
> the /home folder, so that you would still have access to the computer
> via ssh if something goes wrong. What happens if you encrypt the whole
> thing? Any other suggestion to the matter?

Remote access won’t help if you’ve forgotten the password.

--
https://www.greenend.org.uk/rjk/