How do I best (and fastest) format a USB stick with a format
that will allow copying files of any size onto it?
Preferably using a typed in command from the console.
I run Kubuntu 23.04.

--
Dieter Britz

Am 16.08.2023 um 14:02:32 Uhr schrieb db:

> How do I best (and fastest) format a USB stick with a format
> that will allow copying files of any size onto it?
> Preferably using a typed in command from the console.
> I run Kubuntu 23.04.

fsck.<filesystem> /dev/sdX

Be aware that formatting doesn't delete all blocks on the device, some
data will be recoverable.

On 16/08/2023 14:03, Marco Moock wrote:

> Am 16.08.2023 um 14:02:32 Uhr schrieb db:
>
>> How do I best (and fastest) format a USB stick with a format
>> that will allow copying files of any size onto it?
>> Preferably using a typed in command from the console.
>> I run Kubuntu 23.04.
>
> fsck.<filesystem> /dev/sdX

perhaps you mean mkfs.<filesystem> /dev/sdX

> Be aware that formatting doesn't delete all blocks on the device, some
> data will be recoverable.

do you need portability to multiple PCs? if so maybe exfat would be a
suitable filesystem ...

Am 16.08.2023 um 14:10:40 Uhr schrieb Andy Burns:

> On 16/08/2023 14:03, Marco Moock wrote:
>
> > Am 16.08.2023 um 14:02:32 Uhr schrieb db:
> >
> >> How do I best (and fastest) format a USB stick with a format
> >> that will allow copying files of any size onto it?
> >> Preferably using a typed in command from the console.
> >> I run Kubuntu 23.04.
> >
> > fsck.<filesystem> /dev/sdX
>
> perhaps you mean mkfs.<filesystem> /dev/sdX

True, I was too fast and fsck uses the same dotted command, but does
file system checks.

> > Be aware that formatting doesn't delete all blocks on the device,
> > some data will be recoverable.
>
> do you need portability to multiple PCs? if so maybe exfat would be a
> suitable filesystem ...

exfat, ntfs and fat32 are very common and readable on most machines.

On 16/08/2023 14:03, Marco Moock wrote:
> Am 16.08.2023 um 14:02:32 Uhr schrieb db:
>
>> How do I best (and fastest) format a USB stick with a format
>> that will allow copying files of any size onto it?
>> Preferably using a typed in command from the console.
>> I run Kubuntu 23.04.
>
> fsck.<filesystem> /dev/sdX
>
> Be aware that formatting doesn't delete all blocks on the device, some
> data will be recoverable.
>

If you have a NSA style hardware lab, yes.
On and SSD with wear levelling the chances of being able to recover
anything meaningful using just the USB or SATA ports is fairly
negligible, even after a few hours

--
Climate Change: Socialism wearing a lab coat.

On 16/08/2023 14:10, Andy Burns wrote:
> On 16/08/2023 14:03, Marco Moock wrote:
>
>> Am 16.08.2023 um 14:02:32 Uhr schrieb db:
>>
>>> How do I best (and fastest) format a USB stick with a format
>>> that will allow copying files of any size onto it?

>
> do you need portability to multiple PCs? if so maybe exfat would be a
> suitable filesystem ...
>
>
Ext4 for linux, exFAT for old fashioned winshit

--
Climate Change: Socialism wearing a lab coat.

The Natural Philosopher wrote:

> Marco Moock wrote:
>
>> Be aware that formatting doesn't delete all blocks on the device, some
>> data will be recoverable.
>
> If you have a NSA style hardware lab, yes.
> On and SSD with wear levelling the chances of being able to recover
> anything meaningful using just the USB  or SATA ports is fairly
> negligible, even after a few hours

Don't think I've ever tried PhotoRec on a flash drive, but I've used it
to save a few people's bacon on dying (unbacked-up) hard drives ...

On 16/08/2023 18:57, Andy Burns wrote:
> The Natural Philosopher wrote:
>
>> Marco Moock wrote:
>>
>>> Be aware that formatting doesn't delete all blocks on the device, some
>>> data will be recoverable.
>>
>> If you have a NSA style hardware lab, yes.
>> On and SSD with wear levelling the chances of being able to recover
>> anything meaningful using just the USB  or SATA ports is fairly
>> negligible, even after a few hours
>
> Don't think I've ever tried PhotoRec on a flash drive, but I've used it
> to save a few people's bacon on dying (unbacked-up) hard drives ...
>
Those have a 1:1 relationship between logical and physical sectors.
SSD drives do not. And even writing just one bit means erasing a huge block.

Also, under use, TRIM operations will typically result in unused blocks
being erased.

--
For every complex problem there is an answer that is clear, simple, and
wrong.

H.L.Mencken

On 2023-08-16 19:20, The Natural Philosopher wrote:
> On 16/08/2023 14:10, Andy Burns wrote:
>> On 16/08/2023 14:03, Marco Moock wrote:
>>
>>> Am 16.08.2023 um 14:02:32 Uhr schrieb db:
>>>
>>>> How do I best (and fastest) format a USB stick with a format
>>>> that will allow copying files of any size onto it?
>
>>
>> do you need portability to multiple PCs? if so maybe exfat would be a
>> suitable filesystem ...
>>
>>
> Ext4 for linux, exFAT for old fashioned winshit

Remember cameras and TV sets.

Another issue is durability of the stick: they are mostly designed for
FAT, the area where the FAT is, withstands more writes.

For ext4, you should disable the journal.

--
Cheers,
Carlos E.R.

On 8/16/23 8:02 AM, db wrote:
> How do I best (and fastest) format a USB stick with a format
> that will allow copying files of any size onto it?
> Preferably using a typed in command from the console.
> I run Kubuntu 23.04.

Use 'sudo gparted", find the stick, give it a new
partition table using the menu item - MSDOS is
probably good enough for any USB stick but GPT
might serve a tad better. Then create a new
partition in FAT32, or maybe NTFS if you expect
files over 4gb in size.

Yes, an ext3/ext4 partition will also deal with
very large files, but it's less "compatible",
you are basically restricted to the Linux universe.
Winders can deal with FAT32 or NTFS.

540k.1055 wrote:

>   Winders can deal with FAT32 or NTFS.

FAT32 can deal with large devices, but tops out at 4GB files.

The Natural Philosopher <tnp@invalid.invalid> writes:
> On 16/08/2023 14:03, Marco Moock wrote:
>> Am 16.08.2023 um 14:02:32 Uhr schrieb db:
>>
>>> How do I best (and fastest) format a USB stick with a format
>>> that will allow copying files of any size onto it?
>>> Preferably using a typed in command from the console.
>>> I run Kubuntu 23.04.
>> fsck.<filesystem> /dev/sdX

(Should be mkfs.)

>> Be aware that formatting doesn't delete all blocks on the device,
>> some
>> data will be recoverable.
>
> If you have a NSA style hardware lab, yes.
> On and SSD with wear levelling the chances of being able to recover
> anything meaningful using just the USB or SATA ports is fairly
> negligible, even after a few hours

OP didn’t seem to be asking about confidentiality, but:

It depends. mkfs does two things:

1) In some cases, it uses TRIM to set the entire partition to 0. For
instance mkfs.ext4 does this by default. Other variants might or
might not do the same.

If it does attempt this, _and_ if the device supports TRIM, host
reads of every block (except those written in stage 2) will
subsequently return all-bytes-0. So, no way to recovery anything that
was previously in the partition via normal IO operations.

But if those two condition are not met, you’ll be able to read back
most of the old data in the partition with ease, no hardware lab
required.

2) It will write a relatively small nunber of blocks to create an empty
filesystem on the partition. This will overwrite old data from the
point of view of normal IO operations, but leave most of the
partition unaffected.

The OP is using a USB stick; there’s a good chance it doesn’t support
TRIM.

For a device that does support TRIM (e.g. SSD, some modern hard drives),
that’s not the end of the story. TRIM doesn’t zero out the underlying
physical storage; it just updates the logical-to-physical mapping in the
device’s onboard controller.

If the controller is a separate component from the physical storage
(e.g. an SMR hard disk) then presumably that’s relatively easy to
bypass, for someone with the right skills.

If they are the same component then that’s rather harder. I assume you’d
look for a debug interface before you started decapping chips.

Anyway, for anyone who really cares about confidentiality, trying to
figure out how to securely erase the device is the wrong answer. Instead
the solution is to encrypt anything confidential on it from the
start. This is particularly important for USB sticks and external drives
since they are relatively easy to lose.

--
https://www.greenend.org.uk/rjk/

On 17/08/2023 08:19, Richard Kettlewell wrote:
> 2) It will write a relatively small number of blocks to create an empty
> filesystem on the partition. This will overwrite old data from the
> point of view of normal IO operations, but leave most of the
> partition unaffected.

Ah, but will it?

If the disk now 'knows' that e.g. block X has no valid data on it, will
it actually return it's contents if asked?

I.e. if the consider that the disk interface is in terms of logical
sectors that have a random and unknown relationship with actual flash
blocks, that is controlled by the disk internal smarts, than if a read
to a sector the drive *has not actually mapped at all* is requested,
what happens? The data may be in a block, but if the block is marked
'empty' or 'not been written' by the disk firmware..

I think you are assuming more about how the drive responds to formatting
than is necessarily true.

--
Of what good are dead warriors? … Warriors are those who desire battle
more than peace. Those who seek battle despite peace. Those who thump
their spears on the ground and talk of honor. Those who leap high the
battle dance and dream of glory … The good of dead warriors, Mother, is
that they are dead.
Sheri S Tepper: The Awakeners.

The Natural Philosopher <tnp@invalid.invalid> writes:
> I think you are assuming more about how the drive responds to
> formatting than is necessarily true.

No, I’m not. ‘Formatting’ in this case just means mkfs. Apart from the
possible use of TRIM, it’s all normal IO.

--
https://www.greenend.org.uk/rjk/

On 8/17/23 3:11 AM, Andy Burns wrote:
> 540k.1055 wrote:
>
>>    Winders can deal with FAT32 or NTFS.
>
> FAT32 can deal with large devices, but tops out at 4GB files.

I thought I noted that. Not 100% sure what kind of
files the OP has in mind. 4gb might be way beyond
the idea of "big" - or not.

The best general rule is to stick to FAT32 with USB
sticks unless it's just impossible. Alas writing 4+gb
files to a stick, even with USB3+, is NOT very quick.
Sticks are also rather untrustworthy. Might last many
years, might last a few hours .....

On 17/08/2023 18:34, Richard Kettlewell wrote:
> The Natural Philosopher <tnp@invalid.invalid> writes:
>> I think you are assuming more about how the drive responds to
>> formatting than is necessarily true.
>
> No, I’m not. ‘Formatting’ in this case just means mkfs. Apart from the
> possible use of TRIM, it’s all normal IO.
>

But how does the drive respond to 'normal IO?'

--
No Apple devices were knowingly used in the preparation of this post.

db <dieterhansbritz@gmail.com> wrote:
> How do I best (and fastest) format a USB stick with a format
> that will allow copying files of any size onto it?

File of any size ?? You will be limited by disk space.
But if you mean files larger than 4G, read on.
This assumes you do not want an encrypted file system.

> Preferably using a typed in command from the console.
> I run Kubuntu 23.04.

??? Is the device
Be VERY CAREFUL you do not destroy
your hard disk. Login as root.

Plugin the flash drive dmesg(1) should should print
the device as /dev/???.

fdisk /dev/???
You should create 1 partition and set it to type 83
remove whatever partition happens to be there.

mkfs.ext4 -O ^has_journal /dev/???x
'x' is probably the number 1, but that depends upon
what you did using fdisk. Since a usb flash drive,
no need for a journal

I chose ext4, but this will will be only readable on Linux.
ext4 will allow files larger than 4 gig. But any Linux
File System should do.

If you want a Windows type FS, I do not know of any that
will allow file sizes larger than 4 gig. But maybe they
exist now ?

You are done.
HTH

--
[t]csh(1) - "An elegant shell, for a more... civilized age."
- Paraphrasing Star Wars

The Natural Philosopher <tnp@invalid.invalid> writes:
> On 17/08/2023 18:34, Richard Kettlewell wrote:
>> The Natural Philosopher <tnp@invalid.invalid> writes:
>>> I think you are assuming more about how the drive responds to
>>> formatting than is necessarily true.
>> No, I’m not. ‘Formatting’ in this case just means mkfs. Apart from
>> the possible use of TRIM, it’s all normal IO.
>
> But how does the drive respond to 'normal IO?'

You write a block to it, when you read that block back you get what you
wrote. It wouldn’t be very useful as a storage device if it did anything
else.

--
https://www.greenend.org.uk/rjk/

On 18/08/2023 17:04, Richard Kettlewell wrote:
> The Natural Philosopher <tnp@invalid.invalid> writes:
>> On 17/08/2023 18:34, Richard Kettlewell wrote:
>>> The Natural Philosopher <tnp@invalid.invalid> writes:
>>>> I think you are assuming more about how the drive responds to
>>>> formatting than is necessarily true.
>>> No, I’m not. ‘Formatting’ in this case just means mkfs. Apart from
>>> the possible use of TRIM, it’s all normal IO.
>>
>> But how does the drive respond to 'normal IO?'
>
> You write a block to it, when you read that block back you get what you
> wrote. It wouldn’t be very useful as a storage device if it did anything
> else.
>
Yes, but what does it do to blocks that it has *not* written to that the
format process has marked as irrelevant?

The question is whether writing and indeed reading a flash device will
alter data that has been written to *other* logical blocks that are not
marked as 'in use'

--
"Anyone who believes that the laws of physics are mere social
conventions is invited to try transgressing those conventions from the
windows of my apartment. (I live on the twenty-first floor.) "

Alan Sokal

On Fri, 18 Aug 2023 11:46:11 -0400, John McCue <jmccue@magnetar.jmcunx.com> wrote:
<snip>
> I chose ext4, but this will will be only readable on Linux.
> ext4 will allow files larger than 4 gig. But any Linux
> File System should do.
>
> If you want a Windows type FS, I do not know of any that
> will allow file sizes larger than 4 gig. But maybe they
> exist now ?

From https://en.wikipedia.org/wiki/ExFAT
Max volume size 128 PB, 512 TB recommended
Max file size 128 PB

Regards Dave Hodgins

David W. Hodgins <dwhodgins@nomail.afraid.org> wrote:
> On Fri, 18 Aug 2023 11:46:11 -0400, John McCue <jmccue@magnetar.jmcunx.com> wrote:
> <snip>
>> I chose ext4, but this will will be only readable on Linux.
>> ext4 will allow files larger than 4 gig. But any Linux
>> File System should do.
>>
>> If you want a Windows type FS, I do not know of any that
>> will allow file sizes larger than 4 gig. But maybe they
>> exist now ?
>
> From https://en.wikipedia.org/wiki/ExFAT
> Max volume size 128 PB, 512 TB recommended
> Max file size 128 PB

UDF on a USB stick also works with Window and Linux machines (I have no
knowledge regarding it working with Mac's) , and the size limits of UDF
are:

Max volume size 2 TiB (with 512-byte sectors), 8 TiB (with 2 KiB
sectors, like most optical discs), 16 TiB (with 4 KiB sectors)

Max file size 16 EiB

(from https://en.wikipedia.org/wiki/Universal_Disk_Format)

And looking at those numbers above, it appears to allow a maximum file
size that is larger than its max volume size (or Wikipedia has an
error).

Rich <rich@example.invalid> wrote:
>
> UDF on a USB stick also works with Window and Linux machines (I have no
> knowledge regarding it working with Mac's)

Interesting, I hadn't seen that in a non CD/DVD content. It seems
you use "mkudffs" from udftools to create a UDF filesystem in
Linux (it took me a little while to work out that the command isn't
in mkfs.<filesystem> form).

--
__ __
#_ < |\| |< _#

On Fri, 18 Aug 2023 15:37:09 -0400, Rich <rich@example.invalid> wrote:

> David W. Hodgins <dwhodgins@nomail.afraid.org> wrote:
>> On Fri, 18 Aug 2023 11:46:11 -0400, John McCue <jmccue@magnetar.jmcunx.com> wrote:
>> <snip>
>>> I chose ext4, but this will will be only readable on Linux.
>>> ext4 will allow files larger than 4 gig. But any Linux
>>> File System should do.
>>>
>>> If you want a Windows type FS, I do not know of any that
>>> will allow file sizes larger than 4 gig. But maybe they
>>> exist now ?
>>
>> From https://en.wikipedia.org/wiki/ExFAT
>> Max volume size 128 PB, 512 TB recommended
>> Max file size 128 PB
>
> UDF on a USB stick also works with Window and Linux machines (I have no
> knowledge regarding it working with Mac's) , and the size limits of UDF
> are:
>
> Max volume size 2 TiB (with 512-byte sectors), 8 TiB (with 2 KiB
> sectors, like most optical discs), 16 TiB (with 4 KiB sectors)
>
> Max file size 16 EiB
>
> (from https://en.wikipedia.org/wiki/Universal_Disk_Format)
>
> And looking at those numbers above, it appears to allow a maximum file
> size that is larger than its max volume size (or Wikipedia has an
> error).

I read the above as meaning a max file size of 128 PB or the size of the device,
whichever is smaller.

Regards, Dave Hodgins

The Natural Philosopher <tnp@invalid.invalid> writes:
> On 18/08/2023 17:04, Richard Kettlewell wrote:
>> The Natural Philosopher <tnp@invalid.invalid> writes:
>>> On 17/08/2023 18:34, Richard Kettlewell wrote:
>>>> The Natural Philosopher <tnp@invalid.invalid> writes:
>>>>> I think you are assuming more about how the drive responds to
>>>>> formatting than is necessarily true.
>>>> No, I’m not. ‘Formatting’ in this case just means mkfs. Apart from
>>>> the possible use of TRIM, it’s all normal IO.
>>>
>>> But how does the drive respond to 'normal IO?'
>> You write a block to it, when you read that block back you get what
>> you wrote. It wouldn’t be very useful as a storage device if it did
>> anything else.
>
> Yes, but what does it do to blocks that it has *not* written to that
> the format process has marked as irrelevant?

In the case where TRIM isn’t used or isn’t supported by the device, it
does nothing to them. mkfs doesn’t write to them so they do not change.

--
https://www.greenend.org.uk/rjk/

On 2023-08-18 13:14, The Natural Philosopher wrote:
> On 18/08/2023 17:04, Richard Kettlewell wrote:
>> The Natural Philosopher <tnp@invalid.invalid> writes:
>>> On 17/08/2023 18:34, Richard Kettlewell wrote:
>>>> The Natural Philosopher <tnp@invalid.invalid> writes:
>>>>> I think you are assuming more about how the drive responds to
>>>>> formatting than is necessarily true.
>>>> No, I’m not. ‘Formatting’ in this case just means mkfs. Apart from
>>>> the possible use of TRIM, it’s all normal IO.
>>>
>>> But how does the drive respond to 'normal IO?'
>>
>> You write a block to it, when you read that block back you get what you
>> wrote. It wouldn’t be very useful as a storage device if it did anything
>> else.
>>
> Yes, but what does it do to blocks that it has *not* written to that the
> format process has marked as irrelevant?
>
> The question is whether writing and indeed reading a flash device will
> alter data that has been written to *other* logical blocks that are not
> marked as 'in use'

For example, if i/o operations happens in blocks of 32K (arbitrary
number just for this example), and blocks are formatted in chunks of 512
bytes, the operating systems reads the 32K and gives the program just
the 512 bytes. The rest should be cached by the kernel. And when writing
the 512 bytes, the kernel actually writes the 32K.

--
Cheers,
Carlos E.R.