I am planning to use a USB stick for some incremental backups.
The stick will be written to once a day only, and probably not much in
the way of new material will be added (or old material will be removed)
each time.

What is the filesystem recommended for this kind of activity on
this kind of hardware? USB sticks usually come with a VFAT filesystem,
but this filesystem has limitations that will probably make it unsuitable
to my purposes - I have quite large files, with long names that contain
unusual characters.

At Mon, 26 Jul 2021 00:11:51 -0000 (UTC) John Smith <12345@whatismyemailaddress.xyz> wrote:

>
> I am planning to use a USB stick for some incremental backups.
> The stick will be written to once a day only, and probably not much in
> the way of new material will be added (or old material will be removed)
> each time.
>
> What is the filesystem recommended for this kind of activity on
> this kind of hardware? USB sticks usually come with a VFAT filesystem,
> but this filesystem has limitations that will probably make it unsuitable
> to my purposes - I have quite large files, with long names that contain
> unusual characters.

If the stick is never going to used on a MS-Windows system, you can reformat
the stick with a Linux file system, such as ext4.

>
>

--
Robert Heller -- Cell: 413-658-7953 GV: 978-633-5364
Deepwoods Software -- Custom Software Services
http://www.deepsoft.com/ -- Linux Administration Services
heller@deepsoft.com -- Webhosting Services

Le 26/07/2021 à 02:11, John Smith a écrit :
> I am planning to use a USB stick for some incremental backups.
> The stick will be written to once a day only, and probably not much in
> the way of new material will be added (or old material will be removed)
> each time.
>
> What is the filesystem recommended for this kind of activity on
> this kind of hardware?

F2FS was designed for "dumb" flash media such as USB sticks and SD
cards. However I don't know if it is mature and reliable enough for
backup purposes.

On 26/07/2021 02.55, Robert Heller wrote:
> At Mon, 26 Jul 2021 00:11:51 -0000 (UTC) John Smith <12345@whatismyemailaddress.xyz> wrote:
>
>>
>> I am planning to use a USB stick for some incremental backups.
>> The stick will be written to once a day only, and probably not much in
>> the way of new material will be added (or old material will be removed)
>> each time.
>>
>> What is the filesystem recommended for this kind of activity on
>> this kind of hardware? USB sticks usually come with a VFAT filesystem,
>> but this filesystem has limitations that will probably make it unsuitable
>> to my purposes - I have quite large files, with long names that contain
>> unusual characters.
>
> If the stick is never going to used on a MS-Windows system, you can reformat
> the stick with a Linux file system, such as ext4.

But make sure it doesn't have a journal.

--
Cheers,
Carlos E.R.

On 26/07/2021 02.11, John Smith wrote:
> I am planning to use a USB stick for some incremental backups.
> The stick will be written to once a day only, and probably not much in
> the way of new material will be added (or old material will be removed)
> each time.
>
> What is the filesystem recommended for this kind of activity on
> this kind of hardware? USB sticks usually come with a VFAT filesystem,
> but this filesystem has limitations that will probably make it unsuitable
> to my purposes - I have quite large files, with long names that contain
> unusual characters.

Depends.

Problem is, USB have the hardware optimized for FAT. Ie, they know that
this area will be written more often and is structured in the hardware
accordingly.

I don't remember how to find a reference for this, but I had it. I have
it on another computer, I think.

For frequent backups I would not use and USB stick, but a proper SSD
disk, formatted in any Linux filesystem you like. For instance, you
could use btrfs with compression.

If Linux attributes are not an issue, you could use exFAT.

Not NTFS, it is CPU intensive in Linux, and problematic to repair.

Don't use a journal. For ext4, create it like this:

mke2fs -t ext4 -O ^has_journal -L SomeName /dev/sdXY

Don't use ext2, has no extents.

--
Cheers,
Carlos E.R.

On 26/07/2021 10.30, Pascal Hambourg wrote:
> Le 26/07/2021 à 02:11, John Smith a écrit :
>>     I am planning to use a USB stick for some incremental backups.
>> The stick will be written to once a day only, and probably not much in
>> the way of new material will be added (or old material will be removed)
>> each time.
>>
>>     What is the filesystem recommended for this kind of activity on
>> this kind of hardware?
>
> F2FS was designed for "dumb" flash media such as USB sticks and SD
> cards. However I don't know if it is mature and reliable enough for
> backup purposes.

Problem is, there are many versions of it and the kernel can not know
which. For that reason, openSUSE decided to partially remove support for it.

--
Cheers,
Carlos E.R.

John Smith <12345@whatismyemailaddress.xyz> writes:

> What is the filesystem recommended for this kind of activity on
> this kind of hardware? USB sticks usually come with a VFAT filesystem,
> but this filesystem has limitations that will probably make it unsuitable
> to my purposes - I have quite large files, with long names that contain
> unusual characters.

Then maybe use the same file system you use for storing your files? So
no issue with names getting messed up.

Also consider USB sticks aren't very reliable. Better than no backup
though.

Anssi Saari <as@sci.fi> writes:

> John Smith <12345@whatismyemailaddress.xyz> writes:
>
>> What is the filesystem recommended for this kind of activity on
>> this kind of hardware? USB sticks usually come with a VFAT filesystem,
>> but this filesystem has limitations that will probably make it unsuitable
>> to my purposes - I have quite large files, with long names that contain
>> unusual characters.
>
> Then maybe use the same file system you use for storing your files? So
> no issue with names getting messed up.
>
> Also consider USB sticks aren't very reliable. Better than no backup
> though.

I've been doing daily backups to some USB sticks for at least a year.
They're reliable enough.

--
Dan Espen

On 7/26/21 3:36 AM, Carlos E. R. wrote:
> Problem is, USB have the hardware optimized for FAT.

O.o

> I don't remember how to find a reference for this, but I had it. I
> have it on another computer, I think.

I'd be interested in seeing such references.

I also question how much optimization there is, and if the lack of it's
use actually effects the performance enough to render the drive
effectively useless.

It seems rather silly for manufacturers to create a drive that
effectively doesn't work with other file systems.

> For frequent backups I would not use and USB stick,....

Why not?

> Don't use a journal. For ext4, create it like this:

What's wrong with the journal?

I assume that the file system will be unmounted properly, particularly
for something for backups which ideally has as pristine a file system as
possible.

> Don't use ext2, has no extents.

Please elaborate why you say that version 2 of the extent file system
doesn't have any extents.

--
Grant. . . .
unix || die

Grant Taylor <gtaylor@tnetconsulting.net> writes:
> On 7/26/21 3:36 AM, Carlos E. R. wrote:
>> Problem is, USB have the hardware optimized for FAT.
>
> O.o
>
>> I don't remember how to find a reference for this, but I had it. I
>> have it on another computer, I think.
>
> I'd be interested in seeing such references.
>
> I also question how much optimization there is, and if the lack of
> it's use actually effects the performance enough to render the drive
> effectively useless.
>
> It seems rather silly for manufacturers to create a drive that
> effectively doesn't work with other file systems.

I’m skeptical too. Directly optimizing using knowledge of a particular
filesystem layout doesn’t seem likely to be a good use of effort.

I could believe that a (non-bargain-basement) flash controller would be
tuned based on access patterns seen in simulated workloads and that
Microsoft’s implementation of FAT featured in those workloads. That’s
unlikely to cause any real-life problems with other filesystems though.

>> For frequent backups I would not use and USB stick,....
>
> Why not?
>
>> Don't use a journal. For ext4, create it like this:
>
> What's wrong with the journal?
>
> I assume that the file system will be unmounted properly, particularly
> for something for backups which ideally has as pristine a file system
> as possible.

I assume something based on fear of flash wear, which is often
exaggerated on Usenet. Could still be a real issue on a very cheap
device, perhaps?

>> Don't use ext2, has no extents.
>
> Please elaborate why you say that version 2 of the extent file system
> doesn't have any extents.

The ‘ext’ is ‘extended’, not ‘extent’. I’ve don’t know (or care) what an
extent is in fs terms but they seem to be mentioned in fs/ext4 but not
fs/ext2.

--
https://www.greenend.org.uk/rjk/

Grant Taylor <gtaylor@tnetconsulting.net> wrote:
> On 7/26/21 3:36 AM, Carlos E. R. wrote:
>> Problem is, USB have the hardware optimized for FAT.
>
> O.o
>
>> I don't remember how to find a reference for this, but I had it. I
>> have it on another computer, I think.
>
> I'd be interested in seeing such references.
>
> I also question how much optimization there is, and if the lack of it's
> use actually effects the performance enough to render the drive
> effectively useless.
>
> It seems rather silly for manufacturers to create a drive that
> effectively doesn't work with other file systems.

Under an /assumption/ that USB thumb drives contain flash translation
layers (FTLs) [1] one possibility would be that the controller reserves
a large block of spare flash cells to use to remap the FAT table
sectors, and a much smaller block to use for the remainder of the
drive. This would be under the assumption of it being used as a FAT
stick, and so the first X blocks (the FAT table) of the disk receiving
more write traffic than the remainder of the flash cells.

>> Don't use a journal. For ext4, create it like this:
>
> What's wrong with the journal?

A lot of write traffic to the same set of disk sectors. And if the USB
stick does not contain a FTL, the flash cells underlying the journal
wear out much more quickly than the rest of the flash cells that do not
hold the journal.

However, if the stick does contain a FTL, then the only issue is that a
journal increases write traffic (once for journal write, a second write
for the final location) which would wear the flash cells out slightly
faster than without a journal (but at least with an FTL, such wear out
would occur more evenly across the flash cells).

>> Don't use ext2, has no extents.
>
> Please elaborate why you say that version 2 of the extent file system
> doesn't have any extents.

The "e" in ext2 does not stand for "extent". It stands for "extended"
as in "second extended filesystem"
<https://en.wikipedia.org/wiki/Ext2>.

Extents <https://en.wikipedia.org/wiki/Extent_(file_systems)> are
different and the 'ext[234]' series did not add extents until ext4.
As to why the GP believes that lack of extents in ext2 makes it bad for
a USB thumb drive, we have to ask the GP about their reasons why.

[1] the controller on SSD's that sits between the I/O interface to the
computer and the actual flash chips and that remaps flash sectors
around to cover for bad flash sectors.

On 26/07/2021 21.10, Grant Taylor wrote:
> On 7/26/21 3:36 AM, Carlos E. R. wrote:
>> Problem is, USB have the hardware optimized for FAT.
>
> O.o
>
>> I don't remember how to find a reference for this, but I had it. I
>> have it on another computer, I think.
>
> I'd be interested in seeing such references.

Remind me in a month or earlier if you notice I mention that I'm back
home ;-)

I googled "usb sticks are optimized for fat", no luck. Nor "flash media
is optimized for fat"

....

Wait, I found some reference in my interesting mail archive. The web
page has been moved or removed, but I copied the interesting part:

Subject: Re: [opensuse] flash media optimization for FAT filesystem
To: oS-en <opensuse@opensuse.org>
From: "Carlos E. R." <>
Date: Wed, 4 Dec 2019 21:28:20 +0100
....

<https://wiki.linaro.org/WorkingGroups/KernelArchived/Projects/FlashCardSurvey?action=show&redirect=WorkingGroups%2FKernel%2FProjects%2FFlashCa
rdSurvey>
LINARO: Flash memory card design

+++·······························

FAT optimization
==================

Most portable flash media come preformatted as with a FAT32 file system.
This is not only done because there is support for this file system in
all operating systems, it is actually a reasonably good choice for the
media: The data on a FAT32 file system is always written in clusters of
e.g. 32 KB, and the media are normally formatted with a cluster size
matching the optimum write size, as well as aligning the clusters to the
start of internal units, and the access patterns on a FAT32 file system
are relatively predictable, alternating between data blocks, file
allocation table (FAT) and directories.

The cards take advantage of this knowledge by optimizing for the access
patterns that are observed on FAT32, which unfortunately can lead to
worst-case access patterns when using ext3 or other Linux file systems.
In particular, the following (mis-)optimizations are commonly seen on
flash media:

* Most allocation groups are in linear write mode by default, only
the first one or two allocation groups allow efficient write patterns in
smaller units at all times. Since the FAT is known to be in the
beginning of the partition, the controller only needs to allow writing
small updates there, while it can expect other parts of the medium to be
used for large image or video files.

* SDHC cards in particular rely on a specific partition table layout
that guarantees the start of the partition to be aligned to a full
allocation group (typically 4 MB), so that the FAT actually ends up in
the location that is optimized for it. Repartitioning the device with
fdisk usually moves the start of each partition to a cylinder boundary
in C/H/S addressing. Due to legacy reasons and backward-compatibility
with MS-DOS, the cylinder usually has 255 heads and 63 sectors of 512
bytes, which puts the start of the first partition just behind the
optimum area. To make matters much much worse, the alignment of the
partition then becomes just 512 bytes instead of 4 MB, which gives
worst-case behavior when the file system attempts to do aligned writes
to the partition.

* Only a small number of allocation groups is kept open at a time, on
many SD cards only a single one, the largest observed number of open
erase blocks was ten. Writing data to another allocation unit while
having multiple units open causes the least recently used one to go
through garbage collection. In the worst case, this can lead to the card
writing a full allocation unit of multiple megabytes for each 512 byte
block that gets written by the file system. All authentic Sandisk cards
tested so far can write to six allocation units, and keep the most
commonly written ones open, while most cheap cards have a smaller number
and also use a simple one-stage least-recently-used algorithm for
deciding with AU to clean up.

* The smallest write unit is significantly larger than a page.
Reading or writing less than one of these units causes a full unit to be
accessed. Trying to do streaming write in smaller units causes the
medium to do multiple read-modify-write cycles on the same write unit,
which in turn causes multiple garbage collection cycles for writing a
single allocation group from start to end. Small SD (non-SDHC) and MMC
cards, as well as most USB sticks of any size use the page size as write
unit.
·······························++-

More links:

<http://www.h-online.com/open/features/Kernel-Log-Coming-in-3-8-Part-1-Filesystems-and-storage-1788524.html>

Kernel Log - Coming in 3.8 (Part 1) -- Filesystems and storage (F2fs)

<http://lwn.net/Articles/518988/>
LWN: An f2fs teardown

<http://lwn.net/Articles/470553/>
LWN: Improving ext4: bigalloc, inline data, and metadata checksums

> I also question how much optimization there is, and if the lack of it's
> use actually effects the performance enough to render the drive
> effectively useless.

Not performance, but life expectancy. The optimization was basically
reducing the size of the block write.

Every time you write one byte on an USB stick, it actually writes a
block of 16KB (IIRC). Well, the write area where the FAT is expected to
be can be much smaller, say 256 bytes (I don't know). Thus the number of
write operations on the same block gets reduced.

> It seems rather silly for manufacturers to create a drive that
> effectively doesn't work with other file systems.

Nonono, it will work with any system.

>> For frequent backups I would not use and USB stick,....
>
> Why not?

It will not survive long.

>> Don't use a journal. For ext4, create it like this:
>
> What's wrong with the journal?

Too many writes to the same location, which wears out the stick faster.

>
> I assume that the file system will be unmounted properly, particularly
> for something for backups which ideally has as pristine a file system as
> possible.
>
>> Don't use ext2, has no extents.
>
> Please elaborate why you say that version 2 of the extent file system
> doesn't have any extents.

It is just a fact :-)

it is a feature that ext4 has.

Reference:

http://www.sysresccd.org/Sysresccd-manual-en_How_to_install_SystemRescued_on_an_USB-stick

If you are using SystemRescueCD-1.2 or more recent, it's recommended
that you use an ext4 filesystem with the journal turned off (this is
possible with Linux >= 2.6.29). USB sticks are Flash filesystems and
this type of memory only supports a limited number of writes.

Journaling filesystems will make many writes at the same location (where
the journal is stored). Therefore, to extend the lifespan of the
memory we should limit the number of writes. Here is how to use ext4
with the journaling turned off:

mke2fs -t ext4 -O ^has_journal /dev/sdf1

Note 2016-03-12: tune2fs -O ^has_journal <ext3/4-device>.

You could also use ext2 but it does not support extents, and then it
requires more accesses to read/write large files to the disk.

The actual URL may have changed, my notes date from 2016 or earlier.

--
Cheers,
Carlos E.R.

On 26/07/2021 23.04, Rich wrote:
> Extents <https://en.wikipedia.org/wiki/Extent_(file_systems)> are
> different and the 'ext[234]' series did not add extents until ext4.
> As to why the GP believes that lack of extents in ext2 makes it bad for
> a USB thumb drive, we have to ask the GP about their reasons why.

No, simply it is better to have extents that not have them. With ext2
there is no journal, but also no extents, so the recommendation is use
ext4 with no journal on flash media.

--
Cheers,
Carlos E.R.

On 2021-07-26, Carlos E. R. <robin_listas@es.invalid> wrote:
> On 26/07/2021 02.55, Robert Heller wrote:
>> At Mon, 26 Jul 2021 00:11:51 -0000 (UTC) John Smith <12345@whatismyemailaddress.xyz> wrote:
>>
>>>
>>> I am planning to use a USB stick for some incremental backups.
>>> The stick will be written to once a day only, and probably not much in
>>> the way of new material will be added (or old material will be removed)
>>> each time.
>>>
>>> What is the filesystem recommended for this kind of activity on
>>> this kind of hardware? USB sticks usually come with a VFAT filesystem,
>>> but this filesystem has limitations that will probably make it unsuitable
>>> to my purposes - I have quite large files, with long names that contain
>>> unusual characters.
>>
>> If the stick is never going to used on a MS-Windows system, you can reformat
>> the stick with a Linux file system, such as ext4.
>
> But make sure it doesn't have a journal.

So, ext3 then :)

Choobs

On 30/07/2021 14.30, John Dow wrote:
> On 2021-07-26, Carlos E. R. <robin_listas@es.invalid> wrote:
>> On 26/07/2021 02.55, Robert Heller wrote:
>>> At Mon, 26 Jul 2021 00:11:51 -0000 (UTC) John Smith <12345@whatismyemailaddress.xyz> wrote:
>>>
>>>>
>>>> I am planning to use a USB stick for some incremental backups.
>>>> The stick will be written to once a day only, and probably not much in
>>>> the way of new material will be added (or old material will be removed)
>>>> each time.
>>>>
>>>> What is the filesystem recommended for this kind of activity on
>>>> this kind of hardware? USB sticks usually come with a VFAT filesystem,
>>>> but this filesystem has limitations that will probably make it unsuitable
>>>> to my purposes - I have quite large files, with long names that contain
>>>> unusual characters.
>>>
>>> If the stick is never going to used on a MS-Windows system, you can reformat
>>> the stick with a Linux file system, such as ext4.
>>
>> But make sure it doesn't have a journal.
>
> So, ext3 then :)

No. Read my posts. ext4 without a journal.

--
Cheers,
Carlos E.R.

Thanks everybody for your replies. In the end I went for ext4
without a journal.

>> But make sure it doesn't have a journal.
>
> So, ext3 then :)

:)

On 2021-07-30, Carlos E. R. <robin_listas@es.invalid> wrote:
> On 30/07/2021 14.30, John Dow wrote:
>> So, ext3 then :)
>
> No. Read my posts. ext4 without a journal.

Oh dear - someone's sense of humour isn't working today :)

J