I'm noticing places in both Slackware's installer and debian-installer
where a mkfs run is immediately followed by a sync...

---------------SeTpartitions--------------------------------------------
... other fs types ...

elif [ "$ROOT_SYS_TYPE" = "ext4" ]; then
if [ "$DOFORMAT" = "Check" ]; then
make_ext4 $ROOT_DEVICE "y"
else
make_ext4 $ROOT_DEVICE "n"
fi
elif [ "$ROOT_SYS_TYPE" = "reiserfs" ]; then

... yet more fs types ...

fi
fi # DOFORMAT?

# Sync before identifying and mounting the root device:
sync
------------------------------------------------------------------------

And debian-installer...

---packages/partman-basicfilesystems/commit.d/format_basicfilesystems---
case $filesystem in
ext2)
options=''
if [ -f $id/usage ]; then
options="$options -T $(cat $id/usage)"
fi
if [ "$(udpkg --print-os)" = hurd ]; then
options="$options -b 4096 -I 128 -o hurd"
fi
db_progress START 0 3 partman/text/formatting
db_progress INFO $template
db_progress SET 1
if log-output -t partman --pass-stdout \
mkfs.ext2 -F $device $options >/dev/null; then
sync
status=OK
else
status=failed
fi
db_progress STOP
------------------------------------------------------------------------

Is this just a known thing among those who've been around? mkfs isn't
truly done and with all bits on disk when it exits?

- Mike Sm.

Mike Small <smallm@panix.com> wrote:
>
> I'm noticing places in both Slackware's installer and
> debian-installer where a mkfs run is immediately followed by a
> sync...
>
> [snip]
>
> Is this just a known thing among those who've been around? mkfs
> isn't truly done and with all bits on disk when it exits?

It is not just mkfs. Linux caches writes in RAM, with the data
actually being committed to disk some time later.

If you want to be sure a write has hit the disk (note, due to on disk
caches, you can never really be /sure/) you need to tell the kernel to
write the cached RAM data out to disk -- that is what 'sync' does.

Those sync's are not strictly necessary, but are likely present for
safety's sake. Run the mkfs, then try to make certian all the writes
have been at least sent to the disk controller.

Rich <rich@example.invalid> writes:
> Mike Small <smallm@panix.com> wrote:
>> I'm noticing places in both Slackware's installer and
>> debian-installer where a mkfs run is immediately followed by a
>> sync...
>>
>> [snip]
>>
>> Is this just a known thing among those who've been around? mkfs
>> isn't truly done and with all bits on disk when it exits?
>
> It is not just mkfs. Linux caches writes in RAM, with the data
> actually being committed to disk some time later.
>
> If you want to be sure a write has hit the disk (note, due to on disk
> caches, you can never really be /sure/) you need to tell the kernel to
> write the cached RAM data out to disk -- that is what 'sync' does.
>
> Those sync's are not strictly necessary, but are likely present for
> safety's sake. Run the mkfs, then try to make certian all the writes
> have been at least sent to the disk controller.

They are redundant, at least for ext4, since mkfs.ext4 calls fsync on
the target device. I’m not going to check other filesystems but the
situation is likely to be the same.

--
https://www.greenend.org.uk/rjk/

Richard Kettlewell <invalid@invalid.invalid> writes:

> Rich <rich@example.invalid> writes:
>> Mike Small <smallm@panix.com> wrote:
>>> I'm noticing places in both Slackware's installer and
>>> debian-installer where a mkfs run is immediately followed by a
>>> sync...
>>>
>>> [snip]
>>>
>>> Is this just a known thing among those who've been around? mkfs
>>> isn't truly done and with all bits on disk when it exits?
>>
>> It is not just mkfs. Linux caches writes in RAM, with the data
>> actually being committed to disk some time later.
>>
>> If you want to be sure a write has hit the disk (note, due to on disk
>> caches, you can never really be /sure/) you need to tell the kernel to
>> write the cached RAM data out to disk -- that is what 'sync' does.
>>
>> Those sync's are not strictly necessary, but are likely present for
>> safety's sake. Run the mkfs, then try to make certian all the writes
>> have been at least sent to the disk controller.
>
> They are redundant, at least for ext4, since mkfs.ext4 calls fsync on
> the target device. I’m not going to check other filesystems but the
> situation is likely to be the same.

Suppose the ext4 fs was on an encrypted volume on top of a dm raid,
i.e. two levels of device mapper. Is it possible that the fsync on the
crypt device wouldn't carry through to the raid?

My reason for looking at what Slackware and Debian do came from seeing a
script at work fail intermittently. It does the following:

1. create an ext4 fs
2. mount it
3. copy files to it
4. unmount it
5. remount it at a different point
6. access copied files.

This sequence sometimes fails at step 5 and rarely in step 6. Typically
it does not fail. Could be a hardware error or something else (I'm
always looking for a new thing to blame udevd on, for instance), but it
sure would be nice (but not reassuring) if only a sync or two was what
was needed.

Mike Small <smallm@panix.com> writes:
> Richard Kettlewell <invalid@invalid.invalid> writes:
>> Rich <rich@example.invalid> writes:
>>> Mike Small <smallm@panix.com> wrote:
>>>> I'm noticing places in both Slackware's installer and
>>>> debian-installer where a mkfs run is immediately followed by a
>>>> sync...
>>>>
>>>> [snip]
>>>>
>>>> Is this just a known thing among those who've been around? mkfs
>>>> isn't truly done and with all bits on disk when it exits?
>>>
>>> It is not just mkfs. Linux caches writes in RAM, with the data
>>> actually being committed to disk some time later.
>>>
>>> If you want to be sure a write has hit the disk (note, due to on disk
>>> caches, you can never really be /sure/) you need to tell the kernel to
>>> write the cached RAM data out to disk -- that is what 'sync' does.
>>>
>>> Those sync's are not strictly necessary, but are likely present for
>>> safety's sake. Run the mkfs, then try to make certian all the writes
>>> have been at least sent to the disk controller.
>>
>> They are redundant, at least for ext4, since mkfs.ext4 calls fsync on
>> the target device. I’m not going to check other filesystems but the
>> situation is likely to be the same.
>
> Suppose the ext4 fs was on an encrypted volume on top of a dm raid,
> i.e. two levels of device mapper. Is it possible that the fsync on the
> crypt device wouldn't carry through to the raid?

It’d certainly be a bug in whichever layer didn’t propagate it, and it
would (almost certainly) affect sync in the same way, i.e. the sync
wouldn’t help in the presence of that (hypothetical) bug.

> My reason for looking at what Slackware and Debian do came from seeing a
> script at work fail intermittently. It does the following:
>
> 1. create an ext4 fs
> 2. mount it
> 3. copy files to it
> 4. unmount it
> 5. remount it at a different point
> 6. access copied files.
>
> This sequence sometimes fails at step 5 and rarely in step 6. Typically
> it does not fail. Could be a hardware error or something else (I'm
> always looking for a new thing to blame udevd on, for instance), but it
> sure would be nice (but not reassuring) if only a sync or two was what
> was needed.

‘Fails’ isn’t really enough information to even speculate. Looking at
the behavior, any error message, and any kernel logs, might shed some
light on the situation.

--
https://www.greenend.org.uk/rjk/

On Tue, 27 Sep 2022 13:15:25 -0400, Mike Small wrote:
> Richard Kettlewell <invalid@invalid.invalid> writes:
>
>> Rich <rich@example.invalid> writes:
>>> Mike Small <smallm@panix.com> wrote:
>>>> I'm noticing places in both Slackware's installer and
>>>> debian-installer where a mkfs run is immediately followed by a
>>>> sync...
>>>>
>>>> [snip]
>>>>
>>>> Is this just a known thing among those who've been around? mkfs
>>>> isn't truly done and with all bits on disk when it exits?
>>>
>>> It is not just mkfs. Linux caches writes in RAM, with the data
>>> actually being committed to disk some time later.
>>>
>>> If you want to be sure a write has hit the disk (note, due to on disk
>>> caches, you can never really be /sure/) you need to tell the kernel to
>>> write the cached RAM data out to disk -- that is what 'sync' does.
>>>
>>> Those sync's are not strictly necessary, but are likely present for
>>> safety's sake. Run the mkfs, then try to make certian all the writes
>>> have been at least sent to the disk controller.
>>
>> They are redundant, at least for ext4, since mkfs.ext4 calls fsync on
>> the target device. I’m not going to check other filesystems but the
>> situation is likely to be the same.
>
> Suppose the ext4 fs was on an encrypted volume on top of a dm raid,
> i.e. two levels of device mapper. Is it possible that the fsync on the
> crypt device wouldn't carry through to the raid?
>
> My reason for looking at what Slackware and Debian do came from seeing a
> script at work fail intermittently. It does the following:
>
> 1. create an ext4 fs
> 2. mount it
> 3. copy files to it
> 4. unmount it
> 5. remount it at a different point
> 6. access copied files.
>
> This sequence sometimes fails at step 5 and rarely in step 6. Typically
> it does not fail. Could be a hardware error or something else (I'm
> always looking for a new thing to blame udevd on, for instance), but it
> sure would be nice (but not reassuring) if only a sync or two was what
> was needed.
>

I suggest adding test return status after each step to help find the problem.

Example:

if [ $? -ne 0 ] ; then
_code=$?
echo "Return status $_code is not 0 for command
command_executed_here
"
exit $_code
fi

On Tue, 27 Sep 2022 13:15:25 -0400, Mike Small wrote:
> 1. create an ext4 fs
> 2. mount it
> 3. copy files to it
> 4. unmount it
> 5. remount it at a different point
> 6. access copied files.
>
> This sequence sometimes fails at step 5 and rarely in step 6. Typically
> it does not fail. Could be a hardware error or something else (I'm
> always looking for a new thing to blame udevd on, for instance), but it
> sure would be nice (but not reassuring) if only a sync or two was what
> was needed.

Umount is supposed to sync the file system before unmounting and
returning.

As Bit Twister said, it might be a good idea to check the exit status of
all the commands and especially the umount commoand. Also look for error
messages from the commands and most of all, check for warnings and errors
in the output from dmesg when something like this goes wrong.

regards Henrik

Henrik Carlqvist <Henrik.Carlqvist@deadspam.com> writes:

> On Tue, 27 Sep 2022 13:15:25 -0400, Mike Small wrote:
>> 1. create an ext4 fs
>> 2. mount it
>> 3. copy files to it
>> 4. unmount it
>> 5. remount it at a different point
>> 6. access copied files.
>>
>> This sequence sometimes fails at step 5 and rarely in step 6. Typically
>> it does not fail. Could be a hardware error or something else (I'm
>> always looking for a new thing to blame udevd on, for instance), but it
>> sure would be nice (but not reassuring) if only a sync or two was what
>> was needed.
>
> Umount is supposed to sync the file system before unmounting and
> returning.
>
> As Bit Twister said, it might be a good idea to check the exit status of
> all the commands and especially the umount commoand. Also look for error
> messages from the commands and most of all, check for warnings and errors
> in the output from dmesg when something like this goes wrong.
>

Well, we checked at least for exit code 0 both on the cp and the umount,
e.g. umount X || die...

I'm looking more closely now, but getting dmesg output is tricky cause
it runs in an initramfs and reboots on the failure. Likely I'll have to
log debugging with dmesg, etc. Just trying to narrow now the steps to
reproduce.

On Thu, 29 Sep 2022 19:05:04 -0400, Mike Small wrote:
> I'm looking more closely now, but getting dmesg output is tricky cause
> it runs in an initramfs and reboots on the failure. Likely I'll have to
> log debugging with dmesg, etc. Just trying to narrow now the steps to
> reproduce.

Another thing that might be worth checking when suspecting bad hardware
is the output of something like:

smartctl --all /dev/sda

The SMART logged data in the disk is persistent between reboots.

regards Henrik