Situation: Server uses to mirrored drives to save backup data. One drive
was throwing bad sectors. Dreaded 3TB Constellation drives so replaced
with 4TB IronWolf. After transferring 2.2TB data from the one good drive
to the two new drives installed drives in server. Server would only boot
into emergency mode. In emergency mode a manual mount command would
mount the two drives and you can exit to regular login.

Diagnosis: Replacing the new drives triggered fsck upon boot, but
systemd.fsck@dev-disk-by...service failed for each new drive. UUIDs were
correctly updated in fstab. Went and tried manually fsck drives and
e2fsck failed with incompatible version.

Conclusion: (Or the DOH-Moment) That server is still running 18.04 LTS
and I formatted the new drives with a usb drive dock and I am running
23.04! The files are currently accessible and writable so I temporarily
disabled fsck for those two drives. In a live 18.04 session formatted up
a new drive with correct version. Luckily I have a 3rd spare drive to
expedite this. Have another system coping files from old good drive to
this new drive so it can latter be swapped to the server and repeat the
processes for the other drive...Just have to get it done before Sunday
when the backup occurs.

Penalty: Humiliation and time. Just passing on the info and maybe save
someone else of the same mistake.

BTW: have not found what they did to change the format of ext4 from the
different versions OS. Both drives were gpt and same format. One would
think that would not matter...

--
Take care,

Jonathan
-------------------
LITTLE WORKS STUDIO
http://www.LittleWorksStudio.com

Jonathan N. Little wrote:
> Situation: Server uses to mirrored drives to save backup data. One drive
> was throwing bad sectors. Dreaded 3TB Constellation drives so replaced
> with 4TB IronWolf. After transferring 2.2TB data from the one good drive
> to the two new drives installed drives in server. Server would only boot
> into emergency mode. In emergency mode a manual mount command would
> mount the two drives and you can exit to regular login.
>
> Diagnosis: Replacing the new drives triggered fsck upon boot, but
> systemd.fsck@dev-disk-by...service failed for each new drive. UUIDs were
> correctly updated in fstab. Went and tried manually fsck drives and
> e2fsck failed with incompatible version.
>
> Conclusion: (Or the DOH-Moment) That server is still running 18.04 LTS
> and I formatted the new drives with a usb drive dock and I am running
> 23.04! The files are currently accessible and writable so I temporarily
> disabled fsck for those two drives. In a live 18.04 session formatted up
> a new drive with correct version. Luckily I have a 3rd spare drive to
> expedite this. Have another system coping files from old good drive to
> this new drive so it can latter be swapped to the server and repeat the
> processes for the other drive...Just have to get it done before Sunday
> when the backup occurs.
>
> Penalty: Humiliation and time. Just passing on the info and maybe save
> someone else of the same mistake.
>
> BTW: have not found what they did to change the format of ext4 from the
> different versions OS. Both drives were gpt and same format. One would
> think that would not matter...

It is surprising that something that significant would have changed in
the formatting. Perhaps the older fsck is just being cautious, and
refusing to try fixing a partition formatted with a newer version of
ext4, which might use features the older fsck doesn't know about. And
then the rest of the system refuses to mount it because fsck failed.

However, a couple of years ago I did have an issue with a couple of
disks that I'd been using with a USB-SATA adapter, which then couldn't
be read when attached directly to the PC's SATA bus. It turns out that
some USB-SATA adapters misreport the logical block size used by the disk
(seems to be an issue with a commonly used chipset). When I attached a
disk to the USB adapter and formatted it, the adapter reported 4096 byte
logical blocks. So the GPT partition table was placed at block 1, 4096
bytes into the disk, and partition addresses given in 4096 byte blocks.
But when I attached the disk directly to the PC's SATA bus, it reported
512 byte logical blocks. The partition table couldn't be found at block
1 (now 512 bytes into the disk) and, even if could have been found, the
partition addresses would all have been misinterpreted.

Some relevant information I came across while trying to figure this one out:
<https://askubuntu.com/questions/909041/harddrive-on-usb-to-sata-adapter-not-showing-full-size>
<https://bugzilla.redhat.com/show_bug.cgi?id=734015>
<https://www.spinics.net/lists/linux-usb/msg70029.html>

That doesn't seem like the same issue you had, since your problems were
with fsck, so in your case it sounds like the partitions were read
correctly and the filesystem was found.

--
Mark.

On 8/26/2023 8:02 AM, Mark Bourne wrote:
> Jonathan N. Little wrote:
>> Situation: Server uses to mirrored drives to save backup data. One drive
>> was throwing bad sectors. Dreaded 3TB Constellation drives so replaced
>> with 4TB IronWolf. After transferring 2.2TB data from the one good drive
>> to the two new drives installed drives in server. Server would only boot
>> into emergency mode. In emergency mode a manual mount command would
>> mount the two drives  and you can exit to regular login.
>>
>> Diagnosis: Replacing the new drives triggered fsck upon boot, but
>> systemd.fsck@dev-disk-by...service failed for each new drive. UUIDs were
>> correctly updated in fstab. Went and tried manually fsck drives and
>> e2fsck failed with incompatible version.
>>
>> Conclusion: (Or the DOH-Moment) That server is still running 18.04 LTS
>> and I formatted the new drives with a usb drive dock and I am running
>> 23.04! The files are currently accessible and writable so I temporarily
>> disabled fsck for those two drives. In a live 18.04 session formatted up
>> a new drive with correct version. Luckily I have a 3rd spare drive to
>> expedite this. Have another system coping files from old good drive to
>> this new drive so it can latter be swapped to the server and repeat the
>> processes for the other drive...Just have to get it done before Sunday
>> when the backup occurs.
>>
>> Penalty: Humiliation and time. Just passing on the info and maybe save
>> someone else of the same mistake.
>>
>> BTW: have not found what they did to change the format of ext4 from the
>> different versions OS. Both drives were gpt and same format. One would
>> think that would not matter...
>
> It is surprising that something that significant would have changed in the formatting.  Perhaps the older fsck is just being cautious, and refusing to try fixing a partition formatted with a newer version of ext4, which might use features the older fsck doesn't know about.  And then the rest of the system refuses to mount it because fsck failed.
>
> However, a couple of years ago I did have an issue with a couple of disks that I'd been using with a USB-SATA adapter, which then couldn't be read when attached directly to the PC's SATA bus.  It turns out that some USB-SATA adapters misreport the logical block size used by the disk (seems to be an issue with a commonly used chipset).  When I attached a disk to the USB adapter and formatted it, the adapter reported 4096 byte logical blocks.  So the GPT partition table was placed at block 1, 4096 bytes into the disk, and partition addresses given in 4096 byte blocks. But when I attached the disk directly to the PC's SATA bus, it reported 512 byte logical blocks.  The partition table couldn't be found at block 1 (now 512 bytes into the disk) and, even if could have been found, the partition addresses would all have been misinterpreted.
>
> Some relevant information I came across while trying to figure this one out:
> <https://askubuntu.com/questions/909041/harddrive-on-usb-to-sata-adapter-not-showing-full-size>
> <https://bugzilla.redhat.com/show_bug.cgi?id=734015>
> <https://www.spinics.net/lists/linux-usb/msg70029.html>
>
> That doesn't seem like the same issue you had, since your problems were with fsck, so in your case it sounds like the partitions were read correctly and the filesystem was found.
>

That's done on purpose, and the product would have
been <cough> popular during a certain epoch (WinXP).
I'm not really sure such a shenanigan is really necessary
on USB, as the addressing there just might be a SCSI CDB
and capable of addressing large objects without a problem.
But I suppose MSDOS partition table support would be worth
something, and that would be the incentive to screw around.

https://superuser.com/questions/1271871/4k-emulation-sata-usb-controllers

There were some RAID controllers with a similar capability.
Maybe an Areca. Instead of a 2TB max drive size, the Areca
could host a 16TB drive/array and the user could use MSDOS partitioning.
At the time they were doing that, maybe GPT didn't exist yet ?
WinXP did not support GPT, so it was likely introduced after
WinXP.

The MSDOS partitioning table had 32 bit fields in it. And that
may have worked out to 2.2TB (making 2TB the largest disk you
would safely use). Various schemes were invented to handle
larger disks, such as Acronis Capacity Manager and a couple others.

The Areca trick existed before Acronis Capacity Manager.
I only accidentally found out about that, while skimming through
the user manual for an Areca (no, I'm not actually rich enough
to afford an Areca card).

This is a kind of feature, where the PCB should have a jumper to
select or de-select such a feature. It should not be totally automated.
Not everyone wants or needs that.

*******

On a slightly different topic...

There are OSes now that support 4Kn drives. I do not recommend
people search out 4Kn drives on purpose. A hyperstorage site may like
such things, but most people don't own enough good software to handle
4Kn drives. If there's a problem, you could be left holding a bag of
useless bits. The OS information here is likely incorrect, because
there was a patch part way through the W10/W11 era, intended for actual
support. Real support may have been later, than any documentation claims.
Earlier OSes may not have had working support (boot from 4Kn drive).

https://en.wikipedia.org/wiki/Advanced_Format

4Kn drives were showing up at retail at one time. I would expect
so many consumers bought those by mistake and returned for a refund,
the vendors figured out they should hide those. Maybe a site that sells
server accessories would list those SKUs. But if your OSes do not support
them, they would be a bad deal. I like to buy hardware when I can, that
every computer in the house can use, instead of just one computer barely
supporting it. I would not consider a 4Kn, even for experiments.

The highest capacity drive I own, I think it's a 512e, as it needs
no pampering. The only problem that drive has, is the holes are drilled
in the wrong place on it. The drive mounting holes are different
at perhaps 8TB or larger. The 6TB drives still have conventional holes.
At the 8TB level, some could have old-holes, some could have new-holes.
I only own one big drive, and I can tell ya, that hole issue is a
fucking disaster :-/ None of my trays attach properly.

Paul

Mark Bourne wrote:
>
> However, a couple of years ago I did have an issue with a couple of
> disks that I'd been using with a USB-SATA adapter, which then couldn't
> be read when attached directly to the PC's SATA bus.  It turns out that
> some USB-SATA adapters misreport the logical block size used by the disk
> (seems to be an issue with a commonly used chipset).

It can depend on the size of the drive. I have external drive docks for
coping and moving data on drives. My older dual Orico will handle up to
2TB drives but for larger drives I had to get a newer one. I have some
little USB-SATA dongles, one I use with a little PI server, but these
seem to have the older drive limit. All is dependent in the controller
chip used.

--
Take care,

Jonathan
-------------------
LITTLE WORKS STUDIO
http://www.LittleWorksStudio.com

Mark Bourne wrote:
> That doesn't seem like the same issue you had, since your problems were
> with fsck, so in your case it sounds like the partitions were read
> correctly and the filesystem was found.

Yes, the drives were fully accessible and read|writable by the 18.04
server. Just auto mounting failed because the fschk flag was set and why
I temporarily disabled in fstab. Final note is I reformatted drives with
18.04 and now no issue. Also noted that this server desperately needs to
be updated.

--
Take care,

Jonathan
-------------------
LITTLE WORKS STUDIO
http://www.LittleWorksStudio.com