[fedora-arm] Re: Latest Fedora update leaves aarch64 unbootable RPi V3.

On Tue, 2020-02-18 at 20:22 -0500, Michael H. Warfield wrote:
> On Tue, 2020-02-18 at 19:44 -0500, Stuart D. Gathman wrote:
> > On Tue, 18 Feb 2020, Michael H. Warfield wrote:
> >
> > > So the problem is in grubby. This had to have happened in just
> > > the
> > > last week. The update to 5.4.18 did not result in this
> > > carnage. So,
> > > whatever happened with grubby happened recently.
> > > Strangely, the /boot/efi/EFI/fedora/grub.cfg file is in the
> > > aarch64
> > > image but not in the arm7l image. :-?
> > It seems like this bug would affect all EFI systems, not just
> > aarch64.
> > I'm be on the alert before installing 5.4.19 on x86_64 as well.

> I agree. Seems strange but...

> I'm actually wondering if the hang and crash is a buffer overrun. I
> looked on a newly rebuilt system and saw the corruption but it was
> only
> a few lines long and the system still worked. I've got a pile of
> development systems are updated every time a kernel gets updated
> (once
> or twice a week) that that line had hit over 800K long. They're all
> largely mirrors of each other, just with different tasks. Each of
> the
> "corrupt" files were identical but all of the updates have been in
> lock
> step.

> All but one of the six affected systems are back up and the single
> one
> that's not managed to boot the kernel and started but ran into a
> panic.
> But I get a kernel menu on it now.

That's it...

The panic on the odd system out was a missing initramfs but that's one
of the two that hurled chunks and crashed during the update and I got
it back on the previous kernel and reinstalled the newer kernel to fix
that once I had fixed grub.cfg. I then compared the working grub.cfg
file to the resulting one from the reinstall.

This:
--
set default_kernelopts="root=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40 ro cma=192MB"
--
Became this during the install:
--
set default_kernelopts="root=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-b3d2-494a-b4ec-ebb4687d6b40 ro cma=192MB"
--

And became this after the reinstall was complete:
--
set default_kernelopts="root=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-b3d2-494a-b4ec-ebb4687d6b40 ro cma=192MB"
--

It duplicated the root UUID on the default_kernelopts several times.
And rebooting with that bad line still worked. It seems to be the size
of that duplicated root UUID that finally borks it. Each time the
kernel gets updated, it gets another string attached until it blows
chunks. This may have been in there for a long time, I'm just real
aggressive about doing kernel updates.

It's now up on the updated kernel. WIERD. I should probably update my
Bugzilla report now.

No idea why it's not showing up on x86_64 systems.

> Just checked my x86_64 system (a Lenovo Yoga 730-15). No sign of the
> corruption in that file. Very strange but 6 systems impacted at
> nearly
> the same time (hours).

Mike
--
Michael H. Warfield (AI4NB) | (o) +1 706 850-8773 | mhw@WittsEnd.com
/\/\|=mhw=|\/\/ | (c) +1 678 463-0932 | http://www.wittsend.com/mhw/
ARIN whois: MHW9-ARIN | An optimist believes we live in the best of all
PGP Key: 0xC0EB9675674627FF | possible worlds. A pessimist is sure of it!