[fedora-arm] Re: Latest Fedora update leaves aarch64 unbootable RPi V3.

On Wed, 2020-02-19 at 11:25 -0500, Paul Whalen wrote:
> Hi Michael,
>
> Trying to reproduce this issue without success. How did you write the
> images and
> what command was used? After booting, did you confirm the kargs were
> as expected?

The original image booted and worked. Some of the machines had been
updated to 5.4.18 the week before. One newer build was from the
original image and updated directly to 5.4.19 and none of the 6
machines could be rebooted without fixing the
/boot/efi/EFI/fedora/grub.cfg file.

Not sure what you mean by confirming the kargs or what would be
expected. So I guess the answer there was no.

> Once booted, you just 'dnf update' and get the corrupted grub.cfg?
>
> Thanks!
> Paul
>
>
> ----- Original Message -----
> > On Tue, 2020-02-18 at 20:22 -0500, Michael H. Warfield wrote:
> > > On Tue, 2020-02-18 at 19:44 -0500, Stuart D. Gathman wrote:
> > > > On Tue, 18 Feb 2020, Michael H. Warfield wrote:
> > > >
> > > > > So the problem is in grubby. This had to have happened in
> > > > > just
> > > > > the
> > > > > last week. The update to 5.4.18 did not result in this
> > > > > carnage. So,
> > > > > whatever happened with grubby happened recently.
> > > > > Strangely, the /boot/efi/EFI/fedora/grub.cfg file is in the
> > > > > aarch64
> > > > > image but not in the arm7l image. :-?
> > > > It seems like this bug would affect all EFI systems, not just
> > > > aarch64.
> > > > I'm be on the alert before installing 5.4.19 on x86_64 as well.
> > > I agree. Seems strange but...
> > > I'm actually wondering if the hang and crash is a buffer
> > > overrun. I
> > > looked on a newly rebuilt system and saw the corruption but it
> > > was
> > > only
> > > a few lines long and the system still worked. I've got a pile of
> > > development systems are updated every time a kernel gets updated
> > > (once
> > > or twice a week) that that line had hit over 800K long. They're
> > > all
> > > largely mirrors of each other, just with different tasks. Each
> > > of
> > > the
> > > "corrupt" files were identical but all of the updates have been
> > > in
> > > lock
> > > step.
> > > All but one of the six affected systems are back up and the
> > > single
> > > one
> > > that's not managed to boot the kernel and started but ran into a
> > > panic.
> > > But I get a kernel menu on it now.
> >
> > That's it...
> >
> > The panic on the odd system out was a missing initramfs but that's
> > one
> > of the two that hurled chunks and crashed during the update and I
> > got
> > it back on the previous kernel and reinstalled the newer kernel to
> > fix
> > that once I had fixed grub.cfg. I then compared the working
> > grub.cfg
> > file to the resulting one from the reinstall.
> >
> > This:
> > --
> > set default_kernelopts="root=UUID=92567a3e-b3d2-494a-b4ec-
> > ebb4687d6b40 ro
> > cma=192MB"
> > --
> > Became this during the install:
> > --
> > set
> > default_kernelopts="root=UUID=92567a3e-b3d2-494a-b4ec-
> > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-
> > b3d2-494a-b4ec-ebb4687d6b40
> > ro cma=192MB"
> > --
> >
> > And became this after the reinstall was complete:
> > --
> > set
> > default_kernelopts="root=UUID=92567a3e-b3d2-494a-b4ec-
> > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-
> > b3d2-494a-b4ec-ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-
> > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-
> > b3d2-494a-b4ec-ebb4687d6b40=92567a3e-b3d2-494a-b4ec-
> > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-
> > b3d2-494a-b4ec-ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-
> > ebb4687d6b40=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-b3d2-
> > 494a-b4ec-ebb4687d6b40=92567a3e-b3d2-494a-b4ec-
> > ebb4687d6b40=92567a3e-b3d2-494a-b4ec-ebb4687d6b40
> > ro cma=192MB"
> > --
> >
> > It duplicated the root UUID on the default_kernelopts several
> > times.
> > And rebooting with that bad line still worked. It seems to be the
> > size
> > of that duplicated root UUID that finally borks it. Each time the
> > kernel gets updated, it gets another string attached until it blows
> > chunks. This may have been in there for a long time, I'm just real
> > aggressive about doing kernel updates.
> >
> > It's now up on the updated kernel. WIERD. I should probably
> > update my
> > Bugzilla report now.
> >
> > No idea why it's not showing up on x86_64 systems.
> >
> > > Just checked my x86_64 system (a Lenovo Yoga 730-15). No sign of
> > > the
> > > corruption in that file. Very strange but 6 systems impacted at
> > > nearly
> > > the same time (hours).
> >
> > Mike
> > --
> > Michael H. Warfield (AI4NB) | (o) +1 706 850-8773 |
> > mhw@WittsEnd.com
> > /\/\|=mhw=|\/\/ | (c) +1 678 463-0932 |
> > http://www.wittsend.com/mhw/
> > ARIN whois: MHW9-ARIN | An optimist believes we live in the
> > best of all
> > PGP Key: 0xC0EB9675674627FF | possible worlds. A pessimist is sure
> > of it!
> >
> > _______________________________________________
> > arm mailing list -- arm@lists.fedoraproject.org
> > To unsubscribe send an email to arm-leave@lists.fedoraproject.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines:
> > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org
> >
> _______________________________________________
> arm mailing list -- arm@lists.fedoraproject.org
> To unsubscribe send an email to arm-leave@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org
>
--
Michael H. Warfield (AI4NB) | (o) +1 706 850-8773 | mhw@WittsEnd.com
/\/\|=mhw=|\/\/ | (c) +1 678 463-0932 | http://www.wittsend.com/mhw/
ARIN whois: MHW9-ARIN | An optimist believes we live in the best of all
PGP Key: 0xC0EB9675674627FF | possible worlds. A pessimist is sure of it!