[fedora-arm] Re: Latest Fedora update leaves aarch64 unbootable RPi V3.

On Wed, 2020-02-19 at 14:12 -0500, Paul Whalen wrote:
>
> ----- Original Message -----
> > On Wed, 2020-02-19 at 12:04 -0500, Michael H. Warfield wrote:
> > > On Wed, 2020-02-19 at 11:25 -0500, Paul Whalen wrote:
> > > > Hi Michael,
> > > >
> > > > Trying to reproduce this issue without success. How did you
> > > > write
> > > > the
> > > > images and
> > > > what command was used? After booting, did you confirm the kargs
> > > > were
> > > > as expected?
> > > The original image booted and worked. Some of the machines had
> > > been
> > > updated to 5.4.18 the week before. One newer build was from the
> > > original image and updated directly to 5.4.19 and none of the 6
> > > machines could be rebooted without fixing the
> > > /boot/efi/EFI/fedora/grub.cfg file.
> > > Not sure what you mean by confirming the kargs or what would be
> > > expected. So I guess the answer there was no.

> Thanks. I was concerned the issue was from the arm-image-installer
> doing something
> it shouldnt.

> > On the bugzilla ticket one error looks to have been from "grubby-
> > deprecated" which was on the system. Removing it and reproducing
> > the
> > reinstall kernel-core and the problem seems to have gone
> > away. Looks
> > like grubby-deprecated is required by extlinux-bootloader, which is
> > then also removed. 3 of the systems have been tested and no longer
> > show the error.
> >
> > We may have it.
> >
> > But the grubby-deprecated packages are in both the download images
> > for
> > Xfce aarch64 and armfp for Fedora 31. Doesn't seem to affect
> > armfp.

> The armhfp images use extlinux and require grubby-deprecated. The
> aarch64
> images shouldn't have it installed.

> Looking at your bz, I see you mention the aarch64 image has extlinux-
> bootloader
> and grubby-deprecated which I dont see. Did you install anything else
> on the
> image that could have pulled it in?

I don't think so. I'll do a fresh raw image cut and see if there is
something in there without doing any updates. Since I have a mix of
systems, there may have been a cross contamination and most of the 64
bit systems were built to upgrade and replace their 32 bit previous
Fedora versions on earlier systems built before the aarch64 was stable
enough (Fedora 30? 29?). And I built one system from scratch and
updated it from there. Since these are in sort of a cooperative
cluster, it might be possible that cross over between what rpm packages
were installed, but it hit all of the V3 systems and I only have a
couple of V2 systems left on-line (for testing) at this time that
require it and all of them have been updating successfully for months.

Just checked a fresh raw un-updated image and you're right, it's not
there.

Now I gotta figure out how it got on all 6 of my V3 systems and WHEN.
It's even on my reference card. So it got introduced into my build
process very early on. Maybe from an old backup.

What's bad is that it's a required package for the armfp systems but is
in the aarch64 repositories and yet can cause this problem if it's
accidentally installed. Maybe that should be should be removed from
the aarch64 repositories unless there's an actual documented need for
it with the caveats of what could happen?

Mike

> > Mike
> >
> > > > Once booted, you just 'dnf update' and get the corrupted
> > > > grub.cfg?
> > > >
> > > > Thanks!
> > > > Paul
> > > >
> > > >
> > > > ----- Original Message -----
> > > > > On Tue, 2020-02-18 at 20:22 -0500, Michael H. Warfield wrote:
> > > > > > On Tue, 2020-02-18 at 19:44 -0500, Stuart D. Gathman wrote:
> > > > > > > On Tue, 18 Feb 2020, Michael H. Warfield wrote:
> > > > > > >
> > > > > > > > So the problem is in grubby. This had to have happened
> > > > > > > > in
> > > > > > > > just
> > > > > > > > the
> > > > > > > > last week. The update to 5.4.18 did not result in this
> > > > > > > > carnage. So,
> > > > > > > > whatever happened with grubby happened recently.
> > > > > > > > Strangely, the /boot/efi/EFI/fedora/grub.cfg file is in
> > > > > > > > the
> > > > > > > > aarch64
> > > > > > > > image but not in the arm7l image. :-?
> > > > > > > It seems like this bug would affect all EFI systems, not
> > > > > > > just
> > > > > > > aarch64.
> > > > > > > I'm be on the alert before installing 5.4.19 on x86_64 as
> > > > > > > well.
> > > > > > I agree. Seems strange but...
> > > > > > I'm actually wondering if the hang and crash is a buffer
> > > > > > overrun. I
> > > > > > looked on a newly rebuilt system and saw the corruption but
> > > > > > it
> > > > > > was
> > > > > > only
> > > > > > a few lines long and the system still worked. I've got a
> > > > > > pile
> > > > > > of
> > > > > > development systems are updated every time a kernel gets
> > > > > > updated
> > > > > > (once
> > > > > > or twice a week) that that line had hit over 800K
> > > > > > long. They're
> > > > > > all
> > > > > > largely mirrors of each other, just with different
> > > > > > tasks. Each
> > > > > > of
> > > > > > the
> > > > > > "corrupt" files were identical but all of the updates have
> > > > > > been
> > > > > > in
> > > > > > lock
> > > > > > step.
> > > > > > All but one of the six affected systems are back up and the
> > > > > > single
> > > > > > one
> > > > > > that's not managed to boot the kernel and started but ran
> > > > > > into
> > > > > > a
> > > > > > panic.
> > > > > > But I get a kernel menu on it now.
> > > > >
> > > > > That's it...
> > > > >
> > > > > The panic on the odd system out was a missing initramfs but
> > > > > that's
> > > > > one
> > > > > of the two that hurled chunks and crashed during the update
> > > > > and I
> > > > > got
> > > > > it back on the previous kernel and reinstalled the newer
> > > > > kernel
> > > > > to
> > > > > fix
> > > > > that once I had fixed grub.cfg. I then compared the working
> > > > > grub.cfg
> > > > > file to the resulting one from the reinstall.
> > > > >
> > > > > This:
> > > > > --
> > > > > set default_kernelopts="root=UUID=92567a3e-b3d2-494a-b4ec-
> > > > > ebb4687d6b40 ro
> > > > > cma=192MB"
> > > > > --
> > > > > Became this during the install:
> > > > > --
> > > > > set
> > > > > default_kernelopts="root=UUID=92567a3e-b3d2-494a-b4ec-
> > > > > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-
> > > > > ebb4687d6b40=92567a3e-
> > > > > b3d2-494a-b4ec-ebb4687d6b40
> > > > > ro cma=192MB"
> > > > > --
> > > > >
> > > > > And became this after the reinstall was complete:
> > > > > --
> > > > > set
> > > > > default_kernelopts="root=UUID=92567a3e-b3d2-494a-b4ec-
> > > > > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-
> > > > > ebb4687d6b40=92567a3e-
> > > > > b3d2-494a-b4ec-ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-
> > > > > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-
> > > > > ebb4687d6b40=92567a3e-
> > > > > b3d2-494a-b4ec-ebb4687d6b40=92567a3e-b3d2-494a-b4ec-
> > > > > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-
> > > > > ebb4687d6b40=92567a3e-
> > > > > b3d2-494a-b4ec-ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-
> > > > > ebb4687d6b40=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-
> > > > > b3d2-
> > > > > 494a-b4ec-ebb4687d6b40=92567a3e-b3d2-494a-b4ec-
> > > > > ebb4687d6b40=92567a3e-b3d2-494a-b4ec-ebb4687d6b40
> > > > > ro cma=192MB"
> > > > > --
> > > > >
> > > > > It duplicated the root UUID on the default_kernelopts several
> > > > > times.
> > > > > And rebooting with that bad line still worked. It seems to
> > > > > be
> > > > > the
> > > > > size
> > > > > of that duplicated root UUID that finally borks it. Each
> > > > > time
> > > > > the
> > > > > kernel gets updated, it gets another string attached until it
> > > > > blows
> > > > > chunks. This may have been in there for a long time, I'm
> > > > > just
> > > > > real
> > > > > aggressive about doing kernel updates.
> > > > >
> > > > > It's now up on the updated kernel. WIERD. I should probably
> > > > > update my
> > > > > Bugzilla report now.
> > > > >
> > > > > No idea why it's not showing up on x86_64 systems.
> > > > >
> > > > > > Just checked my x86_64 system (a Lenovo Yoga 730-15). No
> > > > > > sign
> > > > > > of
> > > > > > the
> > > > > > corruption in that file. Very strange but 6 systems
> > > > > > impacted
> > > > > > at
> > > > > > nearly
> > > > > > the same time (hours).
> > > > >
> > > > > Mike
> > > > > --
> > > > > Michael H. Warfield (AI4NB) | (o) +1 706 850-8773 |
> > > > > mhw@WittsEnd.com
> > > > > /\/\|=mhw=|\/\/ | (c) +1 678 463-0932 |
> > > > > http://www.wittsend.com/mhw/
> > > > > ARIN whois: MHW9-ARIN | An optimist believes we live in
> > > > > the
> > > > > best of all
> > > > > PGP Key: 0xC0EB9675674627FF | possible worlds. A pessimist
> > > > > is
> > > > > sure
> > > > > of it!
> > > > >
> > > > > _______________________________________________
> > > > > arm mailing list -- arm@lists.fedoraproject.org
> > > > > To unsubscribe send an email to
> > > > > arm-leave@lists.fedoraproject.org
> > > > > Fedora Code of Conduct:
> > > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > > List Guidelines:
> > > > > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > > List Archives:
> > > > > https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org
> > > > >
> > > > _______________________________________________
> > > > arm mailing list -- arm@lists.fedoraproject.org
> > > > To unsubscribe send an email to
> > > > arm-leave@lists.fedoraproject.org
> > > > Fedora Code of Conduct:
> > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > List Guidelines:
> > > > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > List Archives:
> > > > https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org
> > > >
> > --
> > Michael H. Warfield (AI4NB) | (o) +1 706 850-8773 |
> > mhw@WittsEnd.com
> > /\/\|=mhw=|\/\/ | (c) +1 678 463-0932 |
> > http://www.wittsend.com/mhw/
> > ARIN whois: MHW9-ARIN | An optimist believes we live in the
> > best of all
> > PGP Key: 0xC0EB9675674627FF | possible worlds. A pessimist is sure
> > of it!
> >
>
>
--
Michael H. Warfield (AI4NB) | (o) +1 706 850-8773 | mhw@WittsEnd.com
/\/\|=mhw=|\/\/ | (c) +1 678 463-0932 | http://www.wittsend.com/mhw/
ARIN whois: MHW9-ARIN | An optimist believes we live in the best of all
PGP Key: 0xC0EB9675674627FF | possible worlds. A pessimist is sure of it!