----- Original Message -----
> On Wed, 2020-02-19 at 14:12 -0500, Paul Whalen wrote:
> >
> > ----- Original Message -----
> > > On Wed, 2020-02-19 at 12:04 -0500, Michael H. Warfield wrote:
> > > > On Wed, 2020-02-19 at 11:25 -0500, Paul Whalen wrote:
> > > > > Hi Michael,
> > > > >
> > > > > Trying to reproduce this issue without success. How did you
> > > > > write
> > > > > the
> > > > > images and
> > > > > what command was used? After booting, did you confirm the kargs
> > > > > were
> > > > > as expected?
> > > > The original image booted and worked. Some of the machines had
> > > > been
> > > > updated to 5.4.18 the week before. One newer build was from the
> > > > original image and updated directly to 5.4.19 and none of the 6
> > > > machines could be rebooted without fixing the
> > > > /boot/efi/EFI/fedora/grub.cfg file.
> > > > Not sure what you mean by confirming the kargs or what would be
> > > > expected. So I guess the answer there was no.
>
> > Thanks. I was concerned the issue was from the arm-image-installer
> > doing something
> > it shouldnt.
>
> > > On the bugzilla ticket one error looks to have been from "grubby-
> > > deprecated" which was on the system. Removing it and reproducing
> > > the
> > > reinstall kernel-core and the problem seems to have gone
> > > away. Looks
> > > like grubby-deprecated is required by extlinux-bootloader, which is
> > > then also removed. 3 of the systems have been tested and no longer
> > > show the error.
> > >
> > > We may have it.
> > >
> > > But the grubby-deprecated packages are in both the download images
> > > for
> > > Xfce aarch64 and armfp for Fedora 31. Doesn't seem to affect
> > > armfp.
>
> > The armhfp images use extlinux and require grubby-deprecated. The
> > aarch64
> > images shouldn't have it installed.
>
> > Looking at your bz, I see you mention the aarch64 image has extlinux-
> > bootloader
> > and grubby-deprecated which I dont see. Did you install anything else
> > on the
> > image that could have pulled it in?
>
> I don't think so. I'll do a fresh raw image cut and see if there is
> something in there without doing any updates. Since I have a mix of
> systems, there may have been a cross contamination and most of the 64
> bit systems were built to upgrade and replace their 32 bit previous
> Fedora versions on earlier systems built before the aarch64 was stable
> enough (Fedora 30? 29?). And I built one system from scratch and
> updated it from there. Since these are in sort of a cooperative
> cluster, it might be possible that cross over between what rpm packages
> were installed, but it hit all of the V3 systems and I only have a
> couple of V2 systems left on-line (for testing) at this time that
> require it and all of them have been updating successfully for months.
>
> Just checked a fresh raw un-updated image and you're right, it's not
> there.
Thanks for confirming.
>
> Now I gotta figure out how it got on all 6 of my V3 systems and WHEN.
> It's even on my reference card. So it got introduced into my build
> process very early on. Maybe from an old backup.
>
> What's bad is that it's a required package for the armfp systems but is
> in the aarch64 repositories and yet can cause this problem if it's
> accidentally installed. Maybe that should be should be removed from
> the aarch64 repositories unless there's an actual documented need for
> it with the caveats of what could happen?
Perhaps the maintainer can make some tweaks so it doesnt happen again. I think
we'll all remember this one if someone hits it and sends to our list :)
Thanks for chasing it down.
Paul
>
> Mike
>
> > > Mike
> > >
> > > > > Once booted, you just 'dnf update' and get the corrupted
> > > > > grub.cfg?
> > > > >
> > > > > Thanks!
> > > > > Paul
> > > > >
> > > > >
> > > > > ----- Original Message -----
> > > > > > On Tue, 2020-02-18 at 20:22 -0500, Michael H. Warfield wrote:
> > > > > > > On Tue, 2020-02-18 at 19:44 -0500, Stuart D. Gathman wrote:
> > > > > > > > On Tue, 18 Feb 2020, Michael H. Warfield wrote:
> > > > > > > >
> > > > > > > > > So the problem is in grubby. This had to have happened
> > > > > > > > > in
> > > > > > > > > just
> > > > > > > > > the
> > > > > > > > > last week. The update to 5.4.18 did not result in this
> > > > > > > > > carnage. So,
> > > > > > > > > whatever happened with grubby happened recently.
> > > > > > > > > Strangely, the /boot/efi/EFI/fedora/grub.cfg file is in
> > > > > > > > > the
> > > > > > > > > aarch64
> > > > > > > > > image but not in the arm7l image. :-?
> > > > > > > > It seems like this bug would affect all EFI systems, not
> > > > > > > > just
> > > > > > > > aarch64.
> > > > > > > > I'm be on the alert before installing 5.4.19 on x86_64 as
> > > > > > > > well.
> > > > > > > I agree. Seems strange but...
> > > > > > > I'm actually wondering if the hang and crash is a buffer
> > > > > > > overrun. I
> > > > > > > looked on a newly rebuilt system and saw the corruption but
> > > > > > > it
> > > > > > > was
> > > > > > > only
> > > > > > > a few lines long and the system still worked. I've got a
> > > > > > > pile
> > > > > > > of
> > > > > > > development systems are updated every time a kernel gets
> > > > > > > updated
> > > > > > > (once
> > > > > > > or twice a week) that that line had hit over 800K
> > > > > > > long. They're
> > > > > > > all
> > > > > > > largely mirrors of each other, just with different
> > > > > > > tasks. Each
> > > > > > > of
> > > > > > > the
> > > > > > > "corrupt" files were identical but all of the updates have
> > > > > > > been
> > > > > > > in
> > > > > > > lock
> > > > > > > step.
> > > > > > > All but one of the six affected systems are back up and the
> > > > > > > single
> > > > > > > one
> > > > > > > that's not managed to boot the kernel and started but ran
> > > > > > > into
> > > > > > > a
> > > > > > > panic.
> > > > > > > But I get a kernel menu on it now.
> > > > > >
> > > > > > That's it...
> > > > > >
> > > > > > The panic on the odd system out was a missing initramfs but
> > > > > > that's
> > > > > > one
> > > > > > of the two that hurled chunks and crashed during the update
> > > > > > and I
> > > > > > got
> > > > > > it back on the previous kernel and reinstalled the newer
> > > > > > kernel
> > > > > > to
> > > > > > fix
> > > > > > that once I had fixed grub.cfg. I then compared the working
> > > > > > grub.cfg
> > > > > > file to the resulting one from the reinstall.
> > > > > >
> > > > > > This:
> > > > > > --
> > > > > > set default_kernelopts="root=UUID=92567a3e-b3d2-494a-b4ec-
> > > > > > ebb4687d6b40 ro
> > > > > > cma=192MB"
> > > > > > --
> > > > > > Became this during the install:
> > > > > > --
> > > > > > set
> > > > > > default_kernelopts="root=UUID=92567a3e-b3d2-494a-b4ec-
> > > > > > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-
> > > > > > ebb4687d6b40=92567a3e-
> > > > > > b3d2-494a-b4ec-ebb4687d6b40
> > > > > > ro cma=192MB"
> > > > > > --
> > > > > >
> > > > > > And became this after the reinstall was complete:
> > > > > > --
> > > > > > set
> > > > > > default_kernelopts="root=UUID=92567a3e-b3d2-494a-b4ec-
> > > > > > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-
> > > > > > ebb4687d6b40=92567a3e-
> > > > > > b3d2-494a-b4ec-ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-
> > > > > > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-
> > > > > > ebb4687d6b40=92567a3e-
> > > > > > b3d2-494a-b4ec-ebb4687d6b40=92567a3e-b3d2-494a-b4ec-
> > > > > > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-
> > > > > > ebb4687d6b40=92567a3e-
> > > > > > b3d2-494a-b4ec-ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-
> > > > > > ebb4687d6b40=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-
> > > > > > b3d2-
> > > > > > 494a-b4ec-ebb4687d6b40=92567a3e-b3d2-494a-b4ec-
> > > > > > ebb4687d6b40=92567a3e-b3d2-494a-b4ec-ebb4687d6b40
> > > > > > ro cma=192MB"
> > > > > > --
> > > > > >
> > > > > > It duplicated the root UUID on the default_kernelopts several
> > > > > > times.
> > > > > > And rebooting with that bad line still worked. It seems to
> > > > > > be
> > > > > > the
> > > > > > size
> > > > > > of that duplicated root UUID that finally borks it. Each
> > > > > > time
> > > > > > the
> > > > > > kernel gets updated, it gets another string attached until it
> > > > > > blows
> > > > > > chunks. This may have been in there for a long time, I'm
> > > > > > just
> > > > > > real
> > > > > > aggressive about doing kernel updates.
> > > > > >
> > > > > > It's now up on the updated kernel. WIERD. I should probably
> > > > > > update my
> > > > > > Bugzilla report now.
> > > > > >
> > > > > > No idea why it's not showing up on x86_64 systems.
> > > > > >
> > > > > > > Just checked my x86_64 system (a Lenovo Yoga 730-15). No
> > > > > > > sign
> > > > > > > of
> > > > > > > the
> > > > > > > corruption in that file. Very strange but 6 systems
> > > > > > > impacted
> > > > > > > at
> > > > > > > nearly
> > > > > > > the same time (hours).
> > > > > >
> > > > > > Mike
> > > > > > --
> > > > > > Michael H. Warfield (AI4NB) | (o) +1 706 850-8773 |
> > > > > > mhw@WittsEnd.com
> > > > > > /\/\|=mhw=|\/\/ | (c) +1 678 463-0932 |
> > > > > > http://www.wittsend.com/mhw/
> > > > > > ARIN whois: MHW9-ARIN | An optimist believes we live in
> > > > > > the
> > > > > > best of all
> > > > > > PGP Key: 0xC0EB9675674627FF | possible worlds. A pessimist
> > > > > > is
> > > > > > sure
> > > > > > of it!
> > > > > >
> > > > > > _______________________________________________
> > > > > > arm mailing list -- arm@lists.fedoraproject.org
> > > > > > To unsubscribe send an email to
> > > > > > arm-leave@lists.fedoraproject.org
> > > > > > Fedora Code of Conduct:
> > > > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > > > List Guidelines:
> > > > > > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > > > List Archives:
> > > > > > https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org
> > > > > >
> > > > > _______________________________________________
> > > > > arm mailing list -- arm@lists.fedoraproject.org
> > > > > To unsubscribe send an email to
> > > > > arm-leave@lists.fedoraproject.org
> > > > > Fedora Code of Conduct:
> > > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > > List Guidelines:
> > > > > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > > List Archives:
> > > > > https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org
> > > > >
> > > --
> > > Michael H. Warfield (AI4NB) | (o) +1 706 850-8773 |
> > > mhw@WittsEnd.com
> > > /\/\|=mhw=|\/\/ | (c) +1 678 463-0932 |
> > > http://www.wittsend.com/mhw/
> > > ARIN whois: MHW9-ARIN | An optimist believes we live in the
> > > best of all
> > > PGP Key: 0xC0EB9675674627FF | possible worlds. A pessimist is sure
> > > of it!
> > >
> >
> >
> --
> Michael H. Warfield (AI4NB) | (o) +1 706 850-8773 | mhw@WittsEnd.com
> /\/\|=mhw=|\/\/ | (c) +1 678 463-0932 |
> http://www.wittsend.com/mhw/
> ARIN whois: MHW9-ARIN | An optimist believes we live in the best of all
> PGP Key: 0xC0EB9675674627FF | possible worlds. A pessimist is sure of it!
>
_______________________________________________
arm mailing list -- arm@lists.fedoraproject.org
To unsubscribe send an email to arm-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org
No comments:
Post a Comment