CentOS 7 does not start after kernel updates

The latest version of the CentOS 7 has a boot problem on our Xen platform. If you have a VPS with CentOS 7 and you run the kernel below (from 693):

3.10.0-693.xx

Or will update (through yum update) to this kernel, then the VPS will not boot due to this bug: https://bugs.centos.org/view.php?id=13763.

At the time of writing, September 2017, there is a workaround available that we describe below. If a fix is ​​available, we will apply this page.

Does this effect have on your VPS?

The following command shows which CentOS kernels are installed:

rpm -qa kernel

The output is as follows:

kernel-3.10.0-514.21.2.el7.x86_64
kernel-3.10.0-514.26.2.el7.x86_64
kernel-3.10.0-514.21.1.el7.x86_64
kernel-3.10.0-514.26.1.el7.x86_64
kernel-3.10.0-514.16.1.el7.x86_64

If you see a kernel with number 693 or higher in that list, it is already installed and you must complete the workaround by installing the patch. The work around can be find in the instructions below.

If you have completed the workaround, you can check that by using the following commands:

rpm -qa kernel plus

Output:

kernel-plus-3.10.0-693.2.2.el7.centos.plus.x86_64

and:

uname -a

Output:

Linux vps64001.public.cloudvps.com 3.10.0-693.2.2.el7.centos.plus.x86_64 # 1 SMP Tue Sep 12 23:18:54 UTC 2017 x86_64 x86_64 x86_64 GNU / Linux

(there is .plus left, and the kernel plus package is installed). If you have not yet rebooted then "uname -a" will give another output.

(You can see it specifies the ".plus" at the end of the kernel line meaning the package has been installed). If you didn't reboot yet then the command "uname -a" will show a different output.

If you are updating through yum update then you will see if and which kernel is being updated:

Installation:

 kernel x86_64 3.10.0-693.2.2.el7 updates 43 M
 kernel devel x86_64 3.10.0-693.2.2.el7 updates 14 M

[...]

Removing:

 kernel x86_64 3.10.0-514.16.1.el7 @updates 148 M 
 kernel devel x86_64 3.10.0-514.16.1.el7 @updates 34 M

The above output will ensure that the VPS does not boot after a restart or malfunction after execution of the update.

Prevent

If you have not updated yet, it is advisable to wait for updating until the bug is fixed and a new kernel is released. You can exclude kernel updates, so you can perform other updates except for the kernel. This can be achieved by updating the following file:

/etc/yum.conf

And make sure it containts the following line:

exclude=kernel kernel-devel kernel-PAE-*

If an exclude rule already exists, you must add "kernel *".

Workaround

If you have already updated and the VPS is still available then you can install another kernel. This kernel is part of the CentOS project, but is officially not supported. This repository is for faster updates without testing it extensively, which in this case causes the patch to be available for this bug.

After the bug has been resolved, you must undo these steps to turn the normal kernel.

Customize the following file:

/etc/yum.repos.d/CentOS-Base.repo

Change the following section from:

[base]
name = CentOS- $ releasever - Base
mirrorlist = http: //mirrorlist.centos.org/? release = $ releasever & arch = $ basearch & repo = us & infra = $ infra
# baseurl = http: //mirror.centos.org/centos/$releasever/os/$basearch/
enabled = 1
gpgcheck = 1
gpgkey = file: /// etc / pki / rpm-gpg / RPM-GPG-KEY-CentOS-7
#released updates

[updates]
name = CentOS- $ releasever - Updates
mirrorlist = http: //mirrorlist.centos.org/? release = $ releasever & arch = $ basearch & repo = updates & infra = $ infra
# baseurl = http: //mirror.centos.org/centos/$releasever/updates/$basearch
enabled = 1
gpgcheck = 1
gpgkey = file: /// etc / pki / rpm-gpg / RPM-GPG-KEY-CentOS-7

To:

[base]
name = CentOS- $ releasever - Base
mirrorlist = http: //mirrorlist.centos.org/? release = $ releasever & arch = $ basearch & repo = us & infra = $ infra
# baseurl = http: //mirror.centos.org/centos/$releasever/os/$basearch/
enabled = 1
gpgcheck = 1
gpgkey = file: /// etc / pki / rpm-gpg / RPM-GPG-KEY-CentOS-7
exclude = kernel kernel-devel kernel-PAE- *

#released updates
[updates]
name = CentOS- $ releasever - Updates
mirrorlist = http: //mirrorlist.centos.org/? release = $ releasever & arch = $ basearch & repo = updates & infra = $ infra
# baseurl = http: //mirror.centos.org/centos/$releasever/updates/$basearch/
enabled = 1
gpgcheck = 1
gpgkey = file: /// etc / pki / rpm-gpg / RPM-GPG-KEY-CentOS-7
exclude = kernel kernel-devel kernel-PAE- *

Then apply the following section to this file from:

#additional packages that extend functionality of existing package
[centosplus]
name = CentOS- $ releasever - Plus
mirrorlist = http: //mirrorlist.centos.org/? release = $ releasever & arch = $ basearch & repo = centosplus & infra = $ infra
# baseurl = http: //mirror.centos.org/centos/$releasever/centosplus/$basearch/
gpgcheck = 1
enabled = 0
gpgkey = file: /// etc / pki / rpm-gpg / RPM-GPG-KEY-CentOS-7

To:

#additional packages that extend functionality of existing package
[centosplus]
name = CentOS- $ releasever - Plus
mirrorlist = http: //mirrorlist.centos.org/? release = $ releasever & arch = $ basearch & repo = centosplus & infra = $ infra
# baseurl = http: //mirror.centos.org/centos/$releasever/centosplus/$basearch/
gpgcheck = 1
enabled = 1
gpgkey = file: /// etc / pki / rpm-gpg / RPM-GPG-KEY-CentOS-7
includepkgs = kernel *

An "yum update" does not automatically install the patched kernel, you must install it manually:

yum install kernel-plus

After that you can run a reboot, or run the VPS. If you do not perform a reboot, the next reboot (manual or error) will start the new kernel.

I rebooted and now the VPS is broken, help?

You can perform the above actions through our console via the interface ( https://interface.cloudvps.com ). Put the VPS in "Rescue mode". You can also customize your kernel boot configuration (from rescue mode):

vi /boot/grub2/grub.cfg

And remove the relevant kernel configuration:

menu key 'CentOS Linux (3.10.0-693.2.2.el7.centos.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unlimitedricted $ menuentry_id_option 'gnulinux- 3.10.0-123.el7.x86_64-advanced-1c1db3e7-4b32-481f-a520-303b72770c00 '{
    linux16 /boot/vmlinuz-3.10.0-693.2.2.el7.centos.x86_64 root = / dev / xvda1 ro vconsole.keymap = us crashkernel = auto vconsole.font = latarcyrheb-sun16 rhgb quiet LANG = en_US.UTF-8
    initrd16 /boot/initramfs-3.10.0-693.2.2.el7.centos.x86_64.img
}

Please check that there is another kernel in this file, otherwise the VPS can not yet start.

You can also send an email to support@cloudvps.com , after which we can resolve this. We will only solve this if you do not get it yourself, if we see that you have not done anything yourself, we will ask you to do it first. If you do not want to do anything yourself, we are required to count the hourly rate, which is subject to prior authorization including cost indication.

Permanent solution

There is currently no permanent solution available. If available, we will report this here. You can also keep an eye on the upstream bug

More information

More information about the CentOS plus repository can be found here: https://wiki.centos.org/AdditionalResources/Repositories/CentOSPlus

The bug has been reported upstream: https://bugs.centos.org/view.php?id=13763#c30014

Helpcenter

General FAQ

Show all FAQs

OpenStack FAQ

Show all FAQs

Knowledgebase

Show all FAQs