Linux

Auto-reboot Linux after a kernel panic


Linux is a robust and stable operating system kernel, but there are instances where it can panic, be it due to bad hardware or bad software. It does not happen often, but it can happen.

If you're running a server or some other always-on system that you may not have easy access to, a kernel panic typically means an inconvenient trip to reboot a system or a phone call to inconvenience someone else. You can, however, configure Linux to automatically reboot on a kernel panic by making a small modification to /etc/sysctl.conf, a configuration file that tweaks many kernel operating parameters.

Add the following to /etc/sysctl.conf:

kernel.panic = 20

This tells the kernel that if it encounters a panic, it is to reboot the system after a 20 second delay. By default, the kernel will never reboot when it encounters a panic, but with the above setting you can force it to.

Of course, if you enable this, make sure you are using swatch or some other means of observing log files to make sure you aware of the fact when the system panics so you can take appropriate steps to correct the problem.

On local systems, it is also convenient to be able to reboot the system with a key-press in the case of a panic. Instead of having the system reboot automatically on a local system, consider using the magic SysRq keys to reboot your system if X locks up or keyboard entry is being ignored.

To enable magic SysRq support, you must again edit /etc/sysctl.conf; some Linux distributions have this enabled by default whereas others do not.

kernel.sysrq = 1

If the time comes when the SysRq keys are required, use the magic SysRq combination, which is: [ALT]+[SysRq]+[COMMAND], where the [SysRq] key is the "print screen" key and [COMMAND] is one of the following:

  • b - reboot immediately without syncing or unmounting disks
  • e - sends a SIGTERM to all running processes, except for init
  • o - shut down system
  • s - attempt to sync all mounted filesystems
  • u - attempt to remount all mounted filesystems as read-only

These keys need to be pressed together simultaneously to take effect.

Auto-rebooting is great for remote systems, and the magic SysRq combo is very useful for local systems.

About

Vincent Danen works on the Red Hat Security Response Team and lives in Canada. He has been writing about and developing on Linux for over 10 years and is a veteran Mac user.

13 comments
gramacho
gramacho

There's also "kernel.panic_on_oops=5" for a oops kind of kernel panic. I needed this to rebbot my linux automatically and save crash dumps (on /proc/last_log).

lordvan
lordvan

so .. is there some way to enable this in the kernel config somewhere if my kernel locks up while booting after installing a new one (on a remote, rented server) ;)

FXEF
FXEF

For this to reboot an un-attended system, what about entering password?

Systems Guy
Systems Guy

Very useful. However, I can't get kernel.sysrq to work properly. I can sync files and I can shutdown but the "b" option, reboot. Doesn't work. My laptop sits there as though no keys were pressed.

canopic@clear
canopic@clear

Can the need for password on auto-reboot be circumvented in a secure way?

jrdjr
jrdjr

I checked my system and it has no kernel.panic variable in /etc/sysctl.conf but if I run the command sysctl -a and list out the values, kernel.panic is set to 5. What does that mean? I would assume it to be reboot after 5 seconds but I know my system drops to kdb> prompt

iandrei
iandrei

sysctl -w kernel.panic_on_oops=1 this will call panic when there is an oops so...auto-reboot :)

vdanen
vdanen

Password on what? The bootloader? If you're running an unattended system and you have password protected booting, you'd best be really sure on how you implement that. Grub, and lilo too I think, can prompt for a password if you want to change from the defaults (i.e. change which entry to boot, which commands are given to the kernel, etc.). If you require a password to just plain old boot on unattended system, you have more problems than just auto-reboots. How much fun would it be for someone who obtained elevated privileges on your system to issue a reboot command, or a forgetful admin? You'd be stuck at a boot prompt waiting for a password. Talk about a successful DoS.

canopic@clear
canopic@clear

Nobody mentioned grub or lilo or bootloader; we were talking about the logon password. Is it bypassed by an auto-reboot and if so how?