What kill level Cannot be trapped in Linux?

1) If you are on the console and in multi-user mode, you could press CTRL-ALT-Fn and login on another screen, use ps -ef | grep <myprocessname> or pidof <myprocessname> and then kill -9 the process by ID number.

2) If you are connected remotely, do the same via another terminal session.

You could also make life a little easier by installing htop, which is a more versatile version of top that allows you to selectively kill running processes. Most distros have htop in a repo.

3) if you are just stuck in a hung ssh session (to another system, for example), try pressing tilde (~), which is the escape key, and then press CTRL-Z to drop back to the host session, then you can kill the stuck ssh process or wait for it to timeout, which most sill do after a period of inactivity.

kill -9 (SIGKILL) always works, provided you have the permission to kill the process. Basically either the process must be started by you and not be setuid or setgid, or you must be root. There is one exception: even root cannot send a fatal signal to PID 1 (the init process).

However kill -9 is not guaranteed to work immediately. All signals, including SIGKILL, are delivered asynchronously: the kernel may take its time to deliver them. Usually, delivering a signal takes at most a few microseconds, just the time it takes for the target to get a time slice. However, if the target has blocked the signal, the signal will be queued until the target unblocks it.

Normally, processes cannot block SIGKILL. But kernel code can, and processes execute kernel code when they call system calls. Kernel code blocks all signals when interrupting the system call would result in a badly formed data structure somewhere in the kernel, or more generally in some kernel invariant being violated. So if (due to a bug or misdesign) a system call blocks indefinitely, there may effectively be no way to kill the process. (But the process will be killed if it ever completes the system call.)

A process blocked in a system call is in uninterruptible sleep. The ps or top command will (on most unices) show it in state D (originally for “disk”, I think).

A classical case of long uninterruptible sleep is processes accessing files over NFS when the server is not responding; modern implementations tend not to impose uninterruptible sleep (e.g. under Linux, the intr mount option allows a signal to interrupt NFS file accesses).

If a process remains in uninterruptible sleep for a long time, you can get information about what it's doing by attaching a debugger to it, by running a diagnostic tool such as strace or dtrace (or similar tools, depending on your unix flavor), or with other diagnostic mechanisms such as /proc/PID/syscall under Linux. See Can't kill wget process with `kill -9` for more discussion of how to investigate a process in uninterruptible sleep.

You may sometimes see entries marked Z (or H under Linux, I don't know what the distinction is) in the ps or top output. These are technically not processes, they are zombie processes, which are nothing more than an entry in the process table, kept around so that the parent process can be notified of the death of its child. They will go away when the parent process pays attention (or dies).