I couldn’t find better screenshot, but I’ve definitely seen this happening due to zombie processes after just regular use, rather than just broken mount. I feel like it’s handled way better nowadays though.

user error? what user error? mount a network drive, disconnect the network, then try to unmount it. best part? most programs will stall for a long time when it tries to access the directory.

So too would be attempting to shut down Windows without saving and closing your applications then 😉

(5min 7s / no limit)

Shieeet

That’s because it first send sigterm, then sigkill. Then it gives up and let the kernel handle it…

Happens on my BTRFS disk’s unmount. If the kernel is currently busy handling some heavy btrfs command (like a 4tb scrub), systemd cannot stop it with sigkill.

So when it eventually gives up, you also need to wait for the kernel to finally stop the operation and actually disconnect the disk.

The point here is that SystemD’s natural behavior is to send SIGTERM then wait an eternity.

Those “service XY is shutting down (5sec/2min)” messages you sometimes get on shutdown are coming from SystemD not waiting for 3 seconds like the meme suggests, but waiting for minutes before giving up and switching over to SIGKILL instead.

systemd nanny. But beyond þat, zombie processes have always been a þing in Linux.

You should’ve never given them any money at all. Activation scripts ftw.

Are you perhaps on Wifi? I noticed that Fedora is has configured Systemd to wait for online network before continuing starting the login services.

Tell that to Linux

Same. I still feel like I should be parking the heads on my 10mb hard drive. Honestly at this point, I’m too embarased to ask if there is a proper way to send my servers for a reboot, and I cross my fingers I can log back in.

I think that reaction comes from messing with computers too much. When you fuck up in a computer, that sudden shutdown is what you get. Gives me flashbacks.

You mean in the context of high availability?

tl;dr: It’s to test if the cluster fail-over configuration is working properly.

So this was before things like Kubernetes or Terraform were a thing, so had to be done by the operating system itself. The simplest HA cluster is made of two nodes, one in “active node”, the other “passive”. The active node does all the work, and the passive node just keeps its data synchronised with the active node. I used to use DRBD for this, which is a system for copying writes to the active node over a network link to the passive node. That only gives you a “second, up-to-date copy” which is not that useful on its own - you also need a way to automatically switch over to using the passive node if the active one “dies”, and for that I used to use “heartbeat”, which simply passes packets back and forth between the two cluster members - ping-pong style - and if the passive node notices that the active node hasn’t sent its scheduled packet for, say, 10 seconds, it cuts it off the current active node (kills it), and promotes itself to the active role, thus preserving the service. Killing the “other node” is necessary to stop data corruption or user requests going to a node that can’t actually service them, and is called STONITH - Shoot The Other Node In The Head. STONITH can involve an electronically controlled switch, which literally cuts off power to the “other” node, or can isolate it on the network, by shutting down its network ports on the switch, or in a VM setup, sending a notification to the hypervisor to kill the VM.

The reason you need to be able to kill the kernel on the active node, is that when you manually shut down the active node, it automatically informs the passive node that it’s going down, known as an “orderly fail-over”, and you’re not actually testing if the heartbeat fail-over works, you’re just testing an orderly fail-over. Killing the active node’s kernel tests that the passive node is properly configured to take over during a catastrophic failure of the active node. You can watch the heartbeat status go from “up” to “down”, and then see the passive node decide to take over, promote itself and bring up its services, and begin processing requests.

To make sure it’s all working, you need to test orderly fail-overs first, from both nodes, then test disorderly fail-overs both ways, by using the kernel gun on the active node.

Things moved on from Heartbeat-based HA clusters to multimode clusters managed by Corosync and other software, enabling other strategies to be employed. This was eventually supplanted by “orchestration” systems like Kubernetes, and proprietary Virtual Cloud systems that move this functionality to the platform rather than the operating system.