Carlos Mavros

Processes, Jobs & Services

Course 25 min read

You SSH into a compute cluster, start a 6-hour FEM simulation, and close your laptop. Next morning the results are gone. Why? Your session ended, and the process died with it.

What Is a Process?

Every running program is a process. Each process has a PID (process ID), an owner, a parent process, and allocated resources (memory, CPU time, open files).

ps aux
ps aux | grep python
pgrep -l python

top and htop

top
htop

htop is an interactive process viewer with a much better interface than top:

  • Arrow keys to navigate
  • F9 to kill a process
  • F6 to sort by column
  • q to quit

Tip: On a shared cluster, always check htop before launching heavy jobs. You’ll see how much CPU and memory is already in use and avoid overloading the machine.

Killing Processes

kill 12345          # SIGTERM (graceful shutdown)
kill -9 12345       # SIGKILL (force kill, no cleanup)
pkill python
pkill -f "train.py"

Background & Foreground Jobs

python simulate.py       # Runs in the foreground (blocks your terminal)
Ctrl+C                   # Kill the foreground process
Ctrl+Z                   # Suspend (pause) the foreground process
bg                       # Resume the suspended process in the background
fg                       # Bring a background process to the foreground
jobs                     # List all jobs in the current session
python simulate.py &     # Start a process directly in the background

nohup — Survive Logout

When you close your SSH session, the shell sends SIGHUP to all its child processes, killing them. nohup makes your process immune to this signal.

nohup python simulate.py > output.log 2>&1 &

Check on it later:

tail -f output.log
cat output.log
pgrep -fl simulate

Key takeaway: For long-running jobs on any remote server, always use nohup ... & or a proper job scheduler. Otherwise, closing your laptop or losing your connection will kill your work.

tmux — The Professional Solution

tmux creates persistent terminal sessions that survive disconnections. It’s the professional way to manage long-running work on remote servers.

tmux new -s simulation
# run your simulation inside the tmux session
Ctrl+B, D               # Detach from the session (it keeps running)
tmux ls                  # List active sessions
tmux attach -t simulation  # Reattach to the session

Daemons and Services with systemctl

System services (daemons) are background processes managed by systemd. You interact with them using systemctl.

systemctl status docker
systemctl start nginx
systemctl stop postgresql
systemctl restart myapp
systemctl enable docker      # Start automatically on boot
systemctl disable myapp      # Don't start on boot

Use journalctl to read service logs:

journalctl -u docker
journalctl -u nginx -n 50
journalctl -u myapp -f
journalctl -u myapp --since "1 hour ago"

Tip: When a service fails to start or crashes, two commands tell you everything: systemctl status servicename and journalctl -u servicename. Check both before searching the internet.

Try It Yourself

Run a long job in the background and manage it.

First, create a Python script called slow_sim.py:

import time

for i in range(1, 11):
    print(f"Step {i}/10 complete")
    time.sleep(3)

print("Simulation finished!")

Now practice managing it:

  1. Run it with nohup: 
    nohup python slow_sim.py > sim_output.log 2>&1 &
  2. Check that it’s running: pgrep -fl slow_sim
  3. Watch the output live: tail -f sim_output.log (press Ctrl+C to stop watching)
  4. Kill the process: pkill -f slow_sim
  5. Verify it’s gone: pgrep -fl slow_sim

Quick Quiz

You run python run.py & on a remote server and then close your SSH session. What happens to the process?

  • A) It keeps running in the background.
  • B) It pauses and resumes when you reconnect.
  • C) It gets terminated by a SIGHUP signal.
  • D) It moves to a system service automatically.
Answer

C) It gets terminated by a SIGHUP signal. When you close your SSH session, the shell sends SIGHUP (hangup signal) to all child processes. The & only puts the process in the background — it does not protect it from session termination. Use nohup or tmux to keep processes running after logout.