Press the power button on your system, and after few moments you see the Linux login prompt. Show Have you ever wondered what happens behind the scenes from the time you press the power button until the Linux login prompt appears? The following are the 6 high level stages of a typical Linux boot process. 1. BIOS
2. MBR
3. GRUB
4. Kernel
5. Init
6. Runlevel programs
There you have it. That is what happens during the Linux boot process.
Every time you start your Linux computer, its initialization system launches a batch
of processes, from a few dozen to hundreds, depending on how the system is set up.
You can see this on your startup screen (Figure 4-1; press
the Escape key to hide your graphical startup screen and see the startup messages).
In olden times we had the Unix System V initialization system (SysV init), BSD init, and Linux Standard Base (LSB) init for launching processes at startup. SysV init was the most common. Those days are fading away, and now systemd is the shiny new init system for Linux. It has been adopted by all the major Linux distributions, though of course there are a number of distributions that still use the legacy init systems. In this chapter you will learn if your Linux distribution uses systemd. You will learn what processes, threads, services, and daemons are, and how to use systemd to manage services: start, stop, enable, disable, and check status. You will become acquainted with the systemctl command, which is the systemd system and service manager. systemd is designed to provide functionality suited to modern complex server and desktop systems, and it does considerably more than the legacy init systems. It provides complete service management from startup to shutdown, starting processes at boot, on-demand after boot, and shutting down services when they are not needed. It manages functions such as system logging, automounting filesystems, automatic service dependency resolution, name services, device management, network connection management, login management, and a host of other tasks. This sounds like a lot until you realize that processes do everything on a computer, and all of this functionality used to be provided by a large assortment of other programs. systemd brings it all together in an integrated software suite that should operate the same way on all Linux systems, though as always with Linux there are some minor exceptions, such as file locations and service names. Be aware that your particular Linux may have some differences from the examples in this chapter. systemd attempts to decrease boot times and parcel out system resources more efficiently by starting processes concurrently and in parallel, and starting only necessary services, leaving other services to start after boot as needed. A service that is dependent on other services no longer has to wait to start for those services to become available because all it needs is a waiting Unix socket to become available. Recipe 4.9 shows how to find processes that are slowing down your system startup. systemd binaries are written in C, which provides some performance enhancement. The legacy inits are masses of shell scripts, and any compiled language operates faster than shell scripts. systemd is backwards compatible with SysV init. Most Linux distributions retain the legacy SysV configuration files and scripts, including /etc/inittab and the /etc/rc.d/ and /etc/init.d/ directories. When a service does not have a systemd configuration file, systemd looks for a SysV configuration file. systemd is also backward compatible with Linux Standard Base (LSB) init. systemd service files are smaller and easier to understand than SysV init files. Compare a SysV init file for sshd with its systemd service file. This is a snippet of the sshd init file, /etc/init.d/ssh, from MX Linux: #! /bin/sh ### BEGIN INIT INFO # Provides: sshd # Required-Start: $remote_fs $syslog # Required-Stop: $remote_fs $syslog # Default-Start: 2 3 4 5 # Default-Stop: # Short-Description: OpenBSD Secure Shell server ### END INIT INFO set -e # /etc/init.d/ssh: start and stop the OpenBSD "secure shell(tm)" daemon test -x /usr/sbin/sshd || exit 0 umask 022 if test -f /etc/default/ssh; then [...]This goes on for a total of 162 lines. This is a complete systemd service file from Ubuntu 20.04, /lib/systemd/system/ssh.service: [Unit] Description=OpenBSD Secure Shell server Documentation=man:sshd(8) man:sshd_config(5) After=network.target auditd.service ConditionPathExists=!/etc/ssh/sshd_not_to_be_run [Service] EnvironmentFile=-/etc/default/ssh ExecStartPre=/usr/sbin/sshd -t ExecStart=/usr/sbin/sshd -D $SSHD_OPTS ExecReload=/usr/sbin/sshd -t ExecReload=/bin/kill -HUP $MAINPID KillMode=process Restart=on-failure RestartPreventExitStatus=255 Type=notify RuntimeDirectory=sshd RuntimeDirectoryMode=0755 [Install] WantedBy=multi-user.target Alias=sshd.serviceEven without reading the documentation, or knowing anything about systemd, you can understand some of what this file is supposed to do. See Rethinking PID 1 for a detailed introduction to systemd by one of its inventors and maintainers, Lennart Poettering. Rethinking PID 1 details the rationale behind building a new init system, its architecture, advantages, and how it uses existing Linux kernel features in place of duplicating existing functionality. You need to know if your Linux distribution uses systemd or something else.
Look for the /run/systemd/system/ directory. If this exists, then your init system is systemd.
The /run/systemd/ directory may be present on your system if your distribution supports multiple init systems. But systemd is not the active init unless you see /run/systemd/system/. There are several other ways to learn which init system your system is using. Try querying /sbin/init. Originally this was the SysV executable, and now most Linux distributions preserve the name and symlink it to the systemd executable. This example confirms that the init is systemd: $ stat /sbin/init File: /sbin/init -> /lib/systemd/systemd [...]On a system using SysV init, it has no symlink: $ stat /sbin/init File: /sbin/init [...]The /proc pseudofilesystem is an interface to your Linux kernel, and contains the current state of a running system. It is called a pseudofilesystem because it exists only in memory and not on disk. In this example, /proc/1/exe is symlinked to the systemd executable: $ sudo stat /proc/1/exe File: /proc/1/exe -> /lib/systemd/systemd [...]On a SysV system, it links to init: $ sudo stat /proc/1/exe File: /proc/1/exe -> /sbin/init [...]The /proc/1/comm file reports your active init system: $ cat /proc/1/comm systemdOn a SysV system, it reports init: $ cat /proc/1/comm initThe command attached to process ID (PID) 1 is your init. PID 1 is the first process launched at startup, which then starts all other processes. You can see this with the ps command: $ ps -p 1 PID TTY TIME CMD 1 ? 00:00:00 systemdWhen the init is SysV, it looks like this: $ ps -p 1 PID TTY TIME CMD 1 ? 00:00:00 initSee Recipe 4.2 for more information on PID 1. Linux support for systemd varies. Most of the major Linux distributions have adoped systemd, including Fedora, Red Hat, CentOS, openSUSE, SUSE Linux Enterprise, Debian, Ubuntu, Linux Mint, Arch, Manjaro, Elementary, and Mageia Linux. Some popular distributions that do not support systemd, or include it but not as the default init, are Slackware, PCLinuxOS, Gentoo Linux, MX Linux, and antiX.
You want a better understanding of services and processes on Linux.
PID 1 is the mother of all processes on Linux systems. This is the first process to start, and then it launches all other processes. Processes are one or more running instances of a program. Every task in a Linux system is performed by a process. Processes can create independent copies of themselves, that is, they can fork. The forked copies are called children, and the original is the parent. Each child has its own unique PID, and its own allocation of system resources, such as CPU and memory. Threads are lightweight processes that run in parallel and share system resources with their parents. Some processes run in the background and do not interact with users. Linux calls these processes services or daemons, and their names tend to end with the letter D, such as httpd, sshd, and systemd. Every Linux system starts PID 1 first, which then launches all other processes. Use the ps command to list all running processes in PID order: $ ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 10:06 ? 00:00:01 /sbin/init splash root 2 0 0 10:06 ? 00:00:00 [kthreadd] root 3 2 0 10:06 ? 00:00:00 [rcu_gp] root 4 2 0 10:06 ? 00:00:00 [rcu_par_gp] [...]The pstree command organizes this mass of information into a tree diagram. This example shows all processes, their child processes, PIDs, and threads, which are enclosed in curly braces: $ pstree -p systemd(1)─┬─ModemManager(925)─┬─{ModemManager}(944) │ └─{ModemManager}(949) ├─NetworkManager(950)─┬─dhclient(1981) │ ├─{NetworkManager}(989) │ └─{NetworkManager}(991) ├─accounts-daemon(927)─┬─{accounts-daemon}(938) │ └─{accounts-daemon}(948) ├─acpid(934) ├─agetty(1103) ├─avahi-daemon(953)───avahi-daemon(970) [...]The full pstree output is quite large. You can view a single process, identified by its PID, and its parents, children, and threads, like the following example for the Kate text editor: $ pstree -sp 5193 systemd(1)───kate(5193)─┬─bash(5218) ├─{kate}(5195) ├─{kate}(5196) ├─{kate}(5197) ├─{kate}(5198) ├─{kate}(5199) [...]This shows that systemd(1) is Kate’s parent, bash(5218) is Kate’s child, and all the processes in curly braces are Kate’s threads.
Processes always exist in one of several states, and these states change according to system activity. The following pstree example displays the PID, user, state, and command fields: $ ps -eo pid,user,stat,comm PID USER STAT COMMAND 1 root Ss systemd 2 root S kthreadd 32 root I< kworker/3:0H-kb 68 root SN khugepaged 11222 duchess Rl konsole
There are several rarely used states you can read about in man 1 ps.
You want to list all services installed on your system, and you want to know
the states of the services: whether they are running, not running, or in an
error state.
systemctl, the systemd manager command, tells all. Run it with no options to see a detailed list of all loaded units. A systemd unit is any related batch of processes defined in a unit configuration file and managed by systemd: $ systemctlThis prints a giant pile of information: 177 active loaded units on my test system with the full unit names, status, and long descriptions. Redirect the the output to a text file for easier study: $ systemctl > /tmp/systemctl-units.txtTreat yourself to more information overload by listing all units, active and inactive: $ systemctl --allThis results in 349 loaded units listed on my test system, including not-found and inactive units. How many total unit files? The following example shows 5 out of 322: $ systemctl list-unit-files UNIT FILE STATE proc-sys-fs-binfmt_misc.automount static -.mount generated mount generated dev-hugepages.mount static home.mount generated [...] 322 unit files listed.We are interested in service files because Linux users and administrators interact mainly with service files and rarely need to bother with any other type of unit file. How many are installed? Let’s see: $ systemctl list-unit-files --type=service UNIT FILE STATE accounts-daemon.service enabled acpid.service disabled alsa-state.service static alsa-utils.service masked anacron.service enabled [...] 212 unit files listed.The preceding example displays the four most common states that a service can be in: enabled, disabled, static, or masked. List only enabled services: $ systemctl list-unit-files --type=service --state=enabled UNIT FILE STATE accounts-daemon.service enabled anacron.service enabled apparmor.service enabled [email protected] enabled avahi-daemon.service enabled [...] 62 unit files listed.List only disabled services: $ systemctl list-unit-files --type=service --state=disabled UNIT FILE STATE acpid.service disabled brltty.service disabled console-getty.service disabled [email protected] disabled [...] 12 unit files listed.List only static services: $ systemctl list-unit-files --type=service --state=static UNIT FILE STATE alsa-restore.service static alsa-state.service static apt-daily-upgrade.service static apt-daily.service static [...] 106 unit files listed.List only masked services: $ systemctl list-unit-files --type=service --state=masked UNIT FILE STATE alsa-utils.service masked bootlogd.service masked bootlogs.service masked checkfs.service masked [...] 36 unit files listed.
Service unit files are in /usr/lib/systemd/system/ or /lib/systemd/system/, according to where your Linux distribution puts them. These are plain-text files you can read. enabled
This shows that the service has become available and is managed by systemd. When a service is enabled, systemd creates a symlink in /etc/systemd/system/ from the unit file in /lib/systemd/system/. It can be started, stopped, reloaded, and disabled by the user with the systemctl command.
Enabling a service does not immediately start it, and disabling a service does not immediately stop it (see Recipe 4.6). disabledDiabled means that there is no symlink in /etc/systemd/system/, and it will not start automatically at boot. You can stop and start it manually. maskedThis means the service is linked to /dev/null/. It is completely disabled and cannot be started by any means. staticThis means that the unit file is a dependency of other unit files, and cannot be started or stopped by the user. Some less-common service states you will see: Indirect states belong to services that are not meant to be managed by users, but are meant to be used by other services. generatedGenerated states indicate that the service has been converted from a nonnative systemd initialization configuration file, either SysV or LSB init. You want to know the status of one service or a few specific services.
systemctl status provides a nice little bundle of useful status information. The following example queries the CUPS service. CUPS, the Common Unix Printing System, should be on all Linux systems: $ systemctl status cups.service ● cups.service - CUPS Scheduler Loaded: loaded (/lib/systemd/system/cups.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2021-11-22 11:01:48 PST; 4h 17min ago TriggeredBy: ● cups.path ● cups.socket Docs: man:cupsd(8) Main PID: 1403 (cupsd) Tasks: 2 (limit: 18760) Memory: 3.8M CGroup: /system.slice/cups.service ├─1403 /usr/sbin/cupsd -l └─1421 /usr/lib/cups/notifier/dbus dbus:// Nov 22 11:01:48 host1 systemd[1]: Started CUPS Scheduler.Query multiple services with a space-delimited list: $ systemctl status mariadb.service bluetooth.service lm-sensors.service
There is a lot of useful information in this little bit of output (Figure 4-2).
The dot next to the service name is a quick status indicator. It appears in colors on most terminals. White is an inactive or deactivating state. Red is a failed or error state. Green indicates an active, reloading, or activating state. The rest of the information in the output is described in the following: LoadedVerifies that the unit file has been loaded into memory, displays its full path, the service is enabled (see the Discussion about states in Recipe 4.3), and vendor preset: disabled/enabled indicates if the installation default is to start at boot or not. When it is disabled, the vendor default is to not start at boot. This only shows the vendor preference and does not indicate if it is currently enabled or disabled. ActiveTells you if the service is active or inactive, and how long it has been in that state. ProcessReports the PIDs and their commands and daemons. Main PIDThis is the process number for the cgroup slice. TasksReports how many tasks the service has started. Tasks are PIDs. CGroup
Shows which unit slice the service belongs to and its PID. The three default unit slices are user.slice, system.slice, and machine.slice. Linux control groups (cgroups) are sets of related processes and all of their future children. In systemd, a slice is a subdivision of a cgroup, and each slice manages a particular group of processes. Run systemctl status to see a diagram of the cgroup hierarchy. By default, service and scope units are grouped in /lib/systemd/system/system.slice. User sessions are grouped in /lib/systemd/system/user.slice. Virtual machines and containers registered with systemd are grouped in /lib/systemd/system/machine.slice. The remaining lines are the most recent log entries from journalctl, the systemd log manager. You want to stop and start services with systemd.
This is a job for systemctl. The following examples use the SSH service to demonstrate service management. Start a service: $ sudo systemctl start sshd.serviceStop a service: $ sudo systemctl stop sshd.serviceStop and then restart a service: $ sudo systemctl restart sshd.serviceReload the service’s configuration. For example, you made a change to sshd_config and want to load the new configuration without restarting the service: $ sudo systemctl reload sshd.service
All of these commands also work with multiple services, space-delimited, for example: $ sudo systemctl start sshd.service mariadb.service firewalld.serviceIf you’re curious about the commands that systemd runs behind the scenes to start, reload, or stop the individual daemons, look in their unit files. Some services have start, reload, stop, and other instructions in their unit files, like this example for httpd: ExecStart=/usr/sbin/httpd/ $OPTIONS -DFOREGROUND ExecReload=/usr/sbin/httpd $OPTIONS -k graceful ExecStop=/bin/kill -WINCH ${MAINPID}You don’t have to do anything special with this information; it is there when you want to know how systemctl is managing a particular service.
You want a service or services to automatically start at boot, or you want to
prevent a service from starting at boot, or to disable it completely.
Enabling a service configures it to automatically start at boot. Disabling a service stops it from starting at boot, but it can be started and stopped manually. Masking a service disables it so that it cannot be started at all. The following example enables the sshd service: $ sudo systemctl enable sshd.service Created symlink /etc/systemd/system/multi-user.target.wants/sshd.service → /usr/lib/systemd/system/sshd.serviceThe output shows that enabling a service means creating a symlink from the service file in /lib/systemd/system/ to /etc/systemd/system/. This does not start the service. You can start the service with systemctl start, or enable and start the service in one command with the --now option: $ sudo systemctl enable --now sshd.serviceThis command disables the sshd service. It does not stop the service, so you must stop it manually after disabling it: $ sudo systemctl disable sshd.service Removed /etc/systemd/system/multi-user.target.wants/sshd.service $ sudo systemctl stop sshd.serviceOr, disable and stop it with one command: $ sudo systemctl disable --now sshd.serviceThis command reenables the mariadb service, which disables and then enables it. If you have created the symlinks manually for a service, this is useful for quickly resetting them to the defaults: $ sudo systemctl reenable mariadb.service Removed /etc/systemd/system/multi-user.target.wants/mariadb.service. Removed /etc/systemd/system/mysqld.service. Removed /etc/systemd/system/mysql.service. Created symlink /etc/systemd/system/mysql.service → /lib/systemd/system/mariadb.service. Created symlink /etc/systemd/system/mysqld.service → /lib/systemd/system/mariadb.service. Created symlink /etc/systemd/system/multi-user.target.wants/mariadb.service → /lib/systemd/system/mariadb.service.The following command disables the bluetooth service completely by masking it, so that it cannot be started at all: $ sudo systemctl mask bluetooth.service Created symlink /etc/systemd/system/bluetooth.service → /dev/null.Unmasking the bluetooth service does not enable it, so it must be started manually: $ sudo systemctl unmask bluetooth.service Removed /etc/systemd/system/bluetooth.service. $ sudo systemctl start bluetooth.service
When you enable, disable, mask, or unmask a service, it remains in its current state unless you use the --now option. The --now option works with enable, disable, and mask to immediately stop or start the service, but it does not work with unmask. See the Discussion in Recipe 4.3 to learn more about how systemd uses symlinks to manage services.
You want to know how to stop troublesome processes. A certain service may be
unresponsive or running away, spawning forks and causing your system to hang.
Your normal stop command is not working. What do you do?
Stopping a process is called killing the process. On Linux systems with systemd, you should use systemctl kill. On systems without systemd, use the legacy kill command. systemctl kill is preferable because it stops all processes that belong to a service and leaves no orphan processes, nor any processes that might restart the service and continue to make trouble. First, try it with no options other than the service name, then check the status: $ sudo systemctl kill mariadb $ systemctl status mariadb ● mariadb.service - MariaDB 10.1.44 database server Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled) Active: inactive (dead) since Sun 2020-06-28 19:57:49 PDT; 6s ago [...]The service has cleanly stopped. If this does not work, then try the nuclear option: $ sudo systemctl kill -9 mariadbThe legacy kill command does not recognize service or command names, but rather requires the PID of the offending process: $ sudo kill 1234If this does not stop it, use the nuclear option: $ sudo kill -9 1234
Use the top command to identify runaway processes. Run it with no options, and the processes using up the most CPU resources are listed at the top. Press the q key to exit top. $ top top - 20:30:13 up 4:24, 6 users, load average: 0.00, 0.03, 0.06 Tasks: 246 total, 1 running, 170 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.4 us, 0.2 sy, 0.0 ni, 99.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 16071016 total, 7295284 free, 1911276 used, 6864456 buff/cache KiB Swap: 8928604 total, 8928604 free, 0 used. 13505600 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3504 madmax 20 0 99.844g 177588 88712 S 2.6 1.1 0:08.68 evolution 2081 madmax 20 0 3818636 517756 177744 S 0.7 3.2 5:07.56 firefox 1064 root 20 0 567244 148432 125572 S 0.3 0.9 12:54.75 Xorg 2362 stash 20 0 2997732 230508 145444 S 0.3 1.4 0:40.72 Web Content [...]kill sends signals to processes, and the default signal is SIGTERM (signal terminate). SIGTERM is gentle, allowing processes to shut down cleanly. SIGTERM is also ignorable, and processes don’t have to pay attention to it. Signals can be identified by name or number; for most folks the numbers are easier to remember, so spelling out the default looks like this: $ sudo kill -1 1234kill -9 is SIGKILL. SIGKILL stops processes immediately and uncleanly, and also attempts to stop all child processes. Killing services with systemctl kill is easier than with kill, and more reliable. You only need the service name, and you don’t have to hunt down PIDs. It ensures that all processes belonging to the service are stopped, which kill cannot ensure. There are a ton of signals that have accumulated over the years, and you can read all about them in man 7 signal. In my experience, the most relevant signals are SIGTERM and SIGKILL, but don’t let that stop you from learning more about the others. If you are uncomfortable with terminology like kill, parents, children, and orphans, so am I. Maybe someday it will change.
You want to reboot to different system states in a manner similar to using SysV runlevels.
systemd targets are similar to SysV runlevels. These are boot profiles that start your system with different options, such as multiuser mode with a graphical desktop, multiuser mode with no graphical desktop, and emergency and rescue modes to use when your current target will not boot. (See the Discussion for more information on runlevels.) The following command checks if the system is running and reports its state: $ systemctl is-system-running runningWhat is the default target? Get the current runlevel: $ runlevel N 5Reboot to rescue mode: $ sudo systemctl rescueReboot to emergency mode: $ sudo systemctl emergencyReboot to the default mode: $ sudo systemctl rebootReboot to a different target without changing the default: $ sudo systemctl isolate multi-user.targetSet a different default runlevel: $ sudo systemctl set-default multi-user.targetList the runlevel target files and their symlinks on your system: $ ls -l /lib/systemd/system/runlevel*List the dependencies in a runlevel target: $ systemctl list-dependencies graphical.target
SysV runlevels are different states that your system can boot to, for example, with a graphical desktop, without a graphical desktop, and with emergency runlevels to use when your default runlevel has problems and will not boot. systemd targets approximately correspond to the legacy SysV runlevels:
systemctl emergency is a special target that is more restricted than rescue mode: no services, no mount points other than the root filesystem, no networking, root user only. It is the most minimal running system for debugging problems. You may see options to boot into a rescue or emergency mode in your GRUB2 bootloader screen. systemctl is-system-running reports various system states:
systemd promises faster startups, but your system starts up slowly, and you
want to find out why.
You want systemd-analyze blame. Run it with no options to see a list of system processes and how long they took to start: $ systemd-analyze blame 34.590s apt-daily.service 6.782s NetworkManager-wait-online.service 6.181s dev-sda2.device 4.444s systemd-journal-flush.service 3.609s udisks2.service 2.450s snapd.service [...]Analyze only user processes: $ systemd-analyze blame --user 3.991s pulseaudio.service 553ms at-spi-dbus-bus.service 380ms evolution-calendar-factory.service 331ms evolution-addressbook-factory.service 280ms xfce4-notifyd.service [...]
It is useful to review everything that starts at boot and perhaps find services you don’t want starting at boot. My favorite to disable is Bluetooth because I don’t use it on my servers or PCs, but many Linux distros enable it by default. Get Linux Cookbook, 2nd Edition now with the O’Reilly learning platform. O’Reilly members experience live online training, plus books, videos, and digital content from nearly 200 publishers. |