Understanding Service Control with systemd

What is systemd?

Systemd was created by Red Hat's Lennart Poettering and Kay Sievers, provides a standard process for controlling which programs start when a Linux system boots. The system daemaon systemd is the process with id 1. It manages processes and services. Run the top command to see it:

top -p 1

In the output, you can see systemd under the COMMAND column:

top - 21:04:36 up 13:49,  3 users,  load average: 0.00, 0.00, 0.00
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1020484 total,    63668 free,   817432 used,   139384 buff/cache
KiB Swap:   248828 total,   220596 free,    28232 used.    56200 avail Mem 

   PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                              
     1 root      20   0   56960   3156   2252 S  0.0  0.3   0:00.74 systemd 

Press Q to exit.

Where is it?

In my system PID 1 is in /bin/systemd. Run the which command:

which systemd

The output shows you where it is located.

/bin/systemd

It is actually lying to us. To prove it, run the file command on the directory:

file /bin/systemd

The file command determines the file type. The output:

/bin/systemd: symbolic link to /lib/systemd/systemd

shows that the service scripts exist in /lib/system/systemd. It is a sym link.

What's in it?

What is /lib/systemd/systemd ? Use the file command:

file /lib/systemd/systemd

The output:

/lib/systemd/systemd: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=70994edf61ebc37695fb0ed7a82eeb48d41413a0, stripped

shows that it is a shared object. If you list the contents of the directory:

ls -l /lib/systemd
total 5352
-rw-r--r--  1 root root 2217104 Jul  5  2017 libsystemd-shared-232.so
drwxr-xr-x  2 root root    4096 Sep  2  2017 network
-rw-r--r--  1 root root     484 Jul  5  2017 resolv.conf
drwxr-xr-x 19 root root   16384 Sep  2  2017 system
-rwxr-xr-x  1 root root 1141448 Jul  5  2017 systemd
-rwxr-xr-x  1 root root    6400 Jul  5  2017 systemd-ac-power
-rwxr-xr-x  1 root root   18688 Jul  5  2017 systemd-backlight
-rwxr-xr-x  1 root root   14672 Jul  5  2017 systemd-binfmt
-rwxr-xr-x  1 root root   10496 Jul  5  2017 systemd-cgroups-agent
-rwxr-xr-x  1 root root   26888 Jul  5  2017 systemd-cryptsetup
-rwxr-xr-x  1 root root   18696 Jul  5  2017 systemd-fsck
-rwxr-xr-x  1 root root   22864 Jul  5  2017 systemd-fsckd
-rwxr-xr-x  1 root root   10496 Jul  5  2017 systemd-hibernate-resume
-rwxr-xr-x  1 root root   22864 Jul  5  2017 systemd-hostnamed
-rwxr-xr-x  1 root root   14672 Jul  5  2017 systemd-initctl
-rwxr-xr-x  1 root root  117160 Jul  5  2017 systemd-journald
-rwxr-xr-x  1 root root   35152 Jul  5  2017 systemd-localed
-rwxr-xr-x  1 root root  207336 Jul  5  2017 systemd-logind
-rwxr-xr-x  1 root root   14672 Jul  5  2017 systemd-modules-load
-rwxr-xr-x  1 root root  424456 Jul  5  2017 systemd-networkd
-rwxr-xr-x  1 root root   18856 Jul  5  2017 systemd-networkd-wait-online
-rwxr-xr-x  1 root root   10576 Jul  5  2017 systemd-quotacheck
-rwxr-xr-x  1 root root   10496 Jul  5  2017 systemd-random-seed
-rwxr-xr-x  1 root root   14672 Jul  5  2017 systemd-remount-fs
-rwxr-xr-x  1 root root   10496 Jul  5  2017 systemd-reply-password
-rwxr-xr-x  1 root root  321952 Jul  5  2017 systemd-resolved
-rwxr-xr-x  1 root root   18768 Jul  5  2017 systemd-rfkill
-rwxr-xr-x  1 root root   39328 Jul  5  2017 systemd-shutdown
-rwxr-xr-x  1 root root   14672 Jul  5  2017 systemd-sleep
-rwxr-xr-x  1 root root   22864 Jul  5  2017 systemd-socket-proxyd
-rwxr-xr-x  1 root root   14752 Jul  5  2017 systemd-sysctl
-rwxr-xr-x  1 root root    1324 Jul  5  2017 systemd-sysv-install
-rwxr-xr-x  1 root root   26960 Jul  5  2017 systemd-timedated
-rwxr-xr-x  1 root root   39248 Jul  5  2017 systemd-timesyncd
-rwxr-xr-x  1 root root  465680 Jul  5  2017 systemd-udevd
-rwxr-xr-x  1 root root   14672 Jul  5  2017 systemd-update-utmp
-rwxr-xr-x  1 root root   10496 Jul  5  2017 systemd-user-sessions
drwxr-xr-x  2 root root    4096 Sep  2  2017 system-generators
drwxr-xr-x  2 root root    4096 Sep  2  2017 system-preset
drwxr-xr-x  2 root root    4096 Jul  5  2017 system-shutdown
drwxr-xr-x  2 root root    4096 Jul  5  2017 system-sleep

You can see the .so file. It is a shared object (dynamic) library similar to DLL on Windows.

Analyze Startup Time

You can display the system startup time using systemd-analyze:

systemd-analyze
Startup finished in 35.185s (kernel) + 2.291s (userspace) = 37.477s

You can see how long the kernel took to startup and the userspace programs took to startup. You can use blame option to get details about which service took how much time during startup:

pollywog@kafka:/src/messenger/rafk$ systemd-analyze blame
          1.166s zookeeper.service
           878ms vboxadd.service
           746ms keyboard-setup.service
           459ms vboxadd-x11.service
           267ms dev-sda2.device
           243ms networking.service
           172ms kafka.service
            93ms vboxadd-service.service
            84ms ssh.service
            83ms systemd-udev-trigger.service
            57ms systemd-journald.service
            43ms systemd-user-sessions.service
            40ms systemd-logind.service
            37ms rsyslog.service
            34ms systemd-udevd.service
            31ms systemd-tmpfiles-setup-dev.service
            28ms systemd-remount-fs.service
            28ms dev-disk-by\x2dlabel-SWAP.swap
            24ms systemd-sysctl.service
            24ms console-setup.service
            22ms dev-mqueue.mount
            22ms dev-hugepages.mount
            22ms systemd-random-seed.service
            21ms sys-kernel-debug.mount
            19ms systemd-update-utmp-runlevel.service
            18ms systemd-tmpfiles-setup.service
            16ms systemd-update-utmp.service
            15ms kmod-static-nodes.service
            14ms systemd-modules-load.service
            14ms systemd-journal-flush.service
            14ms user@1000.service
             7ms systemd-tmpfiles-clean.service

You can use this data to tune performance. Do we need these services? Can we configure these services to startup quickly?

Targets Not Runlevels

Unlike init which uses runlevels, the systemd uses targets. The run level of the VM created by Vagrant on my Mac is 5:

who -r
         run-level 5  2019-02-04 01:12

The level 5 means: Start the system normally with GUI display manager. This corresponds to the target graphical.target in systemd. We can run the systemctl command to verify:

systemctl get-default

The systemctl is a command used to control the systemd system and service manager. The output:

graphical.target

You can change the default target using the systemctl command:

sudo systemctl set-default multi-user.target
Created symlink /etc/systemd/system/default.target → /lib/systemd/system/multi-user.target.

It created a symlink to change the default target. We can now read the default target:

pollywog@kafka:/src/messenger/rafk$ systemctl get-default
multi-user.target

We can change the runlevel target

sudo systemctl isolate multi-user.target

We are now running at the runlevel 3. We can verify it:

pollywog@kafka:/src/messenger/rafk$ who -r
         run-level 3  2019-02-07 22:03                   last=5

I was in run-level 5 before, now I am at run-level 3. The target units represent the run levels.

Service Units

The service units represent the services. We can enable a service to start at system boot by using systemctl command. For example, we can enable sshd to run on bootup:

systemctl enable sshd

If we were doing it manually, we use start:

systemctl start sshd

We can check the status of a process:

pollywog@kafka:/src/messenger/rafk$ systemctl status sshd

Note that we are not providing the .service extendion to sshd. The output:

● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-02-04 01:12:36 WET; 3 days ago
 Main PID: 525 (sshd)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/ssh.service
           └─525 /usr/sbin/sshd -D

shows that the sshd is running and is active. We can also see it is enabled. To learn more about system unit, run:

man systemd.unit

We can view all properties of a service by running:

systemctl show sshd

The output:

Type=notify
Restart=on-failure

here shows only few to illustrate what happens when the sshd crashes. It is automatically restarted on failure. To list all units:

systemctl -a

This produces lot of output. You can also do:

systemctl list-unit-files

This will produce output that can be hours of fun for the entire family.

TODO : CREATE A SERVICE FILE THAT STARTS A KAFKA CONSUMER

References

Ubuntu Startup Scripts - Run Level Explained


Related Articles