Debian Administration

From Training Material
Revision as of 12:17, 3 September 2021 by Lsokolowski (talk | contribs) (→‎Midnight Commander (MC) ⌘)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search



title
Debian Administration Fundamentals
subtitle
for those who already know RedHat products
author
Alexander Patrakov

Slide conventions ⌘

  • You already know RedHat products, and it's exactly the same in Debian, so I am not focusing on this topic
  • You already know RedHat products, but it's a bit different in Debian, so let me explain the difference
  • Debian-specific topic

Day 1 ⌘

  • Debian as a Linux distro
  • Debian installation
  • System configuration
  • Package management basics

What is Debian? ⌘

Debian (/ˈdɛbiən/) is a Unix-like computer operating system that is composed entirely of free software, most of which is under the GNU General Public License, and packaged by a group of individuals called the Debian Project.

  • Has Constitution and Social Contract
  • Separate repository for non-Free packages
    • Free as in Freedom, defined by DFSG
    • Also "contrib" repository for Free software that cannot be built or cannot function without something non-Free
  • Apt/dpkg for package management

Debian branches and versions ⌘

  • Stable, Testing, Unstable branches
    • Currently, Wheezy = oldstable, Jessie = stable, Stretch = testing
    • After Stretch release, Wheezy = oldoldstable, Jessie = oldstable, Stretch = stable, Buster = testing
    • Unstable is always Sid

When to use which branch (1/2) ⌘

  • Use stable on production servers
    • It is not bug-free, but it never changes
    • Only serious bugs (security issues, data loss, ...) are fixed
    • No new upstream versions ever
      • except for otherwise unsupportable packages (browsers, MySQL)
  • Use testing on desktops, or when developing a new product
    • Reasonably modern – gets packages from unstable after 10 days
    • Sometimes gets bugs, but they are fixed by an upload to unstable
      • But no official security support

When to use which branch (2/2) ⌘

  • Use unstable if you are a Debian developer or want to help
    • All packages get uploaded here first, except for stable fixes
    • Packages there use versions declared "stable" upstream
    • Packages migrate to testing after a short period
    • Packages sometimes get stuck here if a release-critical bug is found, or there is a library version transition
  • Experimental branch exists for packages not fit even for unstable
    • Typical use:
      • "Dear Debian developers, upstream released a new version of FOO, but it breaks a lot of stuff, so it is not going to unstable. Please install it from experimental and fix the issues in your packages"

Exercise ⌘

  • Install Debian Jessie in a virtual machine
    • You will need to download the installation CD, use the netinst variant
    • No desktop environment and no print server (to save time)

Debian support and help ⌘

sudo apt install debian-handbook debian-reference harden-doc apt-doc debian-refcard

System configuration ⌘

  • Console basics
  • The filesystem
  • Important files

Console basics ⌘

  • The default interactive shell is Bash
  • The default /bin/sh shell is dash
    • Reconfigure: dpkg-reconfigure dash
    • Watch for incompatibilities when copying scripts from non-Debian systems
      • Or just stick #!/bin/bash at the top, not #!/bin/sh

Shell startup files ⌘

  • /etc/profile, ~/.profile for login shells
    • Place envronment variables here, export them
  • /etc/bash.bashrc, ~/.bashrc for all shells
    • Place PS1, functions, aliases here

Exercise ⌘

  • Find the place which appends a '#' to the root prompt
  • Find the place which sometimes makes the user command prompt green
  • Make root command prompt red
  • Add /opt/nobleprog/bin to $PATH
  • Install git and clone a repository
  • Make sure that bash prompt shows the current git branch
    • Google for instructions
      • Beware: bad instructions exist
    • Which files did you modify, and why?

Skipped topics ⌘

  • GUI system administration tools
    • The only important one is Synaptic, to be covered later
    • No GUI tools exist due to Debian Release Policy: "Packages must not modify other packages' configuration files except by an agreed upon APIs"
  • Virtual consoles
  • How to leave the command prompt
  • How to shutdown the system
  • Recovering a sane console

Sudo ⌘

  • Debian uses su, not sudo, by default
  • Can be made to use sudo if you don't provide the root password at installation time
  • There is a sudo group, members can use sudo

The filesystem ⌘

  • Debian follows FHS
    • RedHat follows it, too
    • One important difference: shared library location
      • /usr/lib/x86_64-linux-gnu

Midnight Commander (MC) ⌘

  • A console-based filesystem explorer
    • Similar to Norton Commander for DOS
      • Use arrows, Enter and Fx keys to interact
    • Popular in EU and Russia: "How can one manage without it?"
    • Unpopular in USA: "Why are you still using this thing of the past? And typing to this one-line-high command prompt? Looks like you haven't learned ls and cp"
  • Important difference from the default upstream configuration:
    • Debian disabled "use internal edit" by default
    • You get vim or nano
    • Even if you select mcedit, you can't get the shell (Ctrl+O) while editing a file
      • Fix: F9 > Options > Configuration > [X] Use internal edit, OK
  • Tweaking MC

The basic Unix-like work environment ⌘

  • The default shell is bash
    • Don't like it? chsh
  • The easiest editor is nano (for those who don't like mcedit)
    • Change with select-editor
      • The selection is stored in ~/.selected_editor
      • $EDITOR and $VISUAL environment variables still work
      • vi is not vim
    • Here is how to quit:
      • Vim: :q! (without saving), or :wq (save)
      • Nano: Ctrl+x
      • Emacs: Ctrl+x Ctrl+c
      • mcedit: F10, or Esc 0

Important configuration files ⌘

  • /etc/hostname sets the hostname
    • It also has to be mapped to 127.0.1.1 in /etc/hosts
      • Not 127.0.0.1
      • 127.0.0.1 should only map tp localhost and localhost.localdomain (canonical name)
      • An attempt to map the real hostname to 127.0.0.1 would make localhost.localdomain its canonical form, which is wrong
  • /etc/localtime symlink sets the timezone
  • /etc/default/locale sets the locale
  • /etc/default/console-setup sets the console font
  • /etc/default/keyboard sets the keymap (shared between console and X)

Debian package management ⌘

  • Debian package management prerequisites
  • Basic package management operations
  • Examples of aptitude operations
  • Advanced package management operations (on day 2)
  • Debian package management internals (on day 2)
  • Recovery from a broken system
  • Tips for package management

General recommendations ⌘

  • Use packages
  • If possible, use packages from official sources only
  • Update them regularly

Why packages? ⌘

  • Dependency tracking
    • Tracking explicit vs implicit installation
  • Tracking files
    • When was the last time some old file was left over in your web app?
  • Configuration file management
  • Ability to report bugs to Debian and benefit from bug reports made by others
  • Security support

Problems with third party packages ⌘

  • Package conflicts between packages in the Debian archive and packages in third party repositories.
  • Bad package upgrades between Debian releases.
  • Difficulty removing third party packages once installed.
  • Removal of unrelated packages to satisfy dependencies for third party packages.
  • Third party packages overriding/ignoring local system/user configurations.
  • etc.

As such, it is not recommended to install packages from third party repositories.

Text copied from: https://wiki.debian.org/MultimediaCodecs

APT and DPKG ⌘

  • dpkg deals with packages as *.deb files
    • Checks dependencies
    • Installs & updates only given packages
    • Deals with configuration files
    • Runs maintainer scripts
    • Somewhat similar to rpm
  • apt deals with packages in repositories
    • Resolves dependencies
    • Downloads packages and gives to dpkg for installation
    • Somewhat similar to yum or dnf

Repository configuration ⌘

  • /etc/apt/sources.list
  • or /etc/apt/sources.list.d/*.list
    • Handy for third-party repositories

Repository entries ⌘

The traditional format is: 

deb http://httpredir.debian.org/debian/ stretch main
deb-src http://httpredir.debian.org/debian/ stretch main
deb http://security.debian.org/debian-security stretch/updates main
deb-src http://security.debian.org/debian-security stretch/updates main
  • deb or deb-src: are we talking about source or binary packages?
  • 2nd field: base URL
  • 3rd field: suite
  • 4th and further fields: components
    • main contrib non-free
  • An alternative RFC822-based format is also supported in apt 1.1+
  • It is recommended to add repositories by hand

Additional official repositories ⌘

  • Backports
    • Newer versions of certain software, recompiled to run on Debian Stable
    • Violate the "no new versions ever" promise
deb http://ftp.debian.org/debian jessie-backports main
  • Updates
    • Some software (like timezone data and virus scanners) needs to be updated frequently
    • All software from Updates will be included in the next point release anyway
deb http://ftp.debian.org/debian jessie-updates main contrib

Proposed updates ⌘

  • Proposed-updates for stable
    • Preview of what will be included in the next point release
    • Not well-tested, can be buggy
deb http://ftp.debian.org/debian jessie-proposed-updates main contrib
  • For the testing distribution, they start to exist after the freeze
    • Now this repository is empty
deb http://ftp.debian.org/debian stretch-proposed-updates main contrib

Useful third-party repositories ⌘

  • Multimedia packages (not in Debian proper because of patent & license problems in USA)
    • Including libdvdcss2 and aacskeys, and including ffmpeg with more encoders enabled
      • The set of decoders in the default debian package is already complete
deb http://www.deb-multimedia.org jessie main non-free
deb http://www.deb-multimedia.org jessie-backports main
  • VirtualBox
deb http://download.virtualbox.org/virtualbox/debian jessie contrib
  • There are others

The corresponding GPG keys should be imported manually with apt-key add

False lead ⌘

  • There is also apt-add-repository from software-properties-common
    • But the package is geared a lot towards Ubuntu and not Debian
      • Knows about Ubuntu-specific PPAs and keywerver
      • Almost useless for Debian
  • Again, it is recommended to add repositories by hand

Installing packages with apt ⌘

  • Refresh package lists:
    • apt update
  • Install packages:
    • apt install pkg1 pkg2 ...
    • Will upgrade packages if they are already installed
    • Side effect: will mark packages as installed manually
  • Install for non-default architecture (e.g. i386)
    • apt install pkg:i386
      • Secondary architectures will be discussed later

Package configuration ⌘

  • Usually Debian packages have safe default configuration
    • Configuration is stored in configuration files in /etc
  • Unlike in RedHat, daemons are started by default
    • They don't serve private information, and sometimes listen on 127.0.0.1
  • If no sane default configuration exists, Debian has to ask
    • Remember these boxes with blue background and red title?
      • That's debconf asking questions

debconf properties ⌘

  • Debconf will not reask questions unless you reconfigure the package
    • Here is how: dpkg-reconfigure package
    • Answers survive package removal, but not purge
  • It is, by default, safe to edit configuration files created using your answers to debconf questions
  • Want to see more questions for a particular package? dpkg-reconfigure -plow package
  • Want less (or more) questions in general? dpkg-reconfigure debconf
    • This set the minimum priority (importance) of questions to ask
    • The rest are skipped and the default answer is used

Searching for packages ⌘

  • apt search term1 term2 ...
    • Lists package names and short descriptions
  • apt show pkg1 pkg2
    • Displays detailed information about packages
      • It is not necessary for packages to be installed

Removing packages ⌘

  • apt remove pkg1 pkg2
    • Does not remove configuration files
  • apt purge pkg1 pkg2
    • Removes configuration files, too
  • apt [--purge] autoremove
    • Removes packages that were automatically installed but are no longer needed
    • Oops, that's in Debian Testing only. On Jessie, use apt-get [--purge] autoremove
  • apt clean
    • Removes previously downloaded debs

Upgrading the whole system ⌘

  • apt upgrade
    • Safe – never removes packages
    • Can install new packages due to dependencies
    • May result in an incomplete upgrade
  • apt full-upgrade
    • Sometimes removes packages that became outdated, or are in conflict with the new ones
  • There is an instruction how to upgrade between releases
    • This is specific to each Debian version, but the important bit is equivalent to apt upgrade followed by apt full-upgrade

Adding a secondary architecture ⌘

  • Debian aupports multiarch!
  • dpkg --add-architecture i386
    • You can also add armel or other non-x86 architectures if you have qemu-user

About apt-get ⌘

  • In previous Debian versions, you used apt-get update, apt-cache search
  • Had to remember which apt-tool does what
  • Now you don't have to – just use apt
    • But apt autoremove was not in Debian Jessie. We had to use apt-get autoremove

Exercise ⌘

  • Given a Debian Stable VM
    • Install postgresql
    • Install something that uses it (would need a database)
      • Can you give a full list of software in Debian that can use Postgresql?
    • Upgrade to Debian Testing (instruction)

Exercise ⌘

  • Given a Debian Testing VM (NOT the result of the previous exercise)
    • Install redis
      • Connect using redis-cli
    • Install 32-bit wine (wine32)
      • Test it by running winecfg
    • Install mongodb 3.2 from mongodb repository
      • Connect using mongoc
    • Completely remove all of the above

Common dpkg operations ⌘

  • dpkg -i file1.deb file2.deb
    • Install packages from deb files
    • If dependencies are unmet, they will be in half-installed state
  • dpkg -r pkg1 pkg2
    • Remove packages
    • Unlike apt remove, fails if this would break dependencies
      • apt remove would suggest to remove those dependencies
    • Use dpkg -P pkg1 pkg2 to purge
  • dpkg -L pkg
    • List files owned by a package

How to fix broken package installation ⌘

  • Figure out why exactly it is broken
  • Remove the reason of the breakage
    • This usually involves editing a configuration file or removing a bad package using dpkg -r
  • apt -f install will add/remove packages as needed to fix dependencies
    • Will also fix half-installed packages by rerunning their maintainer scripts
    • apt -f install is a good way to install missing dependencies after a dpkg -i ...

Exercise ⌘

  • Given a Debian Stable VM:
  • Install apache
    • The package is actually called apache2
  • Install nginx
    • Yes, there are non-crazy people who run both on the same machine
    • However, on Debian, this will fail
      • Need different ports!
  • Fix it so that Apache listens on port 8080
    • Too easy? Purge both and try again, so that Nginx listens on port 8080
    • Still too easy? "Oh, I don't actually want Apache"

Listing packages and their status ⌘

# dpkg -l \*spell\*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                Version        Architecture   Description
+++-===================-==============-==============-=================================
ii  aspell              0.60.7~2011070 amd64          GNU Aspell spell-checker
un  aspell-bin          <none>         <none>         (no description available)
un  aspell-da           <none>         <none>         (no description available)
un  aspell-dictionary   <none>         <none>         (no description available)
un  aspell-doc          <none>         <none>         (no description available)
ii  aspell-en           2016.06.26-0-0 all            English dictionary for GNU Aspell
un  aspell-hi           <none>         <none>         (no description available)
un  aspell-mr           <none>         <none>         (no description available)
un  aspell-no           <none>         <none>         (no description available)
un  aspell6-dictionary  <none>         <none>         (no description available)
un  aspell6a-dictionary <none>         <none>         (no description available)
un  hunspell            <none>         <none>         (no description available)
un  hunspell-ar         <none>         <none>         (no description available)
un  hunspell-da         <none>         <none>         (no description available)
...

Listing removed but not purged packages ⌘

  • Desired state = removed, Status = conf-files
dpkg -l | awk '/^rc/{print $2}'
  • It's a good idea to purge them before upgrading the whole distribution
dpkg -l | awk '/^rc/{print $2}' | xargs dpkg --purge

Files used by apt and dpkg ⌘

  • /var/lib/apt/lists
    • Package lists
  • /var/cache/apt/archives
    • Downloaded packages
    • Handy for downgrading if anything breaks
  • /var/lib/dpkg/info
    • maintainer scripts (preinst, config, postinst, prerm, postrm)
  • /var/lib/dpkg/status
    • What is installed, what is half-installed, what is removed but not purged

Emergency downgrading ⌘

  • The log of all package operations is in /var/log/dpkg.log
  • Look for "upgrade" lines after the date where everything broke
  • Both old and new versions are listed
    • So now you know what to downgrade to
    • The which-pkg-broke tool from debian-goodies will also list suspects
  • Look for old packages in /var/cache/apt/archives
  • dpkg -i ...

Preventing re-upgrades to broken package versions ⌘

  • Temporary solution: apt hold pkg1 pkg2
  • Then undo the hold: apt unhold pkg1 pkg2

Exercise ⌘

  • You are given a Debian Testing system with MATE desktop
  • It has a bug: MATE Terminal has a weird color scheme and crashes if one changes it
    • This is actually a bug caused by GTK+ 3.21
  • Fix it by downgrading GTK+ to the last version available before 3.21
  • Note to the trainer: this was valid on 2016-09-18. For the next training, find another broken package and update this slide.

Exercise ⌘

  • Given a Debian Testing VM:
  • Install Opera
    • It's a proprietary web browser
  • Which files are installed?

Exercise ⌘

  • Given a Debian Stable VM:
  • Switch to sysvinit instead of systemd
    • apt install sysvinit-core ; reboot
    • We need this in order to reproduce the bug
  • Install kexec-tools
  • Try to remove them
    • You can't (it just reboots)
      • Known bug, but please don't cheat by looking or googling
  • Remove them

Package dependencies – positive ones ⌘

DPKG has more expressive power for dependency specification than RPM

If a is installed...

  • a Depends on b: b absolutely must be installed for a to operate properly
  • a Pre-Depends on b: b absolutely must be installed for a to operate properly
    • also a's pre-installation script can rely on b
  • a Recommends b: b should be installed in most cases when a is installed
  • a Suggests b: b may be useful in some cases when a is installed
    • Mostly just a hint to the user

Unversioned dependencies on so-called Essential packages are implied and should not be declared explicitly

Authoritative source: Debian Policy Manual

Package dependencies – negative ones ⌘

If a is installed...

  • a Breaks b: b should be deconfigured
    • Usually, in order to be upgraded
  • a Conflicts with b: b's files should not even exist in the filesystem

Virtual packages and file moves ⌘

If a is installed...

  • a Provides b: the dependency resolver should think that b is also installed
    • Even if b does not exist at all
  • a Replaces b: two meanings, both encountered together
    • Don't complain if a contains files with names already in b
      • They don't belong to b anymore
    • If there is a package conflict, resolve it by removing b instead of complaining

Deselecting unwanted recommendations ⌘

  • By default, recommended packages are installed, and suggestions aren't
  • Sometimes a recommendation is too heavy and is not really needed
    • Pass its name, append a minus
      • apt install wanted unwanted-
  • To install with no recommendations at all: apt --no-install-recommends foo
    • This is not recommended

aptitude ⌘

  • An alternative to apt
    • Not really recommended, but sometimes useful
    • I use it for cleaning out packages after changing server roles
  • Can install, upgrade, remove, purge, search packages
    • aptitude [install|remove|purge|search|show|hold|unhold] ...
    • aptitude [update|safe-upgrade|full-upgrade]
  • Also can be started without arguments
    • Presents a curses-based interactive interface then

aptitude keybindings (part 1/2) ⌘

  • Enter: expand a category
  • Enter: provide package information
    • q to make it go away
  • +: mark for installation
  • -: mark for removal
  • _: mark for purging
  • :: don't mark
  • g: go!
  • q: don't go
  • F10 or Ctrl+T: menu with even more actions

aptitude keybindings (part 2/2) ⌘

  • u: update package lists
  • U: mark all possible packages for upgrade
  • /: search
    • More info later
  • n: repeat the last search
  • l: search, and hide what doesn't match the criteria
  • m: mark as manually installed
  • M: mark as automatically installed
  • =: hold
  • :: don't hold

aptitude: other useful commands ⌘

  • aptitude why pkg
    • Finds a dependency chain that installs the package
      • Even if the package is not actually installed
        • yes, this is confusing – treat it as "why would I want this package?"

aptitude: dealing with broken dependencies ⌘

  • Breakage happens e.g. when you mark a package for removal without marking its reverse dependencies for removal
  • Broken packages are highlighted in red
  • Aptitude automatically presents the full list of problems at the bottom
    • Press e to solve them
    • Press , or . to navigate the list of proposed solutions
    • Press r to tell aptitude that you reject all solutions with the highlighted action
      • E.g.: don't ever try to install this bloated junk instead of what I just tried to remove
    • Press a to tell aptitude to always do the highlighted action
    • Usually the first solution is too destructive, but the second is OK
    • Apply the solution with the ! key
    • There is always an option to solve broken dependencies manually

Exercise ⌘

  • Given a VM with Debian Stable that was installed earlier:
    • Remember that I asked you to deselect Desktop Environment and Print Server?
    • Look: libgtk2.0-0 is still there
      • And also many other seemingly-junk packages that are related to the graphical session
    • Figure out why they are here
    • Safely remove them

aptitude search syntax ⌘

  • By default, searches for name substring
    • That's only a convenience shortcut for ?name(foo)
  • Other possible terms: ?version(4\.0), ?description(yacht)
    • Versions and other search strings are interpreted as regular expressions, that's why \.
  • Some other special terms: ?installed, ?obsolete, ?broken, ?upgradable
  • Boolean expressions: ?and(a, b), ?or(a, b), ?not(a)
    • Shorthand forms: a b, a | b. !a
  • For loops
  • Full list of possible terms does not it on one screen

Example ⌘

  • Purging removed packages for good:
    • aptitude purge '?config-files'

Day 2 ⌘

  • System initialization
  • Network configuration
  • More apt/dpkg tips
  • Advanced package management topics

The system initialization ⌘

  • Supported init systems: sysvinit, systemd, upstart
    • Systemd is the default in Debian Jessie and Stretch
    • Sysvinit was the default before that
    • We'll cover sysvinit and systemd
      • Upstart is essentially dead

Switching the init system ⌘

  • I intentionally don't answer why you would want this
  • From systemd to sysvinit
apt install sysvinit-core
reboot
apt purge systemd
  • From sysvinit to systemd
apt install systemd-sysv
reboot
apt purge sysvinit
  • Sometimes you'll need to care about systemd-shim
    • Provides some services that originally come from systemd to sysvinit-based system
    • Debian-specific, not supported anywhere else, voids your right to report bugs upstream

Installing without systemd ⌘

  • Add to the installer kernel command line:
preseed/late_command="in-target apt-get install -y sysvinit-core"

Exercise ⌘

  • Does GNOME work in Debian Testing without systemd?

Overview of the bootstrap process ⌘

  • BIOS → MBR → GRUB → Kernel and initrd → /sbin/init → everything else
  • UEFI → GRUB → Kernel and initrd → /sbin/init → everything else

Meaning of the word "runlevel" ⌘

  • Sysvinit: a numbered set of instruction to start and stop services
    • 0 = shutdown, 6 = reboot, 1 = single-user recovery
    • RedHat: 3 = text mode, 5 = graphical session
    • Debian: 2 = normal operation
  • Systemd: "runlevels" do not exist. There are "targets".
    • graphical.target, multi-user.target, ...
      • systemctl get-default
      • systemctl set-default graphical
        • But better don't. You'll break update-rc.d

How sysvinit boots the system ⌘

  • Reads /etc/inittab
  • At system initialization, runs /etc/init.d/rcS
    • So it runs scripts in /etc/rcS.d/ with the "start" parameter
      • see /etc/init.d/rcS, /etc/init.d/rc
  • The default runlevel is 2
    • This runs scripts from /etc/rc2.d/ with the "start" parameter
  • A shutdown or reboot, uses runlevel 0 or 6
    • This runs scripts from /etc/rc{0,6}.d/ with the "stop" parameter
  • Ordering of scripts is determined by their names
    • Names start with numbers generated automatically by update-rc.d according to dependencies in LSB headers

How systemd boots the system ⌘

  • Performs a number of hard-coded tasks like starting systemd-udevd, setting keymap, ...
    • Mostly equivalent to what scripts in /etc/rcS.d/ did
  • Reads service files from /lib/systemd/system and /etc/systemd/system
    • Vendor files are in /lib, custom services and sysadmin overrides are in /etc
  • Resolves dependencies of default.target
  • Generates additional services that run sysvinit scripts
    • Considers only rc2.d, not rcS.d
  • Runs services in parallel
  • On shutdown/reboot, resolves dependencies of {shutdown,reboot}.target

Starting and stopping services manually ⌘

  • Wrong way: /etc/init.d/something start
    • Inherits the environment from the command prompt
  • Systemd way: systemctl start something
    • Works with systemd only
  • Generic and recommended way: service something start
    • Runs sysvinit scripts in a predictable environment
    • If the system has been booted with systemd, delegates all the work there

Configuration of runlevels ⌘

  • Just Don't. It will break when they force systemd on us all.
    • But you can still enable and disable services

Disabling and enabling services ⌘

  • update-rc.d: works both for systemd and sysvinit, synchronizes states
  • systemctl enable/disable: also works
    • Debian-specific feature: also disables the same service for sysvinit
  • systemctl mask
    • like "disable", but will also make it impossible to start the service manually or as a dependency

Exercise ⌘

  • Given a Debian Testing VM:
  • Install memcached
  • Make sure that it doesn't start automatically
    • There is a setting in /etc/default/memcached just for this, right?
    • Why doesn't it work?

Exercise ⌘

  • Given a Debian Testing VM and memcached:
  • Configure it to run two memcacheds on different ports
    • Do it via systemd service files

Network configuration ⌘

  • For desktops: use NetworkManager
    • In some cases, unavoidable
      • Some packages depend on NetworkManager
      • You can disable it and use ifupdown, but then some packages will think that there is no internet
  • For servers: use ifupdown
    • Although NetworkManager developers add relevant features, too!
    • There is also systemd-networkd

Network device support ⌘

  • Common problem: firmware
    • Moved to non-free section due to licensing reasons
    • Affects Broadcom NICs, Intel wireless cards, and many other devices
  • Solution: install firmware
    • e.g. apt install firmware-bnx2
      • Requires network!
  • Alternative installer CDs with firmware

NetworkManager ⌘

  • Supports Ethernet, WiFi, mobile connections, DSL
  • Bonds, VLANs, bridges
    • Going to server market!
  • Various kinds of VPNs

NetworkManager configuration tools ⌘

  • Graphical configuration
  • nmtui
  • nmcli

NetworkManager configuration files ⌘

  • /etc/NetworkManager/system-connections/*
    • Users in the netdev group can change these connections
  • Available fields: see documentation (man nm-settings-keyfile)
  • If you change files by hand, call nmcli con reload

NetworkManager limitations ⌘

  • No 6in4/6to4/teredo tunnels
  • No tinc VPN
    • You can still configure tinc manually
  • No PPPoE over WiFi
    • Still required by some German ISPs
  • Only one VPN active at a time
    • May be fixed soon (network-manager-1.4.0 is already in Unstable)

Exercise ⌘

  • Reconfigure the network so that it becomes a bridge with one port
    • It should still use DHCP
  • Use nmtui or a graphical tool

Exercise ⌘

  • Make sure that the VM's bridge obtains its IP address via DHCP, but uses only 8.8.8.8 as a DNS server
    • Do it by editing system connection files directly and reapplying configuration

ifupdown ⌘

  • See /etc/network/interfaces, /etc/network/interfaces.d/
  • man 5 interfaces
    • Only basics are covered there
  • Add-on packages like bridge-utils and vlan are needed for advanced features
    • e.g. man 5 bridge-utils-interfaces

Static routes ⌘

  • There are guides that add them from "up" and "down statements in the interface definition
  • The proper way is to install ifupdown-extra package
    • It reads /etc/network/routes

resolvconf ⌘

  • There are multiple sources of nameserver configuration
    • Static configuration
    • DHCP
    • VPN
    • Local nameserver
  • Task: coordinate the contents of /etc/resolv.conf
    • Just install the resolvconf package

Exercise ⌘

  • Again, create a bridge with one port
    • Use /etc/network/interfaces

Firewall ⌘

  • Default: no firewall at all
    • Why would one need it?
      • Some packages don't provide IP ACLs on their own
      • Reduces attack surface
      • Required by some regulations
    • Good policy: don't install unneeded network servers
      • Or configure them to listen on 127.0.0.1 if they are for local service only
      • The default install contains rpcbind and nfs-common
        • Purge them if you don't need tem

Available iptables frontends ⌘

  • Ferm
  • UFW
  • apt search firewall

Beyond iptables ⌘

  • ebtables: filter packets passing through a bridge
  • arptables: filter ARP packets
    • Use ip6tables for NDP

Starting firewall at boot ⌘

  • Use one of the frontends
  • Or, iptables-persistent
    • Purpose: restore iptables rules at boot
    • Use netfilter-persistent save to save the initial configuration
    • Or, edit /etc/iptables/rules.v4

Common firewall mistakes ⌘

  • Loading it too late
    • Happens with custom scripts
    • Does not happen with iptables-persistent
  • Forgetting about IPv6
  • Forgetting about IPv6 link-local addresses

Network fine-tuning ⌘

  • MTU (Maximum Transmission Unit)
    • Can be set in /etc/network/interfaces or via NetworkManager
  • Various sysctls
    • Set them in /etc/sysctl.conf or /etc/sysctl.d/yourfile.conf
    • E.g. here is how to disable IPv6 on a particular card:
      • net.ipv6.conf.eth1.disable_ipv6 = 1

Tips for package management ⌘

  • Use deborphan
    • It finds obsolete library and transitional packages
  • Use needrestart
    • It warns if service need restart due to a library upgrade
  • Consider unattended-upgrades
    • It can install security updates automatically
      • But does not restart services due to library upgrades
      • And you risk rebooting the server in the middle of upgrade
      • Relatively safe on Ubuntu (because it's enabled by default there), but I wouldn't use it on Debian
    • Consider using it together with apt-listchanges and apticron
    • Read the excellent LWN article about it

Finding modified files ⌘

  • debsums
    • Checks MD5 checksums of installed files against those from packages
    • Needs initialization after installation: debsums_init
      • debsums is intended primarily as a way of determining what installed files have been locally modified by the administrator or damaged by media errors and is of limited use as a security tool.
  • You can use e.g. tripwire for detecting unauthorized alterations of files

Converting RPMs ⌘

  • Use alien
    • Or better don't

Duplicating package state ⌘

  • Use apt-clone
    • Saves/restores /etc/apt/sources.list, keys and package states

Avoiding debconf prompts ⌘

  • Sometimes it's boring to answer the same debconf prompts on multiple machines
    • Preseed them!
    • On the source machine: apt install debconf-utils
    • On the source machine: debconf-get-selections > selections.txt
    • Remove unneeded lines, copy selections.txt to other machines
    • On target machines: debconf-set-selections < selections.txt

More tips ⌘

  • Install a mail server (exim4 or postfix or something else)
    • So that you are notified about failing cron jobs, bad disks, and other important events
    • exim4 is there by default
  • Read logs
    • logwatch will send you the interesting bits via email
  • Read changelogs of packages that you care about
    • zless /usr/share/doc/package/changelog.Debian.gz
    • apt-listchanges
  • Read README.debian.gz and NEWS.debian.gz

Graphical software installation tools ⌘

  • synaptic
    • Can install, upgrade, configure, search packages
    • Offers features similar to apt
  • gnome-software
    • Not a package manager
      • Focused on Apps, not Packages
      • User should not know about low-level packages
      • Displays only graphical apps
      • Displays stars, reviews and ratings stored on a central GNOME server

Configuring APT ⌘

  • Files in /etc/apt/apt.conf.d
    • *.conf or no extension
    • Full list of options is available in man apt.conf, man apt-get and so on

Excercise ⌘

  • Given a VM with Debian Testing:
  • Limit apt-get download bandwidth to 500 KB/s
  • Disable downloading of incremental differences (PDiffs) of Packages files
    • This will force full downloading of these files
    • Useful on fast connections
  • Make sure that apt shows versions when installing and upgrading packages

How to unpack deb on non-debian system ⌘

  • A deb file is just an ar(1) archive
    • ar comes from binutils
  • Contains data.tar.gz and control.tar.gz
  • data.tar.gz contains package files
  • control.tar.gz contains files with metadata

So:

  • ar x file.deb
  • tar xpf data.tar.gz
  • tar xpf control.tar.gz

Exercise ⌘

What's in debian package filename ⌘

x11vnc_0.9.13-1.2+b2_amd64.deb

  • x11vnc: package name
  • 0.9.13-1.2+b2: version
  • amd64: architecture
    • "all" means "no architecture-dependent files"

What's in debian package version ⌘

0.9.13-1.2+b2

  • 0.9.13: version that was released by the upstream software authors
  • 1.2+b2: debian build version
    • Missing for packages where debian == upstream
    • 1: the number of the last build by the maintainer
      • That's the only debian-specific version component in the case if the last build was indeed by the maintainer
    • 2: the number of the last build by someone else since the last maintainer upload
    • b2: the number of automated no-changes rebuilds since then
    • Other suffixes exist

Building Debian packages ⌘

  • You need build-essential
  • Read Build-Depends
  • Extract source package (see next slide)
  • Build it

Downloading and extracting source packages ⌘

  • You need deb-src in /etc/apt/sources.list
    • apt-get build-dep pkg1 downloads dependencies
    • apt-get source pkg1 downloads sources
  • Alternatively, use dget from devscripts
    • It needs an URL for the dsc file

Exercise ⌘

  • Extract the source package for indent
    • Try to do the same extraction manually (tar + patch)

Building the package ⌘

  • Two layers of tools
    • The build is driven by debian/rules
      • It is a Makefile
      • fakeroot debian/rules binary
    • dpkg-buildpackage -uc -us (from devscripts) offers additional checks

Exercise ⌘

  • Rebuild indent without changes
    • Please bump the version in debian/changelog using dch

The equivs package ⌘

  • Builds a dummy package
    • Useful to satisfy dependencies if you built something from sources
      • Even better is to produce your own package - learn debhelper tomorrow!

Mirroring official archive ⌘

  • ftpsync
    • Alternative: debmirror
    • Want a proxy instead? apt-proxy, apt-cacher, apt-cacher-ng

Day 3 ⌘

  • The mail subsystem
  • SSH configuration
  • PAM
  • SELinux and AppArmor
  • Kernel modules and device permissions
  • Compiling a kernel for Debian
  • Debianization of third-party software
  • Building your own archive of debian packages

The mail subsystem ⌘

  • Default: exim4
  • You can install Postfix or Sendmail
    • Disclaimer: I don't like Exim
      • Very verbose default configuration files
      • Postfix configuration is much more compact (20 vs 1018 non-comment lines)

exim4 in Debian ⌘

  • Packages:
exim4 - metapackage to ease Exim MTA (v4) installation
exim4-base - support files for all Exim MTA (v4) packages
exim4-config - configuration for the Exim MTA (v4)
exim4-daemon-heavy - Exim MTA (v4) daemon with extended features, including exiscan-acl
exim4-daemon-heavy-dbg - debugging symbols for the Exim MTA "heavy" daemon
exim4-daemon-light - lightweight Exim MTA (v4) daemon
exim4-daemon-light-dbg - debugging symbols for the Exim MTA "light" daemon
exim4-dbg - debugging symbols for the Exim MTA (utilities)
exim4-dev - header files for the Exim MTA (v4) packages
exim4-doc-html - documentation for the Exim MTA (v4) in html format
exim4-doc-info - documentation for the Exim MTA (v4) in info format

Light vs heavy exim4 daemon ⌘

Features found in the heavy version:

  • User lookups in third-party stores:
    • LDAP, sqlite, PostgreSQL, MySQL
  • SASL and SPA SMTP authentication
  • embedded Perl interpreter
  • exiscan-acl for integration of virus-scanners and SpamAssassin

How to configure exim4 ⌘

  • In simple cases, dpkg-reconfigure exim4-config should be enough
  • In other cases, edit /etc/exim4/conf.d/*
    • Splitting configuration into small files is recommended for easier upgrades

Rerouting mail ⌘

  • /etc/aliases
    • For non-exim mail servers, also run newaliases
    • |/path/to/script also works

Reading local mail ⌘

  • By default mail is stored in /var/mail/$USER in mbox format
  • Use mail or mutt to read
    • Better, redirect

Configuring ssh ⌘

  • /etc/ssh/sshd_config for the server
  • /etc/ssh/ssh_config for the client
  • By default Debian Jessie (and Stretch) denies root login using password
    • Key-based login is still possible and useful for backups

Limits on connection attempts ⌘

  • Brute-force password-guessing attacks
  • MaxStartups directive
  • Common problem: attacks break legitimate login attempts (usually seen by cron tasks)
  • fail2ban?
    • Problem: not really parseable logs
      • ssh 'nobleprog from 8.8.8.8'@192.168.122.73

Exercise ⌘

  • Create a group "ssh-users"
  • Reconfigure sshd so that only root (with keys) and ssh-users can connect
    • Test key-based login
  • Create an exception for one particular IP address

PAM ⌘

  • Pluggable Authentication Modules
    • Not specific to Debian
      • Useful e.g. for LDAP authentication
      • Often configured together with NSS (/etc/nsswitch.conf)
    • Difference from RedHat: no authconfig
      • There is pam-auth-update
    • You can also edit files in /etc/pam.d
      • Common settings are in common-* files
      • Service-specific settings are in other files

How PAM works ⌘

  • Provides four services
    • auth: how to authenticate user (e.g. how to check password)
    • account: how to check that access is allowed (e.g. enforce time-of-day restrictions)
    • password: how to change the password
    • session: what else to do when the user logs in (e.g. register it with systemd)

PAM directives ⌘

  • In general, modules can succeed, fail, or want to be ignored
    • requisite: module must succeed. Don't look further if it fails.
    • sufficient: Don't look further if it succeeds.
    • required: module must succeed. Go further if it fails, to confuse the attacker.
    • optional: use the result only if everything else wants to be ignored
    • include: include the directives from the other file
    • substack: same as include, but don't short-circuit the whole stack if a "requisite" or "sufficient" works in a subscack

Exercise ⌘

  • Make sure that only members of group root can use su

SELinux ⌘

  • Mandatory access control
  • Based on labels and contexts
  • Prevents processes from accessing files that they shouldn't access
  • Not installed by default
  • Guide: https://wiki.debian.org/SELinux
    • Comes with a recommendation to use Postfix instead of Exim4
  • Recommendation: turn it on in a permissive mode, see if there are any denials
  • Then audit2allow if needed, and then enable with enforcing=1 in /etc/default/grub

Exercise ⌘

  • Try to configure SELinux on Debian Jessie
  • Try again with a graphical session
    • It will fail
    • That's why you use it on servers only, and with sysvinit only

AppArmor ⌘

Kernel command line ⌘

  • Usually set in /etc/default/grub
    • Run update-grub to apply changes
  • Change with e at GRUB boot screen
  • Useful recovery option: init=/bin/bash
    • Want more debug? add debug, remove quiet
  • For initrd debugging:
    • break=premount
    • man initramfs-tools

Kernel modules ⌘

  • For hardware that is present in the system, udev directs kmod to load the necessary modules
    • No action is needed
    • You can fine-tune module arguments using files in /etc/modprobe.d
      • options i915 preliminary_hw_support=1
      • Alternatively use kernel command line: i915.preliminary_hw_support=1

Kernel modules, more tips ⌘

  • Want to disable autoloading of a specific module?
    • blacklist badmodule in /etc/modprobe.d/somefile.conf
    • or blacklist=badmodule1,badmodule2 on the kernel command line
  • Want to load some module that is not loaded by default?
    • Put its name in /etc/modules or in /etc/modules-load.d/somefile.conf

Third-party kernel modules ⌘

  • You need linux-headers package matching the running kernel
  • Debian has module-assistant
    • Can build packaged third-party modules from source
  • There is also dkms

Exercise ⌘

  • Install xtables-addons modules

Custom kernel ⌘

  • You can package it as a deb using make-kpkg from kernel-package
    • The official kernel sources are in the linux-source package
    • nano /etc/kernel-pkg.conf
    • make menuconfig
    • fakeroot make-kpkg --initrd --revision=1.0.custom kernel_image
      • will create something like linux-image-3.16-amd64_1.0.custom_amd64.deb

Storage ⌘

  • Debian installer supports partitions, RAID, LVM, iSCSI
    • Does not support LVM thin provisioning
  • Debian initramfs-tools support more
  • Dracut (originally from RedHat) supports booting from almost everything in the world
  • If you want to install on strange storage, don't use Debian installer
    • Use debootstrap, then install a boot loader
      • Debootstrap is good for creating Debian chroots, too!

Exercise ⌘

  • Create a Debian Testing chroot on a Debian Stable system
    • Configure schroot so that users can safely chroot there

Automated installation ⌘

Why build a custom package ⌘

  • New version (backport)
  • Special build options
  • Fix a bug
  • New and internal development

Useful resources ⌘

Source packages ⌘

  • Source package = *.orig.tar.gz + *.debian.tar.gz + *.dsc
    • *.orig.tar.gz = original release tarball from upstream
    • *.debian.tar.gz = debian patches + debian files that drive the build
    • *.dsc = metadata

Source packages and binary packages ⌘

  • From one source package, many binary packages are built
  • Example: openssl_1.0.1b-1.dsc (+tars) → 5 packages
    • libssl-dev_1.0.1b-1_amd64.deb, libssl1.0.0-dbg_1.0.1b-1_amd64.deb, libssl1.0.0_1.0.1b-1_amd64.deb, openssl_1.0.1b-1_amd64.deb, libssl-doc_1.0.1b-1_all.deb
    • Note: architecture-dependent vs architecture-independent packages

Important files that drive the build ⌘

  • debian/changelog: changelog
    • First line also determines the package version
    • dates are date -R
    • use dch to add entries
  • debian/control: binary package names, dependencies
  • debian/copyright: copyrights
  • debian/rules: Makefile that drives the build
  • debian/source/format: how the sources are organized. *.orig.tar.gz + *.debian.tar.gz = "3.0 (quilt)"
    • For internal packages that have debian directory in the same repository, use "3.0 (native)"
  • debian/patches/series: list of patches to apply

Mechanics of building a package ⌘

  • Targets in debian/rules:
    • build: builds the program. Does not require root.
    • build-arch, build-indep (both optional): builds only architecture-dependent or architecture-independent parts
    • binary: builds all binary packages (typically runs under fakeroot)
    • binary-arch, binary-indep (both required): build only architecture-dependent or architecture-independent binary packages
    • clean: cleans up after build and binary

But it doesn't look like this! ⌘

  • Most debian/rules files are now based on debhelper 7+
    • Simplest possible (and in many cases working) debian/rules file:
#!/usr/bin/make -f
# Uncomment this to turn on verbose mode.
#export DH_VERBOSE=1
%:
        dh $@
  • CDBS was also popular

Debhelper mechanics ⌘

  • Debhelper uses mind-reading technology. Reads developer's mind, not yours!
    • Needs a hint (mind reader version number) in debian/compat. The correct level is 9 in Jessie
      • Don't forget Build-Depends: debhelper (>= 9)
    • Contains simple rules like these:
      • There is a setup.py file ⇒ let's build it like a typical python setuptools/distutils package
      • After the build, there is a shared library in the standard location ⇒ let's fix permissions and add a call to ldconfig to postinst
      • There is debian/$package.init.d ⇒ let's use it as a sysvinit initscript and start it in postinst
      • There is a debian/$package.service ⇒ must be a systemd service, let's install it as such

Fine-tuning debhelper ⌘

  • Sometimes the default build is not good enough
    • Add DH_VERBOSE=1
    • Read what dh_* commands are executed
    • Find which one does the wrong thing (example: dh_what_went_wrong)
    • Read its manual page
    • Add override_dh_what_went_wrong: target
      • call dh_what_went_wrong with the correct arguments, explaining to it what to do
      • or just do it as plain shell commands if that's easier

Exercise ⌘

Producing your apt archive ⌘

Additional exercises

Debian exercises

Retrieved from ‘https://training-course-material.com/index.php?title=Debian_Administration&oldid=83524’