Debian Administration

title

Debian Administration Fundamentals

subtitle

for those who already know RedHat products

author

Alexander Patrakov

Slide conventions

• You already know RedHat products, and it's exactly the same in Debian, so I am not focusing on this topic
• You already know RedHat products, but it's a bit different in Debian, so let me explain the difference
• Debian-specific topic

Day 1

• Debian as a Linux distro
• Debian installation
• System configuration
• Package management basics

What is Debian?

Debian (/ˈdɛbiən/) is a Unix-like computer operating system that is composed entirely of free software, most of which is under the GNU General Public License, and packaged by a group of individuals called the Debian Project.

• Has Constitution and Social Contract
• Separate repository for non-Free packages
  • Free as in Freedom, defined by DFSG
  • Also "contrib" repository for Free software that cannot be built or cannot function without something non-Free
• Apt/dpkg for package management

Debian branches and versions

• Stable, Testing, Unstable branches
  • Currently, Wheezy = oldstable, Jessie = stable, Stretch = testing
  • After Stretch release, Wheezy = oldoldstable, Jessie = oldstable, Stretch = stable, Buster = testing
  • Unstable is always Sid

When to use which branch (1/2)

• Use stable on production servers
  • It is not bug-free, but it never changes
  • Only serious bugs (security issues, data loss, ...) are fixed
  • No new upstream versions ever
    • except for otherwise unsupportable packages (browsers, MySQL)

• Use testing on desktops, or when developing a new product
  • Reasonably modern – gets packages from unstable after 10 days
  • Sometimes gets bugs, but they are fixed by an upload to unstable
    • But no official security support

When to use which branch (2/2)

• Use unstable if you are a Debian developer or want to help
  • All packages get uploaded here first, except for stable fixes
  • Packages there use versions declared "stable" upstream
  • Packages migrate to testing after a short period
  • Packages sometimes get stuck here if a release-critical bug is found, or there is a library version transition

• Experimental branch exists for packages not fit even for unstable
  • Typical use:
    • "Dear Debian developers, upstream released a new version of FOO, but it breaks a lot of stuff, so it is not going to unstable. Please install it from experimental and fix the issues in your packages"

Exercise

• Install Debian Jessie in a virtual machine
  • You will need to download the installation CD, use the netinst variant
  • No desktop environment and no print server (to save time)

Debian support and help

• Limitations and severe problems of the current stable distribution are described on the release pages
• Mailing lists: debian-user@lists.debian.org
  • Subscribe/Unsubscribe: https://lists.debian.org/debian-user/
  • Non-english versions also exist
• IRC channel: #debian on freenode
• Bug reports: https://bugs.debian.org
  • Use reportbug to submit new bugs, then read your email
• Blogs of Debian people: http://planet.debian.org
• Locally in your system

sudo apt install debian-handbook debian-reference harden-doc apt-doc debian-refcard

System configuration

• Console basics
• The filesystem
• Important files

Console basics

• The default interactive shell is Bash
• The default /bin/sh shell is dash
  • Reconfigure: dpkg-reconfigure dash
  • Watch for incompatibilities when copying scripts from non-Debian systems
    • Or just stick #!/bin/bash at the top, not #!/bin/sh

Shell startup files

• /etc/profile, ~/.profile for login shells
  • Place envronment variables here, export them
• /etc/bash.bashrc, ~/.bashrc for all shells
  • Place PS1, functions, aliases here

Exercise

• Find the place which appends a '#' to the root prompt
• Find the place which sometimes makes the user command prompt green
• Make root command prompt red
• Add /opt/nobleprog/bin to $PATH
• Install git and clone a repository
  • apt-get install git
  • git clone https://github.com/git/git.git
• Make sure that bash prompt shows the current git branch
  • Google for instructions
    • Beware: bad instructions exist
  • Which files did you modify, and why?

Skipped topics

• GUI system administration tools
  • The only important one is Synaptic, to be covered later
  • No GUI tools exist due to Debian Release Policy: "Packages must not modify other packages' configuration files except by an agreed upon APIs"
• Virtual consoles
• How to leave the command prompt
• How to shutdown the system
• Recovering a sane console

Sudo

• Debian uses su, not sudo, by default
• Can be made to use sudo if you don't provide the root password at installation time
• There is a sudo group, members can use sudo

The filesystem

• Debian follows FHS
  • RedHat follows it, too
  • One important difference: shared library location
    • /usr/lib/x86_64-linux-gnu

Midnight Commander (MC)

• A console-based filesystem explorer
  • Similar to Norton Commander for DOS
    • Use arrows, Enter and Fx keys to interact
  • Popular in EU and Russia: "How can one manage without it?"
  • Unpopular in USA: "Why are you still using this thing of the past? And typing to this one-line-high command prompt? Looks like you haven't learned ls and cp"
• Important difference from the default upstream configuration:
  • Debian disabled "use internal edit" by default
  • You get vim or nano
  • Even if you select mcedit, you can't get the shell (Ctrl+O) while editing a file
    • Fix: F9 > Options > Configuration > [X] Use internal edit, OK
• Tweaking MC

The basic Unix-like work environment

• The default shell is bash
  • Don't like it? chsh
• The easiest editor is nano (for those who don't like mcedit)
  • Change with select-editor
    • The selection is stored in ~/.selected_editor
    • $EDITOR and $VISUAL environment variables still work
    • vi is not vim
  • Here is how to quit:
    • Vim: :q! (without saving), or :wq (save)
    • Nano: Ctrl+x
    • Emacs: Ctrl+x Ctrl+c
    • mcedit: F10, or Esc 0

Important configuration files

• /etc/hostname sets the hostname
  • It also has to be mapped to 127.0.1.1 in /etc/hosts
    • Not 127.0.0.1
    • 127.0.0.1 should only map tp localhost and localhost.localdomain (canonical name)
    • An attempt to map the real hostname to 127.0.0.1 would make localhost.localdomain its canonical form, which is wrong
• /etc/localtime symlink sets the timezone
• /etc/default/locale sets the locale
• /etc/default/console-setup sets the console font
• /etc/default/keyboard sets the keymap (shared between console and X)

Debian package management

• Debian package management prerequisites
• Basic package management operations
• Examples of aptitude operations
• Advanced package management operations (on day 2)
• Debian package management internals (on day 2)
• Recovery from a broken system
• Tips for package management

General recommendations

• Use packages
• If possible, use packages from official sources only
• Update them regularly

Why packages?

• Dependency tracking
  • Tracking explicit vs implicit installation
• Tracking files
  • When was the last time some old file was left over in your web app?
• Configuration file management
• Ability to report bugs to Debian and benefit from bug reports made by others
• Security support

Problems with third party packages

• Package conflicts between packages in the Debian archive and packages in third party repositories.
• Bad package upgrades between Debian releases.
• Difficulty removing third party packages once installed.
• Removal of unrelated packages to satisfy dependencies for third party packages.
• Third party packages overriding/ignoring local system/user configurations.
• etc.

As such, it is not recommended to install packages from third party repositories.

Text copied from: https://wiki.debian.org/MultimediaCodecs

APT and DPKG

• dpkg deals with packages as *.deb files
  • Checks dependencies
  • Installs & updates only given packages
  • Deals with configuration files
  • Runs maintainer scripts
  • Somewhat similar to rpm
• apt deals with packages in repositories
  • Resolves dependencies
  • Downloads packages and gives to dpkg for installation
  • Somewhat similar to yum or dnf

Repository configuration

• /etc/apt/sources.list
• or /etc/apt/sources.list.d/*.list
  • Handy for third-party repositories

Repository entries

The traditional format is:

deb http://httpredir.debian.org/debian/ stretch main
deb-src http://httpredir.debian.org/debian/ stretch main
deb http://security.debian.org/debian-security stretch/updates main
deb-src http://security.debian.org/debian-security stretch/updates main

• deb or deb-src: are we talking about source or binary packages?
• 2nd field: base URL
• 3rd field: suite
• 4th and further fields: components
  • main contrib non-free
• An alternative RFC822-based format is also supported in apt 1.1+
• It is recommended to add repositories by hand

Additional official repositories

• Backports
  • Newer versions of certain software, recompiled to run on Debian Stable
  • Violate the "no new versions ever" promise

deb http://ftp.debian.org/debian jessie-backports main

• Updates
  • Some software (like timezone data and virus scanners) needs to be updated frequently
  • All software from Updates will be included in the next point release anyway

deb http://ftp.debian.org/debian jessie-updates main contrib

Proposed updates

• Proposed-updates for stable
  • Preview of what will be included in the next point release
  • Not well-tested, can be buggy

deb http://ftp.debian.org/debian jessie-proposed-updates main contrib

• For the testing distribution, they start to exist after the freeze
  • Now this repository is empty

deb http://ftp.debian.org/debian stretch-proposed-updates main contrib

Useful third-party repositories

• Multimedia packages (not in Debian proper because of patent & license problems in USA)
  • Including libdvdcss2 and aacskeys, and including ffmpeg with more encoders enabled
    • The set of decoders in the default debian package is already complete

deb http://www.deb-multimedia.org jessie main non-free
deb http://www.deb-multimedia.org jessie-backports main

• VirtualBox

deb http://download.virtualbox.org/virtualbox/debian jessie contrib

• There are others

The corresponding GPG keys should be imported manually with apt-key add

False lead

• There is also apt-add-repository from software-properties-common
  • But the package is geared a lot towards Ubuntu and not Debian
    • Knows about Ubuntu-specific PPAs and keywerver
    • Almost useless for Debian
• Again, it is recommended to add repositories by hand

Installing packages with apt

• Refresh package lists:
  • apt update
• Install packages:
  • apt install pkg1 pkg2 ...
  • Will upgrade packages if they are already installed
  • Side effect: will mark packages as installed manually
• Install for non-default architecture (e.g. i386)
  • apt install pkg:i386
    • Secondary architectures will be discussed later

Package configuration

• Usually Debian packages have safe default configuration
  • Configuration is stored in configuration files in /etc
• Unlike in RedHat, daemons are started by default
  • They don't serve private information, and sometimes listen on 127.0.0.1
• If no sane default configuration exists, Debian has to ask
  • Remember these boxes with blue background and red title?
    • That's debconf asking questions

debconf properties

• Debconf will not reask questions unless you reconfigure the package
  • Here is how: dpkg-reconfigure package
  • Answers survive package removal, but not purge
• It is, by default, safe to edit configuration files created using your answers to debconf questions
  • Debconf is not a registry
• Want to see more questions for a particular package? dpkg-reconfigure -plow package
• Want less (or more) questions in general? dpkg-reconfigure debconf
  • This set the minimum priority (importance) of questions to ask
  • The rest are skipped and the default answer is used

Searching for packages

• apt search term1 term2 ...
  • Lists package names and short descriptions
• apt show pkg1 pkg2
  • Displays detailed information about packages
    • It is not necessary for packages to be installed

Removing packages

• apt remove pkg1 pkg2
  • Does not remove configuration files
• apt purge pkg1 pkg2
  • Removes configuration files, too
• apt [--purge] autoremove
  • Removes packages that were automatically installed but are no longer needed
  • Oops, that's in Debian Testing only. On Jessie, use apt-get [--purge] autoremove
• apt clean
  • Removes previously downloaded debs

Upgrading the whole system

• apt upgrade
  • Safe – never removes packages
  • Can install new packages due to dependencies
  • May result in an incomplete upgrade
• apt full-upgrade
  • Sometimes removes packages that became outdated, or are in conflict with the new ones
• There is an instruction how to upgrade between releases
  • This is specific to each Debian version, but the important bit is equivalent to apt upgrade followed by apt full-upgrade

Adding a secondary architecture

• Debian aupports multiarch!
• dpkg --add-architecture i386
  • You can also add armel or other non-x86 architectures if you have qemu-user

About apt-get

• In previous Debian versions, you used apt-get update, apt-cache search
• Had to remember which apt-tool does what
• Now you don't have to – just use apt
  • But apt autoremove was not in Debian Jessie. We had to use apt-get autoremove

Exercise

• Given a Debian Stable VM
  • Install postgresql
  • Install something that uses it (would need a database)
    • Can you give a full list of software in Debian that can use Postgresql?
  • Upgrade to Debian Testing (instruction)

Exercise

• Given a Debian Testing VM (NOT the result of the previous exercise)
  • Install redis
    • Connect using redis-cli
  • Install 32-bit wine (wine32)
    • Test it by running winecfg
  • Install mongodb 3.2 from mongodb repository
    • Connect using mongoc
  • Completely remove all of the above

Common dpkg operations

• dpkg -i file1.deb file2.deb
  • Install packages from deb files
  • If dependencies are unmet, they will be in half-installed state
• dpkg -r pkg1 pkg2
  • Remove packages
  • Unlike apt remove, fails if this would break dependencies
    • apt remove would suggest to remove those dependencies
  • Use dpkg -P pkg1 pkg2 to purge
• dpkg -L pkg
  • List files owned by a package

How to fix broken package installation

• Figure out why exactly it is broken
• Remove the reason of the breakage
  • This usually involves editing a configuration file or removing a bad package using dpkg -r
• apt -f install will add/remove packages as needed to fix dependencies
  • Will also fix half-installed packages by rerunning their maintainer scripts
  • apt -f install is a good way to install missing dependencies after a dpkg -i ...

Exercise

• Given a Debian Stable VM:
• Install apache
  • The package is actually called apache2
• Install nginx
  • Yes, there are non-crazy people who run both on the same machine
  • However, on Debian, this will fail
    • Need different ports!
• Fix it so that Apache listens on port 8080
  • Too easy? Purge both and try again, so that Nginx listens on port 8080
  • Still too easy? "Oh, I don't actually want Apache"

Listing packages and their status

# dpkg -l \*spell\*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                Version        Architecture   Description
+++-===================-==============-==============-=================================
ii  aspell              0.60.7~2011070 amd64          GNU Aspell spell-checker
un  aspell-bin          <none>         <none>         (no description available)
un  aspell-da           <none>         <none>         (no description available)
un  aspell-dictionary   <none>         <none>         (no description available)
un  aspell-doc          <none>         <none>         (no description available)
ii  aspell-en           2016.06.26-0-0 all            English dictionary for GNU Aspell
un  aspell-hi           <none>         <none>         (no description available)
un  aspell-mr           <none>         <none>         (no description available)
un  aspell-no           <none>         <none>         (no description available)
un  aspell6-dictionary  <none>         <none>         (no description available)
un  aspell6a-dictionary <none>         <none>         (no description available)
un  hunspell            <none>         <none>         (no description available)
un  hunspell-ar         <none>         <none>         (no description available)
un  hunspell-da         <none>         <none>         (no description available)
...

Listing removed but not purged packages

• Desired state = removed, Status = conf-files

dpkg -l | awk '/^rc/{print $2}'

• It's a good idea to purge them before upgrading the whole distribution

dpkg -l | awk '/^rc/{print $2}' | xargs dpkg --purge

Files used by apt and dpkg

• /var/lib/apt/lists
  • Package lists
• /var/cache/apt/archives
  • Downloaded packages
  • Handy for downgrading if anything breaks
• /var/lib/dpkg/info
  • maintainer scripts (preinst, config, postinst, prerm, postrm)
• /var/lib/dpkg/status
  • What is installed, what is half-installed, what is removed but not purged

Emergency downgrading

• The log of all package operations is in /var/log/dpkg.log
• Look for "upgrade" lines after the date where everything broke
• Both old and new versions are listed
  • So now you know what to downgrade to
  • The which-pkg-broke tool from debian-goodies will also list suspects
• Look for old packages in /var/cache/apt/archives
  • Last resort: download debs from https://snapshot.debian.net
• dpkg -i ...

Preventing re-upgrades to broken package versions

• Temporary solution: apt hold pkg1 pkg2
• Then undo the hold: apt unhold pkg1 pkg2

Exercise

• You are given a Debian Testing system with MATE desktop
• It has a bug: MATE Terminal has a weird color scheme and crashes if one changes it
  • This is actually a bug caused by GTK+ 3.21
• Fix it by downgrading GTK+ to the last version available before 3.21
• Note to the trainer: this was valid on 2016-09-18. For the next training, find another broken package and update this slide.

Exercise

• Given a Debian Testing VM:
• Install Opera
  • It's a proprietary web browser
• Which files are installed?

Exercise

• Given a Debian Stable VM:
• Switch to sysvinit instead of systemd
  • apt install sysvinit-core ; reboot
  • We need this in order to reproduce the bug
• Install kexec-tools
• Try to remove them
  • You can't (it just reboots)
    • Known bug, but please don't cheat by looking or googling
• Remove them

Package dependencies – positive ones

DPKG has more expressive power for dependency specification than RPM

If a is installed...

• a Depends on b: b absolutely must be installed for a to operate properly
• a Pre-Depends on b: b absolutely must be installed for a to operate properly
  • also a's pre-installation script can rely on b
• a Recommends b: b should be installed in most cases when a is installed
• a Suggests b: b may be useful in some cases when a is installed
  • Mostly just a hint to the user

Unversioned dependencies on so-called Essential packages are implied and should not be declared explicitly

Authoritative source: Debian Policy Manual

Package dependencies – negative ones

If a is installed...

• a Breaks b: b should be deconfigured
  • Usually, in order to be upgraded
• a Conflicts with b: b's files should not even exist in the filesystem

Virtual packages and file moves

If a is installed...

• a Provides b: the dependency resolver should think that b is also installed
  • Even if b does not exist at all
• a Replaces b: two meanings, both encountered together
  • Don't complain if a contains files with names already in b
    • They don't belong to b anymore
  • If there is a package conflict, resolve it by removing b instead of complaining

Deselecting unwanted recommendations

• By default, recommended packages are installed, and suggestions aren't
• Sometimes a recommendation is too heavy and is not really needed
  • Pass its name, append a minus
    • apt install wanted unwanted-
• To install with no recommendations at all: apt --no-install-recommends foo
  • This is not recommended

aptitude

• An alternative to apt
  • Not really recommended, but sometimes useful
  • I use it for cleaning out packages after changing server roles
• Can install, upgrade, remove, purge, search packages
  • aptitude [install|remove|purge|search|show|hold|unhold] ...
  • aptitude [update|safe-upgrade|full-upgrade]
• Also can be started without arguments
  • Presents a curses-based interactive interface then

aptitude keybindings (part 1/2)

• Enter: expand a category
• Enter: provide package information
  • q to make it go away
• +: mark for installation
• -: mark for removal
• _: mark for purging
• :: don't mark
• g: go!
• q: don't go
• F10 or Ctrl+T: menu with even more actions

aptitude keybindings (part 2/2)

• u: update package lists
• U: mark all possible packages for upgrade
• /: search
  • More info later
• n: repeat the last search
• l: search, and hide what doesn't match the criteria
• m: mark as manually installed
• M: mark as automatically installed
• =: hold
• :: don't hold

aptitude: other useful commands

• aptitude why pkg
  • Finds a dependency chain that installs the package
    • Even if the package is not actually installed
      • yes, this is confusing – treat it as "why would I want this package?"

aptitude: dealing with broken dependencies

• Breakage happens e.g. when you mark a package for removal without marking its reverse dependencies for removal
• Broken packages are highlighted in red
• Aptitude automatically presents the full list of problems at the bottom
  • Press e to solve them
  • Press , or . to navigate the list of proposed solutions
  • Press r to tell aptitude that you reject all solutions with the highlighted action
    • E.g.: don't ever try to install this bloated junk instead of what I just tried to remove
  • Press a to tell aptitude to always do the highlighted action
  • Usually the first solution is too destructive, but the second is OK
  • Apply the solution with the ! key
  • There is always an option to solve broken dependencies manually

Exercise

• Given a VM with Debian Stable that was installed earlier:
  • Remember that I asked you to deselect Desktop Environment and Print Server?
  • Look: libgtk2.0-0 is still there
    • And also many other seemingly-junk packages that are related to the graphical session
  • Figure out why they are here
  • Safely remove them

aptitude search syntax

• By default, searches for name substring
  • That's only a convenience shortcut for ?name(foo)
• Other possible terms: ?version(4\.0), ?description(yacht)
  • Versions and other search strings are interpreted as regular expressions, that's why \.
• Some other special terms: ?installed, ?obsolete, ?broken, ?upgradable
• Boolean expressions: ?and(a, b), ?or(a, b), ?not(a)
  • Shorthand forms: a b, a | b. !a
• For loops
• Full list of possible terms does not it on one screen

Example

• Purging removed packages for good:
  • aptitude purge '?config-files'

Day 2

• System initialization
• Network configuration
• More apt/dpkg tips
• Advanced package management topics

The system initialization

• Supported init systems: sysvinit, systemd, upstart
  • Systemd is the default in Debian Jessie and Stretch
  • Sysvinit was the default before that
  • We'll cover sysvinit and systemd
    • Upstart is essentially dead

Switching the init system

• I intentionally don't answer why you would want this
• From systemd to sysvinit

apt install sysvinit-core
reboot
apt purge systemd

• From sysvinit to systemd

apt install systemd-sysv
reboot
apt purge sysvinit

• Sometimes you'll need to care about systemd-shim
  • Provides some services that originally come from systemd to sysvinit-based system
  • Debian-specific, not supported anywhere else, voids your right to report bugs upstream

Installing without systemd

• Add to the installer kernel command line:

preseed/late_command="in-target apt-get install -y sysvinit-core"

Exercise

• Does GNOME work in Debian Testing without systemd?

Overview of the bootstrap process

• BIOS → MBR → GRUB → Kernel and initrd → /sbin/init → everything else
• UEFI → GRUB → Kernel and initrd → /sbin/init → everything else

Meaning of the word "runlevel"

• Sysvinit: a numbered set of instruction to start and stop services
  • 0 = shutdown, 6 = reboot, 1 = single-user recovery
  • RedHat: 3 = text mode, 5 = graphical session
  • Debian: 2 = normal operation
• Systemd: "runlevels" do not exist. There are "targets".
  • graphical.target, multi-user.target, ...
    • systemctl get-default
    • systemctl set-default graphical
      • But better don't. You'll break update-rc.d

How sysvinit boots the system

• Reads /etc/inittab
• At system initialization, runs /etc/init.d/rcS
  • So it runs scripts in /etc/rcS.d/ with the "start" parameter
    • see /etc/init.d/rcS, /etc/init.d/rc
• The default runlevel is 2
  • This runs scripts from /etc/rc2.d/ with the "start" parameter
• A shutdown or reboot, uses runlevel 0 or 6
  • This runs scripts from /etc/rc{0,6}.d/ with the "stop" parameter
• Ordering of scripts is determined by their names
  • Names start with numbers generated automatically by update-rc.d according to dependencies in LSB headers

How systemd boots the system

• Performs a number of hard-coded tasks like starting systemd-udevd, setting keymap, ...
  • Mostly equivalent to what scripts in /etc/rcS.d/ did
• Reads service files from /lib/systemd/system and /etc/systemd/system
  • Vendor files are in /lib, custom services and sysadmin overrides are in /etc
• Resolves dependencies of default.target
• Generates additional services that run sysvinit scripts
  • Considers only rc2.d, not rcS.d
• Runs services in parallel
• On shutdown/reboot, resolves dependencies of {shutdown,reboot}.target

Starting and stopping services manually

• Wrong way: /etc/init.d/something start
  • Inherits the environment from the command prompt
• Systemd way: systemctl start something
  • Works with systemd only
• Generic and recommended way: service something start
  • Runs sysvinit scripts in a predictable environment
  • If the system has been booted with systemd, delegates all the work there

Configuration of runlevels

• Just Don't. It will break when they force systemd on us all.
  • But you can still enable and disable services

Disabling and enabling services

• update-rc.d: works both for systemd and sysvinit, synchronizes states
• systemctl enable/disable: also works
  • Debian-specific feature: also disables the same service for sysvinit
• systemctl mask
  • like "disable", but will also make it impossible to start the service manually or as a dependency

Exercise

• Given a Debian Testing VM:
• Install memcached
• Make sure that it doesn't start automatically
  • There is a setting in /etc/default/memcached just for this, right?
  • Why doesn't it work?

Exercise

• Given a Debian Testing VM and memcached:
• Configure it to run two memcacheds on different ports
  • Do it via systemd service files

Network configuration

• For desktops: use NetworkManager
  • In some cases, unavoidable
    • Some packages depend on NetworkManager
    • You can disable it and use ifupdown, but then some packages will think that there is no internet
• For servers: use ifupdown
  • Although NetworkManager developers add relevant features, too!
  • There is also systemd-networkd

Network device support

• Common problem: firmware
  • Moved to non-free section due to licensing reasons
  • Affects Broadcom NICs, Intel wireless cards, and many other devices
• Solution: install firmware
  • e.g. apt install firmware-bnx2
    • Requires network!
• Alternative installer CDs with firmware

NetworkManager

• Supports Ethernet, WiFi, mobile connections, DSL
• Bonds, VLANs, bridges
  • Going to server market!
• Various kinds of VPNs

NetworkManager configuration tools

• Graphical configuration
• nmtui
• nmcli

NetworkManager configuration files

• /etc/NetworkManager/system-connections/*
  • Users in the netdev group can change these connections
• Available fields: see documentation (man nm-settings-keyfile)
• If you change files by hand, call nmcli con reload

NetworkManager limitations

• No 6in4/6to4/teredo tunnels
• No tinc VPN
  • You can still configure tinc manually
• No PPPoE over WiFi
  • Still required by some German ISPs
• Only one VPN active at a time
  • May be fixed soon (network-manager-1.4.0 is already in Unstable)

Exercise

• Reconfigure the network so that it becomes a bridge with one port
  • It should still use DHCP
• Use nmtui or a graphical tool

Exercise

• Make sure that the VM's bridge obtains its IP address via DHCP, but uses only 8.8.8.8 as a DNS server
  • Do it by editing system connection files directly and reapplying configuration

ifupdown

• See /etc/network/interfaces, /etc/network/interfaces.d/
• man 5 interfaces
  • Only basics are covered there
• Add-on packages like bridge-utils and vlan are needed for advanced features
  • e.g. man 5 bridge-utils-interfaces

Static routes

• There are guides that add them from "up" and "down statements in the interface definition
• The proper way is to install ifupdown-extra package
  • It reads /etc/network/routes

resolvconf

• There are multiple sources of nameserver configuration
  • Static configuration
  • DHCP
  • VPN
  • Local nameserver
• Task: coordinate the contents of /etc/resolv.conf
  • Just install the resolvconf package

Exercise

• Again, create a bridge with one port
  • Use /etc/network/interfaces

Firewall

• Default: no firewall at all
  • Why would one need it?
    • Some packages don't provide IP ACLs on their own
    • Reduces attack surface
    • Required by some regulations
  • Good policy: don't install unneeded network servers
    • Or configure them to listen on 127.0.0.1 if they are for local service only
    • The default install contains rpcbind and nfs-common
      • Purge them if you don't need tem

Available iptables frontends

• Ferm
• UFW
• apt search firewall

Beyond iptables

• ebtables: filter packets passing through a bridge
• arptables: filter ARP packets
  • Use ip6tables for NDP

Starting firewall at boot

• Use one of the frontends
• Or, iptables-persistent
  • Purpose: restore iptables rules at boot
  • Use netfilter-persistent save to save the initial configuration
  • Or, edit /etc/iptables/rules.v4

Common firewall mistakes

• Loading it too late
  • Happens with custom scripts
  • Does not happen with iptables-persistent
• Forgetting about IPv6
• Forgetting about IPv6 link-local addresses

Network fine-tuning

• MTU (Maximum Transmission Unit)
  • Can be set in /etc/network/interfaces or via NetworkManager
• Various sysctls
  • Set them in /etc/sysctl.conf or /etc/sysctl.d/yourfile.conf
  • E.g. here is how to disable IPv6 on a particular card:
    • net.ipv6.conf.eth1.disable_ipv6 = 1

Tips for package management

• Use deborphan
  • It finds obsolete library and transitional packages
• Use needrestart
  • It warns if service need restart due to a library upgrade
• Consider unattended-upgrades
  • It can install security updates automatically
    • But does not restart services due to library upgrades
    • And you risk rebooting the server in the middle of upgrade
    • Relatively safe on Ubuntu (because it's enabled by default there), but I wouldn't use it on Debian
  • Consider using it together with apt-listchanges and apticron
  • Read the excellent LWN article about it

Finding modified files

• debsums
  • Checks MD5 checksums of installed files against those from packages
  • Needs initialization after installation: debsums_init
    • debsums is intended primarily as a way of determining what installed files have been locally modified by the administrator or damaged by media errors and is of limited use as a security tool.
• You can use e.g. tripwire for detecting unauthorized alterations of files

Converting RPMs

• Use alien
  • Or better don't

Duplicating package state

• Use apt-clone
  • Saves/restores /etc/apt/sources.list, keys and package states

Avoiding debconf prompts

• Sometimes it's boring to answer the same debconf prompts on multiple machines
  • Preseed them!
  • On the source machine: apt install debconf-utils
  • On the source machine: debconf-get-selections > selections.txt
  • Remove unneeded lines, copy selections.txt to other machines
  • On target machines: debconf-set-selections < selections.txt

More tips

• Install a mail server (exim4 or postfix or something else)
  • So that you are notified about failing cron jobs, bad disks, and other important events
  • exim4 is there by default
• Read logs
  • logwatch will send you the interesting bits via email
• Read changelogs of packages that you care about
  • zless /usr/share/doc/package/changelog.Debian.gz
  • apt-listchanges
• Read README.debian.gz and NEWS.debian.gz

Graphical software installation tools

• synaptic
  • Can install, upgrade, configure, search packages
  • Offers features similar to apt
• gnome-software
  • Not a package manager
    • Focused on Apps, not Packages
    • User should not know about low-level packages
    • Displays only graphical apps
    • Displays stars, reviews and ratings stored on a central GNOME server

Configuring APT

• Files in /etc/apt/apt.conf.d
  • *.conf or no extension
  • Full list of options is available in man apt.conf, man apt-get and so on

Excercise

• Given a VM with Debian Testing:
• Limit apt-get download bandwidth to 500 KB/s
• Disable downloading of incremental differences (PDiffs) of Packages files
  • This will force full downloading of these files
  • Useful on fast connections
• Make sure that apt shows versions when installing and upgrading packages

How to unpack deb on non-debian system

• A deb file is just an ar(1) archive
  • ar comes from binutils
• Contains data.tar.gz and control.tar.gz
• data.tar.gz contains package files
• control.tar.gz contains files with metadata

So:

• ar x file.deb
• tar xpf data.tar.gz
• tar xpf control.tar.gz

Exercise

• Try unpacking a deb file manually
  • http://ftp.us.debian.org/debian/pool/main/r/redis/redis-server_2.8.17-1+deb8u5_amd64.deb

What's in debian package filename

x11vnc_0.9.13-1.2+b2_amd64.deb

• x11vnc: package name
• 0.9.13-1.2+b2: version
• amd64: architecture
  • "all" means "no architecture-dependent files"

What's in debian package version

0.9.13-1.2+b2

• 0.9.13: version that was released by the upstream software authors
• 1.2+b2: debian build version
  • Missing for packages where debian == upstream
  • 1: the number of the last build by the maintainer
    • That's the only debian-specific version component in the case if the last build was indeed by the maintainer
  • 2: the number of the last build by someone else since the last maintainer upload
  • b2: the number of automated no-changes rebuilds since then
  • Other suffixes exist

Building Debian packages

• You need build-essential
• Read Build-Depends
• Extract source package (see next slide)
• Build it

Downloading and extracting source packages

• You need deb-src in /etc/apt/sources.list
  • apt-get build-dep pkg1 downloads dependencies
  • apt-get source pkg1 downloads sources
• Alternatively, use dget from devscripts
  • It needs an URL for the dsc file

Exercise

• Extract the source package for indent
  • Try to do the same extraction manually (tar + patch)

Building the package

• Two layers of tools
  • The build is driven by debian/rules
    • It is a Makefile
    • fakeroot debian/rules binary
  • dpkg-buildpackage -uc -us (from devscripts) offers additional checks

Exercise

• Rebuild indent without changes
  • Please bump the version in debian/changelog using dch

The equivs package

• Builds a dummy package
  • Useful to satisfy dependencies if you built something from sources
    • Even better is to produce your own package - learn debhelper tomorrow!

Mirroring official archive

• ftpsync
  • Alternative: debmirror
  • Want a proxy instead? apt-proxy, apt-cacher, apt-cacher-ng

Day 3

• The mail subsystem
• SSH configuration
• PAM
• SELinux and AppArmor
• Kernel modules and device permissions
• Compiling a kernel for Debian
• Debianization of third-party software
• Building your own archive of debian packages

The mail subsystem

• Default: exim4
• You can install Postfix or Sendmail
  • Disclaimer: I don't like Exim
    • Very verbose default configuration files
    • Postfix configuration is much more compact (20 vs 1018 non-comment lines)

exim4 in Debian

• Packages:

exim4 - metapackage to ease Exim MTA (v4) installation
exim4-base - support files for all Exim MTA (v4) packages
exim4-config - configuration for the Exim MTA (v4)
exim4-daemon-heavy - Exim MTA (v4) daemon with extended features, including exiscan-acl
exim4-daemon-heavy-dbg - debugging symbols for the Exim MTA "heavy" daemon
exim4-daemon-light - lightweight Exim MTA (v4) daemon
exim4-daemon-light-dbg - debugging symbols for the Exim MTA "light" daemon
exim4-dbg - debugging symbols for the Exim MTA (utilities)
exim4-dev - header files for the Exim MTA (v4) packages
exim4-doc-html - documentation for the Exim MTA (v4) in html format
exim4-doc-info - documentation for the Exim MTA (v4) in info format

Light vs heavy exim4 daemon

Features found in the heavy version:

• User lookups in third-party stores:
  • LDAP, sqlite, PostgreSQL, MySQL
• SASL and SPA SMTP authentication
• embedded Perl interpreter
• exiscan-acl for integration of virus-scanners and SpamAssassin

How to configure exim4

• In simple cases, dpkg-reconfigure exim4-config should be enough
• In other cases, edit /etc/exim4/conf.d/*
  • Splitting configuration into small files is recommended for easier upgrades

Rerouting mail

• /etc/aliases
  • For non-exim mail servers, also run newaliases
  • |/path/to/script also works

Reading local mail

• By default mail is stored in /var/mail/$USER in mbox format
• Use mail or mutt to read
  • Better, redirect

Configuring ssh

• /etc/ssh/sshd_config for the server
• /etc/ssh/ssh_config for the client
• By default Debian Jessie (and Stretch) denies root login using password
  • Key-based login is still possible and useful for backups

Limits on connection attempts

• Brute-force password-guessing attacks
• MaxStartups directive
• Common problem: attacks break legitimate login attempts (usually seen by cron tasks)
• fail2ban?
  • Problem: not really parseable logs
    • ssh 'nobleprog from 8.8.8.8'@192.168.122.73

Exercise

• Create a group "ssh-users"
• Reconfigure sshd so that only root (with keys) and ssh-users can connect
  • Test key-based login
• Create an exception for one particular IP address

PAM

• Pluggable Authentication Modules
  • Not specific to Debian
    • Useful e.g. for LDAP authentication
    • Often configured together with NSS (/etc/nsswitch.conf)
  • Difference from RedHat: no authconfig
    • There is pam-auth-update
  • You can also edit files in /etc/pam.d
    • Common settings are in common-* files
    • Service-specific settings are in other files

How PAM works

• Provides four services
  • auth: how to authenticate user (e.g. how to check password)
  • account: how to check that access is allowed (e.g. enforce time-of-day restrictions)
  • password: how to change the password
  • session: what else to do when the user logs in (e.g. register it with systemd)

PAM directives

• In general, modules can succeed, fail, or want to be ignored
  • requisite: module must succeed. Don't look further if it fails.
  • sufficient: Don't look further if it succeeds.
  • required: module must succeed. Go further if it fails, to confuse the attacker.
  • optional: use the result only if everything else wants to be ignored
  • include: include the directives from the other file
  • substack: same as include, but don't short-circuit the whole stack if a "requisite" or "sufficient" works in a subscack

Exercise

• Make sure that only members of group root can use su

SELinux

• Mandatory access control
• Based on labels and contexts
• Prevents processes from accessing files that they shouldn't access
• Not installed by default
• Guide: https://wiki.debian.org/SELinux
  • Comes with a recommendation to use Postfix instead of Exim4
• Recommendation: turn it on in a permissive mode, see if there are any denials
• Then audit2allow if needed, and then enable with enforcing=1 in /etc/default/grub

Exercise

• Try to configure SELinux on Debian Jessie
• Try again with a graphical session
  • It will fail
  • That's why you use it on servers only, and with sysvinit only

AppArmor

• Another mandatory access control system
• Used by default in Ubuntu
• Guide: https://wiki.debian.org/AppArmor/HowToUse

Kernel command line

• Usually set in /etc/default/grub
  • Run update-grub to apply changes
• Change with e at GRUB boot screen
• Useful recovery option: init=/bin/bash
  • Want more debug? add debug, remove quiet
• For initrd debugging:
  • break=premount
  • man initramfs-tools

Kernel modules

• For hardware that is present in the system, udev directs kmod to load the necessary modules
  • No action is needed
  • You can fine-tune module arguments using files in /etc/modprobe.d
    • options i915 preliminary_hw_support=1
    • Alternatively use kernel command line: i915.preliminary_hw_support=1

Kernel modules, more tips

• Want to disable autoloading of a specific module?
  • blacklist badmodule in /etc/modprobe.d/somefile.conf
  • or blacklist=badmodule1,badmodule2 on the kernel command line
• Want to load some module that is not loaded by default?
  • Put its name in /etc/modules or in /etc/modules-load.d/somefile.conf

Third-party kernel modules

• You need linux-headers package matching the running kernel
• Debian has module-assistant
  • Can build packaged third-party modules from source
• There is also dkms

Exercise

• Install xtables-addons modules

Custom kernel

• You can package it as a deb using make-kpkg from kernel-package
  • The official kernel sources are in the linux-source package
  • nano /etc/kernel-pkg.conf
  • make menuconfig
  • fakeroot make-kpkg --initrd --revision=1.0.custom kernel_image
    • will create something like linux-image-3.16-amd64_1.0.custom_amd64.deb

Storage

• Debian installer supports partitions, RAID, LVM, iSCSI
  • Does not support LVM thin provisioning
• Debian initramfs-tools support more
• Dracut (originally from RedHat) supports booting from almost everything in the world
• If you want to install on strange storage, don't use Debian installer
  • Use debootstrap, then install a boot loader
    • Debootstrap is good for creating Debian chroots, too!

Exercise

• Create a Debian Testing chroot on a Debian Stable system
  • Configure schroot so that users can safely chroot there

Automated installation

• Append auto url=... to debian-installer kernel command line
• Here is what to put at the URL: https://www.debian.org/releases/jessie/example-preseed.txt

Why build a custom package

• New version (backport)
• Special build options
• Fix a bug
• New and internal development

Useful resources

• Debian Policy
• Debian New Maintainers' Guide
• Package search
• Package tracking system
• List of other manuals

Source packages

• Source package = *.orig.tar.gz + *.debian.tar.gz + *.dsc
  • *.orig.tar.gz = original release tarball from upstream
  • *.debian.tar.gz = debian patches + debian files that drive the build
  • *.dsc = metadata

Source packages and binary packages

• From one source package, many binary packages are built
• Example: openssl_1.0.1b-1.dsc (+tars) → 5 packages
  • libssl-dev_1.0.1b-1_amd64.deb, libssl1.0.0-dbg_1.0.1b-1_amd64.deb, libssl1.0.0_1.0.1b-1_amd64.deb, openssl_1.0.1b-1_amd64.deb, libssl-doc_1.0.1b-1_all.deb
  • Note: architecture-dependent vs architecture-independent packages

Important files that drive the build

• debian/changelog: changelog
  • First line also determines the package version
  • dates are date -R
  • use dch to add entries
• debian/control: binary package names, dependencies
• debian/copyright: copyrights
• debian/rules: Makefile that drives the build
• debian/source/format: how the sources are organized. *.orig.tar.gz + *.debian.tar.gz = "3.0 (quilt)"
  • For internal packages that have debian directory in the same repository, use "3.0 (native)"
• debian/patches/series: list of patches to apply

Mechanics of building a package

• Targets in debian/rules:
  • build: builds the program. Does not require root.
  • build-arch, build-indep (both optional): builds only architecture-dependent or architecture-independent parts
  • binary: builds all binary packages (typically runs under fakeroot)
  • binary-arch, binary-indep (both required): build only architecture-dependent or architecture-independent binary packages
  • clean: cleans up after build and binary

But it doesn't look like this!

• Most debian/rules files are now based on debhelper 7+
  • Simplest possible (and in many cases working) debian/rules file:

#!/usr/bin/make -f
# Uncomment this to turn on verbose mode.
#export DH_VERBOSE=1
%:
        dh $@

• CDBS was also popular

Debhelper mechanics

• Debhelper uses mind-reading technology. Reads developer's mind, not yours!
  • Needs a hint (mind reader version number) in debian/compat. The correct level is 9 in Jessie
    • Don't forget Build-Depends: debhelper (>= 9)
  • Contains simple rules like these:
    • There is a setup.py file ⇒ let's build it like a typical python setuptools/distutils package
    • After the build, there is a shared library in the standard location ⇒ let's fix permissions and add a call to ldconfig to postinst
    • There is debian/$package.init.d ⇒ let's use it as a sysvinit initscript and start it in postinst
    • There is a debian/$package.service ⇒ must be a systemd service, let's install it as such

Fine-tuning debhelper

• Sometimes the default build is not good enough
  • Add DH_VERBOSE=1
  • Read what dh_* commands are executed
  • Find which one does the wrong thing (example: dh_what_went_wrong)
  • Read its manual page
  • Add override_dh_what_went_wrong: target
    • call dh_what_went_wrong with the correct arguments, explaining to it what to do
    • or just do it as plain shell commands if that's easier

Exercise

• Package dhcdrop

Producing your apt archive

• See the structure at http://ftp.debian.org/debian/
• Use apt-ftparchive or aptly
  • Or any other suitable tool
• Sign your release file with GPG

Additional exercises

Debian exercises