Subversion for Administrators

From Training Material
Revision as of 14:09, 20 May 2022 by Lsokolowski (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search


title
Subversion for Administrators
author
Bernard Szlachta (NobleProg Ltd), Lukasz Sokolowski

Subversion for Administrators

Subversion for Administrators Training Materials

Agenda ⌘

  • Repository Structure (trunk per project, many projects)
  • Protocols (File Access, Standalone, Apache, SSH)
  • Logging
  • Lock administration
  • Removing files from repository (svndumpfilter)
  • Hooks
  • Svnsync
  • Upgrade/verify/dump/load/hotcopy
  • Encrypted datatransfer

Setting up Subversion ⌘

  • Choosing from connection-possiblilities (http://, https://, svn://, svn+ssh://)
  • Choosing a repository structure (one Repository for all or one per project)?
  • Choosing a permission structure (how many users and groups, who may read/write to which path/repository)?

Files structure in repo ⌘

  • conf/
    • This directory is a container for configuration files.
  • db/
    • This directory contains the data store for all of your versioned data.
  • format
    • This file describes the repository's internal organizational scheme.
  • hooks/
    • This directory contains hook script templates and hook scripts, if any have been installed.
  • locks/
    • Subversion uses this directory to house repository lock files,
      used for managing concurrent access to the repository.

Subversion Commands ⌘

  • svn
  • svnadmin
  • svnlook
  • svnserve
  • svn2cl
  • svndumpfilter
  • svnrdump

svnadmin ⌘

Tool for maintenance of repo

Removing dead transactions

svnadmin lstxns myrepos
svnadmin rmtxns myrepos `svnadmin lstxns myrepos`

Packing FSFS filesystems

svnadmin pack /var/svn/repos


svnlook ⌘

Tool for diagnostic purposes

  • for examining the various revisions and transactions (which are revisions in the making) in a repository
  • do not change the repo
  • typically used by the repository hooks for reporting the changes that
    • are about to be committed
    • or that were just committed

svnserve ⌘

simple standalone service

svndumpfilter ⌘

to easily modify streams of Subversion repository history data by acting as a path-based filter

svnrdump ⌘

svnadmin dump and svnadmin load subcommands, rolled up into a separate program

Repository Structure ⌘

SvnArchitAdmin.png

Repo Structure ⌘

Module, Project, Library, etc...

  • Repository per project
  • Modules in root
  • Modules in trunk
    • Modules tagged separately
    • Modules tagged together

Modules in root ⌘

SvnModRootAdmin.png
  • each module has to be checked out separately
  • commits can not span multiple modules
  • tags can not span multiple modules in single commit
  • most common layout

Modules in trunk ⌘

SvnModTrunkAdmin.png

Modules tagged separately

  • checkout of all modules is possible
  • commits can span multiple modules
  • tags can not span multiple modules in single commit (workaround by 3rd party tools)‏

Modules in trunk con't ⌘

SvnModTrunkTogAdmin.png

Modules tagged together

  • checkout of all modules is possible
  • commits can span multiple modules
  • tags can span multiple modules in single commit
  • maybe difficult to determine relevant modules of a tag

SVN Backend ⌘

Header text FileSystem [„fsfs“] Berkeley DB[„bdb“]
Data integrity stable Difficult to deploy, but extremely reliable
Robustness robust fragile (“wedged”)
Usable from a read-only mount yes no
Platform-independent storage yes no
Usable over network filesystems yes no
Repository size smaller larger
Scalability: number of revision trees Depends on OS/FS (fixed in SVN 1.5) No problems
Large commits faster slower
Directories with many files faster slower
Group permissions handling (unix) no problems Umask problem

Creating a repository ⌘

you can create repositories in any empty directory:

>svnadmin create /path/to/repository
>svnadmin create --fs-type fsfs /var/svn/repos
>svnadmin create --fs-type bdb /var/svn/repos

Protocol Comparison ⌘

http:// https:// svn:// svn+ssh://
authentication all apache auth methods CRAM-MD5, SASL SSH
useraccount private userfiles or any other auth method private userfiles or SASL system accounts
authorization path-based authorization path-based authorization no path-based authorization
encryption optional SSL optional SASL always SSH
logging special configured logs no logging no logging
interoperation WebDAV clients svn clients only svn clients only
web access limited support no support no support
speed slower faster faster
complexity fair simple fair

Svnserve daemon ⌘

Setting up Subversion as a standalone service just requires a simple command:

>svnserve –d –r [path to repository|repositories]

TEST:

>svn ls svn://localhost/[reponame]

Windows Service ⌘

>sc.exe create svn 
 binpath="c:\svn\bin\svnserve –-service –r [path to repo]"
 displayname="Subversion Server"
 depend= Tcpip
 start=auto

Svnserve authentication (1/4) ⌘

3 files for authentication (in repo/config directory):

  • svnserve.conf - general configuration file
  • authz – stores path based authentication data
  • passwd – stores user/password combinations

Svnserve authentication (2/4) ⌘

svnserve.conf
[general]
anon-access = read
auth-access = write
password-db = passwd
authz-db = authz
realm = My First Repository

controls what anonym users can do (read, write or none)
controls what authenticated users can do (again, read, write or none)
location of the password database
location of the authz-db
authentication realm

Svnserve authentication(3/4) ⌘

authz
[aliases]
[groups]
devs = user1, user2
[/foo/bar]
username =
[repository:/baz/fuz]
@devs = rw

aliases can provide alternative names 
groups of users 
a path in repository. Next lines will be either aliases(&), groups (@) or users followed by either “r” for read, “rw” for read-write, or nothing for no access
repositoryname can be addressed with colon(:)
Same fileformat as for apache authzfile

Svnserve authentication(4/4) ⌘

passwd

[users]
username = password
username2 = password

only a single user section

username followed by password (in clear text!)

Exercise ⌘

  • Create repo in the /repo directory
  • Create two directories harry and sally (use file protocol)
  • Create two user sally/sally and harry/harry and grant access each user access to write their own directory
  • Test it using svnserve –d and svn protocol

Apache (single repository) ⌘

LoadModule dav_module      modules/mod_dav.so
LoadModule dav_svn_module  modules/mod_dav_svn.so
[...]
<Location /svn>
  DAV svn
  SVNPath path/to/repository
</Location>

dav module must be loaded before dav_svn module (2)
dav svn module translates http-dav requests into svn operations
location on server where repository will be served
activation of svn dav module
path to Subversion repository to be served

Exercise ⌘

Make /repo repository available at http://localhost/repo

apache (multi repos) ⌘

Multi repository(without path based authorization):

LoadModule dav_module      modules/mod_dav.so
LoadModule dav_svn_module  modules/mod_dav_svn.so
[...]
<Location /svn>
  DAV svn
  SVNParentPath path/to/repositories
</Location>

dav module must be loaded before dav_svn module (2)
dav svn module translates http-dav requests into svn operations
location on server where repository will be served
activation of svn dav module
path to folder with Subversion repositories  to be served

apache authentication (1/3) ⌘

Basic authentication (without path based authorization):
<Location /svn>
  DAV svn
  SVNParentPath path/to/repositories
  AuthType Basic
  AuthName "Subversion repository"
  AuthUserFile /path/to/http-auth-file
  Require valid-user
</Location>
selection of authentication algorithm („Basic“ means password will not be encrypted)
the name of the authentication realm (will be shown on password entry)
file with user/passwords (use htpasswd to create file and add user/passwords)
only valid users can access this location

apache authentication (2/3) ⌘

Basic authentication, anonym read access (without path based authorization):
<Location /svn>
  DAV svn
  SVNParentPath path/to/repositories
  AuthType Basic
  AuthName "Subversion repository"
  AuthUserFile /path/to/http-auth-file
<LimitExcept GET PROPFIND OPTIONS REPORT>
    Require valid-user
  </LimitExcept>
</Location>
makes exception for all http-DAV read operations, so anonymous user can read 

apache authentication (3/3) ⌘

Path based authorization :
LoadModule authz_svn_module   modules/mod_authz_svn.so
[...]
<Location /svn>
  DAV svn
  SVNParentPath path/to/repositories
  AuthType Basic
  AuthName "Subversion repository"
  AuthUserFile /path/to/http-auth-file
  Require valid-user
  AuthzSVNAccessFile /path/to/access/file 
</Location>
need to load authz_svn module to enable path based authentication
location of authz accessfile

Exercise ⌘

Enforce the constraint that sally can access only /repo/sally folder and harry can access /repo/harry folder with apache

Logging (apache) ⌘

apache webserver can be configured to log any Subversion high level operation. Place this line outside of a location section:

CustomLog logs/svn_logfile "%t %u %{SVN-ACTION}e" env=SVN-ACTION

The %t and %u are standard apache logging constants meaning time and user.

Logging is only available on apache installations

svn+ssh ⌘

  • No special configuration required
  • There is NO PATH-BASED authorization
  • Good idea to use groups and
    • umask 0002
    • chmod g+ws

*Setting UP svn+ssh ⌘

  1. Install open-ssh server (e.g. apt-get install ssh)
  2. Setup a user if they don’t exists
  3. Create a group (e.g. groupadd grepo)
  4. Add the user to the group
    • e.g. usermod -a -G grepo ubuntu
  5. Change permission and ownership on the repo folders
    • chmod g+ws /repo –R
    • chown :grepo /repo -R
  6. Change default umask (e.g. in /etc/profile) to umask 0002

Exercise ⌘

  • Enable Logging as in the example show on the previous slide
  • Test with commit and checkout commands

Apache PAM authentication* ⌘

<VirtualHost *:80>
        ServerName svn.nobleprog.net
<Location /repo>
        AuthType Basic
        AuthName “Sample Repository"
        AuthPAM_Enabled On
        AuthGROUP_Enabled on
        Require group hitra
        AuthBasicAuthoritative off
        AuthUserFile /dev/null
        Require valid-user
        DAV svn
        SVNPath /repo
    </Location>
</VirtualHost>

lock administration ⌘

As administrator you can always kill locks to all files: To list locked files, you can use the lslocks command:

>svnadmin lslocks /path/to/repo [path/inside/repo]

To remove the locks use rmlocks

>svnadmin rmlocks /path/to/repo path/to/locked/file

changing log messages ⌘

Log Messages are not versioned, so changes to them will overwrite the old log message

Only administrators with file-system access can change log-message using this command:

>svnadmin setlog {path/to/repo} -r REV MSG-FILE --bypass-hooks

--bypass-hooks is important, as it will allow administrator to change revision without calling the hook »pre-revprop-change«

Caution: old data is lost!

You can allow client-side revision property changes by activating pre-revprop-change-hook

Exercise ⌘

Change the last revision log message to “I changed it”

Dump and Load ⌘

svnadmin create /repo1
svnadmin dump /repo > /tmp/dump
svnadmin load /repo1 < /tmp/dump

Removing files from repository ⌘

Subversion cannot remove files from repository, so if you really want to delete any trace of a file,
you need to filter a dump and load it back to a new repository.

SvnDelFileAdmin.png

svndumpfilter problems ⌘

svndumpfilter has problems with trailing slashes:
If your dumpfile has trailing slashes your include/exclude argument
must have trailing slashes also and vice versa.

svndumpfilter has problems with copies: As a copy must show their origin in Subversion it could happen
that just its origin was filtered by svndumpfilter.

This is a system immanent behavior of Subversion and cannot be changed.

However there are a few rewritten dumpfilters with some quirks fixed:

http://furius.ca/pubcode/pub/conf/bin/svndumpfilter3.html

svndumpfilter Example ⌘

  • Create a new repository with 3 Dirs:
    • project_A
    • project_B
    • project_C
  • and in each directory a file:
    • file_a.txt
    • file_b.txt
    • file_c.txt
  • Export repository to dumpfile:
    • svnadmin dump {path/to/repo} > {path/to/dumpfile}
  • run svndumpfilter (special windows syntax!):
    • svndumpfilter exclude project_A < c:\dump.txt > c:\project_a_ex.txt
    • svndumpfilter include project_A < c:\dump.txt > c:\project_a.txt
  • compare the three dumpfiles

Exercise ⌘

  • Split the repository into 2
    • /harry_repo containing file:///repo/harry folder
    • /sally_repo containing rest of the repository

Hooks ⌘

Hooks play an important role in customizing
workflows and adapting Subversion for special needs.

Hooks are triggered by special repository events
and depending on the return value Subversion
approves or cancels the action.

The hook script’s environment is empty so you need absolute paths and set all needed variables explicitly.

Hooks con't ⌘

There are 9 different types of hooks for 4 different events during commits:

commit hooks 		(start-commit, pre-commit, post-commit)
lock hooks 		(pre-lock, post-lock)
unlock hooks 		(pre-unlock, post-unlock)
revprop-change hook 	(pre-revprop-change, post-revprop-change)

Messages sent to stderr will be marshalled to the svn client.

On pre- or start- hooks, returning a non-zero value usually cancels the action.

Commit Hooks ⌘

The commit is the only event which has 3 hooks:

start-commit(REPO_PATH, USER_NAME, CAPS) – from v 1.5
This hook is called right before any data is transferred to the server.
Useful for maintenance lock-down, or extra user-authentication.

pre-commit(REPO_PATH, TXN_NAME)
This hook is called right after data is transferred to the server,
before the transaction is finished.
Here you can check all different semantics by examinig the transaction. Useful for enforcing workflows.

post-commit(REPO_PATH, NEW_REVISION)
This hook is called after transaction is complete, before acknowledge is sent back to client.
Useful for sending notification mails.

Commit hook example ⌘

check for empty commit message:
REPOS="$1"
TXN="$2"

SVNLOOK=/usr/local/bin/svnlook
$SVNLOOK log -t "$TXN" "$REPOS" | \
   grep "[a-zA-Z0-9]" > /dev/null || exit 1

exit 0

Commit hook example in Windows ⌘

"C:\Program Files\TortoiseSVN\bin\svnlook.exe" log -t %2 %1 | FindStr [a-zA-Z0-9]
IF %ERRORLEVEL% EQU 0 GOTO OK
echo "Commit Comments are Required" >&2
exit 1
:OK
exit 0

lock/unlock hooks ⌘

For customizing locking in Subversion you need lock/unlock hooks

lock hooks:
pre-lock(REPOS_PATH, PATH, USER, LOCK_COMMENT, STEAL[1:0])
Here you can define workflows (e.g. disallow stealing locks, allowing locks only for certain files)

post-lock(REPOS_PATH, USER) < locked path via stdin
Useful for notifications.

unlock hooks:		
pre-unlock(REPOS_PATH, PATH, USER, TOKEN, BREAK:UNLOCK[1:0])
define workflow for releasing (breaking) locks, e.g. define administrator 
who is allowed to break locks or notify previous lock-owner

post-unlock()
Useful for notifications.

revprop-change hooks ⌘

We strongly advise against enabling revprop-changes, as backup and recovery will be much more complex.
revprop-change hooks:
pre-revprop-change(REPOS_PATH, REV, USER, PROPNAME, ACTION) 
used for semantics to control who can change the revision properties. On stdin is the old property value

post-revprop-change(REPOS_PATH, REV, USER, PROPNAME, ACTION)
used for notification and backup purposes

revprop-change hook in windows ⌘

@ECHO OFF
:: Set all parameters. Even though most are not used, in case you want to add
:: changes that allow, for example, editing of the author or addition of log messages.
set repository=%1
set revision=%2
set userName=%3
set propertyName=%4
set action=%5

:: Only allow the log message to be changed, but not author, etc.
if /I not "%propertyName%" == "svn:log" goto ERROR_PROPNAME

:: Only allow modification of a log message, not addition or deletion.
if /I not "%action%" == "M" goto ERROR_ACTION

:: Make sure that the new svn:log message is not empty.
set bIsEmpty=true
for /f "tokens=*" %%g in ('find /V ""') do (
set bIsEmpty=false
)
if "%bIsEmpty%" == "true" goto ERROR_EMPTY

goto :eof

:ERROR_EMPTY
echo Empty svn:log messages are not allowed. >&2
goto ERROR_EXIT

:ERROR_PROPNAME
echo Only changes to svn:log messages are allowed. >&2
goto ERROR_EXIT

:ERROR_ACTION
echo Only modifications to svn:log revision properties are allowed. >&2
goto ERROR_EXIT

:ERROR_EXIT
exit /b 1

Exercises ⌘

  • Copy pre-commit.tmpl file to pre-commit and test whether you can commit something without specifying a message
  • Create 'pre-revprop-change' hook and allow only to modify 'svn:log' property

svnsync ⌘

SvnSyncAdmin.png

svnsync con't ⌘

create mirror repository

>svnadmin create /path/to/mirror

create a pre-revprop-change hookscript in mirror repository

>echo exit 0 > /path/to/mirror/hooks/pre-revprop-change
>chmod +x /path/to/mirror/hooks/pre-revprop-change

initialize mirror repository

>svnsync init file:///path/to/mirror file:///path/to/repo 

synchronize repositories

>svnsync sync file:///path/to/mirror 

Backups – hot copy ⌘

copy a repository (even on a running server!) with the hotcopy command:

>svnadmin hotcopy /path/to/repo /path/to/new/repo

Note that hooks will be copied, too.

SvnHotAdmin.png

Hot Copies ⌘

Hotcopy copies the repository along with all other file configurations, permissions, hooks and logs.

  • Hotcopy works on a running system regardless of running transactions
  • Hotcopy creates an exact copy of a new repository which can be used as a backup.
  • Hotcopy uses as much diskspace as the original repository.
  • Hotcopies are faster to restore than dumpfiles.

Repository Integrity ⌘

You should check the integrity of your repositories on a regular basis

Use the verify command:

>svnadmin verify /path/to/repo

Dumps ⌘

  • Dump is the secure way of exporting a repository with all history and versioning information.
  • Human readable fileformat.
  • Binary files exist inline so you should not edit the dump-file with your normal texteditor.
  • Files are quite large but compress well.
  • Through the --incremental switch dumps can be saved from revision to revision.
  • Restoring a broken repository from a dump might take a long time.

Upgrading Repository ⌘

You can use a dump/load cycle to upgrade the repository format:

>svnadmin create /new/repo
>svnadmin dump /path/to/repo | svnadmin load /new/repo

SvnUpgrAdmin.png

Upgrading con't ⌘

Since svn 1.5 you can use the upgrade command to upgrade the repository format

>svnadmin upgrade /path/to/repo

SvnUpgr2Admin.png

This option is much faster, however, you will not get all server-side repository features.

GUI ⌘

  • Plugins
    • subversive (eclipse plugin)
    • netbeans, etc
  • tortoise
  • VisualSVN