conf versioning in pkgsrc

using Version Control Systems to keep track of system configuration changes in package upgrades

name synopsis
Keivan Motavalli
The project will consist in extending the pkgsrc package collection framework to support
automatically storing updated revisions of package-provided system configuration files in
account various kinds of local and networked Version Control Systems, starting with CVS, then
motavalli.keivan allowing administrators to merge site modifications.
(gmail.com)
Once the core is realized, not tied to a specific VCS or use case, additional features could
be implemented, such as support for pulling site and <package,version,range,role> specific
location configuration from a networked repository upon installation, support for version control sys-
omiss
tems other than CVS and tooling to aid administrators in merging local modifications the
omiss
same way independently of the versioning solution in use.
Milan, Italy

reach me community benefit

keivan@motavalli.me
github.com/kmotavalli Many package managers and binary or source software collections exist, made for specific
+39 omiss (mobile & operating systems. These aid users in easily obtaining programs and libraries known to
IMs) work together, and are reckoned on by elaborate and scriptable deployment solutions that
+39 omiss (landline) support a variety of platforms.

time zone is different in that, after being forked from FreeBSD ports to serve as the
UTC+1 package management system and software collection for NetBSD, it has evolved to target
many unix and unix-like operating systems. pkgsrc supports all major BSDs, Gnu/Linux,
working hours QNX, MacOS X, Solaris and Illumos distributions, commercial unixes such as AIX, HP-UX,
Full time after Tru64, posix compatibility layers for Windows NT, MINIX 3, Haiku and even discontinued
semester ends OSs.
(mid June) It is both a wide (17000) software collection, a multiplatform binary package manager, a
full weekends and source-based build framework and a system for creating packaged software repositories
18-22 otherwise ready for use across different operating systems and hardware architectures.
Some third parties have employed pkgsrc to provide a package collection that is coherent
alternate on the many systems they work with, support or develop and distribute.
contact However, pkgsrc currently implements a rudimentary way of handling configuration files and
omiss data resources: it copies said files (often as examples) on the system only if the package has
(gmail.com) never been installed before, and avoids touching them otherwise, to refrain from overwriting
omiss changes and customizations made by the site administrator(s).
+39 omiss
(mobile) This poses a problem because, across updates, changes get delivered by package main-
omiss, Milan tainers: more secure defaults may come in minor revisions. New options are introduced and
documented; different, non backwards compatible configuration formats may be required by
major software releases, rendering systems inoperative for the inattentive.
Portage provides dispatch-conf to handle configuration updates in a Version Control Sys-
tem; it also provides an interactive interface for comparing and merging changes, based
on diff3 and RCS merge. Portage however is different in scope from pkgsrc (it is tailored
for Gentoo Linux and only partially supports binary packages). Furthermore, dispatch-conf
only works with Revision Control System, which is limited in functionality: it doesn’t support
use cases that involve remote repositories and branching multiple files at once to form a
known working configuration.
Debconf from the Debian Project also is capable of keeping, replacing or attempting to
merge configuration files across updates, and even provides a framework to packagers for
interactively setting up software options upon installation, but again it is distribution specific
and does not attempt to keep track of revisions at all.
Infrastructure as Code and Configuration Management solutions, as the name calls, are
very extensible and may come with a management overhead and a learning curve that are
too heavy for small deployments.
Extending pkgsrc functionality, as suggested by the NetBSD project, may cover the needs
of those who like having a coherent (binary or source) package base at their disposal across
many platforms and at the same time may like being able to update, keep track of and deploy
configurations on systems in a simple way: using the same tools that already manage pack-
age installation and upgrades, optionally combined with configuration stores developers are
used interacting with as part of their work routine.
Code written as part of this project deliverables could be further exploited by introducing
ways to easily roll back to a known working configuration for a set of daemons running in a
system.
As mentioned, packages could also automatically search for said working configuration on
remote branches named with a matching software version range, role and/or hostname
upon installation, just by pointing the system to an appropriately populated VCS repository
and setting few other options in pkgsrc.

implementation
The source tree of pkgsrc is composed of
packages package definitions and build instructions themselves, written in BSD Make and
organized in categories
bootstrap the script and infrastructure which creates necessary directories, compiles and/or
installs the basic tools used by pkgsrc on the platform it’s running on, then pro-
ceeds setting appropriate options in mk.conf
pkg_install C source code for binary tools used to handle the binary package format used
by pkgsrc across platforms. Part of the tree under pkgtools/pkg_install/files
mk the bmake infrastructure called by packages makefiles
mk/pkginstall the bmake framework which generates shell scripts included in built pack-
ages, which are then called by pkg_add and pkg_delete

required: bootstrap
The bootstrap script should be extended to also compile, install a default VCS on supported
platforms; for ease of testing and development, this should be cvs because it’s already
part of NetBSD base system, allowing to directly work on one implementation in Phase 1
already. Should Phase 1 development reveal necessary dependencies on other utilities,
those would need to be bootstrapped, too. The bootstrap script should also, as default
behavior, initialize a local cvs repository for use in pkgsrc, either directly of by calling another
tool, if support for handling other default VCSs in the future may be desired.

required: pkginstall framework

Under ”mk” makefiles exist to generate INSTALL and DEINSTALL scripts at compile time, for
each package being built. These scripts get included in binary packages and are then run
by pkg_add upon installation, either locally on the same system where the build took place
or on other systems, even on installations only using third party built binary packages.
Among tasks executed by the INSTALL script is checking for previous installations of the
package and copying $CONF_FILES defined by the packager to their appropriate destination
on the system. This logic should be modified to also support adding config files to a VCS
branch and register the fact that manual modifications are needed, if that’s the case, noti-
fying the operator of the condition. Support for generic versioning operations should not be
coded with a specific software solution in mind, but implemented using a wrapper which will
then gain support for different Versioning Control Systems.
Optionally logic could be written to also support checking, if a remote VCS is set up on the
target system, the existence of a branch compatible with the package version being installed
and the system role, name: if enabled, the feature could pull appropriate configuration and
locally install it, taking precedence over package-provided configuration files.
All CONF_FILES needed should be hierarchically placed in accordance with pkgsrc soft-
ware categories and package naming (eg, under /mail/postfix/) and branched with a name
expression of the system role (eg, myproject-testing_net02) and version information (eg
3.2.5_3.0_any), so that the file at
mail/postfix/main.cf branched as 3.2.5_3.0_any_myproject-testing_net02,
with meanings exactVersion_rangeStart_rangeEnd_role_optionalHostname
would be up for checks disposed this way:
first, all branches whose ranges are compatible with the version of the package being in-
stalled are selected.
Then, if a role other than ”any” is present, it is compared with the role defined in pkgsrc
configuration and selected. The last part of the branch name is optional and, if present,
compared character by character with the system hostname, finally selecting the branches
that best match it.
The checks now further refine the candidates: if a branch exactly corresponds with the
version of the package being installed, that branch gets selected, otherwise the procedure
should choose the next closest one.
The file is then copied on the system. Each branch should contain all the configuration files
the package needs, so that versions don’t get mixed up.

required: pkg_install utilities

Part of pkgsrc are a set of tools, written in C, which handle binary package installation,
removal, querying and various administrative operations over the pkgdb.
To avoid hardcoding options such as the versioning system to be used and network cre-
dentials in the INSTALL script at compile time, new environment variables should be parsed
from configuration and passed by the pkg_add utility to the INSTALL script. It will be required
to extend said tool.

optional: merging tool

Optionally, in Phase 3 a new tool could be devised to allow the operator to interface with
various supported versioning solutions in the same way: it could handle setting up reposi-
tories, taking and rolling back snapshots of system configuration into appropriately named
repositories and branches, and implement guided merging and editing functionality similar
to how dispatch-conf does for Portage. In absence of this functionality the administrator
will have to rely on merge solutions specific of the chosen VCS, losing consistency, not being
able to rely on a workflow tailored for 3 way configuration files merging and making switch-
ing revision systems harder. Optional functionality could also be implemented by extending
existing tools, such as pkg_admin, if considered acceptable by the pkgsrc community.
about me
I am a freshman currently studying Computer Science at the University of Milano-Bicocca.
After some years of lurking around open source projects as a user interested in systems
administration and informatics, this is the first time I get to contribute to a project having
goals that, in case of completion of the optional features, I can see being concretely useful.
Having always thought of myself as interested in matters of technology that don’t pivot
around coding, I got to use some C in the course of my studies and discovered that I also
really like learning that (essential) branch of computer-related-things, so I decided to enroll
in a Computer Science course.
Google Summer Of Code has been discussed and promoted at the university LUG by for-
mer participants and I quickly, aiming to self support my studies, started searching for tasks
that I appreciated, considering my interest for servers and administration. I already used
some self-taught python for personal projects and I am now learning Java and mips32 pro-
gramming. It would be rewarding to start gaining practical experience by coding for a wider
scope than personal utilities and lab assignments.
Yes, NetBSD is one of the projects I used to lurk :)

timeline
April Community Bonding hoping for the best...
This time would be used to further navigate the code base of pkgsrc and
ways to bootstrap and test the required components on supported targets. It
could be useful for learning how contributing to the project works. It would
be necessary to approach the wider pkgsrc/NetBSD community about their
needs in relation to this proposal and on matters related to options definition,
naming and placing. Feedback would be required about how to check for and
get notified of manual interventions when required
May 14 Phase 1 limited-time work phase
In May I will still be attending courses full time for another 30 days, until the
semester ends. Weekends and the evening are free for scripting: I would work
on getting up a prototype on NetBSD, which comes with cvs and pkg_tools
already in the base system, by modifying the pkginstall framework to wrap
around generic CVS functions and then call cvs. pkg_add would need minor
changes, too, in order to pass additional parameters to the now extended
INSTALL script inside binary packages.
June 15 Phase 2 Work full time except on exam days
I would work on modifying the bootstrap script: additional tools may need
installation on some platforms, and a default, local repository should be ini-
tialized by the script when first deploying pkgsrc on a system. Support for
remote cvs repositories would be implemented in the pkginstall framework,
then the solution tested and refined on more platforms supported by pkgsrc
July, August Final phase work full time
If testing goes well in Phase 2, spare time could exist to allow for the ten-
tative implementation of optional deliverables, such as support in pkginstall
for other VCS solutions (Git, Mercurial and the BSD-licensed fossil-scm), the
structuring of a tool similar to dispatch-conf (Portage) usable to interactively
merge local configuration changes the same way independently of the VCS
used, branch and reinstall multiple configuration files to keep track of known
working states, manage non-default VCS configuration and initialization, and
adding support for automatically pulling configuration from remote reposito-
ries